Just an idea
#1
by
distantquant
- opened
Have you thought about merging this with ShinojiResearch/Senku-70B-Full?
It may make a good model.
Great idea, I will definitely try that. I was not aware of Senku yet. Swap merging with a light finetune perfectly fits our hypothesis about attempting to restore quantization error by using weights from other models.
I tried a few merges with MiquMaid x Quartet already, and they were disappointing.
senku beat an early version of GPT4 on eq-bench (https://eqbench.com/) so yeah it may have a good potential