Only small amount of tokens generated in oobabooga
#13
by
synthetisoft
- opened
I got it running in Oobabooga but I can't seem to get it to generate more than 159 tokens. Usually, it's less. I set max_new_tokens to 2000. Any idea what might be going wrong?
synthetisoft
changed discussion title from
Oobabooga token window
to Only small amount of tokens generated in oobabooga
I have same issue. I am not sure if it is 159 tokens, but after it stops at certain points (<200 tokens), it stops generate tokens and Oobabooga says 0 tokens in 0 secs, etc.
I haven't tried it yet, but I just watched a video where a tip for a different model was described: It could be that the model is mimicking length of our prompts. So try giving it really long prompts by over-explaining or whatever.