to bos or not to bos?
The new tokenizer_config.json has "add_bos_token" set true while the previous version this was set false. Generate quality seems degraded with it set to true, is this correct?
Hi
@steampunque
, we tested the new version and it matches the output of the tokenizer from mistral-common. If you find any cases where the HF tokenizer does not match mistral-common for the same input, please let me know.
Thanks for your response. I use a custom autotokenizer on a different inference platform from HF. The original 8x22 model had "add_bos_token" set to false in tokenizer_config.json and generations seemed good without it. The new updated version has "add_bos_token" set to true and generations subjectively don't seem as good now so I was curious if that is an error in the new updated configuration for the model or is expected behavior.