Spaces:

xu-song
/

tokenizer-arena

Running

App Files Files Community

tokenizer-arena / requirements.txt

xu-song's picture

add more tokenizers

f4973d4 10 months ago

No virus

60 Bytes

	transformers==4.33.1
	sentencepiece
	tiktoken
	icetk
	torch
	zhon