Add a notice to warn about the compatibility issues between code and weights
Browse files
README.md
CHANGED
|
@@ -6,6 +6,11 @@ library_name: tf-keras
|
|
| 6 |
|
| 7 |
> `to-kun` took tokens to t-can
|
| 8 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 9 |
Current tokenizers have notorious issues that are bringing all the LLMs down.
|
| 10 |
|
| 11 |
`tokun` is a model specialized in text embedding.
|
|
|
|
| 6 |
|
| 7 |
> `to-kun` took tokens to t-can
|
| 8 |
|
| 9 |
+
```
|
| 10 |
+
The weights in this repository were trained with the architecture v0.11.2 of the package "tokun".
|
| 11 |
+
Later versions are incompatible and need to be trained from scratch.
|
| 12 |
+
```
|
| 13 |
+
|
| 14 |
Current tokenizers have notorious issues that are bringing all the LLMs down.
|
| 15 |
|
| 16 |
`tokun` is a model specialized in text embedding.
|