トークナイザーに全角記号・数字のNKFC正規化を追加 (#2)

- トークナイザーに全角記号・数字のNKFC正規化を追加 (1dc9fb6bc4d845bf6b52587914a953501cad8509)


Co-authored-by: Kanta Hayashi <[email protected]>

tokenizer.json

@@ -124,6 +124,9 @@
   "normalizer": {
     "type": "Sequence",
     "normalizers": [
+      {
+        "type": "NFKC"
+      },
       {
         "type": "Replace",
         "pattern": {