Update README.md
Browse files
README.md
CHANGED
|
@@ -5,6 +5,10 @@ language:
|
|
| 5 |
- vi
|
| 6 |
---
|
| 7 |
|
| 8 |
-
Base Model: LLaMa2 7B Chat HF
|
| 9 |
-
+
|
| 10 |
-
+
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5 |
- vi
|
| 6 |
---
|
| 7 |
|
| 8 |
+
## Base Model: LLaMa2 7B Chat HF
|
| 9 |
+
+ Extend vocab to 44,800 for better Vietnamese understanding
|
| 10 |
+
+ Continual Pre-Train with >2B tokens Vietnamese
|
| 11 |
+
+ Trainning profile: LoRa (rank=32, alpha=128, 16fp), 1 epoch, block size = 512. Takes 300GPU Hours x RXT4090 24GB
|
| 12 |
+
|
| 13 |
+
## Can be better use for
|
| 14 |
+
+ Futher training / Fine-tuning for Vietnamese tasks
|