Commit
•
6233b87
1
Parent(s):
652c05c
Add Dataset
Browse files
README.md
CHANGED
|
@@ -2,6 +2,8 @@
|
|
| 2 |
license: apache-2.0
|
| 3 |
language:
|
| 4 |
- en
|
|
|
|
|
|
|
| 5 |
---
|
| 6 |
|
| 7 |
***<p style="font-size: 24px">Feel free to try out our [OpenChatKit feedback app](https://huggingface.co/spaces/togethercomputer/OpenChatKit)!</p>***
|
|
@@ -185,6 +187,4 @@ Please refer to [togethercomputer/OpenDataHub](https://github.com/togethercomput
|
|
| 185 |
- **Optimizer:** [8bit-AdamW](https://github.com/TimDettmers/bitsandbytes)
|
| 186 |
- **Gradient Accumulations**: 2
|
| 187 |
- **Batch:** 2 x 2 x 64 x 2048 = 524288 tokens
|
| 188 |
-
- **Learning rate:** warmup to 1e-6 for 100 steps and then kept constant
|
| 189 |
-
|
| 190 |
-
|
|
|
|
| 2 |
license: apache-2.0
|
| 3 |
language:
|
| 4 |
- en
|
| 5 |
+
datasets:
|
| 6 |
+
- laion/OIG
|
| 7 |
---
|
| 8 |
|
| 9 |
***<p style="font-size: 24px">Feel free to try out our [OpenChatKit feedback app](https://huggingface.co/spaces/togethercomputer/OpenChatKit)!</p>***
|
|
|
|
| 187 |
- **Optimizer:** [8bit-AdamW](https://github.com/TimDettmers/bitsandbytes)
|
| 188 |
- **Gradient Accumulations**: 2
|
| 189 |
- **Batch:** 2 x 2 x 64 x 2048 = 524288 tokens
|
| 190 |
+
- **Learning rate:** warmup to 1e-6 for 100 steps and then kept constant
|
|
|
|
|
|