Set dropout in config.json to be 0 ?

As far as I'm aware, and according to the GPT-NeoX-20b arXiv paper, the model wasn't trained with dropout. Is there a reason there is dropout in this config on both attention and hiddens? (is this because dropout is recommended for finetuning or something?)

cc

@stellaathena

config.json

@@ -2,11 +2,11 @@
   "architectures": [
     "GPTNeoXForCausalLM"
   ],
-  "attention_probs_dropout_prob": 0.1,
+  "attention_probs_dropout_prob": 0,
   "bos_token_id": 0,
   "eos_token_id": 0,
   "hidden_act": "gelu_fast",
-  "hidden_dropout_prob": 0.1,
+  "hidden_dropout_prob": 0,
   "hidden_size": 6144,
   "initializer_range": 0.02,
   "intermediate_size": 24576,