Model continually pre-trained on MS-MARCO (T5 span corruption task).

Starting model: google-t5/t5-base

Training script: follows nanoT5 using the following huggingface class:  
T5ForConditionalGenerationRoPE: https://github.com/cxcscmu/LongEmbeddingAnalysis/blob/main/OpenMatch/src/openmatch/modeling/rope_t5.py