The bilingual-gpt-neox-4b series are pre-trained from scratch on a mixture of Japanese and English corpora.