esm2_t130_150M-lora-classifier_2024-04-26_10-08-51

This model is a fine-tuned version of facebook/esm2_t30_150M_UR50D on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.4537
• Accuracy: 0.8984

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0008701568055793088
• train_batch_size: 28
• eval_batch_size: 28
• seed: 8893
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 30
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
0.6764	1.0	55	0.6794	0.5820
0.5521	2.0	110	0.6192	0.6777
0.5409	3.0	165	0.5147	0.7383
0.5518	4.0	220	0.3518	0.8672
0.1386	5.0	275	0.3596	0.8574
0.303	6.0	330	0.4030	0.8359
0.1962	7.0	385	0.3143	0.8848
0.1501	8.0	440	0.3232	0.8652
0.2994	9.0	495	0.3014	0.8770
0.0914	10.0	550	0.2980	0.8887
0.2108	11.0	605	0.2854	0.8770
0.2896	12.0	660	0.3684	0.8691
0.0818	13.0	715	0.3349	0.8828
0.3152	14.0	770	0.3530	0.8848
0.0554	15.0	825	0.3371	0.8887
0.1928	16.0	880	0.3347	0.875
0.2658	17.0	935	0.3765	0.8867
0.4242	18.0	990	0.4166	0.8945
0.0964	19.0	1045	0.3400	0.8945
0.0375	20.0	1100	0.3581	0.9004
0.1781	21.0	1155	0.3816	0.8848
0.1563	22.0	1210	0.3940	0.8867
0.017	23.0	1265	0.4098	0.8926
0.1866	24.0	1320	0.4710	0.8770
0.0632	25.0	1375	0.4541	0.8828
0.1501	26.0	1430	0.4645	0.8828
0.109	27.0	1485	0.4434	0.8926
0.0353	28.0	1540	0.4264	0.8984
0.4502	29.0	1595	0.4479	0.8984
0.0341	30.0	1650	0.4537	0.8984

Framework versions

• PEFT 0.10.0
• Transformers 4.39.3
• Pytorch 2.2.1
• Datasets 2.16.1
• Tokenizers 0.15.2