Update README.md
Browse files
README.md
CHANGED
|
@@ -23,6 +23,4 @@ Prompt template:
|
|
| 23 |
GGML quants available [here](https://huggingface.co/TheBloke/Puma-3b-GGML).</br>
|
| 24 |
GPTQ quants available [here](https://huggingface.co/TheBloke/Puma-3b-GPTQ).
|
| 25 |
|
| 26 |
-
Note: Don't expect this model to be good, I was just starting out to finetune. So don't roast me please!
|
| 27 |
-
|
| 28 |
-
Benchmarks:<table><tr><th>arc_challenge</th><td><table><tr><th>acc</th><td>0.3660409556313993</td></tr><tr><th>acc_stderr</th><td>0.014077223108470142</td></tr><tr><th>acc_norm</th><td>0.4121160409556314</td></tr><tr><th>acc_norm_stderr</th><td>0.014383915302225403</td></tr></table></td></tr><tr><th>hellaswag</th><td><table><tr><th>acc</th><td>0.5401314479187412</td></tr><tr><th>acc_stderr</th><td>0.004973683026202171</td></tr><tr><th>acc_norm</th><td>0.7177853017327226</td></tr><tr><th>acc_norm_stderr</th><td>0.004491574539441884</td></tr></table></td></tr><tr><th>hendrycksTest-abstract_algebra</th><td><table><tr><th>acc</th><td>0.21</td></tr><tr><th>acc_stderr</th><td>0.04093601807403326</td></tr><tr><th>acc_norm</th><td>0.21</td></tr><tr><th>acc_norm_stderr</th><td>0.04093601807403326</td></tr></table></td></tr><tr><th>hendrycksTest-anatomy</th><td><table><tr><th>acc</th><td>0.25925925925925924</td></tr><tr><th>acc_stderr</th><td>0.03785714465066653</td></tr><tr><th>acc_norm</th><td>0.25925925925925924</td></tr><tr><th>acc_norm_stderr</th><td>0.03785714465066653</td></tr></table></td></tr><tr><th>hendrycksTest-astronomy</th><td><table><tr><th>acc</th><td>0.28289473684210525</td></tr><tr><th>acc_stderr</th><td>0.03665349695640767</td></tr><tr><th>acc_norm</th><td>0.28289473684210525</td></tr><tr><th>acc_norm_stderr</th><td>0.03665349695640767</td></tr></table></td></tr><tr><th>hendrycksTest-business_ethics</th><td><table><tr><th>acc</th><td>0.34</td></tr><tr><th>acc_stderr</th><td>0.04760952285695236</td></tr><tr><th>acc_norm</th><td>0.34</td></tr><tr><th>acc_norm_stderr</th><td>0.04760952285695236</td></tr></table></td></tr><tr><th>hendrycksTest-clinical_knowledge</th><td><table><tr><th>acc</th><td>0.2528301886792453</td></tr><tr><th>acc_stderr</th><td>0.02674989977124123</td></tr><tr><th>acc_norm</th><td>0.2528301886792453</td></tr><tr><th>acc_norm_stderr</th><td>0.02674989977124123</td></tr></table></td></tr><tr><th>hendrycksTest-college_biology</th><td><table><tr><th>acc</th><td>0.24305555555555555</td></tr><tr><th>acc_stderr</th><td>0.03586879280080339</td></tr><tr><th>acc_norm</th><td>0.24305555555555555</td></tr><tr><th>acc_norm_stderr</th><td>0.03586879280080339</td></tr></table></td></tr><tr><th>hendrycksTest-college_chemistry</th><td><table><tr><th>acc</th><td>0.19</td></tr><tr><th>acc_stderr</th><td>0.03942772444036623</td></tr><tr><th>acc_norm</th><td>0.19</td></tr><tr><th>acc_norm_stderr</th><td>0.03942772444036623</td></tr></table></td></tr><tr><th>hendrycksTest-college_computer_science</th><td><table><tr><th>acc</th><td>0.21</td></tr><tr><th>acc_stderr</th><td>0.040936018074033256</td></tr><tr><th>acc_norm</th><td>0.21</td></tr><tr><th>acc_norm_stderr</th><td>0.040936018074033256</td></tr></table></td></tr><tr><th>hendrycksTest-college_mathematics</th><td><table><tr><th>acc</th><td>0.36</td></tr><tr><th>acc_stderr</th><td>0.04824181513244218</td></tr><tr><th>acc_norm</th><td>0.36</td></tr><tr><th>acc_norm_stderr</th><td>0.04824181513244218</td></tr></table></td></tr><tr><th>hendrycksTest-college_medicine</th><td><table><tr><th>acc</th><td>0.20809248554913296</td></tr><tr><th>acc_stderr</th><td>0.030952890217749877</td></tr><tr><th>acc_norm</th><td>0.20809248554913296</td></tr><tr><th>acc_norm_stderr</th><td>0.030952890217749877</td></tr></table></td></tr><tr><th>hendrycksTest-college_physics</th><td><table><tr><th>acc</th><td>0.2647058823529412</td></tr><tr><th>acc_stderr</th><td>0.04389869956808778</td></tr><tr><th>acc_norm</th><td>0.2647058823529412</td></tr><tr><th>acc_norm_stderr</th><td>0.04389869956808778</td></tr></table></td></tr><tr><th>hendrycksTest-computer_security</th><td><table><tr><th>acc</th><td>0.33</td></tr><tr><th>acc_stderr</th><td>0.047258156262526045</td></tr><tr><th>acc_norm</th><td>0.33</td></tr><tr><th>acc_norm_stderr</th><td>0.047258156262526045</td></tr></table></td></tr><tr><th>hendrycksTest-conceptual_physics</th><td><table><tr><th>acc</th><td>0.33617021276595743</td></tr><tr><th>acc_stderr</th><td>0.030881618520676942</td></tr><tr><th>acc_norm</th><td>0.33617021276595743</td></tr><tr><th>acc_norm_stderr</th><td>0.030881618520676942</td></tr></table></td></tr><tr><th>hendrycksTest-econometrics</th><td><table><tr><th>acc</th><td>0.24561403508771928</td></tr><tr><th>acc_stderr</th><td>0.04049339297748144</td></tr><tr><th>acc_norm</th><td>0.24561403508771928</td></tr><tr><th>acc_norm_stderr</th><td>0.04049339297748144</td></tr></table></td></tr><tr><th>hendrycksTest-electrical_engineering</th><td><table><tr><th>acc</th><td>0.2689655172413793</td></tr><tr><th>acc_stderr</th><td>0.03695183311650232</td></tr><tr><th>acc_norm</th><td>0.2689655172413793</td></tr><tr><th>acc_norm_stderr</th><td>0.03695183311650232</td></tr></table></td></tr><tr><th>hendrycksTest-elementary_mathematics</th><td><table><tr><th>acc</th><td>0.2724867724867725</td></tr><tr><th>acc_stderr</th><td>0.022930973071633366</td></tr><tr><th>acc_norm</th><td>0.2724867724867725</td></tr><tr><th>acc_norm_stderr</th><td>0.022930973071633366</td></tr></table></td></tr><tr><th>hendrycksTest-formal_logic</th><td><table><tr><th>acc</th><td>0.21428571428571427</td></tr><tr><th>acc_stderr</th><td>0.03670066451047181</td></tr><tr><th>acc_norm</th><td>0.21428571428571427</td></tr><tr><th>acc_norm_stderr</th><td>0.03670066451047181</td></tr></table></td></tr><tr><th>hendrycksTest-global_facts</th><td><table><tr><th>acc</th><td>0.32</td></tr><tr><th>acc_stderr</th><td>0.046882617226215034</td></tr><tr><th>acc_norm</th><td>0.32</td></tr><tr><th>acc_norm_stderr</th><td>0.046882617226215034</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_biology</th><td><table><tr><th>acc</th><td>0.24193548387096775</td></tr><tr><th>acc_stderr</th><td>0.024362599693031093</td></tr><tr><th>acc_norm</th><td>0.24193548387096775</td></tr><tr><th>acc_norm_stderr</th><td>0.024362599693031093</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_chemistry</th><td><table><tr><th>acc</th><td>0.26108374384236455</td></tr><tr><th>acc_stderr</th><td>0.030903796952114485</td></tr><tr><th>acc_norm</th><td>0.26108374384236455</td></tr><tr><th>acc_norm_stderr</th><td>0.030903796952114485</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_computer_science</th><td><table><tr><th>acc</th><td>0.27</td></tr><tr><th>acc_stderr</th><td>0.04461960433384739</td></tr><tr><th>acc_norm</th><td>0.27</td></tr><tr><th>acc_norm_stderr</th><td>0.04461960433384739</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_european_history</th><td><table><tr><th>acc</th><td>0.30303030303030304</td></tr><tr><th>acc_stderr</th><td>0.035886248000917075</td></tr><tr><th>acc_norm</th><td>0.30303030303030304</td></tr><tr><th>acc_norm_stderr</th><td>0.035886248000917075</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_geography</th><td><table><tr><th>acc</th><td>0.24242424242424243</td></tr><tr><th>acc_stderr</th><td>0.030532892233932026</td></tr><tr><th>acc_norm</th><td>0.24242424242424243</td></tr><tr><th>acc_norm_stderr</th><td>0.030532892233932026</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_government_and_politics</th><td><table><tr><th>acc</th><td>0.24352331606217617</td></tr><tr><th>acc_stderr</th><td>0.030975436386845426</td></tr><tr><th>acc_norm</th><td>0.24352331606217617</td></tr><tr><th>acc_norm_stderr</th><td>0.030975436386845426</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_macroeconomics</th><td><table><tr><th>acc</th><td>0.2564102564102564</td></tr><tr><th>acc_stderr</th><td>0.022139081103971545</td></tr><tr><th>acc_norm</th><td>0.2564102564102564</td></tr><tr><th>acc_norm_stderr</th><td>0.022139081103971545</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_mathematics</th><td><table><tr><th>acc</th><td>0.2518518518518518</td></tr><tr><th>acc_stderr</th><td>0.026466117538959905</td></tr><tr><th>acc_norm</th><td>0.2518518518518518</td></tr><tr><th>acc_norm_stderr</th><td>0.026466117538959905</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_microeconomics</th><td><table><tr><th>acc</th><td>0.24789915966386555</td></tr><tr><th>acc_stderr</th><td>0.028047967224176892</td></tr><tr><th>acc_norm</th><td>0.24789915966386555</td></tr><tr><th>acc_norm_stderr</th><td>0.028047967224176892</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_physics</th><td><table><tr><th>acc</th><td>0.31125827814569534</td></tr><tr><th>acc_stderr</th><td>0.037804458505267334</td></tr><tr><th>acc_norm</th><td>0.31125827814569534</td></tr><tr><th>acc_norm_stderr</th><td>0.037804458505267334</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_psychology</th><td><table><tr><th>acc</th><td>0.26422018348623855</td></tr><tr><th>acc_stderr</th><td>0.01890416417151019</td></tr><tr><th>acc_norm</th><td>0.26422018348623855</td></tr><tr><th>acc_norm_stderr</th><td>0.01890416417151019</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_statistics</th><td><table><tr><th>acc</th><td>0.2175925925925926</td></tr><tr><th>acc_stderr</th><td>0.028139689444859672</td></tr><tr><th>acc_norm</th><td>0.2175925925925926</td></tr><tr><th>acc_norm_stderr</th><td>0.028139689444859672</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_us_history</th><td><table><tr><th>acc</th><td>0.24019607843137256</td></tr><tr><th>acc_stderr</th><td>0.02998373305591362</td></tr><tr><th>acc_norm</th><td>0.24019607843137256</td></tr><tr><th>acc_norm_stderr</th><td>0.02998373305591362</td></tr></table></td></tr><tr><th>hendrycksTest-high_school_world_history</th><td><table><tr><th>acc</th><td>0.28270042194092826</td></tr><tr><th>acc_stderr</th><td>0.029312814153955924</td></tr><tr><th>acc_norm</th><td>0.28270042194092826</td></tr><tr><th>acc_norm_stderr</th><td>0.029312814153955924</td></tr></table></td></tr><tr><th>hendrycksTest-human_aging</th><td><table><tr><th>acc</th><td>0.3901345291479821</td></tr><tr><th>acc_stderr</th><td>0.03273766725459156</td></tr><tr><th>acc_norm</th><td>0.3901345291479821</td></tr><tr><th>acc_norm_stderr</th><td>0.03273766725459156</td></tr></table></td></tr><tr><th>hendrycksTest-human_sexuality</th><td><table><tr><th>acc</th><td>0.1984732824427481</td></tr><tr><th>acc_stderr</th><td>0.03498149385462473</td></tr><tr><th>acc_norm</th><td>0.1984732824427481</td></tr><tr><th>acc_norm_stderr</th><td>0.03498149385462473</td></tr></table></td></tr><tr><th>hendrycksTest-international_law</th><td><table><tr><th>acc</th><td>0.30578512396694213</td></tr><tr><th>acc_stderr</th><td>0.04205953933884124</td></tr><tr><th>acc_norm</th><td>0.30578512396694213</td></tr><tr><th>acc_norm_stderr</th><td>0.04205953933884124</td></tr></table></td></tr><tr><th>hendrycksTest-jurisprudence</th><td><table><tr><th>acc</th><td>0.24074074074074073</td></tr><tr><th>acc_stderr</th><td>0.0413311944024384</td></tr><tr><th>acc_norm</th><td>0.24074074074074073</td></tr><tr><th>acc_norm_stderr</th><td>0.0413311944024384</td></tr></table></td></tr><tr><th>hendrycksTest-logical_fallacies</th><td><table><tr><th>acc</th><td>0.22085889570552147</td></tr><tr><th>acc_stderr</th><td>0.03259177392742177</td></tr><tr><th>acc_norm</th><td>0.22085889570552147</td></tr><tr><th>acc_norm_stderr</th><td>0.03259177392742177</td></tr></table></td></tr><tr><th>hendrycksTest-machine_learning</th><td><table><tr><th>acc</th><td>0.24107142857142858</td></tr><tr><th>acc_stderr</th><td>0.04059867246952685</td></tr><tr><th>acc_norm</th><td>0.24107142857142858</td></tr><tr><th>acc_norm_stderr</th><td>0.04059867246952685</td></tr></table></td></tr><tr><th>hendrycksTest-management</th><td><table><tr><th>acc</th><td>0.2912621359223301</td></tr><tr><th>acc_stderr</th><td>0.04498676320572922</td></tr><tr><th>acc_norm</th><td>0.2912621359223301</td></tr><tr><th>acc_norm_stderr</th><td>0.04498676320572922</td></tr></table></td></tr><tr><th>hendrycksTest-marketing</th><td><table><tr><th>acc</th><td>0.28205128205128205</td></tr><tr><th>acc_stderr</th><td>0.029480360549541194</td></tr><tr><th>acc_norm</th><td>0.28205128205128205</td></tr><tr><th>acc_norm_stderr</th><td>0.029480360549541194</td></tr></table></td></tr><tr><th>hendrycksTest-medical_genetics</th><td><table><tr><th>acc</th><td>0.31</td></tr><tr><th>acc_stderr</th><td>0.04648231987117316</td></tr><tr><th>acc_norm</th><td>0.31</td></tr><tr><th>acc_norm_stderr</th><td>0.04648231987117316</td></tr></table></td></tr><tr><th>hendrycksTest-miscellaneous</th><td><table><tr><th>acc</th><td>0.30140485312899107</td></tr><tr><th>acc_stderr</th><td>0.016409091097268787</td></tr><tr><th>acc_norm</th><td>0.30140485312899107</td></tr><tr><th>acc_norm_stderr</th><td>0.016409091097268787</td></tr></table></td></tr><tr><th>hendrycksTest-moral_disputes</th><td><table><tr><th>acc</th><td>0.30346820809248554</td></tr><tr><th>acc_stderr</th><td>0.024752411960917202</td></tr><tr><th>acc_norm</th><td>0.30346820809248554</td></tr><tr><th>acc_norm_stderr</th><td>0.024752411960917202</td></tr></table></td></tr><tr><th>hendrycksTest-moral_scenarios</th><td><table><tr><th>acc</th><td>0.24692737430167597</td></tr><tr><th>acc_stderr</th><td>0.014422292204808835</td></tr><tr><th>acc_norm</th><td>0.24692737430167597</td></tr><tr><th>acc_norm_stderr</th><td>0.014422292204808835</td></tr></table></td></tr><tr><th>hendrycksTest-nutrition</th><td><table><tr><th>acc</th><td>0.28431372549019607</td></tr><tr><th>acc_stderr</th><td>0.025829163272757475</td></tr><tr><th>acc_norm</th><td>0.28431372549019607</td></tr><tr><th>acc_norm_stderr</th><td>0.025829163272757475</td></tr></table></td></tr><tr><th>hendrycksTest-philosophy</th><td><table><tr><th>acc</th><td>0.29260450160771706</td></tr><tr><th>acc_stderr</th><td>0.02583989833487798</td></tr><tr><th>acc_norm</th><td>0.29260450160771706</td></tr><tr><th>acc_norm_stderr</th><td>0.02583989833487798</td></tr></table></td></tr><tr><th>hendrycksTest-prehistory</th><td><table><tr><th>acc</th><td>0.2993827160493827</td></tr><tr><th>acc_stderr</th><td>0.02548311560119547</td></tr><tr><th>acc_norm</th><td>0.2993827160493827</td></tr><tr><th>acc_norm_stderr</th><td>0.02548311560119547</td></tr></table></td></tr><tr><th>hendrycksTest-professional_accounting</th><td><table><tr><th>acc</th><td>0.2624113475177305</td></tr><tr><th>acc_stderr</th><td>0.02624492034984301</td></tr><tr><th>acc_norm</th><td>0.2624113475177305</td></tr><tr><th>acc_norm_stderr</th><td>0.02624492034984301</td></tr></table></td></tr><tr><th>hendrycksTest-professional_law</th><td><table><tr><th>acc</th><td>0.24641460234680573</td></tr><tr><th>acc_stderr</th><td>0.011005971399927242</td></tr><tr><th>acc_norm</th><td>0.24641460234680573</td></tr><tr><th>acc_norm_stderr</th><td>0.011005971399927242</td></tr></table></td></tr><tr><th>hendrycksTest-professional_medicine</th><td><table><tr><th>acc</th><td>0.21691176470588236</td></tr><tr><th>acc_stderr</th><td>0.02503584522771125</td></tr><tr><th>acc_norm</th><td>0.21691176470588236</td></tr><tr><th>acc_norm_stderr</th><td>0.02503584522771125</td></tr></table></td></tr><tr><th>hendrycksTest-professional_psychology</th><td><table><tr><th>acc</th><td>0.27941176470588236</td></tr><tr><th>acc_stderr</th><td>0.018152871051538812</td></tr><tr><th>acc_norm</th><td>0.27941176470588236</td></tr><tr><th>acc_norm_stderr</th><td>0.018152871051538812</td></tr></table></td></tr><tr><th>hendrycksTest-public_relations</th><td><table><tr><th>acc</th><td>0.2636363636363636</td></tr><tr><th>acc_stderr</th><td>0.04220224692971987</td></tr><tr><th>acc_norm</th><td>0.2636363636363636</td></tr><tr><th>acc_norm_stderr</th><td>0.04220224692971987</td></tr></table></td></tr><tr><th>hendrycksTest-security_studies</th><td><table><tr><th>acc</th><td>0.37551020408163266</td></tr><tr><th>acc_stderr</th><td>0.031001209039894836</td></tr><tr><th>acc_norm</th><td>0.37551020408163266</td></tr><tr><th>acc_norm_stderr</th><td>0.031001209039894836</td></tr></table></td></tr><tr><th>hendrycksTest-sociology</th><td><table><tr><th>acc</th><td>0.26865671641791045</td></tr><tr><th>acc_stderr</th><td>0.03134328358208954</td></tr><tr><th>acc_norm</th><td>0.26865671641791045</td></tr><tr><th>acc_norm_stderr</th><td>0.03134328358208954</td></tr></table></td></tr><tr><th>hendrycksTest-us_foreign_policy</th><td><table><tr><th>acc</th><td>0.35</td></tr><tr><th>acc_stderr</th><td>0.0479372485441102</td></tr><tr><th>acc_norm</th><td>0.35</td></tr><tr><th>acc_norm_stderr</th><td>0.0479372485441102</td></tr></table></td></tr><tr><th>hendrycksTest-virology</th><td><table><tr><th>acc</th><td>0.3253012048192771</td></tr><tr><th>acc_stderr</th><td>0.03647168523683227</td></tr><tr><th>acc_norm</th><td>0.3253012048192771</td></tr><tr><th>acc_norm_stderr</th><td>0.03647168523683227</td></tr></table></td></tr><tr><th>hendrycksTest-world_religions</th><td><table><tr><th>acc</th><td>0.30409356725146197</td></tr><tr><th>acc_stderr</th><td>0.035282112582452334</td></tr><tr><th>acc_norm</th><td>0.30409356725146197</td></tr><tr><th>acc_norm_stderr</th><td>0.035282112582452334</td></tr></table></td></tr><tr><th>truthfulqa_mc</th><td><table><tr><th>mc1</th><td>0.2460220318237454</td></tr><tr><th>mc1_stderr</th><td>0.015077219200662597</td></tr><tr><th>mc2</th><td>0.3821583918021301</td></tr><tr><th>mc2_stderr</th><td>0.013892826466923648</td></tr></table></td></tr></table>
|
|
|
|
| 23 |
GGML quants available [here](https://huggingface.co/TheBloke/Puma-3b-GGML).</br>
|
| 24 |
GPTQ quants available [here](https://huggingface.co/TheBloke/Puma-3b-GPTQ).
|
| 25 |
|
| 26 |
+
Note: Don't expect this model to be good, I was just starting out to finetune. So don't roast me please!
|
|
|
|
|
|