Hugging Face's logo

ddh0
/
MiquSuperdark-70B-v1

Text Generation
Transformers
Safetensors
llama
mergekit
Merge
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
ddh0 commited on
Commit
16b41bb
•
1 Parent(s): 3e63d0d

Upload folder using huggingface_hub

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. README.md +14 -16
  2. config.json +28 -0
  3. mergekit_config.yml +17 -0
  4. model-00001-of-00081.safetensors +3 -0
  5. model-00002-of-00081.safetensors +3 -0
  6. model-00003-of-00081.safetensors +3 -0
  7. model-00004-of-00081.safetensors +3 -0
  8. model-00005-of-00081.safetensors +3 -0
  9. model-00006-of-00081.safetensors +3 -0
  10. model-00007-of-00081.safetensors +3 -0
  11. model-00008-of-00081.safetensors +3 -0
  12. model-00009-of-00081.safetensors +3 -0
  13. model-00010-of-00081.safetensors +3 -0
  14. model-00011-of-00081.safetensors +3 -0
  15. model-00012-of-00081.safetensors +3 -0
  16. model-00013-of-00081.safetensors +3 -0
  17. model-00014-of-00081.safetensors +3 -0
  18. model-00015-of-00081.safetensors +3 -0
  19. model-00016-of-00081.safetensors +3 -0
  20. model-00017-of-00081.safetensors +3 -0
  21. model-00018-of-00081.safetensors +3 -0
  22. model-00019-of-00081.safetensors +3 -0
  23. model-00020-of-00081.safetensors +3 -0
  24. model-00021-of-00081.safetensors +3 -0
  25. model-00022-of-00081.safetensors +3 -0
  26. model-00023-of-00081.safetensors +3 -0
  27. model-00024-of-00081.safetensors +3 -0
  28. model-00025-of-00081.safetensors +3 -0
  29. model-00026-of-00081.safetensors +3 -0
  30. model-00027-of-00081.safetensors +3 -0
  31. model-00028-of-00081.safetensors +3 -0
  32. model-00029-of-00081.safetensors +3 -0
  33. model-00030-of-00081.safetensors +3 -0
  34. model-00031-of-00081.safetensors +3 -0
  35. model-00032-of-00081.safetensors +3 -0
  36. model-00033-of-00081.safetensors +3 -0
  37. model-00034-of-00081.safetensors +3 -0
  38. model-00035-of-00081.safetensors +3 -0
  39. model-00036-of-00081.safetensors +3 -0
  40. model-00037-of-00081.safetensors +3 -0
  41. model-00038-of-00081.safetensors +3 -0
  42. model-00039-of-00081.safetensors +3 -0
  43. model-00040-of-00081.safetensors +3 -0
  44. model-00041-of-00081.safetensors +3 -0
  45. model-00042-of-00081.safetensors +3 -0
  46. model-00043-of-00081.safetensors +3 -0
  47. model-00044-of-00081.safetensors +3 -0
  48. model-00045-of-00081.safetensors +3 -0
  49. model-00046-of-00081.safetensors +3 -0
  50. model-00047-of-00081.safetensors +3 -0
README.md CHANGED
@@ -1,28 +1,28 @@
1
  ---
2
- base_model:
3
- - sophosympatheia/Midnight-Miqu-70B-v1.5
4
- - NeverSleep/MiquMaid-v3-70B
5
- - maywell/miqu-evil-dpo
6
- - 152334H/miqu-1-70b-sf
7
  library_name: transformers
8
  tags:
9
  - mergekit
10
  - merge
11
- license: other
12
  ---
13
  # MiquSuperdark-70B-v1
14
 
15
- **MiquSuperdark-70B-v1** is a merge of three of the most popular Miqu-derived models, along with Miqu itself. The goal of the merge is to create an strong, well-rounded chat model that picks up desirable traits from its constituent models without sacrificing intelligence.
 
 
 
16
 
17
- This is a DARE Linear merge with the following composition:
18
- - [sophosympatheia/Midnight-Miqu-70B-v1.5](https://huggingface.co/sophosympatheia/Midnight-Miqu-70B-v1.5) at weight 0.4
19
- - [NeverSleep/MiquMaid-v3-70B](https://huggingface.co/NeverSleep/MiquMaid-v3-70B) at weight 0.2
20
- - [maywell/miqu-evil-dpo](https://huggingface.co/maywell/miqu-evil-dpo) at weight 0.2
21
- - [152334H/miqu-1-70b-sf](https://huggingface.co/152334H/miqu-1-70b-sf) at weight 0.2 (used as base model)
22
 
23
- DARE Linear was chosen as the merge method based on [this HF discussion](https://huggingface.co/jukofyork/Dark-Miqu-70B/discussions/2), in which the creator of Midnight-Miqu says "*in my own testing I consistently got the best results from using a dare_linear merge when working with miqu models*".
24
 
25
- ## Merge Configuration
 
 
 
 
 
26
 
27
  The following YAML configuration was used to produce this model:
28
 
@@ -45,5 +45,3 @@ models:
45
  dtype: float16
46
  tokenizer_source: model:/home/dylan/Documents/AI/merge/miqu-1-70b-sf
47
  ```
48
-
49
- The tokenizer is copied from the base model [152334H/miqu-1-70b-sf](https://huggingface.co/152334H/miqu-1-70b-sf).
 
1
  ---
2
+ base_model: []
 
 
 
 
3
  library_name: transformers
4
  tags:
5
  - mergekit
6
  - merge
7
+
8
  ---
9
  # MiquSuperdark-70B-v1
10
 
11
+ This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
12
+
13
+ ## Merge Details
14
+ ### Merge Method
15
 
16
+ This model was merged using the linear [DARE](https://arxiv.org/abs/2311.03099) merge method using /home/dylan/Documents/AI/merge/miqu-1-70b-sf as a base.
 
 
 
 
17
 
18
+ ### Models Merged
19
 
20
+ The following models were included in the merge:
21
+ * /home/dylan/Documents/AI/merge/MiquMaid-v3-70B
22
+ * /media/dylan/SanDisk/LLMs/miqu-evil-dpo/
23
+ * /media/dylan/SanDisk/LLMs/Midnight-Miqu-70B-v1.5
24
+
25
+ ### Configuration
26
 
27
  The following YAML configuration was used to produce this model:
28
 
 
45
  dtype: float16
46
  tokenizer_source: model:/home/dylan/Documents/AI/merge/miqu-1-70b-sf
47
  ```
 
 
config.json ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "LlamaForCausalLM"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 1,
8
+ "eos_token_id": 2,
9
+ "hidden_act": "silu",
10
+ "hidden_size": 8192,
11
+ "initializer_range": 0.02,
12
+ "intermediate_size": 28672,
13
+ "max_position_embeddings": 32764,
14
+ "model_type": "llama",
15
+ "num_attention_heads": 64,
16
+ "num_hidden_layers": 80,
17
+ "num_key_value_heads": 8,
18
+ "pad_token_id": 0,
19
+ "pretraining_tp": 1,
20
+ "rms_norm_eps": 1e-05,
21
+ "rope_scaling": null,
22
+ "rope_theta": 1000000,
23
+ "tie_word_embeddings": false,
24
+ "torch_dtype": "float16",
25
+ "transformers_version": "4.36.0",
26
+ "use_cache": true,
27
+ "vocab_size": 32000
28
+ }
mergekit_config.yml ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ merge_method: dare_linear
2
+ base_model: /home/dylan/Documents/AI/merge/miqu-1-70b-sf
3
+ models:
4
+ - model: /media/dylan/SanDisk/LLMs/Midnight-Miqu-70B-v1.5
5
+ parameters:
6
+ weight: 0.4
7
+ - model: /home/dylan/Documents/AI/merge/miqu-1-70b-sf
8
+ parameters:
9
+ weight: 0.2
10
+ - model: /media/dylan/SanDisk/LLMs/miqu-evil-dpo/
11
+ parameters:
12
+ weight: 0.2
13
+ - model: /home/dylan/Documents/AI/merge/MiquMaid-v3-70B
14
+ parameters:
15
+ weight: 0.2
16
+ dtype: float16
17
+ tokenizer_source: model:/home/dylan/Documents/AI/merge/miqu-1-70b-sf
model-00001-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b0bc44cfc8b0caaa73ca9a20ebc72de37bfda26a3f4dea8f305b25a998e19d10
3
+ size 1988117048
model-00002-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4e7090501cae7b0b35ab2f8b37b3263a9c0394d8b0a6f1d35b76b77c03bac663
3
+ size 1711309856
model-00003-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:383ba84fba3bef358a09e7c612e7b4e644d3236de5aa57a66f0a7aad53d34b3a
3
+ size 1711309856
model-00004-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:68e7f4cce99af7dbf72bc4298773ba0f5436588ea526296ea4f656b3ed1f0b65
3
+ size 1711309864
model-00005-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:55a9dcddcc1cae39fa3ac33dbe8ec9239686fd407ad2d9e010ae499f1e42ee68
3
+ size 1711309864
model-00006-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ff8c88888900436c2076d9b2aacb17a6030cf297a2489a8f588759f509454bee
3
+ size 1711309864
model-00007-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2ce54d1596a6dcf8f9e1d2a134e01123a6cd32d9ca1c6766dda302418919c765
3
+ size 1711309864
model-00008-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e999a9d64877e486b37e65388490e76ebdcc2df5b032f039a47c2fa165200710
3
+ size 1711309864
model-00009-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0330654c6715a4cec27551583267969ba5e1305b36c9cfadfbd1c82e35d11e50
3
+ size 1711309864
model-00010-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:84012fea8f9a1c4b3a45c3a3de932d26e6e1b96a90acd38b64513e0aaf7e9562
3
+ size 1711309864
model-00011-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:98664e29bfeaa08b98e0e61b1cc31ea734ab686ee0562d1875ee6ac6052fa048
3
+ size 1711309864
model-00012-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0bb5df95096ab6451a76ada113d504d74b9d8388c5cbbbeca4031c5636ad362e
3
+ size 1711309864
model-00013-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b55b77aff5d211d4328372bed6167d0268af306f2475b4e76d32252c5281901e
3
+ size 1711309856
model-00014-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9faaa7bbf647c01bee079c5d5f9e663834cb276b9178ac2b2d8b8de8eedf9f17
3
+ size 1711309856
model-00015-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:146a105b230ba34852af7b33e9edb86f120634b21845bcbd686cd12cbf626d99
3
+ size 1711309864
model-00016-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d2db1c7f533c43a9d20c9fc3c97cdd8a8f7ee13bd9ab26b2e3ab80f49615641e
3
+ size 1711309864
model-00017-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ae60ff350d4ce02d68f2698b78dd9b3db9860d46dc4e08fdb0bf95a7472eac72
3
+ size 1711309864
model-00018-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:35c24049577e17c8ae3441825d4bb51dd8803ddf31f062ab6b77138df1e8f542
3
+ size 1711309864
model-00019-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c3ee60d166c9932cef7433a6df20792be146ec1941807a79348204251333a0af
3
+ size 1711309864
model-00020-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0aa07a5a8fd60b7c074e585927866b50cf5612e8d4bed73165008fa0e36359c3
3
+ size 1711309864
model-00021-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:98ded078f151dc2eac615214d1f563f31596bab89ff0bc1680adb01f9e79eba2
3
+ size 1711309864
model-00022-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9bc562141c958ea526ec5e7f301fa7c2968b5c48886a94cc7a6d2b77d72ef18f
3
+ size 1711309864
model-00023-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d10317c4bc2a62281b0ae5aad373e8cdd73261ce43d05f3eed61b3240bcf19a4
3
+ size 1711309864
model-00024-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d8bb3c7ef05bec8a8d1fba6e3c09778b961a3ab4cd5b2d689e14193fcba73ead
3
+ size 1711309856
model-00025-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0a1f406ffa2a2e346c04364aea3f155a7f75212d86e12163bfe4dda320accc9c
3
+ size 1711309856
model-00026-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ade9d8dfecc1433fbb19dffb88c400ccd34f42f213bdc99accc757abea979ed0
3
+ size 1711309864
model-00027-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c0f5800913053850ee793e8895124187f3d540cf9deb5965d38006680f500468
3
+ size 1711309864
model-00028-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:686915f31b6f4c898f20faa2e5d5ccfbf2d209a7848cd048449302ab7d72f329
3
+ size 1711309864
model-00029-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:478acd77b84b6ffb3bcc9002910034b578db2a44e8286d405bdc956db82ce4c1
3
+ size 1711309864
model-00030-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:46fad18b4b17bba82306de14da6b26825ba9713ccc343531840058458d8a6505
3
+ size 1711309864
model-00031-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:219158cf9bbaaa4a4ece380f49015428223cc54198b41cc37f8cb809a66e7d11
3
+ size 1711309864
model-00032-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8b85ce2801bf7a10559b2d3617c6e01252e71c1a0fd9a5aa757d4faca91c77ec
3
+ size 1711309864
model-00033-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b2a6df7cb60cfb7931b66ba79feb8fa3d26bfca470af68d5105827ed64227047
3
+ size 1711309864
model-00034-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:43ffd2f008138380cf55cbcc4740aa7cf28f667ae2dc3bfcd0ef87d0c38f25e0
3
+ size 1711309864
model-00035-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:44dca8c133b27ab59ebe6c2bde9b138e538ee66e3e79b6cae1511949e5415730
3
+ size 1711309856
model-00036-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:93e3ffe10a51fa17f1d0b31db8924d5554fe4b134542c484ab18f3763f0bbecd
3
+ size 1711309856
model-00037-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4650f6dbc9badc41bc5756584b9a084da2a23565bd463e741339f61ad045e759
3
+ size 1711309864
model-00038-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a2dbb3250ddc45643b62ae3bf2406206417f68fbf28f435f74addcac36ba5828
3
+ size 1711309864
model-00039-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:beb9d845096cb973ff27d48ce148435c5940aeb20337b8d7612e455a3cb3f145
3
+ size 1711309864
model-00040-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f7fdaa14f8b6a3f8fccc017215fc57b4aa3375f8e4f29c55bd0496421ee3cbe8
3
+ size 1711309864
model-00041-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a94295f40e3a93083ab11756fc0c53a49fec1160d32ecb72fd58ea9b0c848f38
3
+ size 1711309864
model-00042-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:33c6217194ea34ef04fad00a2d760833646d8d6507f355b6bd2556123a8059f6
3
+ size 1711309864
model-00043-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2a7083d206465589e0b81ac467250588e1ef6bdca905c4cfedf177d6e6a29de0
3
+ size 1711309864
model-00044-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c8df90c99310dd115e18b0ca9c09f2265d4d07b31827f7180dddf02620054a34
3
+ size 1711309864
model-00045-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d78e1fc039a804cf08ab9fbc9f3fcf5990ed1b1a6523122f56eea372cc7a502d
3
+ size 1711309864
model-00046-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:06dddee03932d104e2332258379749b8736b44419447ac24f39783ce9f36c55c
3
+ size 1711309856
model-00047-of-00081.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7f483db92e40ef7b2bdc3ae5275e88cc0633f699cce3f377da33da46d4f96809
3
+ size 1711309856