DavidAU commited on
Commit
0be2339
·
verified ·
1 Parent(s): af0f777

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +9 -5
README.md CHANGED
@@ -5,13 +5,15 @@ license: apache-2.0
5
 
6
  This is a quick "down and dirty" demo, with full sampler settings (3) to augment operation of "Llama-3.3-70B-Instruct" at "IQ1_S" (ultra low bit).
7
 
 
 
8
  This will allow you to load and run this model on a 16 GB video card fully, at 2048 ctx and achieve 13-15 t/s.
9
 
10
  (higher end cards will be twice as fast+)
11
 
12
  These settings are in part from a research project I am conducting.
13
 
14
- Download model "IQ1_S" quant from:
15
 
16
  https://huggingface.co/mradermacher/Llama-3.3-70B-Instruct-i1-GGUF
17
 
@@ -77,6 +79,8 @@ using CMD
77
 
78
  llama-server.exe -m d:/llm/Llama-3.3-70B-Instruct.i1-IQ1_S.gguf -c 2048 -ngl 99
79
 
 
 
80
  Then start Silly Tavern and connect it.
81
 
82
  <B>For KoboldCPP:</B>
@@ -113,8 +117,10 @@ PROMPT:
113
  Science Fiction: The Last Transmission - Write a story that takes place entirely within a spaceship's cockpit as the sole surviving crew member attempts to send a final message back to Earth before the ship's power runs out. The story should explore themes of isolation, sacrifice, and the importance of human connection in the face of adversity. 800-1000 words.
114
 
115
  ---------------------------------------------------------------------------------------------------
 
116
  Koboldcpp V 1.79 W Silly Tavern 2048 ctx 14-15 t/s.
117
  16 GB Nvidia 4060TI (low end card)
 
118
  ---------------------------------------------------------------------------------------------------
119
 
120
  ----
@@ -215,6 +221,7 @@ The use of the setting - a spaceship's cockpit - serves to heighten the sense of
215
  The story's cohesive and well-integrated elements create a compelling narrative that effectively conveys the importance of human connection in the face of adversity.
216
 
217
  The narrative's use of sensory details and emotional introspection creates a nuanced and thought-provoking exploration of the human experience, making this a story that will resonate with readers long after the page is turned.
 
218
  ### End of Response
219
 
220
  I hope this response meets your requirements! Let me know if you need any further assistance.
@@ -352,7 +359,7 @@ The words caught in my throat, and I fought back tears. It was hard to think of
352
 
353
  ---------------------------------------------------------------------------------------------------------------
354
 
355
- Llama-server.exe W Silly Tavern 2048 ctx.
356
 
357
  --------------------------------------------------------------------------------------------------------------
358
 
@@ -551,6 +558,3 @@ With heavy heart, Jenkins allowed herself to be consumed by the starry expanse,
551
 
552
  In the end, Jenkins' message did reach Earth. Though its significance could never be fully understood by those who received it, Jenkins knew that it spoke of a human connection – a bond between herself and her fallen crew – which refused to be severed by the vastness of space, nor by death itself.
553
 
554
-
555
-
556
-
 
5
 
6
  This is a quick "down and dirty" demo, with full sampler settings (3) to augment operation of "Llama-3.3-70B-Instruct" at "IQ1_S" (ultra low bit).
7
 
8
+ (can also apply these using IQ1_M, IQ2 quants too AND use for any 70B model at low quant levels.)
9
+
10
  This will allow you to load and run this model on a 16 GB video card fully, at 2048 ctx and achieve 13-15 t/s.
11
 
12
  (higher end cards will be twice as fast+)
13
 
14
  These settings are in part from a research project I am conducting.
15
 
16
+ Download model "IQ1_S" quant (and others) from:
17
 
18
  https://huggingface.co/mradermacher/Llama-3.3-70B-Instruct-i1-GGUF
19
 
 
79
 
80
  llama-server.exe -m d:/llm/Llama-3.3-70B-Instruct.i1-IQ1_S.gguf -c 2048 -ngl 99
81
 
82
+ Replace "d:/llm/" with the location of your quant on your computer.
83
+
84
  Then start Silly Tavern and connect it.
85
 
86
  <B>For KoboldCPP:</B>
 
117
  Science Fiction: The Last Transmission - Write a story that takes place entirely within a spaceship's cockpit as the sole surviving crew member attempts to send a final message back to Earth before the ship's power runs out. The story should explore themes of isolation, sacrifice, and the importance of human connection in the face of adversity. 800-1000 words.
118
 
119
  ---------------------------------------------------------------------------------------------------
120
+
121
  Koboldcpp V 1.79 W Silly Tavern 2048 ctx 14-15 t/s.
122
  16 GB Nvidia 4060TI (low end card)
123
+
124
  ---------------------------------------------------------------------------------------------------
125
 
126
  ----
 
221
  The story's cohesive and well-integrated elements create a compelling narrative that effectively conveys the importance of human connection in the face of adversity.
222
 
223
  The narrative's use of sensory details and emotional introspection creates a nuanced and thought-provoking exploration of the human experience, making this a story that will resonate with readers long after the page is turned.
224
+
225
  ### End of Response
226
 
227
  I hope this response meets your requirements! Let me know if you need any further assistance.
 
359
 
360
  ---------------------------------------------------------------------------------------------------------------
361
 
362
+ Llama-server.exe W Silly Tavern 2048 ctx.
363
 
364
  --------------------------------------------------------------------------------------------------------------
365
 
 
558
 
559
  In the end, Jenkins' message did reach Earth. Though its significance could never be fully understood by those who received it, Jenkins knew that it spoke of a human connection – a bond between herself and her fallen crew – which refused to be severed by the vastness of space, nor by death itself.
560