Update README.md
Browse files
README.md
CHANGED
@@ -5,13 +5,15 @@ license: apache-2.0
|
|
5 |
|
6 |
This is a quick "down and dirty" demo, with full sampler settings (3) to augment operation of "Llama-3.3-70B-Instruct" at "IQ1_S" (ultra low bit).
|
7 |
|
|
|
|
|
8 |
This will allow you to load and run this model on a 16 GB video card fully, at 2048 ctx and achieve 13-15 t/s.
|
9 |
|
10 |
(higher end cards will be twice as fast+)
|
11 |
|
12 |
These settings are in part from a research project I am conducting.
|
13 |
|
14 |
-
Download model "IQ1_S" quant from:
|
15 |
|
16 |
https://huggingface.co/mradermacher/Llama-3.3-70B-Instruct-i1-GGUF
|
17 |
|
@@ -77,6 +79,8 @@ using CMD
|
|
77 |
|
78 |
llama-server.exe -m d:/llm/Llama-3.3-70B-Instruct.i1-IQ1_S.gguf -c 2048 -ngl 99
|
79 |
|
|
|
|
|
80 |
Then start Silly Tavern and connect it.
|
81 |
|
82 |
<B>For KoboldCPP:</B>
|
@@ -113,8 +117,10 @@ PROMPT:
|
|
113 |
Science Fiction: The Last Transmission - Write a story that takes place entirely within a spaceship's cockpit as the sole surviving crew member attempts to send a final message back to Earth before the ship's power runs out. The story should explore themes of isolation, sacrifice, and the importance of human connection in the face of adversity. 800-1000 words.
|
114 |
|
115 |
---------------------------------------------------------------------------------------------------
|
|
|
116 |
Koboldcpp V 1.79 W Silly Tavern 2048 ctx 14-15 t/s.
|
117 |
16 GB Nvidia 4060TI (low end card)
|
|
|
118 |
---------------------------------------------------------------------------------------------------
|
119 |
|
120 |
----
|
@@ -215,6 +221,7 @@ The use of the setting - a spaceship's cockpit - serves to heighten the sense of
|
|
215 |
The story's cohesive and well-integrated elements create a compelling narrative that effectively conveys the importance of human connection in the face of adversity.
|
216 |
|
217 |
The narrative's use of sensory details and emotional introspection creates a nuanced and thought-provoking exploration of the human experience, making this a story that will resonate with readers long after the page is turned.
|
|
|
218 |
### End of Response
|
219 |
|
220 |
I hope this response meets your requirements! Let me know if you need any further assistance.
|
@@ -352,7 +359,7 @@ The words caught in my throat, and I fought back tears. It was hard to think of
|
|
352 |
|
353 |
---------------------------------------------------------------------------------------------------------------
|
354 |
|
355 |
-
Llama-server.exe
|
356 |
|
357 |
--------------------------------------------------------------------------------------------------------------
|
358 |
|
@@ -551,6 +558,3 @@ With heavy heart, Jenkins allowed herself to be consumed by the starry expanse,
|
|
551 |
|
552 |
In the end, Jenkins' message did reach Earth. Though its significance could never be fully understood by those who received it, Jenkins knew that it spoke of a human connection – a bond between herself and her fallen crew – which refused to be severed by the vastness of space, nor by death itself.
|
553 |
|
554 |
-
|
555 |
-
|
556 |
-
|
|
|
5 |
|
6 |
This is a quick "down and dirty" demo, with full sampler settings (3) to augment operation of "Llama-3.3-70B-Instruct" at "IQ1_S" (ultra low bit).
|
7 |
|
8 |
+
(can also apply these using IQ1_M, IQ2 quants too AND use for any 70B model at low quant levels.)
|
9 |
+
|
10 |
This will allow you to load and run this model on a 16 GB video card fully, at 2048 ctx and achieve 13-15 t/s.
|
11 |
|
12 |
(higher end cards will be twice as fast+)
|
13 |
|
14 |
These settings are in part from a research project I am conducting.
|
15 |
|
16 |
+
Download model "IQ1_S" quant (and others) from:
|
17 |
|
18 |
https://huggingface.co/mradermacher/Llama-3.3-70B-Instruct-i1-GGUF
|
19 |
|
|
|
79 |
|
80 |
llama-server.exe -m d:/llm/Llama-3.3-70B-Instruct.i1-IQ1_S.gguf -c 2048 -ngl 99
|
81 |
|
82 |
+
Replace "d:/llm/" with the location of your quant on your computer.
|
83 |
+
|
84 |
Then start Silly Tavern and connect it.
|
85 |
|
86 |
<B>For KoboldCPP:</B>
|
|
|
117 |
Science Fiction: The Last Transmission - Write a story that takes place entirely within a spaceship's cockpit as the sole surviving crew member attempts to send a final message back to Earth before the ship's power runs out. The story should explore themes of isolation, sacrifice, and the importance of human connection in the face of adversity. 800-1000 words.
|
118 |
|
119 |
---------------------------------------------------------------------------------------------------
|
120 |
+
|
121 |
Koboldcpp V 1.79 W Silly Tavern 2048 ctx 14-15 t/s.
|
122 |
16 GB Nvidia 4060TI (low end card)
|
123 |
+
|
124 |
---------------------------------------------------------------------------------------------------
|
125 |
|
126 |
----
|
|
|
221 |
The story's cohesive and well-integrated elements create a compelling narrative that effectively conveys the importance of human connection in the face of adversity.
|
222 |
|
223 |
The narrative's use of sensory details and emotional introspection creates a nuanced and thought-provoking exploration of the human experience, making this a story that will resonate with readers long after the page is turned.
|
224 |
+
|
225 |
### End of Response
|
226 |
|
227 |
I hope this response meets your requirements! Let me know if you need any further assistance.
|
|
|
359 |
|
360 |
---------------------------------------------------------------------------------------------------------------
|
361 |
|
362 |
+
Llama-server.exe W Silly Tavern 2048 ctx.
|
363 |
|
364 |
--------------------------------------------------------------------------------------------------------------
|
365 |
|
|
|
558 |
|
559 |
In the end, Jenkins' message did reach Earth. Though its significance could never be fully understood by those who received it, Jenkins knew that it spoke of a human connection – a bond between herself and her fallen crew – which refused to be severed by the vastness of space, nor by death itself.
|
560 |
|
|
|
|
|
|