Create README.md

README.md

@@ -0,0 +1,126 @@
+---
+language:
+- ja
+---
+
+## 本モデルについて about this model.
+[Qwen/Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct)を[日本語が多く含まれる重要度行列(iMatrix)](dahara1/imatrix-jpn-test)を使って量子化し、長文(128K)要約を可能にしたgguf版です。日本語対応能力が多めに保持されている事を期待しています。  
+This is a gguf version of [Qwen/Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct) that has been quantized using [importance matrix (iMatrix) that contains a lot of Japanese](dahara1/imatrix-jpn-test) to enable summarization of long texts (128K). We hope that it retains a large amount of Japanese support.
+
+少なくともQwen2.5-3B-Instruct-gguf-japanese-imatrix-128K/Qwen2.5-3B-Instruct-Q8_0-f16.ggufが32Kトークンを超える超長文を正しく要約できる事を確認済です。  
+It has been confirmed that at least Qwen2.5-3B-Instruct-gguf-japanese-imatrix-128K/Qwen2.5-3B-Instruct-Q8_0-f16.gguf can correctly summarize extremely long texts exceeding 32K tokens.  
+
+128Kコンテキスト延長については[unsloth/Qwen2.5-Coder-32B-Instruct-128K-GGUF]の指摘を参考にしています。ありがとう。  
+Regarding the 128K context extension, I have taken note of the suggestion made by [unsloth/Qwen2.5-Coder-32B-Instruct-128K-GGUF]. Thank you.  
+
+
+## For ollama users
+ollama ユーザーは[FAQ](https://github.com/ollama/ollama/blob/main/docs/faq.md)を参考にしてcontext window sizeパラメーターを修正してください。  
+If you use ollama, check [FAQ](https://github.com/ollama/ollama/blob/main/docs/faq.md) and set context window size parameter like below.
+
+```
+/set parameter num_ctx 40960
+```
+or API 
+```
+curl http://localhost:11434/api/generate -d '{
+  "model": "llama3.2",
+  "prompt": "Why is the sky blue?",
+  "options": {
+    "num_ctx": 40960
+  }
+}'
+```
+
+あなたが他のツールを使っている場合、同様にあなたの使っているツールのマニュアルを調べて、コンテキストウインドウサイズを延長する事を忘れないでください  
+ただし、コンテキストサイズを必要以上に大きくするとモデルの実行速度が低下するので注意してください  
+本モデルは理論上、最大値131072に設定できますが、実行速度と品質に影響が出る事が考えられます  
+
+If you are using other tools, be sure to extend the context window size as well, by consulting the manual of your tool.  
+But please note that increasing the context window size more than necessary will slow down the model's execution speed.  
+In theory, this model can be set to the maximum value of 131072, but this may affect execution speed and quality.  
+
+
+## Sample llama.cpp script
+
+以下は、Wikipediaの約50000文字の記事を取得して内容を要約するサンプルです  
+Below is a sample that retrieves a Wikipedia article of about 50,000 characters and summarizes its contents.  
+
+
+llama.cpp server command sample.
+```
+./llama.cpp/build/bin/Release/llama-server.exe -m ./Qwen2.5-3B-Instruct-Q8_0-f16.gguf -c 40960
+```
+
+
+llama.cpp client script sample.
+```
+import transformers
+import requests
+import json
+from transformers import AutoTokenizer
+tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")
+
+url = "https://ja.wikipedia.org/wiki/%E7%94%B7%E3%81%AE%E5%A8%98"
+
+def get_wikipedia_text(url):
+    response = requests.get(url)
+    if response.status_code == 200:
+        from bs4 import BeautifulSoup
+        soup = BeautifulSoup(response.text, 'html.parser')
+        paragraphs = soup.find_all('p')
+        text = "\n".join([p.get_text() for p in paragraphs])
+        return text
+    else:
+        raise Exception(f"Failed to fetch the article. Status code: {response.status_code}")
+        
+if __name__ == "__main__":
+
+    html_text = get_wikipedia_text(url)
+    #html_text = html_text[:40000]
+
+    instruct = "### 指示\n\n上記の文章を日本語で３行で要約してください"
+
+    # instruct first version
+    messages  = [
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": instruct + "\n\n" + html_text},
+    ]
+
+    # instruct last version
+    messages  = [
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": html_text  + "\n\n" + instruct},
+    ]
+    
+    prompt = tokenizer.apply_chat_template(
+            messages,
+            add_generation_prompt=True,
+            tokenize=False
+    )
+    print(prompt)
+
+    payload = {
+            "prompt": prompt,
+            "n_predict": 512
+    }
+
+    url = "http://localhost:8080/completion"
+    headers = {
+        "Content-Type": "application/json"
+    }
+
+    response = requests.post(url, headers=headers, data=json.dumps(payload))
+    if response.status_code != 200:
+        print(f"Error: {response.text}")
+
+    response_data = response.json()
+
+    response_content = response_data.get('content', '').strip()
+    print(response_content)
+```
+
+#### 出力結果(output sample)
+
+
+