Update README.md
Browse files
README.md
CHANGED
@@ -54,12 +54,34 @@ cae7b4ee8d1ad4e4402632bb0600cc17 ./tokenizer_config.json.ef7ef410b9b909949e96f1
|
|
54 |
848005d07146c31e73a10020b3a3099a ./tokenizer.model.9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347.enc
|
55 |
```
|
56 |
|
57 |
-
2. Decrypt the files using https://github.com/LianjiaTech/BELLE/tree/main/models
|
|
|
|
|
|
|
|
|
|
|
58 |
```
|
59 |
for f in "encrypted"/*; do if [ -f "$f" ]; then python3 decrypt.py "$f" "original/7B/consolidated.00.pth" "result/"; fi; done
|
60 |
```
|
61 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
62 |
3. Check md5sum
|
|
|
|
|
63 |
```
|
64 |
md5sum ./*
|
65 |
0fa6ff8379308d40f090878593f085a9 ./config.json
|
|
|
54 |
848005d07146c31e73a10020b3a3099a ./tokenizer.model.9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347.enc
|
55 |
```
|
56 |
|
57 |
+
2. Decrypt the files using the scripts in https://github.com/LianjiaTech/BELLE/tree/main/models
|
58 |
+
|
59 |
+
This command for conversion requires approximately 60 GB of RAM.
|
60 |
+
In case you have insufficient memory, you can create a sizeable swap file,
|
61 |
+
which permits the operating system to utilize the disk as virtual memory.
|
62 |
+
|
63 |
```
|
64 |
for f in "encrypted"/*; do if [ -f "$f" ]; then python3 decrypt.py "$f" "original/7B/consolidated.00.pth" "result/"; fi; done
|
65 |
```
|
66 |
|
67 |
+
After executing the aforementioned command, you will obtain the following files.
|
68 |
+
|
69 |
+
```
|
70 |
+
./config.json
|
71 |
+
./generation_config.json
|
72 |
+
./pytorch_model-00001-of-00003.bin
|
73 |
+
./pytorch_model-00002-of-00003.bin
|
74 |
+
./pytorch_model-00003-of-00003.bin
|
75 |
+
./pytorch_model.bin.index.json
|
76 |
+
./README.md
|
77 |
+
./special_tokens_map.json
|
78 |
+
./tokenizer_config.json
|
79 |
+
./tokenizer.model
|
80 |
+
```
|
81 |
+
|
82 |
3. Check md5sum
|
83 |
+
You can verify the integrity of these files by performing an MD5 checksum to ensure their complete recovery.
|
84 |
+
Here are the MD5 checksums for the relevant files:
|
85 |
```
|
86 |
md5sum ./*
|
87 |
0fa6ff8379308d40f090878593f085a9 ./config.json
|