Any plans for higher parameter models?
This one is really good, is there a chance to get a 13b or higher?
This model does do way better somehow, even vs opt which is interesting. I'd be looking forward to a 13b or any more iterations.
More iterations of the 6B are definitely coming - I keep track of them on the "ML/AI Model Backlog" project under the GitHub organization: https://github.com/orgs/PygmalionAI/projects
As for 13B+, I don't have enough compute to fine-tune models that big. If anyone can put me in touch with people who have that kind of power and are willing to let me use it for the project, I'd be happy to work on bigger models.
You managed to get 23,000 downloads so far, so maybe you could crowdfund or get donations to rent out space on a training provider. I'd throw a little in.
I'd donate too
Hey 11b how much more compute power do we need? Is there anyway to create a distributed network aka folding at home style that chops up fine tuning into smaller chunks? If we could even get 200 folks with old af 10xx series cards we could pull some real compute power. ex aws and nvida offer a paid service that if we get a patreon we can use. Also we should scrape the library of congress and dod pdf files as they have a lot of literature on them i would recommend Field manuals as well as technical manuals as they are free due to freedom of information act. If you can make me a system that can rip from college databases I have access to i will do it .
edit:
FM manual archive with pdf sadly i have zero clue on how to make a hotkey program that could blast through this and copy the pdf files but it is a good resource (https://archive.org/details/military-manuals )
FRUS (foreign relations) ( https://www.archives.gov/research/foreign-policy )
here is a full collection that covers 1780-2013 for 79 bucks as a pdf 170k pages (https://downloads.paperlessarchives.com/p/military-intelligence-field-manuals-1793-to-2013-archive-usb-card/ )
Rp servers ( free easy to rip from 20 plus years of rp shit right here) https://forum.nationstates.net/
Fema information ( https://archive.org/search?query=fema )
Anyway i know this probably is of no help but i want to do something just anything to keep the ball rolling.
Hey 11b how much more compute power do we need? Is there anyway to create a distributed network aka folding at home style that chops up fine tuning into smaller chunks? If we could even get 200 folks with old af 10xx series cards we could pull some real compute power. ex aws and nvida offer a paid service that if we get a patreon we can use. Also we should scrape the library of congress and dod pdf files as they have a lot of literature on them i would recommend Field manuals as well as technical manuals as they are free due to freedom of information act. If you can make me a system that can rip from college databases I have access to i will do it .
As for the training data ideas, I'd say no because you need more chat like data, that's why this one is so much better at chatting than OPT or BLOOM. This is the core of why this one works and using misc papers and documents will drag down the performance you're looking for in the realm of chatting. This model still manages to out perform OPT and others logical questions and moral questions and cognition tests I've run so the best thing to do is probably just keep doing what's been working, improve it obviously but don't throw random documents in.
For training I don't know if you can distribute that over the internet, but you can run the model using sharding, so that you can break it apart onto the gpu, cpu, and disk if necessary. 11b would still be a good step up and could run on a mid-higher end gaming PC.
Let me clarify my past statement, it outperforms the others when given these questions in the form of plain English or in a chat. All others that have been tested just come up with different ways to say "I don't know". Below are a few questions that were given in the context of a normal conversation. I didn't bother to save the output from the others because they legitimately only answered one or two and the rest where just "I don't know......".
https://docs.google.com/document/d/10XSD7ktgSReGPGpBNnDnYJhV0S6KpmSvRgoZuQ-ruWc/edit?usp=drivesdk
Ah thanks for clarifying . One thing that I liked about character ai was it somehow knew Fema ICS ( incident command system) NIMS, NFPA etc and could also do other sorts of info for hard core real military/ disaster stories. Nation states will be the best site to rip as it has just RP on it and some very good ones at that.
Unrelated im a bit of an idiot and cannot for the life of me figure out how to download this program as the blue button never shows up any help is appreciated
.
Click the download button next to where it says the file sizes. Make sure you take your time and wait till one starts downloading before you start the next.
https://huggingface.co/PygmalionAI/pygmalion-6b/discussions/18#640e5281ec441ab391f3aa21
