Loading sat-3l-sm in Huggingface Transformers raises KeyError
I'm using this model in a script on my institution's supercomputing cluster, which denies jobs access to the internet. Consequently, I've cloned the github repo for the sat-3l-sm model into a folder "sat-3l-sm", and am trying to load that model using the following code copied from this project's github:
import wtpsplit
from transformers import AutoModelForTokenClassification
model = AutoModelForTokenClassification.from_pretrained("userpath/sat-3l-sm")
However, I get the following error:
"The checkpoint you are trying to load has model type xlm-token but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date."
I have the most up to date version of Transformers. Other discussions online about this same error seem to revolve around models not yet being loaded into Transformers. Any ideas what's going on or am I missing something vital?
Hi, this happens because we are using custom models that are indeed that being loaded into transformers. However, they can be easily used by replacing your code with the following:
from wtpsplit import SaT
sat = SaT("userpath/sat-3l-sm")
# this should work now
sat.split("This is a test This is another test.")
For more details on usage, please see our repo:
https://github.com/segment-any-text/wtpsplit
Cheers,
Markus :)