Cour de Cassation automatic titrage prediction model

Model for the automatic prediction of titrages (keyword sequence) from sommaires (synthesis of legal cases). The models are described in this paper. If you use this model, please cite our research paper (see below).

Model description

The model is a transformer-base model trained on parallel data (sommaires-titrages) provided by the Cour de Cassation. The model was intially trained using the Fairseq toolkit, converted to HuggingFace and then fine-tuned on the original training data to smooth out minor differences that arose during the conversion process. Tokenisation is performed using a SentencePiece model, the BPE strategy and a vocab size of 8000.

Intended uses & limitations

This model is to be used to produce titrages for those sommaires that do not have them or to complement existing (manually) created titrages.

How to use

Model input is the matière (matter) concatenated to the text from the sommaire separated by the token <t>. Each example should be on a single line. E.g. bail <t> La recommendation du tribunal selon l'article... (fictive example for illustrative purposes. The maximum input length of the model is 1024 input tokens (after tokenisation).

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokeniser = AutoTokenizer.from_pretrained("rbawden/CCASS-auto-titrages-base")
model = AutoModelForSeq2SeqLM.from_pretrained("rbawden/CCASS-auto-titrages-base")

matiere = "matter"
sommaire = "full text from the sommaire on a single line"
inputs = tokeniser([matiere + " <t> " + sommaire], return_tensors='pt')
outputs = model.generate(inputs['input_ids'])
tokeniser.batch_decode(outputs, skip_special_tokens=True, clean_up_tokenisation_spaces=True)

Limitations and bias

The models' predictions should not be taken as ground-truth titrages and should always be indicated as being automatically generated. They were designed not to be used as such, but to improve search coverage for improved similarity prediction between different cases (the predicted titrages being used to predict the similarity).

The model is not constrained to predict titres that have previously been seen, so this should be taken into account in the deployment of this model as a titrage tool in order to avoid the multiplication of different titres.

Training data

Training data is provided by the Cour de Cassation (the original source being Jurinet data, but with pseudo-anonymisation applied). For training, we use a total of 159,836 parallel examples (each example is a sommaire-titrage pair). Our development data consists of 1,833 held-out examples.

Training procedure

Preprocessing

We use SentencePiece, the BPE strategy and a joint vocabulary of 8000 tokens. This model was converted into the HuggingFace format and integrates a number of normalisation processes (e.g. removing double doubles, apostrophes and quotes, normalisation of different accent formats, lowercasing).

Training

The model was initialised trained using Fairseq until convergence on the development set (according to our customised weighted accuracy measure - please see the paper for more details). The model was then converted to HuggingFace and training continued to smooth out incoherences introduced during the conversion procedure (incompatibilities in the way the SentencePiece and NMT vocabularies are defined, linked to HuggingFace vocabularies being necessarily the same as the tokeniser vocabulary, a constraint that is not imposed in Fairseq).

Evaluation results

Full results for the initial Fairseq models can be found in the paper.

Results on this converted model coming soon!

BibTex entry and citation info

If you use this work, please cite the following article:

Thibault Charmet, Inès Cherichi, Matthieu Allain, Urszula Czerwinska, Amaury Fouret, Benoît Sagot and Rachel Bawden, 2022. Complex Labelling and Similarity Prediction in Legal Texts: Automatic Analysis of France’s Court of Cassation Rulings. In Proceedings of the 13th Language Resources and Evaluation Conference, Marseille, France.]

@inproceedings{charmet-et-al-2022-complex,
  tite = {Complex Labelling and Similarity Prediction in Legal Texts: Automatic Analysis of France’s Court of Cassation Rulings},
  author = {Charmet, Thibault and Cherichi, Inès and Allain, Matthieu and Czerwinska, Urszula and Fouret, Amaury, and Sagot, Benoît and Bawden, Rachel},
  booktitle = {Proceedings of the 13th Language Resources and Evaluation Conference},
  year = {2022},
  address = {Marseille, France},
  pages = {4754--4766},
  url = {http://www.lrec-conf.org/proceedings/lrec2022/pdf/2022.lrec-1.509.pdf}