rbawden/CCASS-semi-auto-titrages-base

Cour de Cassation semi-automatic titrage prediction model

Model for the semi-automatic prediction of titrages (keyword sequence) from sommaires (synthesis of legal cases).

The models are similar to the automatic models described in this paper and to the model available here. If you use this semi-automatic model, please cite our research paper (see below).

Model description

The model is a transformer-base model trained on parallel data (sommaires-titrages) provided by the Cour de Cassation. The model was intially trained using the Fairseq toolkit, converted to HuggingFace and then fine-tuned on the original training data to smooth out minor differences that arose during the conversion process. Tokenisation is performed using a SentencePiece model, the BPE strategy and a vocab size of 8000.

Intended uses & limitations

This model is to be used to help in the production of titrages for those sommaires that do not have them or to complement existing (manually) created titrages.

How to use

Contrary to the automatic titrage prediction model (designed to predict the entire sequence), this model is designed to help in the manual production of titrages, by proposing the next titre (keyword) in the sequence given a sommaire and the beginning of the titrage.

Model input is the matière (matter) concatenated to the titres already decided on (separated by ), concatenated to the text from the sommaire separated by the token <t>. Each example should be on a single line. E.g. bail <t> résiliation <t> causes <t> La recommendation du tribunal selon l'article... (fictive example for illustrative purposes, where the matter=bail, the beginning of the titrage=résiliation causes. The maximum input length of the model is 1024 input tokens (after tokenisation).

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokeniser = AutoTokenizer.from_pretrained("rbawden/CCASS-semi-auto-titrages-base")
model = AutoModelForSeq2SeqLM.from_pretrained("rbawden/CCASS-semi-auto-titrages-base")

matiere_and_titrage_prefix = "matter <t> titre"
sommaire = "full text from the sommaire on a single line"
inputs = tokeniser([matiere_and_titrage_prefix + " <t> " + sommaire], return_tensors='pt')
outputs = model.generate(inputs['input_ids'])
tokeniser.batch_decode(outputs, skip_special_tokens=True, clean_up_tokenisation_spaces=True)

Limitations and bias

The models' predictions should not be taken as ground-truth titrages and the final decision should be the expert's. The model is not constrained to predict titres that have previously been seen, so this should be taken into account in the deployment of this model as a titrage tool in order to avoid the multiplication of different titres.

Training data

Training data is provided by the Cour de Cassation (the original source being Jurinet data, but with pseudo-anonymisation applied). For training, we use a total of 159,836 parallel examples (each example is a sommaire-titrage pair). Our development data consists of 1,833 held-out examples.

Training procedure

Preprocessing

We use SentencePiece, the BPE strategy and a joint vocabulary of 8000 tokens. This model was converted into the HuggingFace format and integrates a number of normalisation processes (e.g. removing double doubles, apostrophes and quotes, normalisation of different accent formats, lowercasing).

Training

The model was initialised trained using Fairseq until convergence on the development set (according to our customised weighted accuracy measure - please see the paper for more details). The model was then converted to HuggingFace and training continued to smooth out incoherences introduced during the conversion procedure (incompatibilities in the way the SentencePiece and NMT vocabularies are defined, linked to HuggingFace vocabularies being necessarily the same as the tokeniser vocabulary, a constraint that is not imposed in Fairseq).

Evaluation results

Full results for the initial (automatic) Fairseq models can be found in the paper.

Results on this semi-automatic model coming soon!

BibTex entry and citation info

If you use this work, please cite the following article:

Thibault Charmet, Inès Cherichi, Matthieu Allain, Urszula Czerwinska, Amaury Fouret, Benoît Sagot and Rachel Bawden, 2022. Complex Labelling and Similarity Prediction in Legal Texts: Automatic Analysis of France’s Court of Cassation Rulings. In Proceedings of the 13th Language Resources and Evaluation Conference, Marseille, France.]

@inproceedings{charmet-et-al-2022-complex,
  tite = {Complex Labelling and Similarity Prediction in Legal Texts: Automatic Analysis of France’s Court of Cassation Rulings},
  author = {Charmet, Thibault and Cherichi, Inès and Allain, Matthieu and Czerwinska, Urszula and Fouret, Amaury, and Sagot, Benoît and Bawden, Rachel},
  booktitle = {Proceedings of the 13th Language Resources and Evaluation Conference},
  year = {2022},
  address = {Marseille, France},
  pages = {4754--4766},
  url = {http://www.lrec-conf.org/proceedings/lrec2022/pdf/2022.lrec-1.509.pdf}