igorsterner/AnE-NER

Multilingual Identification of English Code-Switching

AnE-NER (Any-English Code-Switching Named Entity Recognition) is a token-level model for detecting named entities in code-switching texts. It classifies words into two classes: I (inside a named entity) and O (outside a named entity). The model shows strong performance on both languages seen and unseen in the training data.

Usage

You can use AnE-NER with Huggingface’s pipeline or AutoModelForTokenClassification.

Let's try the following example (taken from this paper)

input = "My Facebook, Ig & Twitter is hellaa dead yall Jk soy yo que has no life!"

Pipeline

from transformers import pipeline
classifier = pipeline("token-classification", model="igorsterner/AnE-NER", aggregation_strategy="simple")
result = classifier(input)

which returns

[{'entity_group': 'I',
  'score': 0.95482016,
  'word': 'Facebook',
  'start': 3,
  'end': 11},
 {'entity_group': 'I',
  'score': 0.9638739,
  'word': 'Ig',
  'start': 13,
  'end': 15},
 {'entity_group': 'I',
  'score': 0.98207414,
  'word': 'Twitter',
  'start': 18,
  'end': 25}]

Advanced

If your input is already word-tokenized, and you want the corresponding word NER labels, you can try the following strategy

import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer

lid_model_name = "igorsterner/AnE-NER"
lid_tokenizer = AutoTokenizer.from_pretrained(lid_model_name)
lid_model = AutoModelForTokenClassification.from_pretrained(lid_model_name)

word_tokens = ['My', 'Facebook', ',', 'Ig', '&', 'Twitter', 'is', 'hellaa', 'dead', 'yall', 'Jk', 'soy', 'yo', 'que', 'has', 'no', 'life', '!']

subword_inputs = lid_tokenizer(
    word_tokens, truncation=True, is_split_into_words=True, return_tensors="pt"
)

subword2word = subword_inputs.word_ids(batch_index=0)
logits = lid_model(**subword_inputs).logits
predictions = torch.argmax(logits, dim=2)

predicted_subword_labels = [lid_model.config.id2label[t.item()] for t in predictions[0]]
predicted_word_labels = [[] for _ in range(len(word_tokens))]

for idx, predicted_subword in enumerate(predicted_subword_labels):
    if subword2word[idx] is not None:
        predicted_word_labels[subword2word[idx]].append(predicted_subword)

def most_frequent(lst):
    return max(set(lst), key=lst.count) if lst else "Other"

predicted_word_labels = [most_frequent(sublist) for sublist in predicted_word_labels]

for token, label in zip(word_tokens, predicted_word_labels):
    print(f"{token}: {label}")

which returns

My: O
Facebook: I
,: O
Ig: I
&: O
Twitter: I
is: O
hellaa: O
dead: O
yall: O
Jk: O
soy: O
yo: O
que: O
has: O
no: O
life!: O

Word-level language labels

If you also want the language of each word, you can additionaly run AnE-LID. Checkout my evaluation scripts for examples of using both at the same time, as we did in the paper: https://github.com/igorsterner/AnE/tree/main/eval.

For the above example, you can get:

My: English
Facebook: NE.English
,: Other
Ig: NE.English
&: Other
Twitter: NE.English
is: English
hellaa: English
dead: English
yall: English
Jk: English
soy: notEnglish
yo: notEnglish
que: notEnglish
has: English
no: English
life: English
!: Other

Citation

Please consider citing my work if it helped you

@inproceedings{sterner-2024-multilingual,
    title = "Multilingual Identification of {E}nglish Code-Switching",
    author = "Sterner, Igor",
    editor = {Scherrer, Yves  and
      Jauhiainen, Tommi  and
      Ljube{\v{s}}i{\'c}, Nikola  and
      Zampieri, Marcos  and
      Nakov, Preslav  and
      Tiedemann, J{\"o}rg},
    booktitle = "Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024)",
    month = jun,
    year = "2024",
    address = "Mexico City, Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.vardial-1.14",
    doi = "10.18653/v1/2024.vardial-1.14",
    pages = "163--173",
    abstract = "Code-switching research depends on fine-grained language identification. In this work, we study existing corpora used to train token-level language identification systems. We aggregate these corpora with a consistent labelling scheme and train a system to identify English code-switching in multilingual text. We show that the system identifies code-switching in unseen language pairs with absolute measure 2.3-4.6{\%} better than language-pair-specific SoTA. We also analyse the correlation between typological similarity of the languages and difficulty in recognizing code-switching.",
}