AI-NLP / PUBLISHED

EN/FR ↔ Swahili NMT Dataset

Contribute language resources for machine translation to and from Swahili.

STACK
PythonpandasPyTorchHugging Face TransformersM2M100-418MGoogle Colab
STATUS

published

Parallel French, English and Swahili language data
01 / CONTEXT

Context

RHOpenLabs develops Kitabu, a platform that could benefit from a machine translation module between English, French and Swahili. However, every neural translation system depends on parallel corpora: texts aligned sentence by sentence in both languages, from which the model learns.

For African languages such as Swahili, these corpora are scarce, heterogeneous and of uneven quality: misaligned segments, mixed languages, duplicates. A model trained on such data reproduces its flaws. Before any model could be considered, reliable parallel data was therefore needed.

IMG-NMT-01 · CONTEXT

EN/FR ↔ Swahili NMT Dataset — context

Step 01 / 06 · Paused

Kitabu could integrate a translation module.

01 · SITUATION02 · IMPLICATIONS03 · FRICTION04 · NEEDKitabu platformRHOPENLABSNeural translationEN / FR ↔ SWParallel corporaALIGNED SENTENCESFlawed model⚠ REPRODUCES FLAWSUneven quality⚠ MISALIGNMENT · DUPLICATESReliable dataBEFORE THE MODEL
Paused6 nodes · 8 links · Hover a node
  • Actor / entity
  • Component / data
  • Friction point
  • Need
Hover or select a component to see its role and connections.
From the field to the need: the situation behind the project.
02 / PROBLEM

Problem

How can parallel data be selected, prepared and structured reliably enough to experiment with translation between English, French and Swahili?

03 / SOLUTION

Solution

A three-stage research effort: comparison of pretrained models and selection of M2M100-418M, assessment of available corpora, then data preparation and partial fine-tuning experiments. These experiments revealed data-quality issues incompatible with production use. In agreement with the company, the project was redirected towards building and publishing an EN/FR ↔ Swahili dataset on Hugging Face.

IMG-NMT-02 · SOLUTION

EN/FR ↔ Swahili NMT Dataset — solution

Step 01 / 06 · Paused

Scarce, uneven corpora.

01 · PROBLEM02 · BUILD03 · HOW IT WORKS04 · OUTCOMEUneven corpora⚠ SCARCEModel comparisonM2M100-418MPreparation pipelineFILTERSExperimentsPARTIAL FINE-TUNINGDataset EN/FR ↔ SWBUILTDocumented limits✓ NOT PRODUCTIONPublished on Hugging Face✓ REUSABLE
Paused7 nodes · 7 links · Hover a node
  • My contribution
  • Friction point
  • Outcome
Hover or select a component to see its role and connections.
Individual project: every component was designed and built by me.
04 / CONTRIBUTION

Contribution

Research internship at RHOpenLabs. I led the model comparison, built the data preparation pipeline, ran the M2M100-418M experiments, then assembled, documented and published the dataset.

05 / FEATURES

Pipeline steps

  • Languages and translation directions are defined.
  • Available corpora are explored and compared.
  • Formats are converted, notably from Moses to JSONL.
  • Each segment's language is identified and unsuitable segments are filtered out.
  • Data is cleaned and deduplicated, and alignment is checked.
  • The dataset serves as the basis for M2M100-418M experiments.
  • It is documented and published on Hugging Face.
06 / ARCHITECTURE

Architecture

Each source corpus is first inspected and converted into a common format. Its segments then pass through successive filters: language identification, cleaning, deduplication and alignment checks. Only pairs that pass every stage enter the final dataset, which feeds the experiments and is then published.

IMG-NMT-03 · ARCHITECTURE

EN/FR ↔ Swahili NMT Dataset — overall architecture

Step 01 / 07 · Paused

Corpora are inspected and converted.

01 · SOURCES02 · PREPARATION03 · FILTERING04 · DATASET05 · NEXTSource corporaMOSESConversionMOSES → JSONLFormat normalisationCOMMON STRUCTURELanguage IDFILTERINGCleaning · dedupDUPLICATESAlignment checksPAIRSDataset EN/FR ↔ SWJSONLM2M100-418MEXPERIMENTALHugging Face✓ PUBLICATION
Paused9 nodes · 8 links · Hover a node
  • Actor / entity
  • My contribution
  • Outcome
Hover or select a component to see its role and connections.
Individual project: every component was designed and built by me.
07 / STACK

Stack

Python, pandas, PyTorch, Hugging Face Transformers, M2M100-418M, Moses / JSONL, Google Colab.

08 / BENEFITS

Benefits

  • A reusable parallel resource enriches the Swahili ecosystem.
  • The project shows that data quality determines translation quality.
  • Experimental limits are documented honestly, without claiming a production-ready model.
  • The dataset paves the way for future work on African languages.
EDITORIAL NOTE

This case study distinguishes the overall project scope from my personal contribution. No undocumented result or metric is presented.