MAFAND-MT Edge Auditor
Projects — — Links: GitHub Repo — Paper (arXiv)
MAFAND-MT Edge Auditor is a deterministic, lightweight pipeline for auditing and sanitizing parallel corpora for low-resource African language translation. It is built for the Hausa subset of MAFAND-MT and runs locally, with no neural models or external APIs.
The Problem
Parallel corpora for low-resource languages are often scraped from the web, where truncated or incomplete translations are common. Standard filters miss content that is silently left out, such as a number or hashtag that appears in the source but not in the translation. Neural filters can catch more, but they are heavy to run and hard to inspect.
Key Contributions
- Three rule-based gates: language and script integrity, length-ratio validation, and numeric/hashtag alignment between source and target.
- Hausa-aware language ID: pairs with Hausa hooked characters (ƙ, ɗ, ɓ, ƴ) or common syntactic markers are protected from statistical misclassification.
- An audit ledger: the status of every sentence pair is logged to SQLite, so each decision can be inspected.
- Evaluated on real data: on the web-scraped Hausa split of MAFAND-MT (5,865 pairs) it ran in under a second on a consumer CPU and flagged 350 pairs (6.0%), all confirmed as true errors by manual review. In fine-tuning experiments with M2M-100, removing the flagged pairs gave small gains for English→Hausa and no meaningful change for Hausa→English.
Technology Used
- Languages and libraries: Python and pycld2.
- Storage: SQLite.
- Evaluation: M2M-100 (418M) fine-tuning, scored with BLEU and chrF++.
