Building the dataset that Somali language AI has been missing.
Somali is spoken by 25 million people across the Horn of Africa and the diaspora — but almost no AI language tools support it. We're assembling an open, high-quality text corpus and the modeling pipeline to train the first open Somali dialect classifier.
How much text we've collected, which sources it comes from, and what passed quality review.
Compiling latest ingestion snapshot…
Records per orchestrated run (latest 8 ingestion cycles)
Actual share vs target mix
Retention through collected → curated
Five sources, each with distinct licenses, acquisition methods, and quality profiles.
Mapping current corpus composition…
Building source readiness overview…
—
Counting active feeds…
—
Reviewing roadmap…
—
Compiling pipeline coverage…
—
Calculating variance…
Share of records at each collection stage per source
Relative volume by acquisition approach
Tracking latest orchestrations…
Daily quota usage and limits per source
Orchestration run history and statistics
| Run ID | Timestamp | Records | Quota Hits |
|---|
Filter by acquisition method or SLA to focus on workstreams.
| Source | Records | Share | Avg Length | Acquisition | Refresh SLA | Owner | Stage | Last Updated |
|---|
Upcoming sources and sunset decisions
We apply 5 filters to every collected record. Only text that passes all criteria enters the corpus.
Compiling retention story…
No runs yet
No runs yet
No runs yet
No runs yet
Filter families by source with share of rejections
Records collected to curated corpus, with quality filter annotations
Retention funnel updates after each run.
Ranked rejections with policy context
Rejections by filter reason
Track retention stability by run
Record-weighted pass rate over recent runs
Per-source stability across runs
Active alerts and recommended actions
| Severity | Source | Alert | Recommendation |
|---|---|---|---|
| Loading alerts… | |||
Active waivers and manual review queues
Data collection is complete. Here's the plan for building and releasing the dialect classifier.
25,959 records collected from 5 sources. Quality filters applied. Curated corpus ready for preprocessing.
Tokenization, train/validation/test splitting (80/10/10), and dialect label assignment for annotated samples.
Fine-tune XLM-R and multilingual BERT as dialect classification baselines. Target: weighted F1 ≥ 0.80 per dialect class (Northern, Southern, Central).
Per-dialect confusion matrices, error analysis on misclassified samples, and bias audit across data sources.
Model card, HuggingFace Hub release, citation guide, and full reproducibility documentation. All training code, data, and model weights open-sourced.
The corpus needs more text and the models need annotation. Here's where you can help:
The Somali Dialect Classifier is an open-source project that thrives on community contributions. Whether you're a developer, linguist, researcher, or simply passionate about Somali language technology, there are many ways to get involved and make an impact.
Share Somali text samples, articles, or documents to expand our corpus.
Contribute code to enhance data collection, processing, or quality checks.
Identify and integrate new high-quality Somali language data sources.
Use the corpus for research and share insights on Somali dialects.