An Open, End-to-End Somali Dialect ML Platform
The Somali NLP Initiative coordinates data acquisition, quality pipelines, annotation planning, modeling, and deployment pathways so researchers and partners can build dialect-aware language tools from a single open source foundation.
Low-resource languages like Somali face a critical barrier in NLP: lack of quality training data. We built an automated, production-grade data pipeline to systematically collect, clean, and unify Somali text from the web's most reliable sources. This is the foundation for advancing Somali language AI and making dialect classification accessible to researchers worldwide.
Through systematic crawling and API integration, we're assembling an open Somali language corpus, providing researchers with access to diverse linguistic data.
Our robust pipeline architecture ensures high-quality data through intelligent filtering, validation, and source-specific quality checks across all pipeline types.
By combining encyclopedic knowledge, journalism, ML datasets, and academic corpora, we ensure comprehensive coverage of Somali dialects, registers, and domains.
Released under an open license, this corpus democratizes access to Somali NLP research, enabling global collaboration and accelerating innovation for low-resource languages.
Compiling latest ingestion snapshot…
Records per orchestrated run (latest 8 ingestion cycles)
Actual share vs target mix
Retention through discovery → silver dataset
Mapping current corpus composition…
Building ETL readiness overview…
—
Counting active feeds…
—
Reviewing roadmap…
—
Compiling pipeline coverage…
—
Calculating variance…
Share of records at Discovery → Extraction → Silver per source
Relative volume by acquisition approach
Tracking latest orchestrations…
Daily quota usage and limits per source
Orchestration run history and statistics
| Run ID | Timestamp | Records | Quota Hits |
|---|
Filter by acquisition method or SLA to focus on workstreams.
| Source | Records | Share | Avg Length | Acquisition | Refresh SLA | Owner | Stage | Last Updated |
|---|
Upcoming sources and sunset decisions
Compiling retention story…
Awaiting data.
Awaiting data.
Awaiting data.
Awaiting data.
Filter families by source with share of rejections
Discovery to Silver with guardrail annotations
Retention funnel updates after each run.
Ranked rejections with policy context
Rejections by filter reason
Track retention stability by run
Record-weighted pass rate over recent runs
Per-source stability across runs
Active alerts and recommended actions
| Severity | Source | Alert | Recommendation |
|---|---|---|---|
| Loading alerts… | |||
Active waivers and manual review queues
Records processed per minute over the last 10 runs with 7-day moving average
Percentage of records passing validation checks per data source
Time spent across discovery, fetch, extract, quality, and write stages
Breakdown of time spent in each pipeline stage for the latest run
Live throughput vs SLA for each ingestion source
Recent orchestrations with duration, throughput, and retries
Duration and retry status for the last 10 pipeline executions
Sources vs error types to spotlight hotspots
Worker concurrency, queue depth, and bandwidth
Operator actions required for upcoming runs
| Severity | Scope | Alert | Recommendation |
|---|---|---|---|
| Loading alerts… | |||
Known throttles and operational notes
Access technical reports, data schemas, API documentation, and export options.
Complete metrics data including success rates, processing times, and quality statistics for all sources.
Download JSON →Comprehensive schema documentation for the Somali dialect corpus including field definitions and data types.
View Schema →REST API endpoints for programmatic access to the corpus, including authentication and rate limits.
View API Docs →Download the corpus in multiple formats: CSV, JSON, Parquet, or HuggingFace Datasets format.
Export Options →Step-by-step guide for loading and using the corpus in popular ML frameworks like PyTorch and TensorFlow.
Read Guide →Academic citation information for referencing this corpus in research papers and publications.
Copy Citation →The Somali Dialect Classifier is an open-source project that thrives on community contributions. Whether you're a developer, linguist, researcher, or simply passionate about Somali language technology, there are many ways to get involved and make an impact.
Share Somali text samples, articles, or documents to expand our corpus.
Contribute code to enhance data collection, processing, or quality checks.
Identify and integrate new high-quality Somali language data sources.
Use the corpus for research and share insights on Somali dialects.