LLM Evaluation · मराठी
Marathi Data for LLM Evaluation
Human evaluation of large language model output in Indian languages, including cultural and factual fit. In Marathi, the binding constraint is usually dialect coverage and code-mixing, not raw hours.
- Language
- Marathi
- Primary metric
- Rubric scores with confidence intervals
- Typical volume
- 500-2,000 hours
Data profile required
- Native-speaker rater panels per language
- Rubric-based scoring with calibration
- Overlapping assignments for agreement
What Marathi adds to the requirement
- Retains the retroflex lateral ळ, which has no Hindi or English equivalent and is frequently substituted with ल by non-native transcribers
- Affricates च and ज have both alveolar and palatal realisations depending on the word, a distinction lost in Devanagari orthography
- Dialects to cover: Standard (Puneri), Varhadi (Vidarbha), Marathwadi, Konkani-influenced coastal Marathi
- Mumbai and Pune speech mixes Marathi, Hindi, and English in the same sentence. Marathi-only recordings collected in Pune under-represent the Mumbai reality of tri-lingual switching.
Metrics to track
- Rubric scores with confidence intervals
- Inter-rater agreement
- Failure-mode distribution
Failure modes
- Raters who are fluent but not native in the variety
- Rubrics written in English and applied to non-English output without localisation
For Marathi specifically: Available Marathi speech data is dominated by standard Puneri read speech. Vidarbha, Marathwada, and coastal Konkan varieties are severely under-collected, which is exactly where deployed voice products lose accuracy.
Recommended cohort
A representative Marathi cohort should be split roughly 40% western Maharashtra, 25% Vidarbha, 20% Marathwada, 15% Konkan rather than concentrated in Pune.
| Dimension | Typical split | Why it matters for Marathi |
|---|---|---|
| Gender | 50 / 50 | Pitch range differences change acoustic model behaviour; unbalanced cohorts bias recognition |
| Age | 18-25: 30%, 26-40: 40%, 41-60: 30% | Older speakers retain conservative Marathi forms that younger urban speakers have lost |
| Region | Maharashtra / Goa / parts of Karnataka and others | Dialect spread across 6 recognised varieties |
| Education | Mixed, including below-graduate | Prompt-reading fluency correlates with education and skews prosody |
| Condition | Studio / quiet room / field | Match the noise profile of your deployment |
Suggested programme shape
Start with an evaluation set of 120 speakers spread across every Marathi dialect in scope, collected before training data. Then field 500-2,000 hours of training data from disjoint speakers.
This ordering is what makes the improvement measurable rather than assumed.
Frequently asked
Is there usable public Marathi data for llm evaluation?
Available Marathi speech data is dominated by standard Puneri read speech. Vidarbha, Marathwada, and coastal Konkan varieties are severely under-collected, which is exactly where deployed voice products lose accuracy.
How many Marathi speakers do we need?
1,000-3,000 speakers for a training corpus, plus a disjoint evaluation cohort covering each dialect. Speaker count matters more than hours for generalisation.
Can you run this across multiple languages at once?
Yes. Multi-language programmes run to one master specification so per-language results stay comparable.
Scope Marathi data for llm evaluation
Send the target metric and the languages in scope.