Use case
Indian Language Data for LLM Evaluation
Human evaluation of large language model output in Indian languages, including cultural and factual fit.
- Primary metric
- Rubric scores with confidence intervals
- Data shape
- Native-speaker rater panels per language
- Languages
- 14 + Indian English
What the data has to look like
- Native-speaker rater panels per language
- Rubric-based scoring with calibration
- Overlapping assignments for agreement
How the result is measured
- Rubric scores with confidence intervals
- Inter-rater agreement
- Failure-mode distribution
Where these projects go wrong
- Raters who are fluent but not native in the variety
- Rubrics written in English and applied to non-English output without localisation
Each of these is a defect that only becomes visible after training, when re-collection costs a release cycle.
How we scope it
A llm evaluation programme starts from the metric you need to move, not from an hour count. We work backwards: target metric, evaluation set design, then the training volume and speaker spread needed to reach it.
That means the evaluation set is specified and collected first, from speakers who never appear in the training data.
The numbers we hold ourselves to
- 100% of delivered files pass automated technical QA for SNR, clipping, duration and silence
- 5-25% of files pass a second native-speaker content review, stratified by city, dialect and transcriber, and escalating to 100% on any batch that fails the agreed threshold
- Accepted yield runs 85-90% for scripted speech, 60-70% for spontaneous, 55-65% for conversational and 50-60% for telephony
- Default cohort quotas: 50/50 gender, with age bands at 30% (18-25), 40% (26-40) and 30% (41-60)
- 48 kHz / 24-bit capture, delivered as 16-bit PCM WAV, with studio sessions held below a -50 dBFS noise floor
- First response within one working day; a scoped, fixed quote within two to three
These are the figures a delivery is measured against, not aspirations. A batch that misses them is re-recorded at our cost rather than repaired.
Frequently asked
How much data does llm evaluation need?
It depends on whether you are training from scratch or adapting a base model. Adaptation typically needs a tenth of the volume, but needs tighter matching to your deployment conditions.
Can you build the evaluation set too?
Yes, and it should be collected from disjoint speakers before training data collection finishes, so you can measure improvement rather than memorisation.
Which languages do you support for this?
All 14 languages in the network plus Indian English accent bands. Multi-language programmes run to one shared specification so results are comparable.
Scope a llm evaluation dataset
Tell us the metric you need to move and the languages in scope.