One API call.
Real human ratings.
Plug human evaluation into your pipeline. Audio in, structured results out.
API-first · Built for ML pipelines
Plug human eval into your pipeline
Input
Output
{
"study_id": "std_8f3a...",
"status": "complete",
"responses": [{
"sample_id1": {
"naturalness": {
"values": [2, 4, 4]
},
"voice_match": {
"values": [4, 3, 4]
}
}
}]
}Study types
Three core study types
Designed around the questions voice and conversational AI teams actually ask.
Live conversations
Real human conversations with AI models. Participants hold live conversations with speech-to-speech systems; structured post-call feedback is collected automatically.
- Test experimental checkpoints with real users
- Gather rapid human feedback during development
- Iterate faster on conversational speech systems
- Benchmark and compare to other voice AI models
Single-sample rating
Participants evaluate individual audio clips using your custom rating questions.
- Display transcripts, tags, prompts, and speaker info alongside audio
- Free-response questions for human transcription, qualitative feedback, and error analysis
Multi-sample comparison
Participants compare audio samples and express preferences across custom dimensions.
Why our ratings are higher quality
High-quality ratings, by design
High-quality participants
Pre-screened raters from multiple trusted pools. Active fraud and AI-response detection.
Speed
Data delivered in hours, not days, closing the human eval loop at the pace your models move.
Unique incentive framework
Performance-based bonuses reward thoughtful, conscientious work, tuned per study and task type.
Fully managed ops
We handle payments, comms, support, and logistics. You never touch a crowd platform.
Global participant network
Many countries, languages, and demographics. Coverage expanding daily.
Subjective metrics
A few good places to start
These are subjective qualities that automated metrics can't capture. They need real human ears and real human judgment.
Does this voice sound like your ideal commentator?
Get feedback on how well a voice fits its intended use case: commentator, podcaster, teacher, assistant, and more.
Could you listen to this voice for hours and not get tired of it?
Ask participants how much they would enjoy listening to the voice at length, the kind of judgment only a human can make.
Built by experts in emotion science & voice AI
The same stack behind the world's most human-aligned TTS and speech-to-speech models
Proven in production
Powering evaluation for Hume's own frontier voice models, used daily across training, alignment, and release.
Frontier voice expertise
Built by the team behind Hume's emotionally intelligent voice models, deep expertise in speech quality, prosody, and emotional perception.
Customize or templates
Bring your own questions, or start from the expert-designed templates we use to evaluate our own state-of-the-art models.
Custom integrations, evaluations, and question types
Collaborate with us to start running human evaluations on your next models. Our research engineering team is ready to help.