Hume Audio Decision Benchmarks

Predicting a listening test in one API call

Before shipping a voice, codec, or speech pipeline, teams need to know how human the voice sounds. That usually means running a listening test: recruiting raters, collecting judgments, and waiting for results. Even a straightforward question like “Does this voice sound natural?” takes time and money to answer.

Our new decision model, Hume Audio Decision, estimates how people would judge an audio clip without needing a transcript. Send a clip and a question, and the model predicts the share of human raters who would choose each answer, such as each rating on a five-point naturalness scale. The output is a distribution over answers, showing how listeners split when they disagree.

In our tests on four public human-preference benchmarks (AudioMOS-AES, NISQA, SOMOS, and DataPoint), Hume Audio Decision scored highest across 11 models. The comparison includes Google’s newest general model, Gemini 4 Hydrogen, and Google’s decision model, Gemini Decision 1, which also returns a distribution of answers.

We also compared Hume Audio Decision with individual human raters. In six of seven comparisons, the model agreed with the rest of the crowd more closely than a single held-out rater did.

Hume Audio Decision had the lowest mean latency we measured, at 0.22 seconds per question, and it costs less than every model that scored 0.73 or higher overall. On general audio understanding, it comes within 0.043 of the leading model on MMSU and within 0.057 on MMAU, and it leads ParaPair and EmphAssess.

Comparing predictions with human ratings

Two examples show what the output looks like. In the first, raters judged speech quality. In the second, they judged whether a speaker emphasized one word. Each example compares the model’s predicted distribution with the raters’ votes, and the model outputs were recorded in advance.

Benchmark results

We tested Hume Audio Decision and ten other models on eleven public benchmarks in three groups: human preference, audio understanding, and tool calling. Every model received the same audio and questions. Most models return a single answer. Hume Audio Decision and Gemini Decision 1 return a distribution, and we score their top answer for accuracy.

Hume Audio Decision leads all four human-preference benchmarks, ParaPair, and EmphAssess. Gemini 4 Hydrogen leads MMSU, MMAR, and Auto-SLURP. Qwen3.8-Omni-Flash leads MMAU, and Gemini 3.8 Flash leads Audio2Tool.

On the overall score, the mean of the three group means, Hume Audio Decision ranks first at 0.800. Gemini 3.8 Flash follows at 0.770 and Gemini 4 Hydrogen at 0.755. Highlighted values mark the leader in each column.

Human preference

NISQA measures audio quality using mean opinion scores (MOS). SOMOS is a large-scale MOS dataset for text-to-speech (TTS), covering 200 systems built on one voice. DataPoint collects A/B judgments on modern TTS systems. AudioMOS-AES rates production complexity, production quality, content usefulness, and content enjoyment.

Hume Audio Decision leads all four, by margins from 0.025 on DataPoint to 0.179 on NISQA.

Understanding

The understanding benchmarks evaluate what is in the audio and how it is spoken. MMSU tests spoken-language understanding and reasoning. MMAU covers questions about speech, environmental sounds, and music, while MMAR tests deeper reasoning across those audio types. ParaPair compares speaking rate, emphasis, and style between clips, and EmphAssess tests whether particular words are emphasized.

Hume Audio Decision leads ParaPair and EmphAssess. It comes within 0.043 of the leader on MMSU and within 0.057 on MMAU. Gemini 4 Hydrogen leads MMSU and MMAR, and Qwen3.8-Omni-Flash leads MMAU.

Tool calling: more to come

We’re continuing to develop tool-calling capabilities and will share more soon. Current results are included in the overall benchmark table.

Comparisons with human raters

The benchmarks above compare models with one another. The next three tests compare Hume Audio Decision with people: a single held-out rater, panels of several raters, and the crowd’s agreement at each confidence level.

Hume Audio Decision vs. a single rater

We start with a single rater. We hold out one rater and measure how often that person agrees with the rest of the crowd. Then we measure Hume Audio Decision against the same crowd. We repeat this for every rater on every question, using public sets that publish each rater’s votes.

On DataPoint, a single held-out rater picks the crowd’s majority answer 56.7% of the time, and Hume Audio Decision picks it 68.6% of the time. SOMOS raters agree poorly with one another because ratings vary little between clips. The one axis where a single rater scores higher is AudioMOS-AES production complexity (0.738 vs. 0.435).

How many human raters match Hume Audio Decision?

Listening tests usually average several raters, so we also asked how many raters it takes to match Hume Audio Decision. We build panels of randomly selected raters, average each panel’s votes, and compare that average with the held-out crowd. Larger panels agree more closely. The panel size at which human agreement equals Hume Audio Decision’s is the model’s rater equivalent.

Across all axes, Hume Audio Decision matches about 2.0 raters on NISQA and 1.7 raters on AudioMOS-AES. On DataPoint and SOMOS, the model agrees with the crowd more closely than the largest panels we could build: 5 raters on DataPoint and 4 on SOMOS.

Results vary by axis. On NISQA loudness, Hume Audio Decision matches about 6.3 raters. On AudioMOS-AES production complexity, it matches fewer than one.

The panel sizes have a limit. Each dataset publishes a fixed number of votes per item, so the largest panel we can build is small. On DataPoint, the largest panel is 5 raters, so “more than 5” is as far as we can measure.

SOMOS clips have 8 to 19 votes, with a median of 11. We hold out 4 votes per clip and cap panels at 4 raters, so every clip appears at every panel size.

How confidence compares with crowd agreement

Because Hume Audio Decision returns a distribution, each answer comes with a confidence. That confidence is useful for routing requests or flagging clips for human review only if it is accurate. When the model says 75%, the crowd should agree about 75% of the time.

To check calibration, we group predictions by how confident the model is in its top answer. For each group, we measure how often the crowd’s top answer matches.

On Humeval dev, our internal held-out set of 6,849 questions, confidence and agreement line up. When Hume Audio Decision is 75% confident, the crowd agrees 74% of the time. In one confidence group, Qwen3-Omni-30B-A3B-Instruct, which also returns a distribution, is 97% confident but matches the crowd only 46% of the time.

Expected calibration error (ECE) is the gap between confidence and agreement, averaged over the groups and weighted by their size. Lower is better. The table compares ECE on our internal set and four public sets.

Cost and latency

We also measured latency and cost through each model’s API. Hume Audio Decision returns its distribution without generating text, and its mean latency was 0.22 seconds per question, the lowest we measured. Gemini Decision 1 was next at 0.42 seconds. The other models took 0.68 to 10.23 seconds, including 5.22 seconds for Gemini 4 Hydrogen.

At $0.28 per 1,000 questions, Hume Audio Decision costs less than every model that scores 0.73 or higher overall: Qwen3.8-Omni-Flash ($0.34), Gemini 3.8 Flash ($1.00), and Gemini 4 Hydrogen ($4.43). Gemini Decision 1 and Gemini 3.5 Flash Lite cost less, and both score lower on all four human-preference benchmarks.

Try it

Hume Audio Decision is available through the Hume API at $0.47 per million Hume tokens.

We are hiring.

FAQ

How does Hume Audio Decision differ from a mean opinion score (MOS) predictor? A MOS predictor learns one score for one kind of rating. Hume Audio Decision takes the question as input, so one model handles rating scales, A vs B comparisons, multiple-choice questions, and yes/no questions. On a single dataset with its own training split, a dedicated predictor can score higher.

Do you train on these benchmarks? DataPoint and AudioMOS-AES are used only for evaluation. NISQA and SOMOS test files are excluded from training.

Why does Hume Audio Decision predict a distribution instead of one answer? A distribution estimates the share of listeners who would choose each answer, including when preferences are closely divided.

How is calibration calculated for models that return only one answer? For API models that do not return logits, this evaluation assigns 100% probability to the predicted choice when calculating calibration error. Their ECE on the four public sets is 0.42 to 0.85.

Methodology

• Benchmarks. AudioMOS-AES (12,240 questions, 4 axes), NISQA test sets (3,560 questions, 5 quality dimensions), SOMOS clean test (2,047 clips), and DataPoint naturalness (6,300 pairs). The comparisons with single raters use NISQA LIVETALK only, the one NISQA set that publishes each rater’s votes. Those comparisons use panels of 10 raters for DataPoint and AudioMOS-AES, 24 for NISQA LIVETALK, and a median of 11 for SOMOS.
• Metrics. Rating scales use Spearman r between the model’s score and the crowd mean. A vs B uses the share of pairs where the model picks the crowd’s majority. SOMOS per-clip scores are correlations over clips. Benchmark scores use the full evaluation sets, and the 200-request samples per benchmark are used only to measure latency and cost. ParaPair is the one exception: it is scored on non-tie pairs only. When we reviewed pairs that raters marked as ties, many still showed a clear difference between the two clips, and the ParaPair authors also report non-tie accuracy separately.
• Gemini models. Gemini Decision 1 was evaluated on November 11, 2026, through the Gemini Developer API. It returns a full answer distribution, and accuracy uses its top answer. Gemini 4 Hydrogen was evaluated on November 12, 2026, on Vertex AI for accuracy and through OpenRouter, with Google as the provider, for latency and cost. It ran at its default thinking level, and its single answer is scored.
• Latency and cost. Means over a fixed sample of 200 requests from each launch benchmark, sent one at a time. The other models were measured on 10 benchmarks and Hume Audio Decision on 8, through the Hume API. The two pairwise benchmarks are excluded for Hume Audio Decision because the current API does not support paired audio inputs. Latency is server-side processing time for every model: provider time reported by OpenRouter, OpenAI, and Google, and gateway server time for the Hume API. API cost is the billed cost. Gemini Decision 1’s cost is estimated from its input tokens at the list price of $0.10 per million input tokens, recorded on November 11, 2026.
• Confidence intervals. We report 95% bootstrap confidence intervals for the comparisons with human raters but not for the main benchmark scores. Single-rater comparisons use 1,000 bootstrap resamples over questions. Panel-curve bands are 95% bootstrap intervals over requests.