NewIntroducing the Hume Audio Decisions APIOne model, any question, straight from audio
hume.ai logo
Blogs
Research

When low word error rate is not enough: Measuring entity accuracy in ASR

David Ayllon
Nathan Leung
David Ayllon and Nathan Leung
5 min read

When low word error rate is not enough: Measuring entity accuracy in ASR

People give voice agents the details that make a task work: their name, an email address to verify an account, a street address for a delivery, or a date and time to book an appointment. In each case, one wrong letter in a name or email address, or one wrong digit in an account number, can cause the entire task to fail.

These details are named entities. Because they make up only a small part of a sentence, overall word error rate can look strong even when the system gets this critical information wrong. In our new Real World VoiceEQ evaluation, two systems with almost the same WER were seven points apart in entity accuracy, and even the leading system missed nearly one in five entities.

To make these failures visible, we added entity accuracy to the Real World VoiceEQ ASR leaderboard. Alongside the benchmark, we are offering the Hume ASR Entity Dataset* for licensing to teams that want to train on these failure modes across languages, accents, and real recording conditions, and we can collect custom data for additional languages and domains.

Why named entities are difficult for ASR

Named entities are often rare in training data. A model may encounter a common word millions of times while seeing an uncommon surname, product code, or street name only a few times, if at all. Language-model priors can then pull the transcription toward more familiar words, turning a name like “Nguyen” into “new win.”

The audio may not provide a single obvious written form either. “Four oh two” could mean 402, 4:02, or 4/02. “Jon,” “John,” and “Jean” can sound alike. An alphanumeric code offers little context for recovering a missed letter or digit.

Entities also cross languages. A Spanish speaker may say a German brand, and an English speaker may give a Japanese name or a Mexican address. The ASR system has to handle the speaker’s accent and the borrowed pronunciation at the same time.

How we measured performance

We built a held-out* English evaluation of 3,292 recordings with 3,257 tagged entities across 27 categories, covering 6.1 hours of speech from 711 speakers living in 25 countries.

The test set includes up to 155 recordings per category, ensuring that names, codes, addresses, and amounts all contribute to the results. Recordings are distributed across participants in six groupings to ensure accent diversity: the United States; the United Kingdom and Ireland; Canada, Australia, and New Zealand; India; Latin America; and other countries.

We evaluated 45 commercial and open systems, with open models ranging from 110 million to 27 billion parameters. We measured both word error rate (WER) and entity accuracy. Entity accuracy requires the entity to appear exactly as written, ignoring case, punctuation, and spacing; one incorrect letter or digit counts as a miss.

The two measures answer different questions. WER shows how much of the sentence was transcribed incorrectly. Entity accuracy shows whether the system captured the information the task depends on.

Scatter plot of 45 ASR systems: entity accuracy against WER. Google Chirp 3 (WER 5.8%, 80.3% of entities right) and Modulate velma-2-english-vfast (WER 5.7%, 73.0%) are highlighted.

Results

Even the strongest systems miss roughly one entity in five

ElevenLabs Scribe v2 achieved the highest entity accuracy at 81.7%, followed closely by MAI-Transcribe 2 at 80.9%, Google Chirp 3 at 80.3%, and Azure Speech at 79.6%. The leading systems perform similarly, and only three pass 80%. Even the strongest model misses nearly one in five of the details that voice-agent tasks often depend on.

Low word error rate does not guarantee entity accuracy

SOV-Transcribe-1, a new 27-billion-parameter open model from Salesforce, recorded the lowest WER of all 45 systems at 4.5%, but ranked sixth on entity accuracy at 78.9%. Google Chirp 3 and Modulate’s velma-2-english-vfast have almost the same WER, 5.8% and 5.7%, yet Chirp 3 transcribed 80.3% of entities correctly and Velma only 73.0%. Across the 37 systems with WER below 10%, entity accuracy ranged from 58% to 82%. A sentence can therefore be more than 90% correct and still contain the wrong account number, name, or address. Measuring entities separately exposes failures that WER hides.

Foreign entities are harder for every system

Entity accuracy fell for every system when the entity originated from another locale. Across all systems, accuracy dropped by an average of 14 percentage points. Google Chirp 3 fell from 81.9% on native entities to 71.8% on foreign entities. Deepgram Nova-3 fell from 70.6% to 54.0%. These results reflect a common real-world challenge: the language a person speaks does not determine the origin of every name, brand, address, or product they mention.

Performance varies sharply by entity type

The leading systems are relatively strong on some numerical entities. Phone numbers, identifiers, and codes reach between 93% and 97% accuracy across the top five systems. That performance is not consistent across the field or across every type of numerical entity. Across all 45 systems, alphanumeric strings average only 53% accuracy and version numbers average 66%.

Bar chart of entity accuracy by category. Bars show the average of the five best systems, ticks the average of all 45. Person names are lowest at 43.7%; codes and abbreviations highest at 96.6%.

Names and formatting remain open problems

Names are substantially harder. Across the five leading systems, average accuracy is 44% for person names, 53% for people in media, and 61% for products. For example, “Tim Peacock” is mistranscribed by many top ASR models as “Team Peacock,” highlighting the challenge voice agents face when looking up customers in a CRM.

Here are six entities from the benchmark and how systems we tested transcribed them:

EntityTranscribed as
Tim PeacockTeam Peacock; chimpeacock
Lee Byung-hunLee Byron Hoon; Lee Baram Hoon
Bostik Glu Dot Dispenser Packboasted glue dot Dispenser Pack; boasted glue dog Dispenser Pack
Al-Malazall malas; all my last
San Luis PotosíSt. Louis potosi; Saint Louis potosi
Dong Xuan Marketdong wan market; dongguan market

Listen to the six recordings:

Person name: Tim Peacock
0:00

“I may have misunderstood the passenger details, so could you confirm whether Tim Peacock is listed on tomorrow’s booking?”

Person in media: Lee Byung-hun
0:00

“Could we schedule a discussion about Lee Byung-hun for next week?”

Product: Bostik Glu Dot Dispenser Pack
0:00

“Please confirm whether the Bostik Glu Dot Dispenser Pack has shipped and advise its expected arrival for our receiving team.”

Street address: Al-Malaz
0:00

“Please arrange delivery to Al-Malaz District by Friday.”

Geographic location: San Luis Potosí
0:00

“Let us move the client check-in to San Luis Potosí.”

Point of interest: Dong Xuan Market
0:00

“Could you confirm whether Dong Xuan Market opens before our morning tour?”

An ASR system must therefore do more than hear the correct sounds. It must also convert them into the precise written form required by the application. Email addresses average 63% accuracy across the top five systems and only 16% across all systems. Many open models produce phrases such as “john dot doe at gmail dot com” instead of the formatted address a downstream system expects.

Why entity-specific evaluation matters

Word error rate remains useful, but it cannot show whether a model’s mistakes occur on incidental words or on the information that determines whether a task succeeds. Entity-specific evaluation makes that distinction visible.

For teams selecting ASR systems, this means testing the entity types and language combinations their application actually encounters. A system that performs well on phone numbers may still struggle with customer names, email addresses, or foreign-origin entities.

For model developers, breaking performance down by entity category and locale identifies where targeted training data is needed. The public benchmark provides a shared measure of the problem. The separate Hume ASR Entity Dataset provides about 800 hours of speech across eight languages, with more than 350,000 tagged entities in 28 categories, designed around the same core failure modes.

Explore the results and dataset

Explore the complete results on the Real World VoiceEQ ASR leaderboard. You can also find them on Hugging Face in the ASR leaderboard and Real World VoiceEQ spaces.

If you are developing an ASR model or voice agent, talk to the Hume team about licensing the Hume ASR Entity Dataset.

*No entity tested in the benchmark appears anywhere in the English portion of the Hume ASR Entity Dataset.

Related topic

View more

Build voice models that understand more than words.