Editor's note: This post covers the science, development and evaluation of Hume’s state of the art Expression API, that enables developers to enrich their applications or research pipelines with real-time measurement of voice and facial expressions.

Facial expressions and vocal tone are central to how we communicate. They convey emotions such as amusement, interest, frustration, and surprise, often without naming those feelings in words. Understanding these signals is part of how we connect with one another and respond to what others express.

Expression measurement turns these signals into structured scores that can be used to annotate data, evaluate AI outputs, and explore expressive audio and images. It gives us a way to describe the emotional meaning people perceive, compare expressions across recordings, and follow how an interaction changes over time.

Consider someone saying, “I can’t believe it.” After good news, the phrase might sound excited. After a setback, it might convey frustration. During a playful exchange, it might express amusement. The words stay the same, while the voice and facial expression contribute different meanings.

At Hume, our work on expression measurement is grounded in emotion science. Our research has explored the range of meanings people associate with facial and vocal expressions, including nuanced and blended expressions. This scientific perspective guides our approach: measure expression through a detailed vocabulary and evaluate those measurements against human judgments. Read more about the science behind Hume’s expression measurements.

An expression can communicate several emotional qualities at once, such as amusement blended with disbelief. Our speech emotion model captures this detail through separate scores for 414 tags: 23 core emotions and 391 finer-grained descriptions. These include amusement, anger, interest, joy, sadness, and surprise, alongside more specific expressions such as admiration, frustration, nervous laughter, and relief.

The voice descriptor model measures 190 characteristics across 27 families, describing qualities such as raspy, breathy, fast, monotone, whispering, and theatrical. These measurements complement emotional expression, allowing us to describe combinations such as frustration expressed quietly or amusement in a deep, raspy voice. Our speech models support Arabic, Bengali, German, English, Spanish, French, Hebrew, Indonesian, Italian, Japanese, Korean, Polish, Portuguese, Russian, Turkish, and Vietnamese.

Our facial expression model recognizes emotional expression across 48 Hume categories, including admiration, amusement, anger, anxiety, concentration, confusion, excitement, joy, and sympathy. It captures nuanced expressions and distinguishes positive from negative surprise. The model also measures 27 visible facial descriptions, such as Smile, Frown, and Jaw drop, providing a complementary view of an expression’s visible features.

Data and human ratings behind expression measurement

For facial expression, our training data includes more than 370,000 images from our human-rated FaceMimic collection. Across the collection, more than 13,000 raters contributed more than one million emotion ratings, describing both the emotional meaning of an expression and its intensity. Participants photograph themselves mimicking a reference expression, then rate their own and others’ expressions. The collection includes participants from the United States, South Africa, India, China, Venezuela, and Ethiopia.

Our speech emotion and voice descriptor models share a speech training resource with more than 260,000 recordings. Its emotion-annotated portion includes more than 210,000 recordings, representing more than 270 hours of mostly performed speech, with multiple emotion annotations for most clips. The resource captures both emotional expression and vocal characteristics, supporting measurements of what a voice conveys and how it sounds.

Evaluating agreement with human perception

We evaluated our voice models using our Hume Feedback API, collecting structured judgments from listeners, including native speakers for multilingual studies. We asked whether the models’ descriptions agreed with human ratings and whether higher scores identified stronger examples of a particular expression or vocal characteristic.

For each model, we begin with evaluations on our internal human-rated data, then examine results on public datasets. These studies assess how well the models recognize expression and capture the strength of the qualities people perceive.

Within each study, the compared systems were assessed on the same evaluation samples.

Speech Emotion Recognition Results

We compared 11 systems on the same set of more than 9,000 recordings from 32 public datasets, including Gemini 3.8 Flash and specialized systems such as EmotionThinker, EmpathicInsight, and VoiceCLAP. All systems made zero-shot predictions: we applied the models directly, without task-specific fine-tuning for this benchmark. Our speech emotion model achieved the highest overall AUC, with statistically significant advantages over each of the other systems shown. Its advantage over Gemini 3.8 Flash also remained statistically significant when we evaluated the full test sets of more than 43,000 recordings and checked different combinations of datasets.

We use one metric throughout the public benchmark tables: emotion-ranking score, also called AUC. Imagine taking two recordings: one labeled sad and one without sadness. The score measures how reliably the model gives the sad recording a higher sadness score, averaged across the evaluated emotions. Higher is better: 50 represents chance-level ranking and 100 represents perfect ranking. This measures how well recordings are ordered, rather than the percentage of labels guessed correctly. The overall table shows all 11 systems ordered by this score. EmpathicInsight and VoiceCLAP share the same rounded score.

The table below groups all 32 public datasets by spoken language and uses the same emotion-ranking score as the overall comparison. Each dataset receives equal weight within its language group, and every model is evaluated on the same recordings. The final row gives the overall mean across all 32 datasets, with each dataset weighted equally. Bold values mark the highest displayed score in each row, including ties at the shown precision.

Scores are averaged across the public datasets for each language, with each dataset given equal weight.

We then compared our speech emotion model with Gemini on internal human ratings collected through our Hume Feedback API. This evaluation covers finer-grained emotional expressions across 16 languages, excluding the broad parent emotion categories. We use the same emotion-ranking metric as in the public benchmark: AUC, shown on a 0-100 scale, with higher scores indicating better ranking. The mean gives each language equal weight.

On our internal human-rated evaluation of finer-grained emotional expressions across 16 languages, our model performed comparably to Gemini.