Bachelor Thesis · UX Research · Human–Robot Interaction
Giving Robots
a Face
Designing and evaluating an expressive facial-animation system for an AI-driven social dialogue robot.
Designing and evaluating an expressive facial-animation system for an AI-driven social dialogue robot.
JARVIS is an AI-based dialogue robot developed as a platform for research into verbal and non-verbal human–robot interaction. The long-term concept is a social robot that could act as a conversational partner in settings such as care, where interaction and emotional support matter.
My thesis investigated whether adding virtual facial expressions could measurably improve the quality of those interactions — not only how people described JARVIS, but how long and how actively they chose to engage with it.
I researched the design space, created the emotion framework, developed the facial system and animations, designed the experimental control, prepared and ran the user study, and evaluated the qualitative and quantitative data.
I first collected 66 robots that people associated positively, drawing from film, television, games, science and robotics. I grouped them by visual design to understand how “friendly” robots communicate personality without needing realistic human faces.
The final direction draws especially from the display-as-face category: simple, luminous facial elements that are clearly technological but still expressive. The research also supported using a bright cyan-blue face, large rounded eyes and a wide, minimal mouth.
I deliberately avoided realism. Research on anthropomorphism and the uncanny valley suggested that JARVIS did not need a human-like face to be treated as a social actor; a simplified face could communicate emotional information clearly while remaining recognisably robotic.
The Jarvis Wheel of Emotions is based on six core emotional directions. Neutrality sits at the centre; each direction contains weaker, core and more intense emotional states.
I separated emotional states from continuous visual mechanisms. Emotions communicate a state such as happiness, fear or anger; visual mechanisms support dialogue itself, including thinking, neutral talking and emotion-specific talking animations.
The wheel wasn't only conceptual. Its position became part of each animation's identifier. For example, em121_hap means emotion → first direction → second intensity → first item → happiness.
I established a standardised naming convention using German and English nouns plus a unique file/animation ID. This kept the growing expression library consistent in programming and later allowed animation usage to be identified in the interaction logs.
| ID / filename | Name DE | Name EN |
|---|---|---|
em0_neu.mp4 | Neutralität | Neutrality |
em111_cal.mp4 | Ruhe | Calmness |
em112_sat.mp4 | Zufriedenheit | Satisfaction |
em121_hap.mp4 | Freude | Happiness |
em131_exc.mp4 | Aufregung | Excitement |
em211_emb.mp4 | Verlegenheit | Embarrassment |
em212_con.mp4 | Verwirrung | Confusion |
em221_sur.mp4 | Überraschung | Surprise |
em311_shy.mp4 | Schüchternheit | Shyness |
em312_ner.mp4 | Nervosität | Nervousness |
em313_con.mp4 | Sorge | Concern |
em321_fea.mp4 | Angst | Fear |
em331_sho.mp4 | Schreck | Shock |
em411_dis.mp4 | Enttäuschung | Disappointment |
em421_sad.mp4 | Trauer | Sadness |
em431_dep.mp4 | Deprimiertheit | Depression |
em511_ann.mp4 | Genervtheit | Annoyance |
em521_dis.mp4 | Ekel | Disgust |
em611_irr.mp4 | Irritiertheit | Irritation |
em621_ang.mp4 | Zorn | Anger |
em631_fur.mp4 | Wut | Fury |
The IDs became particularly useful during evaluation: JARVIS' logs could record which expression had been played and I could count expression use alongside interaction duration, topics and participant reactions.
Based on emotion research, robot references, emoji conventions and the neutral base face, I created a reusable specification for the left eye, right eye and mouth across emotional and talking states.
| Expression | Left eye | Right eye | Mouth |
|---|---|---|---|
| Sprechen Neutralität | open & blinking | open & blinking | slight upward curve, talking |
| Denken | diagonally upward | diagonally upward | narrow downward curve, angled upward |
| Sprechen Freude | slightly closed, tilted upward | slightly closed, tilted upward | upward curve, talking |
| Sprechen Überraschung | wide open | wide open | narrow, wide open, talking |
| Sprechen Trauer | outer corners contracted | outer corners contracted | downward curve, talking |
| Sprechen Angst | outer corners contracted + wide open | outer corners contracted + wide open | open, narrower at top, talking |
| Sprechen Ekel | inner corner contracted | slightly closed | asymmetric downward curve |
| Sprechen Zorn | inner corners contracted | inner corners contracted | downward curve, talking |
| Neutralität | open & blinking | open & blinking | slight upward curve |
| Ruhe | slightly closed | slightly closed | medium upward curve |
| Zufriedenheit | slightly closed, tilted downward | slightly closed, tilted downward | slight upward curve |
| Freude | open | open | upward curve |
| Aufregung | wider open | wider open | open upward curve |
| Verlegenheit | wide open | wide open | thin linear mouth |
| Verwirrung | slightly closed | inner corner contracted | right side slightly open/downward |
| Überraschung | wide open | wide open | narrow and wide open |
| Schüchternheit | diagonally downward | diagonally downward | slight upward curve |
| Nervosität | open, moving left/right | open, moving left/right | slightly open at outer edge |
| Sorge | outer corners contracted | outer corners contracted | linear |
| Angst | outer corners contracted + wide open | outer corners contracted + wide open | slight downward open curve |
| Schreck | outer corners contracted + wide open | outer corners contracted + wide open | wide open, narrower at top |
| Enttäuschung | closed, tilted downward | closed, tilted downward | slight downward open curve |
| Trauer | outer corners contracted | outer corners contracted | downward curve |
| Deprimiertheit | closed, tilted downward | closed, tilted downward | slight downward curve |
| Genervtheit | flat upper edge, slightly right | flat upper edge, slightly right | slight downward curve |
| Ekel | inner corner contracted | slightly closed, tilted upward | asymmetric downward curve |
| Irritiertheit | inner corners slightly contracted | inner corners slightly contracted | slight downward curve |
| Zorn | inner corners contracted | inner corners contracted | downward curve |
| Wut | strongly contracted inward | strongly contracted inward | strong open downward curve |
This specification became the central reference during animation, helping me keep expressions visually consistent instead of designing every emotion independently.
I modelled eyes and mouth as separate planes in Blender and used the neutral face as the shared base state. Shape Keys stored alternative forms for each facial component, while keyframes controlled their influence over time so Blender could interpolate between states.
For symmetric expressions I could animate one eye and mirror it; asymmetric expressions used a separate file where both eyes could be controlled independently. I also created talking variants for core emotions, so the face could remain expressive while JARVIS spoke.
Black background · emissive cyan elements · modular eyes / mouth / optional brows · frontal camera for a 2D display effect.
em0_neuem121_hapem221_surem321_feaem421_sadem521_disem621_angem212_conThinking and talking were represented through the animated face, alongside context-specific emotional expressions.
The control retained movement: the square rotated while thinking and pulsed while talking. This reduced the chance that movement alone would explain differences between conditions.
I prepared and conducted the study with 12 participants aged 23–64, split into two groups of six. Each participant spoke with one JARVIS condition and completed a post-interaction questionnaire.
I combined subjective measures — Likert-scale and open questions — with objective observations including interaction duration, discussed topics, conversational turn-taking, participant expressions and the facial-animation IDs produced by JARVIS.
The study evaluated six dimensions of interaction quality: sympathy, naturalness, comfort, perceived intelligence, emotional connection and engagement.
I also reviewed session video, screen recordings, conversation logs and questionnaire exports, consolidating the analysis in a structured dataset.
Average ratings on a 1–5 Likert scale consistently favoured the facial-expression condition.
Participants using expressive JARVIS spoke for more than twice as long on average and discussed more topics. Positive expressions such as joy and excitement were also observed more frequently in the expressive condition.
Participants described expressive JARVIS as friendlier and more inviting, but the face also affected perceived competence, trust, conversational flow and willingness to continue interacting.
The study supported the thesis hypothesis: targeted facial animation improved the quality of human–robot interaction compared with the non-facial control. The strongest difference appeared in engagement, and the objective interaction data pointed in the same direction.
The project ultimately combined user-centred research, interaction design, animation, technical system thinking and empirical evaluation. It received a final thesis grade of 1.0.
Important research was conducted along the way.
