Measuring Human-AI Interaction Quality
Protected dissertation research
Dissertation Study 1
This case study contains ongoing dissertation work. Enter the shared password to view the full project.
UX Measurement / Human-AI Interaction / Scale Development
I developed and initially validated HEAIQS, a 29-item scale that measures four distinct qualities people may perceive when they interact with in-vehicle AI.
Project Objective
AI interfaces can feel social, personal, aware of the shared situation, or clear about what the system is doing. These qualities are related, but they are not the same. A single trust, usability, or human-likeness score cannot show which part of the interaction changed.
The research goal was to create a clear measurement tool for studying these separate qualities without assuming that more human-like AI is always better.
Research Timeline
* Approximate active project time from IRB preparation through reporting.
Research Process
Method Alternatives and Choice
Alternative: One overall rating
A single trust, usability, or human-likeness score would be simple, but it would hide whether a design changed social response, personalization, shared presence, or operational clarity.
Alternative: Reuse a general AI scale
An existing general scale would reduce development time, but available measures did not cover the four interaction qualities together or fit an embodied passenger experience closely enough.
Alternative: Interviews only
Interviews could explain how people described the AI, but they would not provide a standardized measure for comparing interface conditions or tracking factor-level change.
Chosen: Scale development plus experiment
This was the best fit because item development, EFA, CFA-guided refinement, and a 3 x 2 experiment could test the measurement structure and show whether each quality responded differently to design cues.
What HEAIQS Measures
Socio-emotional agency
Whether the AI appears able to understand and respond in a socially meaningful way.
Perceived personalization
Whether the interaction appears adapted to the individual passenger.
Situated social presence
Whether the AI appears present and involved in the shared vehicle situation.
Operational communication
Whether the AI clearly communicates vehicle state, actions, and relevant information.
Evidence
The analysis supported reporting four separate scores rather than one total score. This matters because an interface change may improve personalization while leaving operational communication unchanged.
Experimental Application
Relational message cues
Compared message packages designed to change how the AI presented its relationship with the passenger.
Adaptive interface cues
Compared static and adaptive interface presentations.
Factor-level analysis
Tested each HEAIQS dimension separately to avoid hiding different responses inside one average.
Sensitivity checks
Compared alternative factor structures and checked whether the measurement conclusions were stable.
Study Figures
What This Means for UX Teams
Measure the intended design effect
Choose a measure that matches the quality a design is expected to change.
Do not use human-likeness as a quality score
A more social AI is not automatically clearer, safer, or easier to use.
Report a profile, not one total
Separate scores show where an experience is strong and where it needs work.
Add outcome measures
Use HEAIQS with trust, usability, comprehension, or behavior measures that fit the product goal.
My Contribution
Dissertation Study 1 of 3. Part of Understanding Human-AI Relationships in Autonomous Vehicles.