The user interview process for usability testing isn't just about asking questions—it's about asking the right questions at the right time. Traditional methods often miss crucial insights because they lack real-time adaptation and flexibility. In this article, I’ll show you the best questions for usability testing and how modern AI tools—like conversational surveys and in-product probes—take your research beyond surface-level answers.
Core questions every usability test needs
The best usability testing interviews are built around a few essential question categories. Each category helps you uncover specific types of friction, opportunity, or unmet need in your product. With AI survey tools, we can now go deeper on these themes in real time. Here are the categories—and why they matter:
Task completion
Can you walk me through how you completed this task?
Was there anything that stopped you or made you hesitate?
These questions reveal not just what works, but where users get stuck.
Pain points
What was the most frustrating or confusing part of this process?
If you could change one thing about this journey, what would it be?
The goal here is to capture moments of real friction.
Expectations vs. reality
Was the experience what you expected? Why or why not?
What did you hope would happen when you clicked [X]?
Exploring misalignment between user mental models and what actually happens uncovers gaps in your UI.
Why are these questions game-changing? Because they expose not just surface-level reactions, but the root causes behind user struggles—a must for meaningful design change. And it’s hard to do this without being able to probe deeper based on each answer, which is why automatic AI follow-ups are so powerful.
Surface-level questions | Insight-driving questions |
|---|---|
Did you like this feature? | What made you like or dislike this feature? Describe a specific moment. |
Was anything confusing? | Can you tell me about a time when you weren’t sure what to do? |
Did you complete the task? | What, if anything, almost prevented you from completing the task? |
It’s no surprise that 70% of UX professionals believe AI will significantly transform their workflows in the next five years—mainly due to how much richer and more scalable deep-dive probing can now become [1].
Turning tasks into conversational insights
Traditionally, a usability test is structured like this: I give the user a task, observe what they do, and then ask follow-up questions. But this rigid flow often breaks the natural rhythm and misses in-the-moment thoughts. The alternative? Make the whole thing conversational by embedding questions directly into the task flow. Here’s how it works—and why it’s so effective:
Tasks and probes happen side by side (not just at the end)
Clarifying questions are triggered by specific actions or hesitations
The conversation adapts as the user engages—just like a great in-person moderator would
Here are three example prompts you can use to get richer data:
Imagine you want users to upload a profile photo.
Prompt: “Can you upload a new profile picture? Let me know how you decide which button to click first.”
Probe: “What made you choose that button over the others?”
Testing a new dashboard? Try:
“Can you find the monthly sales report? Tell me if anything on this page surprises you.”
Probe: “What were you expecting to see but didn’t?”
Exploring conversion flow drop-off:
“Go ahead and try to upgrade your plan. Was anything unclear or did you pause at any point?”
Probe: “What would have made the next step more obvious?”
Conversational surveys make this process scalable and real-time, with the ability to trigger rich follow-up questions based on actual task performance—not just memory. This is where in-product AI-powered conversational testing, like in-product conversational surveys, shines for task-level insights.
Example prompt for analyzing task data: “Review these responses—what are the top three blockers users face while completing the upgrade flow?”
The art of the follow-up: probing for hidden usability issues
Let’s be honest: most user answers don’t tell the full story. I’ve found that the best questions for usability testing are often just the start. Real revelations come from following up, exploring context, and uncovering what isn’t said outright. This is where modern AI shines. AI-powered follow-ups can, in many cases, match a skilled interviewer for relevant, dynamic probing—especially at scale.
There are three main types of effective probes you can use:
Emotional Probes: Tap into the user’s feelings about the experience.
Initial response: “I was frustrated when the upload failed.”
Probe: “Can you describe what made you feel that way? Was it just the error, or something else?”Contextual Probes: Dig into external circumstances or device/environment influences.
Initial response: “It took me longer on my phone.”
Probe: “What was different about doing this on mobile versus desktop for you?”Comparative Probes: Encourage comparison with other tools or past experiences.
Initial response: “This wasn’t as easy as I expected.”
Probe: “When’s the last time you found this task easy in another app? What made it different?”
Emotional probes uncover frustration levels so your team can prioritize fixes that drive satisfaction. Contextual probes reveal environment factors so you can tailor mobile or accessibility improvements. Comparative probes highlight what users love (or dislike) about competitors, often pointing to significant gaps or missed opportunities.
With automatic AI follow-up questions, you don’t need to be in the room to have this “aha!”-generating back-and-forth—these clarifications can happen for every respondent, not just a lucky handful.
Remote testing challenges and AI-powered solutions
Remote usability testing introduces a new set of hurdles. When I can’t observe my users directly, it’s easy to miss hesitation, confusion, or context. Here’s where traditional manual remote testing falls short:
No access to body language or facial cues
Delayed or incomplete follow-ups due to scheduling or user fatigue
Heavy reliance on user memory rather than in-the-moment insight
But AI-driven surveys offer new ways to overcome these limits. With context-aware, in-product triggers, I can capture feedback based on what’s actually happening—immediately after a user hits an error, finishes onboarding, or tries a new feature. These AI-powered flows have been a game changer: 60% of UX research professionals see AI as a tool to analyze large datasets faster, making large-scale qualitative usability tests viable [2].
Traditional remote testing | AI-powered conversational testing |
|---|---|
Requires invites and scheduling | Triggers instantly during real user activity |
Follow-ups often missing or late | Clarifying questions adapt in real time |
Manual analysis post-interview | Automatic grouping and theme detection via AI |
Feedback based on memory | Feedback based on fresh, in-the-moment actions |
Common in-product triggers include:
After a feature is used for the first time
Upon encountering an error or unexpected state
During onboarding—right after a critical milestone is reached
AI survey tools like Specific’s AI survey generator make it easy to create contextual surveys that adapt to each user’s unique journey. The key is treating the survey like a real conversation, with automatic clarifying follow-ups that capture the “why” behind every action.
That’s why “follow-ups make your survey a conversation”—so it works as a true conversational survey, not just a form with static questions.
From questions to actionable insights
Even with the best interview and probe techniques, I know the hardest challenge is turning all that unstructured feedback into decisions. This is where AI-augmented analysis saves weeks of labor. AI can now cluster, synthesize, and surface top issues across thousands of responses—instantly highlighting themes I’d miss otherwise. 58% of UX designers say AI analysis increases their research accuracy [3]. Here’s how to move from raw notes to change-making insights:
Analysis goal: Find top usability pain points for mobile users.
Prompt: “What are the most commonly reported frustrations from mobile survey respondents?”
Insight: Top three mobile-specific blockers with direct user quotes.
Analysis goal: Uncover biggest delighters in onboarding flow.
Prompt: “Which onboarding elements do users consistently describe as easier or more helpful than they expected?”
Insight: List of top positive moments with context.
Analysis goal: Compare first-time users vs. power users on navigation ease.
Prompt: “How do first-time and returning users describe their experience finding the dashboard?”
Insight: Segment-level findings for targeted improvements.
With AI survey response analysis, you can even create multiple analysis threads for different focus areas—say, analyzing one segment for friction in a new feature while tracking another for loyalty signals.
Start your conversational usability testing today
If you want to transform your user interview process, nothing matches the power of real-time, adaptive conversations. Get faster, truer, and more actionable insights—plus scalable, contextual understanding for every user flow. Every day without conversational testing is a day of missed insights. Create your own survey and make every response count.