Why AI Personas Can't Replace Real User Research

Why AI personas and synthetic users are so tempting

It is easy to see why "AI users" have caught on. Type a product description into a chatbot, ask it to role-play five different customer types, and within a minute you have feedback, quotes, even a satisfaction score. No recruitment, no scheduling, no incentive payments, no waiting for a research team to come back with findings.

For teams under pressure to ship, that speed is genuinely appealing. A real research study takes days or weeks to plan, recruit for, and run. A synthetic one takes minutes. When budgets are tight and deadlines are fixed, it is tempting to treat the two as interchangeable, just at different price points.

They are not interchangeable. They answer different questions, and mistaking one for the other is where the risk lies.

What an AI persona actually is

An AI persona is text. Specifically, it is a large language model predicting what someone matching a description you gave it would plausibly say, based on patterns in the text it was trained on. When you ask it "would a busy parent in their 30s find this onboarding flow confusing?", it is not consulting a busy parent. It is generating a statistically likely answer to that question, based on everything written by and about busy parents that happened to be in its training data.

That can be a useful starting point for a hypothesis. It is not evidence. The model has no body, no device, no Wi-Fi that drops out, no five other tabs open, no toddler interrupting halfway through a form. It has never actually used your product, because it cannot use anything. It can only describe what using something is generally like, in the abstract, based on what people have written about similar things in the past.

Where synthetic research breaks down

It gives you the average, not your users. An AI model trained on a huge volume of internet text will tend to produce a kind of consensus view: the most commonly expressed opinion, smoothed across millions of sources. Your users are not "the internet on average". They are a specific group of people, in a specific context, with specific needs that your product or service has to meet. Synthetic feedback tells you what a generic person might generally think. Real research tells you what your people actually think.

It can't surface what you didn't think to ask. Some of the most valuable findings in real research come from things participants say that nobody on the team anticipated: a workaround they have built, a step they skip every time, a reason for not using a feature that has nothing to do with the feature itself. An AI persona can only respond within the frame of the question you give it. It cannot show you the thing you didn't know was there to look for, because it isn't actually looking at anything.

It carries the bias of its training data. Large language models are trained on text that is disproportionately written by people who are online, younger, more digitally confident, and more likely to write in English about technology. The communities that are hardest to reach in standard research, older people, people with disabilities, people on lower incomes, people who don't use digital services much, are also the communities most underrepresented in the data these models learned from. Ask an AI to simulate "a range of users" and you are likely to get a narrower range than you think. We write more about this in our guide to inclusive research recruitment.

There is no one to be accountable to. If a synthetic panel tells you a feature will land well and it doesn't, there is no participant whose time was wasted, no person whose needs were misread, and no one you can go back to and ask "what did we miss?". Real participants give you a relationship: people you recruited, who gave informed consent, who you can follow up with, and whose experience you have a duty of care towards. That accountability is part of what keeps research honest.

Where AI tools genuinely help research

None of this means AI tools have no place in research. Used well, they can make real research faster and sharper. We use them to draft discussion guides and interview questions, which a researcher then reviews and adapts for the specific study and participants. We use them to help organise and summarise large volumes of interview transcripts and open survey responses, so a researcher can spend more time on interpretation and less on transcription.

AI tools are good at the parts of research that are mechanical: structuring, summarising, spotting recurring words and phrases across a large dataset that a human can then dig into. They are not good at the parts that require actually being a person: noticing a participant's hesitation, following up on something unexpected, or understanding why a workaround exists in the first place.

A hybrid approach: AI for speed, people for truth

The useful split is this: let AI tools speed up the parts of a research project that are about handling information, and keep real people for the parts that are about understanding people. Draft your interview guide with AI support, then run the interviews with real participants. Use AI to help cluster themes across fifty transcripts, then have a researcher check those themes against what was actually said and decide what they mean.

This matters even more when the thing you are testing is itself an AI feature. If you are building a chatbot, copilot, or recommendation engine, testing it with another AI playing the role of your user tells you how two AI systems interact with each other. It does not tell you whether a real person trusts the chatbot's answer, understands why a recommendation was made, or knows what to do when the AI gets it wrong. That is the gap our AI product testing service is built to close: real recruited participants using your AI features, so you find out how people actually respond before they do it on your live product.

How we work with real participants

Everything we do starts with recruiting the right people: not a generic panel, but participants who genuinely match the people your product, service, or research is for, including the groups that standard panels tend to miss. Our community of research participants, The Collective, gives us a diverse, fairly paid pool to recruit from across the UK.

From there, the method depends on the question. For questions about why people behave the way they do, we run in-depth user research: interviews, contextual inquiry, and longitudinal studies that get past the surface answer. For questions about whether a specific design or feature works, we run moderated usability testing, watching real people use real products and noticing exactly where they hesitate, succeed, or give up.

AI tools have a place in how we prepare for and analyse this work. They do not replace the work itself. The findings you get back are grounded in what actual people did and said, because that is the only kind of finding that tells you what will actually happen when your product meets the real world.