Catching the defects that would have undermined an alpha study, before it began
A wellbeing app was preparing to open its build to real participants for the first time. An independent review of the whole product, run before that fieldwork began, found technical defects that would have generated misleading data throughout the study rather than genuine signal about how people experience the design. Fixing the right things first, in the right order, changed what the alpha would be able to tell the team.
A build about to meet real participants, with no independent check first
The client had built a wellbeing app designed to support people through a recovery journey - daily planning, guided exercises, progress tracking and a supportive AI coach - and was preparing to open it to real participants for an alpha study: a diary study, tracking how people actually used it over time in their own lives.
Before committing participant time and budget to that fieldwork, the team wanted an independent check on the build itself. The risk they were alert to is a familiar one in early-stage products: a participant who abandons a task because a required field silently blocks them, or whose progress is lost on a screen they didn't expect to reset, looks identical in research data to a participant who is genuinely struggling with the design. Without ruling out the first kind of failure, conclusions drawn from the second wouldn't be trustworthy.
For a wellbeing product specifically, the stakes were a little higher than usual. Participants using an app to support them through recovery are not well served by an experience that behaves unpredictably, and accessibility barriers that would be a minor inconvenience elsewhere can stop someone from completing a task they specifically need to complete.
An expert-led review of the whole build, before any participant saw it
This was a UX audit rather than a piece of user research: an independent, expert-led review of the complete product, evaluated against established usability and accessibility principles rather than observed through participant sessions. No recruitment was needed, which meant the review could run immediately and return findings before the alpha's own recruitment and scheduling had to begin.
The review covered the full journey a real participant would take - from first opening the app and completing onboarding and consent, through daily use of its core planning, check-in and coaching features, into the more specialist tools further into the product. Each screen was assessed on its own terms and against the journey around it, since some of the most consequential issues were about how screens connected to each other rather than any single screen in isolation.
Findings were separated into three kinds: confirmed bugs - behaviour that was reproducibly broken regardless of anyone's opinion about the design; usability issues - design choices likely to slow participants down or confuse them even though the underlying functionality worked; and open questions, where a product decision rather than a fix was needed from the team. Keeping these distinct mattered, because they call for different kinds of response and different levels of urgency.
What the review involved
Findings that would have looked like usability failures, but weren't
Several confirmed defects would have surfaced during the alpha as though they were usability problems, when they were nothing of the kind. One flow could be marked complete without the data behind it ever being captured, quietly generating records of engagement that hadn't actually happened. Another could lose a participant's progress entirely if they navigated away and back. A third left participants stuck on a required step with no indication of what was blocking them. Left in place, each of these would have shown up in the alpha's data as drop-off or disengagement - exactly the signal a diary study exists to interpret - without the team ever knowing the real cause wasn't the design at all.
The review didn't only flag problems. It confirmed a set of things the product was already doing well - a warm, non-clinical tone, generous touch targets, and features that stood out as genuine points of difference - and recommended those strengths be preserved deliberately as other parts of the product were simplified, rather than lost in the process of fixing everything else.
Usability and accessibility issues that fell short of being launch-blocking were still documented in full, but sequenced separately: addressed alongside the confirmed bugs where the effort was small, held for public launch where it wasn't, and explicitly deferred where they represented genuine but lower-priority improvements. That separation meant the client could act immediately on what mattered most, without waiting for a single, larger release to fix everything at once.
A delayed alpha, and a diary study now underway that means something
On the strength of the findings, the client made the call to delay the alpha rather than run it against a build that would have compromised its own results. The highest-priority defects were fixed first, usability and accessibility improvements followed on the schedule the review had set out, and only then did the diary study begin.
That sequencing paid for itself directly. An alpha run against the original build would likely have needed to be repeated once the defects were found in the data anyway, at far greater cost in participant time, incentives and delay than fixing the build first. A delay measured in weeks avoided losing months.
Participation Studio is now running that alpha - a diary study with real participants, tracking their experience over time - for the same client. The data coming back describes how people actually experience the product's design, not artefacts of bugs that should never have reached a participant in the first place.
We fully recognise the effort your team invested in recruitment, moderation, and analysis, and we genuinely appreciate the quality of the discussions and reporting.
- Usability problems in an insulin device surfaced before manufacturing locked anything in, with human factors evidence supporting the regulatory submission.
- An NHS patient portal team got a prioritised, evidenced list of adoption barriers, and the highest-leverage fix wasn't the one they expected.
- Patient involvement changed a trial's primary endpoint before the protocol was finalised, cited by the ethics committee as a model for evidencing PPIE.
Trusted PPI and PPIE delivery partner to the NIHR HealthTech Research Centre in Accelerated Surgical Care.
Facing a similar decision?
Tell us what you need to decide and who you need to hear from - we will come back with a clear, proportionate proposal.
Get a QuoteWe reply within one working day.
Prefer to talk it through first?
Book a free 20-minute call. No obligation, no sales pitch. We'll tell you honestly whether research is worth it for your decision, and what it would cost.