QAbyAI logo
Email Testing Is the Unsung QA Pain — What Real Teams Actually Build

Email Testing Is the Unsung QA Pain — What Real Teams Actually Build

59 email-flow steps across 5 users and 4 teams on QAby.AI. The niche QA pain no vendor markets to, with real telemetry on OTP, magic-link, and password-reset testing.

Himanshu Saleria
••
Email TestingOTPAI TestingTest AutomationData

Published 2026-06-13 · Last updated 2026-06-13 · 10-minute read

We did not predict that email testing would matter. It does. Here is the data.

When we shipped the wait-for-email and extract-from-email step types last year, the internal bet was that a handful of POC teams might use them. Twelve months in, a small but consistent slice of real teams reaches for them every week.

This post is a niche-data POV. The volumes are small. The pattern is real.

TL;DR

  • 33 wait-for-email + 26 extract-from-email steps = 59 email-flow steps total on QAby.AI, across 5 users and 4 teams. That is 0.65% of the 9,103 authored steps in our production telemetry.
  • Email testing is the unsung pain. Nobody writes a tooling RFP for it. Almost every signup, magic-link login, password reset, and payment receipt depends on it.
  • The four real use cases: OTP login, email-triggered workflows, email assertions, and multi-step auth flows with an email gate.
  • A healthcare-platform engineering team in our dataset hit Gmail-login brittleness, conditional popups, and optional-step branching the day they tried to automate signup.
  • Most stacks either mock the email (and skip the real flow) or hit a shared inbox (and live with the flake). Neither tests the thing the user actually experiences.

The 40–60 word answer. Email and OTP testing shows up as 59 of 9,103 authored steps across 5 users and 4 teams on QAby.AI. It is niche by volume, real in pattern. The four use cases are OTP login, email-triggered workflows, email assertions, and multi-step auth flows. Most testing tools either mock email or fight a flaky shared inbox; first-class wait-for-email and extract-from-email steps replace both.

The broader telemetry context (n=14 teams, 9,103 step events, and the full step-type mix) lives in The State of AI QA in Mid-Market SaaS 2026. This post zooms into the slice almost every vendor demo skips.

How big is the email-testing pain in real QA work?

Email-flow testing is 0.65% of all authored steps on QAby.AI but appears in 4 of 14 teams. Tiny volume, repeating signal.

The full counts from our production analytics: 33 wait-for-email plus 26 extract-from-email adds up to 59 email-flow steps out of the 9,103 in the dataset. Five users across four teams reached for them at least once. Compared with click (3,618 steps) or assert-ai (762 steps), the absolute count is tiny. What is interesting is the team-count ratio. Almost a third of teams on the platform built at least one email-aware test, even though the average team did not.

That ratio matches what we hear in customer calls. A healthcare-platform engineering team we worked with last quarter (call him Mike, the founding engineer) ran into Gmail-login brittleness on day one. The signup flow fired an email. The test had to wait for it, open it, pull the OTP, and feed it back into the app. Conditional popups added more branches: sometimes a security prompt, sometimes a marketing modal, sometimes neither. The "happy path" branched into four real paths the moment a real inbox got involved.

Mike's team did not call this email testing. They called it "the signup mess." The vendor-side label does not exist yet, because vendors do not market to it.

What does email testing look like in practice?

Email testing in practice is four named workflows, all of them mixed into signup, login, and post-purchase flows.

The four use cases below cover almost every real email-flow test we see authored on the platform. The volume is concentrated in the first two. The last two are smaller but matter more when they fail, because failures here cost a customer.

OTP login flows

The single most common email-testing workflow. A user enters a phone number or email; the app sends a one-time code; the test has to wait, read the code, and type it back into the OTP input.

Sounds simple. It is not. The test has to know which inbox the code went to, wait long enough without hanging, and extract the right token out of a template that marketing changed last Tuesday. It also has to avoid sharing the inbox with the next test, or yesterday's stale OTPs get picked up first.

This is the workflow that drove most of the 33 wait-for-email events in our data. Teams build it once, then reuse it everywhere signup, magic-link, or password reset shows up.

Email-triggered workflows

The second-most-common case. A workflow starts in the app and finishes in an email: invitation accept, payment confirmation, password reset, transactional receipt. The test cannot verify the workflow without actually receiving and parsing the email.

A small payments SaaS in our dataset has a refund flow that fires three emails: one to the buyer, one to the seller, one to the internal accounting team. The test that proves the refund worked is not the click on the "Refund" button. It is the email the buyer receives 10 seconds later.

Email assertions

A subtler use case. The test does not need to act on the email; it needs to check the email. Did the user receive the right one? Does the subject line match the templated string? Did the body contain the correct order ID, customer name, or unsubscribe link?

This is where extract-from-email does the same work assert-ai does for on-page content. Pull the value out of the body, then assert against it. The longer breakdown of the assertion mix lives in The Anatomy of an AI-Authored Test.

Multi-step auth flows

The fourth pattern. OAuth or SSO logins that route through an email gate: Google or Microsoft account, a verification link, then the redirect back to the app. Sometimes the email is the second factor; sometimes the only one.

The healthcare-platform team above hit this twice in one test. Once for Gmail login (Google sends a security email if the IP looks new). Once for the in-app email verification. Two email steps inside one signup test. No "demo flow" in any tooling pitch covers that shape.

Key Takeaways

  • Email and OTP testing exists in 4 of 14 teams on the platform. 59 steps total, 0.65% of all authoring. Niche by volume, real in pattern.
  • The four use cases are OTP login, email-triggered workflows, email assertions, and multi-step auth flows with an email gate.
  • Most stacks either mock the email or share a flaky inbox. Both skip the real flow. First-class wait-for-email and extract-from-email steps replace both.
  • Healthcare-platform-style signup flows can fire two email steps inside one test. No standard demo covers that shape.

How do you test OTP flows without faking the email?

Test OTP flows by polling a real inbox with a clean wait-for-email primitive, then extracting the code with a regex or AI parser. Mocking the email skips the flow that actually breaks.

The pragmatic recipe most teams converge on:

  1. Use a dedicated test inbox per run. Shared inboxes accumulate stale messages. Tools like Mailosaur, MailSlurp, and the open-source MailHog all build around this idea: give every test an inbox it owns and tear it down afterward.
  2. Wait, do not sleep. A wait-for-email primitive that polls until an email arrives (with a sane timeout) beats sleep(10) in every dimension. It is faster on the happy path and explicit about the failure mode.
  3. Extract with the template in mind. If the OTP lives between two known strings, regex works. If the template changes regularly, an AI extract step survives the change. The 26 extract-from-email events in our data lean on the AI version because templates drift.
  4. Make the email-step a first-class assertion target. If the test fails because the email never arrived, the failure message should say "email did not arrive," not "selector not found on a confirmation page that never loaded." That distinction shaves hours off triage. We unpack the signal-quality lens in The Debugging Ladder.

The teams that ship email-aware tests confidently do these four. The teams that do not still call it "the flaky test we always retry."

Why does most tooling skip the email step?

Most tooling skips email steps because email is not a browser primitive, and the cost of building a clean primitive is higher than vendors want to absorb for a 1%-of-steps feature.

A browser-only framework can fake clicks, type into inputs, and assert on DOM. It cannot, on its own, open an SMTP inbox, parse a multipart MIME message, and pull a token out of a templated body. That work happens off the browser, and most frameworks leave it to the team.

The result: email testing becomes a project-by-project DIY. Some teams set up a side service to listen to incoming mail. Some buy Mailosaur or MailSlurp for the inbox layer. A few stub the email entirely and discover, two releases later, that the real template diverged from the stub.

Because the cost is borne project by project, the pain stays invisible at the category level. Buyers do not put "email testing" on their RFP because they have already lost the argument and built around it. The buyer-side check we run with prospects, documented in How to Evaluate AI Testing Tools, now includes one question: can your tool wait for an email and extract a value out of it without custom code? Most cannot. A few can.

Why is the OTP and reset flow such a big share of "the what-to-test gap"?

The OTP and reset flow shows up in The What-to-Test Gap because it is functionally critical and operationally invisible. Nobody asks "do we test signup?" until signup breaks for one cohort and nobody notices for a week.

The lifecycle of an OTP bug runs like this. The flow worked yesterday. A marketing team updates the email template (new branding, different button text). The selector in the parsing regex breaks. The test never gets a code, then fails. The team marks it flaky and re-runs. Next sprint, a new user finishes signup with the broken template and silently churns.

This is the gap. Not "we did not have an automated test for signup." We had one. It became unreliable in a way that looked normal. The flaky bucket grew, the retries hid the signal, and a real regression slipped through. The same dynamic that drives the broader N-3 Lag in the dataset (where automation is always three sprints behind dev) lives in microcosm in email-aware tests, because email is the most template-volatile surface in the entire app.

The buyers we hear from describe the symptom, never the cause. The cause is that email testing was never given a clean primitive, so the workaround drifts every time the template does. The fix is platform-level: a wait-for-email and extract-from-email step that the tool maintains, not one the team rebuilds every sprint.

What changes when you treat email testing as a first-class step?

Three things change when email testing becomes a first-class step type instead of a custom-glued helper.

First, the test reads like the user flow. Sign up, wait for email, extract code, type code, assert dashboard. Anyone on the team can read the test and understand what it does. The same readability bar that the broader anatomy work documents in The Anatomy of an AI-Authored Test applies here.

Second, the failure message stops being a lie. "Email did not arrive in 30 seconds" is a real signal. "Selector .otp-input not found" is what the same failure used to look like, and it sent half the triage time in the wrong direction.

Third, the team starts writing tests for flows they previously skipped. The friction of email automation kept signup, magic-link, invitation accept, and reset off the regression list at most of the teams we talked to. Drop the friction, and those flows move on-suite. That is the part of the data that surprised us: the 5 users who used these steps used them across 26 distinct test cases. Once they had the primitive, they reused it constantly.

The locked positioning we run with applies here too. Devs ship faster than QA tests. We close the gap. Email flows are where the gap is widest because the workaround was always somebody else's problem. Release confidence at engineering velocity, without hiring SDETs to babysit the inbox.

If your suite has zero email-flow tests today and signup is in your top-3 user journeys, the gap is not your team's discipline. It is the missing primitive.

Run My Audit →

Frequently asked questions

How common is email testing in real QA automation suites?

In our production telemetry, 0.65% of authored steps are email-flow steps (59 of 9,103). That sounds tiny until you see the team-count: 4 of 14 teams use them. Email testing is a niche by volume and a normal pattern by adoption. Almost a third of active teams build at least one email-aware test, even though the average team builds many more click and type steps.

What is an OTP test and why does it break so often?

An OTP test verifies the one-time code flow used in signup, password reset, and magic-link login. It breaks often because the test has to coordinate across the browser and a real inbox, plus parse a templated email that marketing or product can change without telling QA. Most failures are template drift, inbox race conditions, or stale messages from a previous run.

Can I just mock the email instead of testing it?

You can, and the test will pass. The flow will not. Mocking the email skips the part that breaks in production: SMTP delivery, template rendering, parser logic, and inbox state. Mocked tests give you false confidence on a critical signup or auth flow. The 33 wait-for-email steps in our data are teams who decided false confidence was the bigger risk.

Which testing tools handle email and OTP flows natively?

Few do. Mailosaur and MailSlurp provide commercial inbox APIs, and MailHog offers an open-source SMTP catcher. On the test-authoring side, native primitives are rarer; QAby.AI offers wait-for-email and extract-from-email as first-class step types, which removes the inbox-plus-parser glue most teams write themselves.

How do you test magic-link login without a real inbox?

You do not, cleanly. Magic-link login is the canonical workflow that proves email testing belongs on-suite. The test signs in with an email address, waits for the link, opens it, follows the redirect, and asserts the authenticated state. Every step except "wait for the link" works in a browser-only framework. The wait is what the tool either handles for you or pushes back onto your team.

Why is email testing not on most RFPs?

Because the workaround happened a long time ago. Teams that needed email automation built a side service, integrated Mailosaur or MailSlurp, or accepted flake. They moved on. Email testing fell off the buyer's evaluation criteria because the problem felt solved-enough. The 0.65% share in our data is what happens when the friction drops: real teams use it across many tests, even though almost no one shopped for it.

What does QAby.AI do for email testing specifically?

Two first-class step types: wait-for-email polls a dedicated inbox until a message arrives (with a timeout), and extract-from-email pulls a value out of the body using either a regex or an AI extraction. Together they cover OTP login, magic-link auth, email assertions, and multi-step flows with an email gate. The full step-type mix and where these fit is documented in The Anatomy of an AI-Authored Test.

About the author

Himanshu Saleria, Founder, QAby.AI. Background in QA-led product engineering at scale; running QAby.AI's customer research, telemetry analysis, and product. LinkedIn.

Sources and further reading

Internal:

External: