Project brief: Validate demand for the product concept, identify design issues, and make design recommendations based on research

Methods: False door testing, usability testing, A/B testing

Tools: InVision, Sketch, Usertesting.com, Shield (Firefox’s A/B testing platform)

Team: Myself (UX Research) + UX Designer, Content Strategist, Data Scientist, Engineers

Early Concept Validation

We learnt from some earlier research that identity theft was the biggest threat that people feared online. To address this threat, the Firefox team created a concept for a service that would alert Firefox users when they navigated to a site that had been part of a security breach. Initial research focused on validating the problem-solution fit for this concept.

After taking action on the breach, users were then asked if they would be willing to sign up for a Firefox account to receive notifications of future breaches related to their email address. The attempted sign-up rate was higher than any other Account offering seen to date. This gave us the signal that the breach alerts service was indeed perceived as high value. As a result, Mozilla created a formal relationship with the website haveibeenpwned.com to utilize their breach data for the Firefox Monitor service.

Answering Design Questions Through Iterative Product Testing

Following the early concept validation, we now focussed attention on design issues. In particular, the product team had the following questions that they needed research input on:

• What is the right tone and urgency for the alert?

• How much information should be exposed about the breach?

• What goals and actions should be supported upon being breached?

• How to make sure that people don’t attribute blame to the site?

• How to make the service seem trustworthy?

To address these questions, we undertook multi-stage iterative product testing cycles at increasing levels of realization, starting with paper prototypes and culminating in A/B testing in a product trial with 200,000+ Firefox users.

Usability Tests: Improving The Basic Flow

The basic user flow that was tested for the initial usability tests incorporated the following steps:

1. User navigates to breached website and receives Breach Alert

2. User navigates to Firefox Monitor Landing Page and enters email address to check for breaches

3. User receives Scan Results for email address

Initial usability tests uncovered the following core issues:

• Context-switching: after receiving a site-specific Breach Alert, users were confused to arrive at the general Firefox Monitor Landing Page, which makes no mention of the specific breach

• Trustworthiness: concerns over entering email address and the legitimacy of the service

• Unclear calls to action: in the Scan Results, users are unclear about the impact of being breached and what remedial action to take

These findings led to the following recommended design changes:

A/B Tests: Choosing Between Variants

Following the initial usability tests and the design improvements that resulted from it, we still had concerns that the overall flow was too long and that users would drop out along the way. We hypothesized that shortening the flow would lead to a better conversion rate for increased sign-ups to Firefox accounts. To test this, we deployed several variants of the initial breach alert on Firefox’s A/B testing platform and measured conversion rates during the trial. We also ran concurrent usability tests during the trial and gathered qualitative feedback on the different design variants.

The results of the tests revealed that the variants with an email field did indeed lead to better conversion rates, and so our hypothesis was correct. Nevertheless, the usability tests that ran concurrently to the A/B tests showed that including an email field in the alert heightened users’ fears of being phished and, on reflection, the team realized that by including an email field we were creating the possibility for websites to spoof the alert and phish users’ email addresses. We therefore took the difficult decision to eliminate all variants that included an email field.

The usability tests also revealed that including specific information about the breach, as in variant 3, made the alert appear more legitimate and trustworthy. However, the usability tests also showed that the amount of text in all the variants was too long and left people confused about what had happened. The combination of these findings led to the next iteration of the breach alert.

The Firefox Monitor service was officially launched on Sep 25, 2018. As of March 2023, Firefox Monitor has over 10 million registered users and has notified them of over 10 billion compromised records.