SOP 060: How to Structure, Run, and Analyze an A/B Test on your Website
Flawlessly plan, execute and analyze an A/B test on your website and determine whether a variant should be implemented definitively on your website or not. This is based on actual data.
Last Updated / Reviewed: 03/03/2025
Execution Time: ~30 min - 2 hours
🚀 Goal: To properly structure and implement an A/B testing culture across your business.
🤞 Ideal Outcome: You’ve flawlessly planned, executed and analyzed an A/B test on your website and are able to determine whether a variant should be implemented definitively on your website or not. This is based on actual data.
🤔 Why this is important: Without a proper A/B testing process in place, you are not able to confidently make business or UI/UX decisions.
🧐 Where this is done VWO (Free and recommended).
🗓️ When this is done: Every time you want to test a new hypothesis on your website.
👨 Who does this: The person responsible for Conversion Rate Optimization.
__⚠️Prerequisites or requirements: __
• You need to have VWO set up on your website. This is free and recommended.
• Follow SOP 059 ([web version](https://members.clickminded.com/sop/059/)).
• You should also have Google Analytics conversion goals set up on your website.
• Follow SOP 168 ([web version](https://members.clickminded.com/sop/168/)).
Click here to get the most up-to-date version of this SOP
Resource for this SOP
Before we begin reviewing the SOP, we have provided a structured template for documenting A/B tests.
- Template 021: A/B Test Documentation Spreadsheet (web version)
This template is designed to ensure a systematic and consistent approach to A/B testing by capturing key details such as test objectives, hypotheses, variations, success metrics, and final results. By using this framework, teams can maintain clear records of past experiments, track performance insights, and optimize future testing strategies based on data-driven decisions.
Now, let’s go through the SOP in detail.
Determining what you are going to test
Before running an A/B test, you need to identify the key element(s) on your landing page that could impact user engagement and conversions. The goal is to test changes that could improve performance based on data, observations, and user behavior insights.
01▸ Select the Element to Test
Start by choosing a specific element on your landing page that you suspect could influence conversions.
Common test elements include:
- Headlines & Subheadlines
Example: Change “Just another WordPress site” to “5x your ROI today through custom emojis.”
- Form Positioning & Fields
Example: Move the lead generation form from the bottom to the top of the page.
- Media (Images & Videos)
Example: Replace the background image on your hero section with a more engaging visual.
- Call-to-Action (CTA) Design
Example: Change the CTA text from “Submit” to “Get your free emoji cheat-sheet now.”
- Sales Copy
Example: Test a completely new sales message or a rewritten section of your existing copy.
- Authority & Trust Elements
Example: Add customer reviews or testimonials to build trust.
- Pricing Strategy
Example: Adjust product pricing from $97 to $79 and measure its impact.
02▸ Note: While website changes can often lead to fluctuations in conversion rates, this is not always the case. Conversion rate changes may also be influenced by external factors such as seasonality, traffic quality, competition, or even weather conditions, which may coincide with your website updates but are not directly caused by them.
Identify Business-Specific Testing Opportunities
Consider recent changes or customer insights that might have affected conversions:
- Recent Website Changes
Identify adjustments that might have influenced conversions (positively or negatively).
Example: If you recently removed the “Pricing” page link from your landing page and saw a drop in conversions, test whether adding it back improves performance.
- Customer Feedback & FAQs
Review customer inquiries and support tickets for recurring questions or concerns.
Example: If users frequently ask, “How does feature X work?” test adding a section explaining that feature on your landing page.
03▸ Use Behavioral Analysis Tools to Identify Test Ideas
- Heatmap & Clickmap Insights
If you have implemented heatmap and clickmap tools (e.g., through SOP 049: Implementing and Analyzing Heatmaps on Your Website), review user engagement patterns. - Identify elements users engage with the most and least. - Test improvements to underperforming areas (e.g., a CTA button that gets little attention). - If you haven’t implemented heatmaps yet, refer to Template 017: Clickmap & Scrollmap Diagnosis Cheat Sheet to spot usability issues.
- Session Recordings
If available, analyze session recordings (e.g., Hotjar) to understand user behavior. - Watch at least 10 sessions where users converted and 10 where they didn’t. - Identify friction points (e.g., users struggling with form fields, missing critical content). - Test variations that address these issues.
Example: If non-converting users leave after struggling with the password requirements, simplify the password creation process in a test variant.
By systematically selecting and validating test elements using data-driven insights, you can ensure meaningful improvements in your website’s conversion rates.
Note: If you have followed SOP 049: Implementing and Analyzing Heatmaps on your Website (web version) for implementing Heatmaps on your Website, you can use Hotjar to record user sessions on your website (This is outside of the scope of this SOP).
Defining how you are going to test it
01▸ Open the ClickMinded Template 021: A/B Test Documentation Spreadsheet (web version).
Note: This spreadsheet should be able to be used with most of the A/B testing tools available. If your A/B testing tool does not offer a specific feature (e.g. targeting specific audiences) you can always remove or edit that column to fit your specific A/B testing tool.
02▸ Fill out the spreadsheet.
- Test #: Incremental number; this should be used internally. It is useful to communicate with your designer, programmer, copywriter, or whenever you want to mention a specific A/B test during a discussion or a Project Management tool.
E.g. 001
Image: Example of an Asana task with the Test# as a reference;
- Start Date / End Date: Add the Start Date whenever you start running your experiment, and update the End Date once the experiment is over. This will allow you to quickly overview which experiments are still running.
Note: The spreadsheet will automatically update the status of the “Running Days” column and set it to “Still Running” if no end date was added yet.
Note 2: If the experiment ran for less than 14 days the “Running Days” cell will turn red to warn you that the test might not have ran for enough time for your results to be meaningful (although this will ultimately depend on how many people were exposed to your experiment during that period of time.)
- Created by: The person responsible for this experiment.
- Running days: Leave empty; this cell contains a formula to calculate how many days your A/B test ran and also to let you know of the A/B tests that are still ongoing.
- Purpose: Clearly define the purpose of this test. The purpose should identify what you are going to test and why. You can use this template to fill out that cell if you don’t have any other ideas:
To test if [INSERT CHANGE HERE] has a positive impact on [INSERT METRIC HERE];
E.g. to test if personalizing the headline with the user’s location has a positive impact on the signup conversion rate.
Important: Do not run A/B tests without having a clear purpose in mind. There is only a limited amount of A/B tests that you can run on a given time (since you are limited by how much traffic you have to experiment on). There is also a big chance that a random A/B test without any purpose will end up underperforming the control, and therefore, temporarily decrease your revenue/sign ups or your business performance in general.
- Testing Element: Define which element on your page you are going to be testing.
E.g. Hero Headline
Important: You should only test one element at a time. Testing multiple elements (E.g. Changing the Headline, the Sub-headline, the copy, and the form location) at once will leave you wondering which of the changes actually had a positive impact and which of those had a negative impact since your results will only show the aggregated data. If you want to have multiple elements changed at once you should run a Multivariate Test (MVT) (outside of the scope of this SOP).
- Audience: Define which audience you are going to be targeting for your experiment. You can run experiments only for a specific group of people. Depending on which tool you are using,g you might be able to target specific Devices, Countries, Traffic Referrals and Traffic Sources, Browsers, etc.
E.g. All US Visitors
- Success Metrics #1, #2, #3: Define which metrics you want to use to evaluate the success/failure of an experiment. You should add the metrics in order of importance to the given experiment (the most important metric should go first, and the least important should go last).
E.g.
Metric #1: Signup Conversion Rate
Metric #2: Revenue
Metric #3: Bounce Rate
- Version A, Version B: Insert a URL with a screenshot of your control version (Version A) and add a URL with a screenshot of your test version (Version B).
If you don’t have a tool to screenshot your page yet, you can use the Awesome Screenshot Chrome Extension. It’s free, and it offers you a way to screenshot the entire page in a single click:
- Testing Page URL: This is the URL of the landing page where your test will be running.
E.g. http://www.asiteaboutemojis.com/
- Experiment URL: This is your A/B testing tool’s experiment URL. Typically (depending on which tool you are using), this is the URL that will allow you to configure your experiment and check the experiment’s results. If you haven’t set up your experiment yet, leave it blank and come back to the spreadsheet to update it once done.
- Results: Once your A/B test has ended, this is where you should log your results so that in a few months, you can look back and understand how your previous experiments went or share it with your team so they are all aware of them.
E.g. personalizing the headline with the user's location increased signups by 36% in the US. It also increased revenue by 12%, and the Bounce Rate decreased by 10%. The test ran for 3 weeks, and a statistically significant result was reached, with version B having a probability to be the best of 95%+ on all metrics.
Finalizing your A/B test using VWO
Note: If you haven’t set up VWO yet, you can do it now by following SOP 059: Set up an A/B test on your Website (web version). This is free and recommended. Once your test is live, return to this SOP to proceed with monitoring and analysis.
Monitor Your Test
Once the experiment is running, keep track of its progress:
01▸
On the navigation menu, go to "Testing" → "A/B" → click on your A/B test.
- Check Traffic Split: Ensure traffic is evenly distributed between variations.
- Monitor Test Duration: Let the test run for at least 14 days (or until statistical significance is reached).
02▸
Go to "Insights" → "Metric Reports" → "Track Metrics" to track performance over time.
Review Key Metrics in VWO: Primary metric (e.g., conversion rate) Secondary metrics (e.g., bounce rate, revenue)__Note: __Watch for external factors. Consider if seasonality, marketing campaigns, or technical issues are affecting results. If you notice uneven traffic distribution or an unexpected drop in conversions, troubleshoot or pause the test.
Analyze Results in VWO
After the test has gathered enough data, follow these steps:
Check Statistical Significance
03▸ Go to "Testing" → select the A/B text you want to analyze → "Reports" tab → look for the Probability to Be Best (PBB) metric.
You need to ensure that the confidence level is 95%+ before making decisions. VWO provides a Probability to Be Best (PBB) metric to determine the likelihood that a variation is better than the control. Find the PBB value for each variation:
-
- If PBB is 95% or higher, the variation is statistically significant.
- If PBB is below 95%, the result is inconclusive, and you may need to extend the test duration or collect more data.
Compare Version A vs. Version B
Look at performance across key metrics:
-
- Did the test variation improve conversion rates?
- How did secondary metrics (e.g., revenue, engagement) change?
Segment the Data
Use VWO’s segmentation features to analyze results by:
-
- Device (desktop vs. mobile)
- Traffic source (organic, paid, referral)
- Geographic location
04▸ Decide Next Steps
05▸ If Version B performs better, implement changes permanently. If results are inconclusive, consider running the test longer or tweaking variations. If Version A wins, document findings and brainstorm new test ideas.
Document & Share Findings
Update your A/B Test Documentation Spreadsheet with the results. Share key insights with your team to inform future experiments. Plan your next test based on your learnings. Repeat the A/B testing process to continually optimize your website performance.