Experimentation and A/B Testing with Optimizely
The marketing team suspects that users don’t click on the “Shop Now” button as often as they would if it was more prominent on the page. Even though A/B Testing is comparing performance between two pages, in this lab we will experiment and create multiple variations of PartsUnlimited website and see which performs better using Optimizely.
Note: To get a reliable results in A/B testing, the experiment should run long enough to collect data to draw a conclusion. Optimizely lets you know this on their “Results” page.
Note: Both website variations created in this HOL are relatively simple and the main purpose of them is to show an idea of how useful A/B Testing can be. A more complicated website variations might require writing/modifying JavaScript and more complicated CSS modifications.
DevOps MPP Course Source
- This lab is used in course Application Monitoring and Feedback Loops - Module 4.
Pre-requisites:
- Complete the lab Continuous Integration with Visual Studio Team Services
- Complete the lab Continuous Deployment with Visual Studio Team Services
- Visual Studio 2017
Lab Tasks:
- Setup first variation in Optimizely
- Integrating Optimizely into PartsUnlimited
- Setup second variation in Optimizely
- Updating settings of your experiment
- Viewing results of your experiment
Task 1: Setup first variation in Optimizely
Step 1. Create an account with Optimizely. Login after confirming your email address and entering password.
Step 2. After being navigated to a “Welcome to Optimizely” page, click on “Create Web Project”.
Step 3. Enter your production
site URL as a name for the new project and click on “Create Project”.
Step 4. Create a new A/B Test.
-
Click on “Create New” then “A/B Test”.
-
Give experiment a name, then enter your
production
site URL as the “Pages” (you may need to select “Create New Page” if it hasn’t appeared) and Set “Audiences” to Everyone which is the default value. click on “Create Experiment”. -
Then Set the Metrics. Audiences and Metrics will be changed later. Click on “Create Experiment”.
Step 5. Dismiss tutorial if it pop-ups. Let’s define the first variation where we will change the styling of “Shop Now” button.
-
Click on
Champaigns > Variation #1
button in the right bottom corner. -
Click on
Create
ThenElement Change
- Enter the selector as “.carousel-link”.
-
Enter the color
rgba(28,54,124,1)
or#1c367c
- Enter the
Enable Event Tracking
Track click on this element: check
Name: Shop Button
Category: Converted
- Click on “Save”.
Task 2: Integrating Optimizely into PartsUnlimited
Step 1. Add the snippet of code you took a note of from Optimizely’s website to the head tag of _Layout.cshtml
file in the PartsUnlimitedWebsite/Views/Shared/ directory.
Step 2. Commit changes using the following commands:
git add .
git commit -m "Added integration with Optimizely"
git push
Step 3. Once the CI build and deployment to dev
slot is completed, push your changes to the staging
slot and finally production
environment.
Task 3: Setup second variation in Optimizely
Let’s go back to Optimizely website. Navigate Champaigns”, click on the experiment created in this lab.
Step 1. Click on “+ Add Variation”, ignore a warning about making changes while the experiment is running.
Step 2. Let’s remove “Fork me on Github” element in the top left corner from this variation. Click on it.
Then chagne Layout > Visibility to “Removed”.
Step 3. In this variation background color of all “Shop Now” buttons will be changed.
- Change background color of the “Shop Button” in the carousel the same way as in variation 1.
-
Enter the
Enable Event Tracking
the same way as in variation 1 as well. - If you scroll down you will see that all elements in “Arrivals” and “Top selling” groups also contain their own “Shop Now” buttons. Their color hasn’t been changed because
.carousel-link
class is not used for these buttons. -
We hope the color is like this.
-
If you were to look at the DOM, you would see that all those “Shop Now” buttons are in div tags with class “shop-now”, i.e. changing the background color of that class will change background color of all “Shop Now” buttons.
-
To change the color of these buttons, select the selector
.shop-now
and change color the same way as in variation 1. - Optionally: Create another tracking goal for every of these buttons.
Task 4: Updating settings of your experiment
Let’s look at the main settings that could be adjusted for an experiment. Navigate “Home” and click on the experiment created in this lab.
Step 1. Traffic Allocation:
- Click on “Edit” next to “Traffic Allocation: 100%”.
- Adjust amount of traffic to participate in this experiment.
- Next, adjust the chance of seeing a particular variation of the website for the users included in this experiment.
-
Click on “Save”.
Note: If 0% are included then none of the users will see any changes.
Step 2. Metrics:
- Click on “Metrics”.
- Set “Shop Button” goal as a primary goal for this experiment.
- Add any other goals you want to include in this experiment.
-
Click on “Save”.
Step 3. Audiences:
- Click on “Audiences”.
-
Click on “Create a New Audience”
-
Enter Name for your audience. Drag “Device” condition to “Audience Conditions” or any other condition you want to use and set it up as you wish. Click on “Save”
Note: It’s possible to create logical expressions using these conditions by dragging them into “and” or “or” sections.
- Click on “Save”.
Step 4. Pages:
- Click on “Pages”.
- Let’s add
dev
andstaging
slots to this experiment. -
Click on “Save”
Step 5. Schedule:
- Click on “Schedule”.
- Select time when experiment will start
- Select time when experiment will end
- Select time zone.
-
Click on “Save”
Task 5: Viewing results of your experiment
Let’s view some data collected by Optimizely.
Note: Data can be generated by opening website in Incognito mode a few times and clicking on “Shop Now” button
Step 1. Click on “Results” button.
Step 2. Once navigated to the “Results” page, you will be able to see and break down all of the collected information. This data can be shared with stakeholders by clicking on “Share” button near the top right corner.
Step 3. This particular data indicates that users are more likely to click on the “Shop Now” button in “Variation #2” of the website than in any other variations.
Note: Keep in mind that if the sample size of this experiment is not large enough then this data would most likely be meaningless.
Summary
In this lab, you have learned how to integrate with Optimizely, create different variations of your website, set audience and traffic allocation for an experiment, view and export results.