In post 1 in the series we set up a CMS manageable Search Algorithm Settings Page and plugged that into our Search & Navigation query.

Experimenting with your Keyword Search Algorithm – 1 of 2

We’ll now build on that to create an Experiment that will help us determine the optimal algorithm configuration.

Extend the Search Algorithm Settings Page

Firstly we will extend our Algorithm Settings Page with the following properties

        [CultureSpecific]
        [Display(Name = "Is Default", Description = "Indicates that this content represents the default Search Algorithm configuration", GroupName = TabNames.Experimentation, Order = 200)]
        public virtual bool IsDefault { get; set; }

        [CultureSpecific]
        [Display(Name = "Experiment Key", Description = "Matches the variation key of the Optimizely Experiment", GroupName = TabNames.Experimentation, Order = 210)]
        public virtual string ExperimentKey { get; set; }

When we set up our experiment these culture specific properties will allow us to set a default configuration and experiment with algorithms across language sites.

Setting up the Experiment

In my previous Optimizely Experiments series, I talked about setting up and running an Experiment which covers the basics of the Optimizely Full Stack product. Rather than revisit I’m going to assume the foundations and concepts explained in this series are understood.

My Experience with Optimizely Fullstack (via Rollouts) – 1 of 4

Experiment Event Metrics

Log into your Optimizely Full Stack account to begin configuration.

Define a new metric for this Experiment Called “Keyword Search Click Thru”. This event will be registered when a customer clicks a search result from a keyword search. Higher numbers of these events will indicate that the algorithm is returning relevant results to our customers.

If you are using Optimizely Search and Navigation typed search, you will most likely have Click Tracking implemented in your solution. Read more on that here:

Is Click Tracking working in your EPiServer Find implementation?

Assuming you are using Optimizely Search and Navigation you could push the event to the Experiments API at the same point that you register the search click tracking.

Experiment Set Up

Lets now publish variations of our Search Algorithm Settings page in the CMS, setting the Experiment Key to unique values for each instance of the page.

Then log into Optimizely Full Stack and configure your Experiment Variations noting that we will use the “experiment_key” variable to map to the settings page instance.

Finally we can configure our experiment to push to AB test instances of our Algorithm Settings page

Executing the Experiment

To retrieve the Experiment Key in code using the Experiment Service detailed in my previous blog series you simply need a few lines of code

                var decision = _experimentationService.Decide(HttpContext, "keyword_search_algorithm_experiment");
                if (decision != null && decision.Enabled)
                {
                    return _experimentationService.GetFeatureVariableString(HttpContext, decision.FlagKey, "experiment_key");
                }

Now that you have the key, you can use this to load the appropriate Algorithm Settings page and AB test the effectiveness of your search configuration.

Conclusion

Search Algorithms are tricky and the optimal configuration for each Commerce application depends on too many moving parts to predict such as catalog size, data, user intent, market specific preferences etc.

Using Experimentation to measures your algorithm success, tweak, learn and measure again is the best possible way to move towards the sweet spot and grow your conversions.

During this series of posts we have talked about working with Optimizely Fullstack from a development perspective and integrating with a Commerce application.

In this post we will get into the fun stuff and implement a simple experiment.

The Experiment

For one of our E-Commerce clients, there are differing theories on what Category Listing Page sorting algorithm will result in customers seeing products they are more likely to buy.

The default sorting for listing pages on this website was by Newest products. The theory is that customers who interact with this brand prefer to see the latest products.

A hypothesis is that more sales would convert if products were displayed in order of Best Seller. Best Seller is a numeric value which contains the accumulated purchases of that products variants over 2 years. It will favour older and well known products.

Category listing pages perform well so any change to a Newest algorithm as default needs to be evidence driven. A perfect first Experiment!

Optimizely Experiment Configuration

With the Optimizely Commerce integrations discussed in the previous posts in place, we can proceed to configure our first experiment as follows:

Audience

Attributes and Segmentation

You can target particular segments of your audience with an experiment through setting “Audience Attributes”. These attributes are synced to Optimizely in the User Context of each Optimizely tracking request.

A number of audience attributes were already integrated in David’s initial Experiments project which I extended and can be viewed in the User Retriever – GetUserAttributes method.

We decided to roll this experiment out across all markets and devices so set our Audience to the default “Everyone”.

Even though we are not using Audience Attributes to target a subset of our audience for the experiment, they are still very important. Optimizely Experiment Reporting allows you to filter by these attributes to see how experiment results might differ across segments of the audience. It’s very powerful and I’ll talk about this more in the fourth and final post in this series.

Audience Percentage

We decided to target 50% of Everyone as this will give us large enough sample size to accurately measure the results of the experiments. This means that 50% of all visitors will be included in the experiment. The other 50% will simply not be included so the site will function on the current defaults.

Metrics

Metrics are events which are integrated in Events Tracking Service covered in the previous post.

These metrics are what allows you to define what success is for your experiment. Quite a few come out of the box with this implementation so it’s worth checking out. The metrics for our experiment are:

• Overall Revenue – The most important metric. This tells us what variation of the experiment is generating the most revenue.
• Over Value – An interesting metric which will tell us if there is a difference is average order value across variations.
• Product – In which variation of the experiment is a customer more likely to click through to view a product details page.

Variations

Here we define what variations of the experiment we want customers included in the Experiment audience to get. In our example we are running:

Best Sellers – 50%

Newest – 50%

You could easily add more supported sort orders to the experiment.

Experiment Code

In the following commit I added an Experimentation Service class that can be used to easily call Optimizely to decide on Experiment Results. The Decide method integrates with the User Context so you don’t need to repeat this code from your Web Application layer.

https://github.com/johnnymullaney/Optimizely-Experiments/commit/39de71f0135984e9ce1b79100c04cf0cc40a4ddf

In this snippet from the Category Listing Page controller, if no sortBy parameter is included as a query string parameter, Optimizely will decide if the customer session should be included in the Experiment.

If Optimizely decides to Enable the experiment, you can then extract the variation from the OptimizelyExperiment object.

With our Experimentation code layer in place, adding the experiment was as easy as these few lines of code.

public async Task<ActionResult> Index(ProductCategoryListingPage currentPage, string sortBy = null)
        {

            if (sortBy == null)
            {
                // integrate with optimizely experiment
                var decision = _experimentationService.Decide(HttpContext, "plp_sort_order_experiment_experiment");
                if (decision.Enabled)
                {
                    // retrieve Experiment Variation
                    sortBy = decision.Variables.GetValue<string>("default_sort_order");
                }
            }

            // proceed to get results from Find

Next and Final Post

We now have set up and configured our first experiment.

In the next and final post we will talk about Optimizely Reports, the insights gained from this particular experiment and what it means for the roadmap for this Commerce application going forward.

There are some interesting findings. Stay tuned! 🙂