Experimenting with your Keyword Search Algorithm – 2 of 2

In post 1 in the series we set up a CMS manageable Search Algorithm Settings Page and plugged that into our Search & Navigation query.

We’ll now build on that to create an Experiment that will help us determine the optimal algorithm configuration.

Extend the Search Algorithm Settings Page

Firstly we will extend our Algorithm Settings Page with the following properties

        [Display(Name = "Is Default", Description = "Indicates that this content represents the default Search Algorithm configuration", GroupName = TabNames.Experimentation, Order = 200)]
        public virtual bool IsDefault { get; set; }

        [Display(Name = "Experiment Key", Description = "Matches the variation key of the Optimizely Experiment", GroupName = TabNames.Experimentation, Order = 210)]
        public virtual string ExperimentKey { get; set; }

When we set up our experiment these culture specific properties will allow us to set a default configuration and experiment with algorithms across language sites.

Setting up the Experiment

In my previous Optimizely Experiments series, I talked about setting up and running an Experiment which covers the basics of the Optimizely Full Stack product. Rather than revisit I’m going to assume the foundations and concepts explained in this series are understood.

Experiment Event Metrics

Log into your Optimizely Full Stack account to begin configuration.

Define a new metric for this Experiment Called “Keyword Search Click Thru”. This event will be registered when a customer clicks a search result from a keyword search. Higher numbers of these events will indicate that the algorithm is returning relevant results to our customers.

If you are using Optimizely Search and Navigation typed search, you will most likely have Click Tracking implemented in your solution. Read more on that here:

Assuming you are using Optimizely Search and Navigation you could push the event to the Experiments API at the same point that you register the search click tracking.

Experiment Set Up

Lets now publish variations of our Search Algorithm Settings page in the CMS, setting the Experiment Key to unique values for each instance of the page.

Then log into Optimizely Full Stack and configure your Experiment Variations noting that we will use the “experiment_key” variable to map to the settings page instance.

Finally we can configure our experiment to push to AB test instances of our Algorithm Settings page

Executing the Experiment

To retrieve the Experiment Key in code using the Experiment Service detailed in my previous blog series you simply need a few lines of code

                var decision = _experimentationService.Decide(HttpContext, "keyword_search_algorithm_experiment");
                if (decision != null && decision.Enabled)
                    return _experimentationService.GetFeatureVariableString(HttpContext, decision.FlagKey, "experiment_key");

Now that you have the key, you can use this to load the appropriate Algorithm Settings page and AB test the effectiveness of your search configuration.


Search Algorithms are tricky and the optimal configuration for each Commerce application depends on too many moving parts to predict such as catalog size, data, user intent, market specific preferences etc.

Using Experimentation to measures your algorithm success, tweak, learn and measure again is the best possible way to move towards the sweet spot and grow your conversions.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: