Johnny Mullaney

This series will discuss an approach to experimenting with your Keyword Search Algorithm using the Optimizely Search & Navigation, Commerce Cloud and Experimentation products.

Search & Navigation

At a basic level the following happens on a keyword search:

• Customer attempts a keyword search on your Commerce Cloud website
• Commerce Cloud code builds a Search & Navigation query and posts to API
• Search & Navigation executes the query on it’s Elasticsearch index taking into account factors such as Statistics gathered through Click Tracking to assign a relevancy score to the products determined to be the best search results

Assuming click tracking is already implemented, the variable we have control over to tweak in optimising search performance is the search query sent to Search & Navigation.

Search Query Optimisation

To CMS manage the query, we create an Algorithm Settings Page that can integrate with the search query sent to the API.

Algorithm Settings Page

The following is an example of a Search Algorithm Settings page PoC from my Github:

https://github.com/johnnymullaney/Foundation/blob/search-algorithm-experiment/src/Foundation/Features/Search/Search/SearchAlgorithmSettingsPage.cs

I’ve divided the content type into three tabs representing important aspects of the query.

1 Property Weightings

Property weightings specify what perceived value we assign to a keyword match in various content property’s.

For example we may specify in the query that a keyword match in a Product Title should be twice as relevant as a match in the Description. This will be a key factor when Search & Navigation assigns relevancy score to search results based on that request.

2 Boosting

Your query can instruct Search & Navigation to boost the relevancy scores further of search results that meet a pre-defined conditions. For example a query may include a relevancy boosting for products that are marked in the CMS as “Popular”.

3 Query Optimisation

I am categorising query optimisations as anything else that may improve the quality of search results such as Synonyms, Fuzzy Search, And/Or operators etc.

There is a very useful Labs Github repository which contains a number of extensions that can be integrated with your Search & Navigation solution to improve search performance.

https://github.com/episerver/EPiServer.Labs.Find.Toolbox

Integrate Settings Page with Search Query

We can integrate our Search Algorithm Settings Page into the query that we build in code.

var query = _findClient.Search<GenericProduct>()
             .For(filterOptions.Q)
             .MinimumShouldMatch(searchAlgorithmSettings.MinimumShouldMatch)
                .InField(x => x.DisplayName,     searchAlgorithmSettings.DisplayNameWeighting)
                .InField(x => x.Brand, searchAlgorithmSettings.BrandWeighting)
                .InField(x => x.Description, searchAlgorithmSettings.DescriptionWeighting);

            if (searchAlgorithmSettings.EnableSynonymsImproved)
            {
                query = query.UsingSynonymsImproved();
            }
            else
            {
                query = query.UsingSynonyms();
            }

            if (searchAlgorithmSettings.EnableDisplayNameRelevanceImproved)
            {
                query = query.UsingRelevanceImproved(x => x.DisplayName);
            }

            if (searchAlgorithmSettings.EnableDisplayNameFuzzyMatch)
            {
                query = query.FuzzyMatch(x => x.DisplayName);
            }

            if (searchAlgorithmSettings.EnableDisplayNameWildcardMatch)
            {
                query = query.WildcardMatch(x => x.DisplayName);
            }

            if (searchAlgorithmSettings.PopularProductBoostingValue > 0)
            {
                typeSearch = typeSearch.BoostMatching(x => x.PopularProduct.Match(true), searchAlgorithmSettings.PopularProductBoostingValue);
            }

Next Post

In the next post, we’ll create an Optimizely Experiment to AB test the success of search query variations.

Maximising traffic acquisition through good SEO practices is a key strategic goal of any commercial website.

Recently I’ve been working with some brands on SEO optimisation from a technical perspective and have spent some time trying to map out what exactly “Good SEO” means.

So what is Good SEO?

That’s a tough question!

In my opinion Good SEO goes far beyond any particular discipline. It means great health and performance across a number of overlapping broad areas from technical to the quality of content and beyond. To execute effectively in each area entails strong process, execution and ongoing health checks.

The broad area’s I look at SEO through are as follows:

1. Technical Best Practices
2. Crawlability / Indexation  optimisations so crawlers can efficiently discover the entire site
3. On Page HTML optimisations so crawlers can understand the intent and content of your page displaying relevant information in Search Engine Results Pages
4. International domain strategy linking content across languages and markets
5. Off Page brand health contributing to a brands authority

I attempted to map SEO considerations in these areas out in the diagram below. For diagramming simplicity each point is allocated to a particular area but I acknowledge that many of the points can overlap multiple areas. Please feel free to leave a comment below if you see something I have omitted.

SEO with Optimizely

Optimizely CMS and Commerce Clouds provides the framework to enable really strong SEO practices.

By practices I mean both high quality performant technical implementations coupled with the tooling to enable good content planning & management processes across your organisation.

There are great community add-ons that are extendable to provide really strong SEO foundations.

Site Health Checks

Websites we work with often have tens of thousands of pages or more. A stray piece of code released on production or content management mistakes can cause real SEO issues. These issues will often be invisible… until they are discovered. So the challenge is discovering the issue before they cause a commercial impact.

I’ve worked with and recommend SEMRush as a tool. You can automate SEO Site Audits to execute regularly and email metric driven reports detailing areas like HTML Markup errors, International SEO errors, Crawlability, Site Performance and a number of other metrics. It also plugs into Google Search Console and Google Analyics so you can view current performance within the reports.

Regularly tracking these metrics means you can identify issues early as you improve your site health scores over time.

Next Post

In this post I’m simply sharing details about something I have been working on in the hope that someone will find it useful.

This is a really big topic so am happy to extend to posts on more specific areas with regard to good technical practices. If it’s useful, let me know 🙂

There are lots of good quality Blog and Forum posts about differing ways to implement Block level Donut style Output caching in Optimizely.

The below is the approach I prefer to use. It keeps things simple while making sure that cache is unique for visitor groups and languages. Using this method cache will also be invalidated when a new version of a block is published.

Block controller

Add the Output Cache attribute to the Block Controller index method.

    [OutputCache(VaryByParam = "currentBlock", VaryByCustom = "language", Duration = 600)]
    public ActionResult Index(ProductListBlock currentBlock)

VaryByParam: Setting this to “currentBlock” will cache based on the ToString() method of the BlockData model returning the cache key. We add our logic there to make sure cache is refreshed appropriately.

VaryByCustom: Used to extend output caching with custom requirements. You must handle logic for the custom string by overriding the GetVaryByCustomString method in the Global.asax. We use this to cache language specific versions of our block.

Block Model

Override the Block model ToString() method to return unique cache keys for published versions of the block.

    public override string ToString()
    {
        var content = this as IContent;
        var changeTrackable = this as IChangeTrackable;

        if (content != null && changeTrackable != null)
            return $"{content.ContentGuid}-{changeTrackable.Saved.Ticks}-{EPiServer.Editor.PageEditing.PageIsInEditMode}";

        return base.ToString();
    }

Global.asax

Override the GetVaryByCustomString method in the Global.asax for the custom string passed into the OutputCache declaration. In our case this makes sure that language versions of the block are uniquely cached.

    public override string GetVaryByCustomString(HttpContext context, string custom)
    {
        if (custom == "language")
        {
            return HttpContext.Current.Request.Cookies["Language"]?.ToString();
        }
        return base.GetVaryByCustomString(context, custom);
    }

Personalising Headless CMS content across channels can be achieved with Optimizely Visitor Groups when a property in the http request can identify the source.

For example a mobile app request sent to a Headless CMS could include the following Http Header key value: “mobile-app-request”:true

Request Header Visitor Group Criterion

A visitor group criterion is required that can can match a request if it sees a header with a specified value.

We have pushed the following GitHib repo containing source code for a Request Header Visitor Group Criterion packaged so that it can be added to a nuget feed.

https://github.com/made-to-engage/EPiServer-Request-Header-Visitor-Group

Once installed, CMS Editor’s have the ability to personalise content for channels.

In this final post of the series, we will investigate the result of a real world experiment and discuss what the results meant to the roadmap of this project.

So far we have talked about working with Optimizely Fullstack from a development perspective and integrating with a Commerce application. Then we configured and coded our first experiment in a few lines of code.

A Real World Experiment

Just like the Experiment we have talked through setting up in this series, we ran an experiment for an E-Commerce website to determine the optimal Sort Order for category pages.

This websites category pages defaulted to display Newest products first. The theory was that customer interact with this website to purchase new release products.

However there was a hypothesis that:

If customers view products on category pages with Best Selling products first, the following success criteria will be proven:

• Customers will click to view more products
• Average Order Value will increase
• Revenue will increase

Real World Results

With Event Tracking enabled as discussed in the previous posts in the series, Optimizely provided us with the following results on each of our success criteria.

Overall Results

We rolled the experiment out to just 10% of total sessions. Even with a low percentage we began to see interesting trends.

At this early point in the Experiment, Customers who were served Best Sellers by default:

• Were as likely to click through to view a product details page on either New or Best selling products. No notable difference
• Had an Average Order Value 10% behind New Products
• Were 19% behind on generating revenue

Of course you need much more sessions and to run the experiment over a longer period of time for the result to achieve significance but these early findings were interesting.

It showed the original hypothesis for the Experiment was likely wrong. The Experiment was failing – and that in itself was a game changer.

What else did we find out

Using Optimizely Reports, you can refine your Results by any Audience Attribute.

Refining the reports per language and/or market showed that some Markets had a huge preference for New products. It didn’t seem to matter for other Markets where the difference was insignifant.

Another game changer. We need to refine and improve our algorithms to best serve products to users of different markets.

And how do we do that?

1. We are going to create our hypothesis for what customers prefer in each market based on analytics and experimentation results
2. We are going to run experiments for each market as we work through a program of experimentation, analysis and continued optimisation.

And this is just for the Category page sort order!

Conclusions

Starting is Low Cost & Easy!

Sign Up for a Free Rollout Accounts.

Use the code bases discussed in this series to get started really quickly.

Start with a simple experiment.

1 Experiment leads to many more

We picked a simple Experiment – Sort Order on Category pages. From that experiment came data on customer behaviour.

From that data came questions. A big finding was that customers seem to have quite different expectations across markets.

Let’s run more targeted experiments to find out more on this…

We should also run experiments on all key functionality as we continue to optimise conversions.

Strategic Process Changes

Well designed Experiments are a low cost way of validating assumptions around how your customers interact either with the current site or may interact with new features being thought up.

Run simple Experiments with MVP implementations to prove value before proceeding with full delivery of an expensive feature. Failed experiments are valuable. This ensures budget is used effectively.

Also Experiment to optimise existing user journeys and functionality to hit the sweet spot across all segments of users. And then continue to measure. It is guaranteed that sweet spots will move over time.

In post 1 of 4 we talked about getting started with Optimizely Experimentations on Commerce.

I have taken David’s original experiments project, pushed a version to my GitHub repository and added some updates that may be helpful for some others working with Commerce applications.

In this post I’ll discuss some of the specifics around this.

The Github repository is available:

johnnymullaney/Optimizely-Experiments (github.com)

Experiment Tracking Service

The foundation experiments code base does a really nice job of minimising library dependencies by using EPiServer Tracking to intercept the payload and send tracking events to Optimizely.

My situation was a little different. Visitor Intelligence is obsolete since the Zaius CDP acquisition .

I decided to add the EPiServer.Commerce.Core library as a dependency to the project. This allowed me to write a Tracking Service class that could be easily called from the main application.

Added Tracking Service to handle events called directly from the web … · johnnymullaney/Optimizely-Experiments@fd19b94 (github.com)

Page View Tracking

To improve maintainability in you web application project, you can create a page view tracking attribute like below. The code below could easily be extended to CMS:

    public class ExperimentPageViewTrackingActionFilter : ActionFilterAttribute
    {
        public override void OnResultExecuting(ResultExecutingContext filterContext)
        {
            base.OnResultExecuting(filterContext);

            var viewResult = filterContext.Result as ViewResult;
            if (viewResult == null)
            {
                return;
            }

            var experimentTrackingService = ServiceLocator.Current.GetInstance<ITrackingService>();
            var currentLanguage = ServiceLocator.Current.GetInstance<ICurrentLanguage>();
            var httpContext = new HttpContextWrapper(HttpContext.Current);

            if (viewResult.Model is ProductListingPageViewModel listingPageViewModel)
            {
                experimentTrackingService.TrackProductListingEvent(httpContext, listingPageViewModel.CurrentPage.Name, currentLanguage.GetCurrentLanguage());
            }

            // check for Commerce Page types
            if (viewResult.Model is ProductDetailPageViewModel)
            {
                experimentTrackingService.TrackProductPageView(httpContext, currentLanguage.GetCurrentLanguage());
            }
        }
    }

Tracking User Interactions

User Interactions like Adding to Cart and Creating an Order can easily be tracked using the following code snippets

_experimentTrackingService.TrackOrderEvent(HttpContext, cart, _currentLanguage);

 _experimentTrackingService.TrackBasketEvent(HttpContext, lineItem, _currentCurrency, _currentLanguage);

Bot Filtering

Bot Filtering will exclude tracking events triggered from bots in your reports. Although the capability to filter these requests is not included in the free Rollouts plan, it’s a good to be extend your integration so it can easilt be turned on in future.

Optimizely’s documentation specifies that you should pass the reserved $opt_user_agent attribute in the Track, Activate, Is Feature Enabled, and Get Enabled Features functions.

How to filter out bots – Optimizely Full Stack

To enable bot filtering capabilties, I made the following update to the User Retiever class which generates the user context sent on each tracking request:

Added user agent reserved attribute https://docs.developers.optimizel… · johnnymullaney/Optimizely-Experiments@b2e5df7 (github.com)

Next Post

In the next post in the series, we’ll use the Experiments project to execute a simple experiment and discuss the impact of that experiment in a real world example.

I’ve been excited to recently get the opportunity to work with the Optimizely Experimentation platform for the first time. My goal has been to analyse the platform technically and demonstrate to clients how experimentation is a game changer in proving what generates results.

Optimizely Rollouts

Rollouts was the plan I started the journey on. At that stage we wanted to demonstrate the potential through running some simple experiments. The results would speak for themselves and open the door to move to a Full Stack plan.

Rollouts allows you to run an experiment on the free plan but you don’t get everything understandably. However what’s available for free is more than enough to start demonstrating results.

Be aware of the following free plan limitations:

Experiment Limits

The Rollouts limit is 1 active experiment at a time in each environment including Production. This is fair on a free plan!

Bot Filtering

Bot Filtering is not available in the free plan. This could skew the reporting metrics somewhat.

Experiments Rest API

The Experiments Rest API endpoint does not work in Rollouts which may limit some clever integrations. However if your going to start pushing the boundaries of a platform integration, you’re probably going to be investing in a paid plan!

Let’s Get Started

Sign Up for a free Rollouts account here.

After that you need to extend your Commerce code base to integrate with Optimizely Experiments through the C# SDK.

This series of posts by David Knipe is a great place to start if you are integrating with the Optimizely CMS or Commerce platforms.

Integrating Optimizely Full Stack with Episerver | david-tec.com

His Foundation Experiments branch on GitHub can easily be added to your solution. It provides some really neat integrations out of the box such as with Optimizely Projects and Visitor Groups. His blog series will bring you through all this.

episerver/foundation-experiments: Foundation Experiments offers a starting point for integrating Optimizely Full Stack into an Episerver project (github.com)

Some Minor Code Base Updates

My project was an Optimizely Commerce application without Visitor Intelligence so I made the following updates to make the Optimizely Tracking and Decision integration seamless.

Added Commerce Core Library

I extended the code to integrate directly with EPiServer Commerce core library. This simplified the integration with my Commerce code base so I could pass classes like IOrderGroup for tracking.

User Retriever

Extended IUserRetriver to include a method that will return an object with the User Id and Attributes stored in one object.

I also added an extra reserved attribute to the user context which would enable bot filtering in reporting tools on a paid version.

Tracking Service

Added a Tracking Service class that can be called directly from the main Web Application to track various events.

Experimentation Service

Added an Experimentation Service class that can be called directly from the main Web Application to get experiment decisions and variables.

Next Post

In the next post I will link to my GitHub repository where I have pushed these changes and will talk through specifics of integrating into your Optimizely Commerce application.