In this final post of the series, we will investigate the result of a real world experiment and discuss what the results meant to the roadmap of this project.

So far we have talked about working with Optimizely Fullstack from a development perspective and integrating with a Commerce application. Then we configured and coded our first experiment in a few lines of code.

A Real World Experiment

Just like the Experiment we have talked through setting up in this series, we ran an experiment for an E-Commerce website to determine the optimal Sort Order for category pages.

This websites category pages defaulted to display Newest products first. The theory was that customer interact with this website to purchase new release products.

However there was a hypothesis that:

If customers view products on category pages with Best Selling products first, the following success criteria will be proven:

• Customers will click to view more products
• Average Order Value will increase
• Revenue will increase

Real World Results

With Event Tracking enabled as discussed in the previous posts in the series, Optimizely provided us with the following results on each of our success criteria.

Overall Results

We rolled the experiment out to just 10% of total sessions. Even with a low percentage we began to see interesting trends.

At this early point in the Experiment, Customers who were served Best Sellers by default:

• Were as likely to click through to view a product details page on either New or Best selling products. No notable difference
• Had an Average Order Value 10% behind New Products
• Were 19% behind on generating revenue

Of course you need much more sessions and to run the experiment over a longer period of time for the result to achieve significance but these early findings were interesting.

It showed the original hypothesis for the Experiment was likely wrong. The Experiment was failing – and that in itself was a game changer.

What else did we find out

Using Optimizely Reports, you can refine your Results by any Audience Attribute.

Refining the reports per language and/or market showed that some Markets had a huge preference for New products. It didn’t seem to matter for other Markets where the difference was insignifant.

Another game changer. We need to refine and improve our algorithms to best serve products to users of different markets.

And how do we do that?

1. We are going to create our hypothesis for what customers prefer in each market based on analytics and experimentation results
2. We are going to run experiments for each market as we work through a program of experimentation, analysis and continued optimisation.

And this is just for the Category page sort order!

Conclusions

Starting is Low Cost & Easy!

Sign Up for a Free Rollout Accounts.

Use the code bases discussed in this series to get started really quickly.

Start with a simple experiment.

1 Experiment leads to many more

We picked a simple Experiment – Sort Order on Category pages. From that experiment came data on customer behaviour.

From that data came questions. A big finding was that customers seem to have quite different expectations across markets.

Let’s run more targeted experiments to find out more on this…

We should also run experiments on all key functionality as we continue to optimise conversions.

Strategic Process Changes

Well designed Experiments are a low cost way of validating assumptions around how your customers interact either with the current site or may interact with new features being thought up.

Run simple Experiments with MVP implementations to prove value before proceeding with full delivery of an expensive feature. Failed experiments are valuable. This ensures budget is used effectively.

Also Experiment to optimise existing user journeys and functionality to hit the sweet spot across all segments of users. And then continue to measure. It is guaranteed that sweet spots will move over time.

In post 1 of 4 we talked about getting started with Optimizely Experimentations on Commerce.

I have taken David’s original experiments project, pushed a version to my GitHub repository and added some updates that may be helpful for some others working with Commerce applications.

In this post I’ll discuss some of the specifics around this.

The Github repository is available:

johnnymullaney/Optimizely-Experiments (github.com)

Experiment Tracking Service

The foundation experiments code base does a really nice job of minimising library dependencies by using EPiServer Tracking to intercept the payload and send tracking events to Optimizely.

My situation was a little different. Visitor Intelligence is obsolete since the Zaius CDP acquisition .

I decided to add the EPiServer.Commerce.Core library as a dependency to the project. This allowed me to write a Tracking Service class that could be easily called from the main application.

Added Tracking Service to handle events called directly from the web … · johnnymullaney/Optimizely-Experiments@fd19b94 (github.com)

Page View Tracking

To improve maintainability in you web application project, you can create a page view tracking attribute like below. The code below could easily be extended to CMS:

    public class ExperimentPageViewTrackingActionFilter : ActionFilterAttribute
    {
        public override void OnResultExecuting(ResultExecutingContext filterContext)
        {
            base.OnResultExecuting(filterContext);

            var viewResult = filterContext.Result as ViewResult;
            if (viewResult == null)
            {
                return;
            }

            var experimentTrackingService = ServiceLocator.Current.GetInstance<ITrackingService>();
            var currentLanguage = ServiceLocator.Current.GetInstance<ICurrentLanguage>();
            var httpContext = new HttpContextWrapper(HttpContext.Current);

            if (viewResult.Model is ProductListingPageViewModel listingPageViewModel)
            {
                experimentTrackingService.TrackProductListingEvent(httpContext, listingPageViewModel.CurrentPage.Name, currentLanguage.GetCurrentLanguage());
            }

            // check for Commerce Page types
            if (viewResult.Model is ProductDetailPageViewModel)
            {
                experimentTrackingService.TrackProductPageView(httpContext, currentLanguage.GetCurrentLanguage());
            }
        }
    }

Tracking User Interactions

User Interactions like Adding to Cart and Creating an Order can easily be tracked using the following code snippets

_experimentTrackingService.TrackOrderEvent(HttpContext, cart, _currentLanguage);

 _experimentTrackingService.TrackBasketEvent(HttpContext, lineItem, _currentCurrency, _currentLanguage);

Bot Filtering

Bot Filtering will exclude tracking events triggered from bots in your reports. Although the capability to filter these requests is not included in the free Rollouts plan, it’s a good to be extend your integration so it can easilt be turned on in future.

Optimizely’s documentation specifies that you should pass the reserved $opt_user_agent attribute in the Track, Activate, Is Feature Enabled, and Get Enabled Features functions.

How to filter out bots – Optimizely Full Stack

To enable bot filtering capabilties, I made the following update to the User Retiever class which generates the user context sent on each tracking request:

Added user agent reserved attribute https://docs.developers.optimizel… · johnnymullaney/Optimizely-Experiments@b2e5df7 (github.com)

Next Post

In the next post in the series, we’ll use the Experiments project to execute a simple experiment and discuss the impact of that experiment in a real world example.

I’ve been excited to recently get the opportunity to work with the Optimizely Experimentation platform for the first time. My goal has been to analyse the platform technically and demonstrate to clients how experimentation is a game changer in proving what generates results.

Optimizely Rollouts

Rollouts was the plan I started the journey on. At that stage we wanted to demonstrate the potential through running some simple experiments. The results would speak for themselves and open the door to move to a Full Stack plan.

Rollouts allows you to run an experiment on the free plan but you don’t get everything understandably. However what’s available for free is more than enough to start demonstrating results.

Be aware of the following free plan limitations:

Experiment Limits

The Rollouts limit is 1 active experiment at a time in each environment including Production. This is fair on a free plan!

Bot Filtering

Bot Filtering is not available in the free plan. This could skew the reporting metrics somewhat.

Experiments Rest API

The Experiments Rest API endpoint does not work in Rollouts which may limit some clever integrations. However if your going to start pushing the boundaries of a platform integration, you’re probably going to be investing in a paid plan!

Let’s Get Started

Sign Up for a free Rollouts account here.

After that you need to extend your Commerce code base to integrate with Optimizely Experiments through the C# SDK.

This series of posts by David Knipe is a great place to start if you are integrating with the Optimizely CMS or Commerce platforms.

Integrating Optimizely Full Stack with Episerver | david-tec.com

His Foundation Experiments branch on GitHub can easily be added to your solution. It provides some really neat integrations out of the box such as with Optimizely Projects and Visitor Groups. His blog series will bring you through all this.

episerver/foundation-experiments: Foundation Experiments offers a starting point for integrating Optimizely Full Stack into an Episerver project (github.com)

Some Minor Code Base Updates

My project was an Optimizely Commerce application without Visitor Intelligence so I made the following updates to make the Optimizely Tracking and Decision integration seamless.

Added Commerce Core Library

I extended the code to integrate directly with EPiServer Commerce core library. This simplified the integration with my Commerce code base so I could pass classes like IOrderGroup for tracking.

User Retriever

Extended IUserRetriver to include a method that will return an object with the User Id and Attributes stored in one object.

I also added an extra reserved attribute to the user context which would enable bot filtering in reporting tools on a paid version.

Tracking Service

Added a Tracking Service class that can be called directly from the main Web Application to track various events.

Experimentation Service

Added an Experimentation Service class that can be called directly from the main Web Application to get experiment decisions and variables.

Next Post

In the next post I will link to my GitHub repository where I have pushed these changes and will talk through specifics of integrating into your Optimizely Commerce application.

Guide to setting up secure Content Approval Workflows – 1 of 2 – Johnny Mullaney

Guide to setting up secure Content Approval Workflows – 2 of 2 – Johnny Mullaney

Related to my previous series on setting up Secure Content Approval Workflows, I had a discussion with someone who followed the steps but their ERP Integration API was sporadically throwing the following error when updating content:

System.ComponentModel.DataAnnotations.ValidationException: Content is locked by ‘Epi Admin’ with lock identifier ‘contentapproval’

What Causes the Error?

This error is thrown when a version of content is in the middle of an approval workflow and the EPiServer Content Repository attempts to update the content. The error is saying that EPiServer will not allow this content to get updated while it is going through an approval workflow.

This makes perfect sense as if content is partially approved, we don’t really want to be updating it.

However the scenario explained to me by the developer and agreed with the client, was that this content should be updated regardless of the approval sequence.

Content Lock Evaluator

EPiServer’s IContentLockEvaluator determines if content is locked for editing under given circumstances. The default implementation is this internal class is as below.

  internal class ContentApprovalLockEvaluator : IContentLockEvaluator
  {
    public static string Identifier = "contentapproval";
    private readonly IContentVersionRepository _contentVersionRepository;
    private readonly IApprovalRepository _approvalRepository;

    public ContentApprovalLockEvaluator(
      IContentVersionRepository contentVersionRepository,
      IApprovalRepository approvalRepository)
    {
      this._contentVersionRepository = contentVersionRepository;
      this._approvalRepository = approvalRepository;
    }

    public ContentLock IsLocked(ContentReference contentLink)
    {
      ContentVersion contentVersion = this._contentVersionRepository.Load(contentLink);
      if (contentVersion == (ContentVersion) null)
        return (ContentLock) null;
      if (contentVersion.Status != VersionStatus.AwaitingApproval)
        return (ContentLock) null;
      ContentApproval result = this._approvalRepository.GetAsync(contentLink).Result;
      if (result == null)
        return (ContentLock) null;
      return result.Status != ApprovalStatus.InReview ? (ContentLock) null : new ContentLock(contentLink, result.StartedBy, ContentApprovalLockEvaluator.Identifier, result.Started);
    }
  }

Content in an approval workflow will return a lock status and block any updates.

Solution

The solution for this use case was to inject our custom implementation of the Content Lock Evaluator to allow content in an approval workflow to be updated. Our implementation would simply return null meaning the content is not locked.

   public class CustomContentLockEvaluator : IContentLockEvaluator
    {
        public ContentLock IsLocked(ContentReference contentLink)
        {
            return null;
        }
    }

Warning

Proceed with caution. This was the solution in a very specific integration where the risks of removing the Content Lock logic was considered and understood by all.

Depending on your situation, consider maintaining as much of the logic in the default EPiServer implementation as possible.

This post follows on from my previous post on setting up secure content approval workflows where we looked at the code updates necessary to consider. In this post we will complete the configuration of our content approval workflow to meet the requirements detailed in the original post.

User Groups

First set up the necessary groups in the Optimizely CMS Administrative interface. These groups will later be used to add users to the Content Approval Workflow.

• 

Create the following groups to match those matched to our virtual roles.

• ContentReviewers
• ContentPublishers

Then create three further roles which we will use to configure our language specific content approval workflows:

• EnglishReviewers
• FrenchReviewers
• SwedishReviewers

Catalog Access Rights

Next assign catalog access rights appropriately to our user groups.

Navigate to the Catalog in Commerce and Click the “Manage” button beside “Visible to Everyone”

Grant reviewers only access to change catalog content.

Users

All Content Reviewers will be added to the ContentReviewers group. This group gives them the appropriate catalog permissions.

We will then add a Content Reviewer to the appropriate language reviewer group. The language reviewer group will be used to configure the language specific Approval Sequence workflow. So for example the French reviewer will be added to both ContentReviewers and FrenchReviewers.

Content Approval Workflow Configuration

Finally we can use the groups created to configure the language specific content approval workflow required. The example below has a Content Reviewer group assigned to each language.

Members of these groups will be notified when a version of the content is assigned to them for review. They will then be able to either:

1. Approve
2. Decline -> Edit previous version -> Approve

Importantly a content reviewer will not have access to publish content or override an approval sequence. Any content that is not directly assigned via an approval workflow sequence will be read only.

On approval, members of the Content Publishers group will be notified. They can then do a final review across all languages before marking as Ready for Publish.

Optimizely Content Approvals are a mature and highly configurable feature. However every project is different and in designing an optimal workflow for our customers – it is important to plan accordingly to ensure a clean user experience while adhering to security principles when dealing with access rights.

The principle of least privilege (PoLP) refers to an information security concept in which a user is given the minimum levels of access – or permissions – needed to perform his/her job functions.

This is a key principle that we will take forward in designing our workflow.

Planning Content Approval Workflows

The key to planning an Approval workflow is defining the types of user roles who will be involved in a sequence.

For each user role you define, consider the “Principle of least privilege” in granting them permissions to your Optimizely system. We only want to give each role access that is absolutely necessary to the functioning of your optimal approval workflow.

Consider the following for each user role you are planning.

• Should members of this user role have access to CMS or Commerce content or both?
• Will the user role be responsible for approving or publishing content or both?
• Can users in a role override the Approval sequence to publish content that has not gone through it’s full workflow?

Working Example

In the rest of this series we’ll work through setting up an optimal workflow to meet a requirement.

The Requirement

• The Approval Workflow is to manage Commerce Content only
• Products are added programmatically though an API integration and should enter the approval sequence automatically
• Content to be approved only by designated language specific approvers (English, Spanish, French). Spanish approvers can only review Spanish content.
• The approvers have the ability to edit content during the review process
• Content in all languages is published by a user with publishing permissions

Our User Roles

Given this requirement we can define 2 distinct roles

Content Reviewers

• Edits and approves content assigned through a workflow
• Cannot publish content

Content Publishers

• Publish content in any language once assigned in the workflow after approval by a Content Approver
• Does not approve content
• However can override an approval sequence for a product to force the publishing even if it has not yet been approved by a Content Approvers.

Code Base Updates

Virutal Roles

If you’re not familiar – this page will explain Optimizely virtual roles: Virtual roles | Optimizely Developer Community (episerver.com)

The “CatalogManagers” virtual role grants access for the Catalog system in Commerce only.

We will define two roles for our system which both map to this CatalogManager virtual role:

ContentReviewers – Can review content that has been assigned

ContentPublishers – full permission to publish content. They have the ability override approval sequences and force publish if required

In the web.config map these roles to the “CatalogManagers” as follows:

      <add name="CatalogManagers" type="EPiServer.Security.MappedRole, EPiServer.Framework" roles="ContentReviewers, ContentPublishers" mode="Any" />

  

Avoid adding these roles to the “CommerceAdmins” virtual role. That should be kept for WebAdmins and Administrators only.

Content Repository Save Actions

The wrong Content Repository Save Action can cause the approval sequence to be overridden.

Review your code base to make sure that  content programmatically created that should go through an approval sequence uses the “Request Approval” save action.

  _contentRepository.Save(writableContent, SaveAction.RequestApproval, AccessLevel.NoAccess);

Next Post

In the next post we will proceed to configure Optimizely Access Rights, User Groups, Roles and finally the Approval Sequence to meet our requirement while adhering the principles outlined at the beginning of the post.