A/B Testing in EXM is a great way to learn from and optimise email campaigns to improve engagement results.  However I came across an an edge case that had some unintended consequences (at least in Sitecore 9.0 Update 1, perhaps in later releases?).

Very specifically, this issue is only apparent when you set up an EXM campaign to dispatch with the following:

• A multivariate (A/B) test with the winner selected Automatically
• It is scheduled for the future
• It is scheduled in a Timezone that is not UTC

We were seeing the test begin as usual, however once the test was complete the winner was not selected and the remainder of the campaign was not sent after the designated delay (stuck in the “A/B Testing” status).  This was odd, but I slept on it.

Investigating again in the morning, the campaign had completed in the middle of the night. 🤔  Odd.  So I set up another test with the winner to be selected just 1 hour after dispatch.  Same result.  Stuck in the “A/B Testing” after the winner should’ve been selected.

After some digging I got a better understanding of how the scheduling of EXM mail works.  At the time of scheduling the dispatch, a “Dispatch” item is created in the /sitecore/system/Settings/Email/Tasks node of the Sitecore tree.  This item has the scheduled time (stored as UTC with appropriate timezone info) and various bits of info to associate the task with a campaign.

When scheduling an A/B test, there is an additional task item created called “Select Winner” that is supposed to have a scheduled time (in this case 1 hour after the dispatch task starts) in line with the settings you created.

A close look revealed that the “Dispatch” Task got created as expected, with the appropriate scheduled time.  However the Select Winner task was created at the expected time, but in the UTC timezone.  In our case, being UTC +8, this meant the Winner was selected 8 hours after we would have expected. Not a HUGE issue, but undesirable.  What happens to a test scheduled in a UTC -6 timezone? Well…I would expect the Select winner task gets triggered almost immediately.

The workaround is to always schedule your A/B tests in UTC.  This will ensure everything is triggered at the correct time.  However, this can be error prone due to PEBKAC errors and our content editors have enough on their plate without always having to do time conversions.

After reporting to Support, there is now also a patch available, which Sitecore support should be able to provide guidance on whether it suits your solution.  Worked a treat for us.