Head-to-Head Performance Test Review: Salesforce Einstein AI

by | Jun 7, 2022 | eCommerce, Hospitality & Vacation Rentals, Success Stories

Contending Salesforce

Salesforce is… well, you know, Salesforce. 

A reminder: “Salesforce provides customer relationship management (CRM) software and applications focused on sales, customer service, marketing automation, analytics, and application development.” 

Thank you Wikipedia. We’re all familiar with the titan technology company that has grown to dominate the world of core business apps and workflow tools.

Our client had been using their suite of products for quite some time: CRM, marketing automation, and email services. Looking to improve marketing performance, they layered on another product: Interaction Studio, Salesforce’s content-personalization brain for website and email marketing.

But as we’re constantly reminded, personalization is central to today’s digital buying experience. Not only do customers expect it, but personalized recommendations are responsible for an enormous proportion of online activity:

  • 40% of Amazon’s click-through is from product recommendations
  • 75% of time spent watching Youtube were based on personalized experiences
  • 80% of Netflix hours viewed are from tailored movie & tv suggestions
  • 90% of Tripadvisor Bookings were recommended to the travelers *

Given the engagement, conversion, and revenue impact that custom-tailored buyer journeys can have, our client’s marketing team made a bold decision: challenge the tech giant’s personalization capabilities. Put Salesforce “Einstein” artificial intelligence to the test. 

A real head-to-head performance bake off. The new contender: Jarvis ML’s machine learning engine.

If you’re keeping score at home, we missed one introduction. In the other corner: Jarvis ML is an intelligence engine that packages and delivers the power of enterprise-grade machine learning to any business.

Breakdown of the Experiment

Let’s talk about how the test was designed.

The assessment was simple and straightforward. Why? So that the results could be determined easily. In fact, our client hired a 3rd party analyst consulting company to run non-biased data analytics for results comparison.

The goal was to determine the efficacy and accuracy of personalized recommendations to individual website visitors; the method was to use dynamic widgets on the website. 

In order to maintain simplicity, these widgets lived only on the homepage – the primary landing page for new traffic. 

The split in traffic looked almost exactly like a normal A/B test. The client set up their webpage to send 50% of web traffic to Salesforce’s personalization widget, and 50% to that of Jarvis ML.

The success criteria was a simple metric of site engagement, to measure the relevance of recommendations. 

We used click-through rate in two flavors: 

  • A) clicks on any recommendation provided by one widget or the other, and 
  • B) the accuracy of each tool’s recommendation, understood by calculated rank.

To make that clear, each personalization engine calculated top recommendations for each individual on the site. The tool then ordered the recommendations from most-highly-likely to least. For example: clicks on the number one recommendation indicated the prediction was more accurate, versus the 2nd or 3rd ranked recommendation.

The evaluation of this test, as stated, was done by a neutral 3rd party – hired for analytical rigor.

Reviewing the Test

The several-week test generated improved clickthrough for our client – the underlying aim of the whole exercise! And for that, everyone was happy.

What happened more specifically? The results looked like this:

  • Jarvis ML generated better daily performance over Salesforce for 93% of the duration of the test.
  • Jarvis ML outperformed Salesforce by 250% in clickthrough rate on recommendations for repeat website visitors and past customers. These are what we call the “known audience” – where data stored in the CRM (like Salesforce), in addition to all of the transaction data and marketing engagement data was available for both tools to use in algorithms and predictions.
  • Notably, for the rest of the web audience – those site visitors who are completely anonymous and unknown – Jarvis ML drove an additional 12% more traffic down the conversion funnel compared to Salesforce.
  • Finally – and greatly appreciated by our client – Jarvis ML’s recommendations continued improving over time. This is one of the desirable attributes of machine learning after all. So after 2 additional months of personalization widgets on the clients website, the Jarvis CTR results were winning against Salesforce’s precedent by 5.3x on click-through for anonymous site visitors.

Why did it work out this way?

In a head-to-head performance test like this (where most variables are kept equal), several factors come into play that can be differentiators:

  • The audience target
  • Which data each tool uses to inform calculations
  • The infrastructure running behind the scenes
  • The accuracy and intricacy of the predictive models
  • The manner of deploying the predicted output

It’s hard to pin-point exactly “why” the performance had the stated difference, except to say it was a combination of the above.

For example, we know Salesforce had access to all the same data sources – website, CRM, marketing, transactions – but we can’t peek under the hood to see what the Salesforce artificial intelligence is prioritizing. 

That said, we can highlight a major focus point of the Jarvis engine: machine learning infrastructure and operational plumbing.

But the bottom line… this is the world of marketing, which can be summarized as one huge experiment to find out what works. We do more of what performs and reduce waste from what doesn’t.


As a result of the uncontested experiment testing Jarvis ML vs Salesforce on the website experience, our client replaced Salesforce Interaction Studio as the personalization engine behind both the website experience and another facet of their revenue funnel: dynamic email campaigns.

While Salesforce is an incredibly powerful suite of tools – and firmly remains their CRM and marketing automation platform – Jarvis ML is the engine layer that sits on top of the marketing, website, CRM, and sales data to generate intelligence, power personalization, and optimize all digital customer experiences.

The head of marketing for our client recognized the ongoing upward-trend as a differentiator between the two tools:

“I think there was a meaningful difference between Salesforce and Jarvis results because the Jarvis machine learning continuously retrained and improved its models.”

While Jarvis ML doesn’t currently run advertising optimization in ad-publishers, our team was excited to hear their CEO mention the ROAS (return on ad spend)-impact of using Jarvis ML. 

He told us that the efficiency of their website throughput increased to the degree where their cost savings on ad spend were more than the Jarvis ML’s annual software fees. (And this was only over the first few months of the calendar year!)

An Underdog Story?

We all love a good underdog story, overcoming the odds. David versus Goliath. The little guy taking on the obvious favorite. 

However, part of the secret behind Jarvis ML’s proficiency in delivering world-class machine learning infrastructure is in the pedigree. Much of the team are alumni – even leadership – at Google, possibly the most widely trained and used machine learning on the planet. 

So while a win against Salesforce is extremely exciting, the team at Jarvis has pretty high standards already.

The next step on the journey, democratize the power of ML. Our mission is to be the most accessible and easy to use end-to-end machine learning platform. As a fully managed MLaaS solution, we enable companies to deploy a revenue generating intelligence engine to their businesses and products within minutes.

Interested in learning more? You can contact us at sales@jarvisml.com


* Source: webinar

Check Out Our Other Posts

How To Map Shoppers To Buying Intent Using Your Data & Machine Learning

How To Map Shoppers To Buying Intent Using Your Data & Machine Learning

Getting to Win-Win With the majority of eCommerce brands focusing on personalization and customer journey ...
Building Your Email Campaign Library: Planned Promotions vs Behaviorally-Triggered

Building Your Email Campaign Library: Planned Promotions vs Behaviorally-Triggered

Skip the Generic Intro We won’t bore you with generalities of how email as a marketing channel is tried and true, ...
Why Live Streaming Should Be In Your Marketing Strategy

Why Live Streaming Should Be In Your Marketing Strategy

What exactly is live streaming and why should you do it?  The Jarvis ML team recently attended The Lead ...