Data Doesn’t Speak for Itself: A Three Pillar Framework to Effectively Measure Virtual & Hybrid Experiences

Jack Blog

April 20th, 2021 By Joe Panepinto PhD

One of the greatest advantages virtual platforms offer experience designers is the ability to collect data on how a participant acts during an event – if they leave, points when they stop watching the content, and the number of times they react or participate during sessions.

Measure VirtualBut after 18 months working with dozens of platforms, we’ve found a common problem that should be familiar to anyone who works with or teaches analytics – the data doesn’t speak for itself. So we’ve been using a simple three-pillar framework for evaluating the data capabilities of virtual and hybrid platforms. We use the same framework to help us interpret the data once an event is over.

Why do you need a framework? Because interpreting the data with a consistent point of view is going to tell you much more than the data alone, which doesn’t tell you why your audience reacted the way it did.  The framework we outline below is simple, and will help you to make sense of the wide variability you’re likely to see in data for virtual experiences. For example, we’ve seen interaction numbers, the number of people who engage in chat, polls, or downloads, range from 70% to less than half that. But clients deemed both a success. Same with the number of chat messages exchanged which has ranged from nearly zero to more than two per attendee; and retention, the percentage of your audience that stays for the entire duration, which has ranged from 100% to about 70% on average.

But numbers don’t provide the entire picture.

The key to understanding the success of your virtual or hybrid experience is in understanding the extent to which you’ve engaged the audience, how much brand love you’ve generated, and whether that has led them to act differently toward your business in some positive way. In other words, you need to look at rational, emotional, and impact measures.

Pillar 1: Rational Measures

Rational measures are the simplest to track and most often cited. They tell you in a straightforward way what actually happened. For example, how long attendees spent in the experience, when they logged in, when they left, what percentage registered vs. attended, and more.

But you need more than averages to actually understand what happened. In order to use the data to help you design more engaging and interactive experiences, you need to be able to match what the audience was doing with what happened at different points within the experience  In other words, you need a platform that will help you to match the rational measures with the agenda items that make up the experience.

This ability of some platforms to deliver rational measures against specific agenda items has enabled us to learn that you need to change up the visual treatment, set, or angle every 3 to 4 minutes if you want to hold attention and deliver more than a glorified Zoom call.

Pillar 2:  Emotional Measures

What did people think of the experience, and did you succeed in generating positive sentiment around your brand? To understand all of this you should evaluate the content of the chat and questions provided by attendees. Score each as positive, negative, or neutral. This may seem obvious, but your experiences should have a higher number of positive responses.

You should also collect emotional measures by utilizing the polling function of your platform to ask questions. This doesn’t have to be elaborate, but it has to be done if you want to understand the effectiveness of your experience.  For example, you can ask a single net-promoter-score type question, such as “would you recommend this experience to others?”. If a survey isn’t an option follow-up with key participants and ask them a few open-ended questions so you can get a sense of how the experience impacted their feelings toward your brand.

Pillar 3: Impact Measures

This is the most important aspect of experience measurement whether its virtual, live or hybrid: What was the impact to the business of the experience? This starts right from the beginning of your experience design – what do you want to achieve and what are the 3-5 measurable objectives agreed to by all major stakeholders? Is the goal to drive leads or to build familiarity with products? Is it to drive employee engagement in general, or to participate in a particular internal initiative?

Impact measures for virtual experiences– like their live counterparts – requires you to collect data from multiple sources and track them over time. For example, an event designed to drive new leads into your business require partnership and integration with your sales automation platform and lead-scoring team. An event designed to increase customer and prospect consideration of your brand requires you to ask specific questions of attendees about their feelings and intentions. A sales event requires integration with your ecommerce solution (for immediate sales), or tracking in your sales automation platform so you can understand the business opportunity that was created.

Get Started with Data-Driven Experience Design

We all want to create the most successful experiences – ones that capture the hearts and minds of our audiences and drive the business forward in a positive way. Understanding whether we’ve been successful starts with your platform’s capabilities but relies just as much on the way your teams’ approach and make sense of the data they collect.

*data reported from Jack ethos virtual engagements in 2021