We Are Experimenting On You

by Forest Conner · April 1, 2016 · Comments

I have a confession to make. If you purchased a video from a site powered by VHX from September 10th to September 14th of last year, I ran an experiment on you. I — Forest Conner, VHX Data Scientist — designed and executed changes to our site that were applied to a random set of our customers. 

Does that feel strange to read? It certainly feels strange for me to say, so I wanted to take an opportunity to delve into the what and, more importantly, the why of our experimenting.

Here’s an example of an A/B test — the one from September. We were comparing UX experiences for customers purchasing a video via VHX, and split-tested the design of the checkout screen. Half of customers interacted with the regular checkout page, and the other half saw an alternate version (see that screenshot above). We hoped the alternate would be more readable and result in less confusion. The results? No less confusion. Our product team used the data from the UX split test to determine permanent design changes on the checkout page. In other words, the experiment contributed directly to making the product better for people using it. 

There’s a wide and important discussion taking place regarding the collection of data by internet companies and user privacy in general. So I feel it is important to state unequivocally that, when we run experiments (commonly called A/B tests) on a subset of our user base, we anonymize our data so that it cannot be associated with an identifiable user. This protects that person’s identity and allows them to perform any number of actions that we can observe, but are never tied to them directly. I may know that 15 people clicked a button on our site without ever knowing who they were. We are as committed to preserving your privacy as we are to improving our product.

This is one way we protect privacy rights in our data collection, but lately I have found myself asking a more difficult question: Is that enough? Much of the conversation about using data revolves around what we can do with the data, but leaves out whether it is ethical to collect it without permission in the first place.

Consider the following example: If someone participates in an experiment at a university for any reason, they would be thoroughly educated about the scope of the research, sign a release before participating, and have the opportunity to ask any questions about the procedure before committing to be involved. Furthermore, if that experiment involved them being misled or deceived in any way, the researchers would be required to explain the nature of the deception and why it was necessary for obtaining of valid results. In the internet world, we are removed from these strict research requirements. Can we still run ethical experiments?

The September checkout experiment that I mentioned above was tested by 20,303 individuals, so it would be nearly impossible for me to claim the consent of each one for testing. It would be equally difficult to obtain unbiased results if I interrupted their purchase flow to inform them of the changes. 

Rather than affect each test that we run, I think the more appropriate solution is to lay down ground rules of what we will and won’t do as part of our testing, and communicate those openly to our users so they can give feedback. Below are rules by which I feel we should operate:

  • When we run experiments, user data will be anonymized completely. Results will only be presented in aggregate and we will not have the ability to attribute an action to any one person.
  • Experiments well be inherently designed to aid our customers and reduce struggles with our product, not solely and indiscriminately to increase VHX revenue. We believe that if we are using your data, it should be for your benefit.
  • Most times, split tests are not that interesting and end up showing no significant difference between the versions being tested. As much of a bummer as this can be, it makes the finding of results even more exciting, so we will make these results public and place them in context of the product whenever possible.
  • If it is ever necessary for our tests to deceive users (see the classic Asch experiments for a fascinating example), we will opt for in-house user testing as opposed to online split testing. This allows us to effectively communicate the reasoning behind any deception. 

This is going to be a work in progress at VHX. By sticking to these simple guidelines, we can stand behind all of the tests we run, always putting our users’ best interest first. Data-collection is powerful! Everyone should make sure to have the ethical framework to back it up.