Why User Testing Sucks | UXPUB
Have you ever wondered why we have such a surprising number of articles that say “User testing is important”, “You can t release products without user testing” and “User test or death”?
I mean, if it s so important and obvious, why convince us like that?
You might think it s about money. As if startups can t afford to do user testing the right way:
The instruction is very simple:
- Find five potential customers from your target audience
- Create a scenario with realistic objectives
- Be calm and silent
It doesn t look so expensive.
All the same, it s not about the money. Maybe it s laziness? I have always suspected that entrepreneurs are the laziest people. Especially in the first few years, when they work 63% longer than the average 40-hour worker.
So why not do some user testing? Why do most people still choose to listen to their intuition?
My observation is that user testing doesn t always work right. Why? Oddly enough, because it engages users. There is an inherited flaw in user testing – an internal contradiction: we try so hard to understand human nature, but at the same time we try to eliminate it as much as possible.
The drama is growing …
We want to capture the sincere reactions of users to our products, while investing so much effort in creating artificial scenarios and environments that have nothing to do with the real world.
More drama …
We try to deeply explore and understand what drives our customers. Instead, we end up creating generic imagery that has less in common with real people using our products than “Married … with kids,” and so on.
Cue in “Time” by Hans Zimmer …
We live with the belief that each user s experience is valuable, but we collect as much data and analytics as possible to minimize the impact of the individual.
Okay, enough drama. Without making this article a philosophical rant, I would like to touch on some gray areas of user testing. I have been doing user testing on Icons8 for the past few years. [1,2,3] This article tells me what is in this I do not like…
Bias
Every time people get involved, there are biases. In user testing, people are on both sides of the equation: users and user testers.
The first bias comes from the users themselves. When we observe people, we think we understand other people. This gives us confidence that the user test results can be interpreted the way we want. To prevent this, we try to be as impartial as possible. We are trying to suppress our own point of view. We practice empathy. So now we are completely dependent on the point of view of our users. But then a problem arises …
Observer expectation effect
No matter how realistic test scenarios are, users behave differently when they know they are being watched. This will never allow us to get a real picture of how people interact with our products. We are now dependent on the inconsistent behavior of our users. We depend on other people s bias.
What do we do when so many biases are involved on both sides? We trust our intuition to make the right decision.
There is, of course, another way. And it becomes more and more popular every year.
Ambiguous data interpretation
We try to observe people in their natural environment. There is only one way to do this: include all kinds of trackers in our software and then create behavior maps. Now we can see exactly how our users interact with our product.
The problem is that each user is now a collection of numbers, a point on a graph. There are no more users, we are watching the numbers.
Now numbers give us confidence that we know something. They give us confidence that the user test results can be interpreted the way we want.
I ll use this example from my old article:
Let s say I have an egg farm. If I receive a report that 10% of our eggs are broken, what should I do?
a) increase the number of chickens to cover the deficit
b) focus on the safety of existing chicks to reduce losses
c) fire my cousin
Big data makes us very confident, but it doesn t save us from ambiguous interpretations.
To interpret numbers objectively, we need to suppress our point of view again. Awaken empathy. There is only one problem with this. There is no one else to empathize with. There are only numbers. And generalized images and characters.
No matter how much objective data we have collected, there will always be some kind of subjective interpretation. Business owner, user tester, analyst – and trust me, intuition will be at play in every case.
In a nutshell: User testing will allow you to see problems, but it does not guarantee that you will make the right choice. Kill the hype.
Explanations … After.
Gee, wouldn t we like to explain all these things …
Imagine watching a movie with a friend. In the middle of the film, an unexpected plot twist occurs. “She killed him.” Your eyes are shaking with fear, you are breathing hard, your mental state is like that embarrassment of an elderly woman in a supermarket who is told that her coupon is out of date.
Agitated, you slowly turn your head to the left where your friend is sitting, expecting that he or she will be as shocked as the one you are experiencing. However, this is not what you see. Your friend is completely relaxed, sitting without showing any emotion. Their lips move slowly, and after a short pause, you hear: “I knew it would be.”
Dazed, you go to Youtube and also see ten more explanatory videos on why the twist was not a twist at all, and a hundred more comments broadcasting the same thing.
Suddenly you feel like you are the only person in the world who didn t expect to see anything.
You are probably thinking now: what does this have to do with user testing?
Well, in the scenario mentioned, you are you. A movie is your web service or application. And your Youtube analyst friend and countless commentators are all testers.
You see, after something happens in the movie (with your service), for some reason, there are always people who say, “We knew this from the very beginning.” However, they always do it after the movie ends.
In user testing, everyone loves to explain everything. After the testing phase, someone has to explain and understand everything that happened. After all, people get paid to do it. At the same time, explaining things is something that our brain loves very much. This makes us comfortable.
This is why user testing sucks.
Everything is explained only after everything has already happened. Why did we lose users? Because we have changed the design of the site. Because we added this one button. Because our competitor has made a new product. Because our site has been down for 12 minutes.
Bring five more people, show them the numbers. Everyone will have an explanation. Just different.
Anyone with a number is now telling the truth, but their own truth. This is why there are so many stories of successful entrepreneurs who seem to know everything about success. Explaining success is different from achieving it. And this is the same reason why these stories don t help everyone achieve the same result. Explanations are just … explanations. Drugs for the brain.
Why has Twitter become successful? I am sure that all articles are dated post-2006.
In short: User testing has a lot of explanations. Go figure it out.
User testing goals contradict each other
The single goal of user testing is artificial. Something like this: “make our products easy to use,” “improve the user experience,” and so on. User testers have their own goals. Business owners have theirs. Even worse, these goals are constantly changing due to trends, technical breakthroughs, or weather. What can you expect from an area that has only been popularized by one company?
Since the release of the iPhone, everyone has become obsessed with simplicity. After the Kindle, versatility and accessibility. Every popular product creates new rules, new explanations of what UX is. UX should be an organic experience, so let s make our buttons mimic real objects. Wait no. Let s keep them simple and get rid of all the unnecessary details.
So the new rules are the rules now (for the moment). Simple, flat, mobile first. Trends have replaced goals. However, we must ignore services like LinkedIn, Facebook and Skype, which violate most of the rules we set. And yet they are still popular. Slack s simple and UX interface didn t kill Skype. Yet it killed simple IRC.
What we are actually doing with all the rules and explanations is trying to replicate the success of the products released before us.
Then comes another great product that completely undermines everything that was said earlier. Just like the iPhone did. And that becomes the next big thing. And again we return to new explanations and theories.
What s the real purpose of UX? Make users happy? Make them productive? Make your business generate more money? Go figure it out.
In short: user tests involve people with different plans, making it nearly impossible to find a solution that works for everyone. At the same time, UX trends may be more volatile than Bitcoin in 2017.
Why do people choose by trusting their intuition?
After everything I ve mentioned, this is what the user testing should be doing:
- Eliminate all kinds of inherited human prejudices among yourself, users, stakeholders, analysts and business owners. All the prejudices that we have had for thousands of years. At the same time, make sure you are still a living, breathing person.
- Ignore any explanations, as most of them seem to be a subjective interpretation of what has already happened. In other words, an opinion.
- Find a compromise in a bowl of conflicting goals: follow current superficial UX best practices, make users happy, make a business successful, change the world, whatever. Don t lose your job.
The whole structured process of simply finding five leads and giving them assignments gives a false impression of how things are working correctly. The structure is there, and it gives false comfort. Yes, you will find problems, but also solutions. There is no guarantee that your decisions will be correct. Too many things are getting out of hand. Too many explanations will be false.
Despite many resources on the importance of user testing …
After all, besides money or laziness, there is a reason why people sometimes choose to trust their intuition.
“In many ways, implementing Amazon Prime has been an act of trust. The company had little concrete understanding of how the program would affect orders or the likelihood of shoppers making purchases in categories other than the media …
… But Bezos was in the thick of things. ”-Brad Stone, The Everything Store: Jeff Bezos and the Age of the Amazon
In April 2018, Amazon finally announced that more than 100 million people are using the service.