Do you you start your user testing at the wrong point? I really enjoy watching users interact with my software products. When I get behind the two way mirror I never know what is going to happen, but I do know that I will learn a lot. It turns out that time and time again users think very differently as they try to get tasks done. Seemingly small changes can have large effects, and vice versa.
As you see more and more user tests you also learn the art of how to set them up. One common mistake I have made is seeding the wrong starting point, and thus missing out on a lot of learning.
One recent example revolved around some developer products. I asked a developer to add a feature to an application and set them up with the documentation of said feature. This initial web page anchored them for the entire process. They stayed within the walls of the website to get the job done. There were many learnings to be had on the information architecture of the site and how a lot of great content was locked away from the developer. For example, some of the best content for a particular problem that she ran into was not exposed in the main documentation, but rather in a codelab, and even a Udacity course. The mental model of the developer wasn’t “now I am going to do a codelab”, nor “I will sign up for a course to learn this first”, but rather “I just want the answer to my damn question to keep progressing!”
I have re-run tests like these, and finally I noted my mistake. Why was I anchoring the developer? The next time around the test subject started with a blank slate. What happened next was obvious: they started to google around for the answer. This lead them to solutions all over the web (for good and ill) and gave us a vastly different view of the problem.
We then opened things up once again by changing the request to involve their own systems. Now we got to observe beyond “how to add feature X to a skeleton” and instead see how the feature fits into particular contexts. You quickly uncover the fact that your documentation may not take into account the pain of having to glue together “feature X” with the context of “getting that setup on AWS”.
Getting into the right mindset can be key. This often comes up on the product side when you are making comparisons. When creating a shopping list app you may focus on competing with $OTHER_SHOPPING_LIST_APP when you should be thinking about how to compete with a pen and paper. With payment solutions you are competing with the features intrinsic to credit cards and cash.
There is a time to go narrow and iterate in focused areas, but it is often important to widen the lens.