• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

Dion Almaer

Software, Development, Products

  • @dalmaer
  • LinkedIn
  • Medium
  • RSS
  • Show Search
Hide Search

UX

We keep confusing efficacy for effectiveness

September 21, 2022

I often see the same debates occur, especially around web technology. One pattern I see repeated in these debates is how focus is often on the efficacy of a certain technical approach (or tool or library or…) and the practical effectiveness is ignored.

What the eff are you talking about?

Before we get into web performance and tech, what’s the actual difference between efficacy and effectiveness?

“Efficacy can be defined as the performance of an intervention under ideal and controlled circumstances, whereas effectiveness refers to its performance under ‘real-world’ conditions.”

A Primer on Effectiveness and Efficacy Trials

I remember when Peter Attia spoke about this in regards to his fish oil consumption. Is It better to take fish oil liquid, or fish oil tablets?

“The fish oil liquid is more potent and more concentrated in a higher dose, and if I take it every single day, I get higher levels in my red blood cell membranes (how we measure levels of EPA and DHA), which is the desired effect. But on average I forget to take it at least 2 times per week because the bottle needs to sit in the fridge and I forget to take it. Conversely, the fish oil capsules are easy for me to put in my pill pack next to my sink, which I never forget to take, but they are not as potent. So, while the liquid is more efficacious, the capsules may be more effective.”

— Peter Attia

Yup, we have all been there. What’s key here is both the fact that Peter actually measures the output (red blood cell membranes) and also the practical input. The details matter. How much is the difference between the two? Is the practical approach so weak as to require doing everything possible to make sure to suck it up and take the liquid? Good to go deeper, but our gut can also tell us that taking something is better than nothing.

You quickly see the efficacy vs. effectiveness pattern everywhere. Let’s bring it into our day jobs and how we debate the usage of technology and the outcomes at the other end.

The reason that benchmarks are so hard to do well, is that they measure a pure path that isn’t actually realistic at all. They are like running traffic directions algorithms without taking into account the other vehicles and traffic patterns on the road, or weather, or the time of the day, etc.

We are debating options while missing many many variables, and ones that really do matter. A subset of them are:

How easy is it to stay on a well lit path?

It is one thing for the creator of the technology to build something they have done before using their creation. But how is it for the average engineer? How variable are the outcomes based on the constraints of the system, the tooling, the documentation, the community, and on and on.

How does your app scale?

The performance of hello world is ~insignificant. How fast if your typical information, and what happens as you add features (code)? I would much develop with a system that allows me to add functionality that balances business value with performance. Compare this to something where the initial scaffolding is faster, but everyone slows over time.

What are you actually building?

The Web in particular is a general framework that has a large spectrum of use cases deployed on top of it (the spectrum mentioned here). The characteristics vary wildly as you look at the relative importance of initial load, amortizing costs for longer app sessions, requirements for collaboration, and more.

With Active Recall, which is an offline-first rich application, data is sync’d in a way that means you are basically always seeing optimistic UI operations. You are logged in by default, and the focus is on very fast actions while you use the application. These requirements land you with a very different architecture where the initial bundle isn’t the focus.

DX vs. UX

So, the next time you see another debate popup pitting DX and UX, remember that they aren’t binary variables at odds with each other. Developer productivity matters, can result in better UX, and make sure that you measuring effectiveness as it relates to your precise context.

And If you work on a platform, or on tooling, get real and do everything you can to help developers where they are, and don’t let perfection become the enemy of progress.

Expectations

September 6, 2016 Leave a Comment

Someone sits looking at their home screen and goes to push on an app icon. What do they expect to happen?

Congrats to the housing team on their new pwa!

That seems pretty obvious. They expect the app to load quickly and get them into a reliable experience.

But you quickly get caught on the particular context:

  • Did the user just add this to the home screen and has expectations on how it is “the same” as the browser version?
  • e.g. “I have added mobile twitter to the home screen because I prefer that experience”
  • Is the user coming back to this after several months and just wants “something that feels right”?
  • Does the user want a URL bar at the top, or accessible?
  • Jeremy Keith and other users may want one, but many will not
  • Does the user experience this service on desktop and beyond? Do they feel connected and understandable?
  • Is this a brand where the user has a lot of engagement and understands that context vs. ending up on a new site where they prefer to fast track through “this is how I buy something, and this site maps to that mental model?”
  • If the experience was launched via a URL vs. tapping on the icon what would the user want to see?
  • …

How the user has been trained by their usage (both in general, and by this service) will greatly change their expectations. If the app hasn’t worked well offline in the past the user may know that and not even go to launch it when in a spotty area. I find myself doing this all the time. For example, I would record into the default notes app and then transcribe over to the other app later. I don’t do that for “web” apps, I do it for any app that has been flaky for me. E.g. Asana used to not work well offline, but it has since been fixed, and I am building up trust to go directly to it.

As you watch people using apps there is a complicated set of expectations at play, and many of these are changing.

  • The platform is adding new behaviors and ways of working that apps need to consider (it is somewhat obvious that core capability such as being able to manage a PWA vs. a “native” app should be unified)
  • The developer needs to both add capability and also nudge users when appropriate (e.g. how early offline capable web apps are more explicit about their connectivity)
  • The user keeps changing and learning.

One thing users aren’t doing when they go to tap on the icon is think “I wonder what technology is used to build this app?” They care about output vs. input. Does it feel good for them and solve their problems. Is it fast enough and accessible.

As a community we are working on the challenges of new platforms and the fact that users are coming to us from many angles. We have browsers and apps and bots and various screens and on and on. We engage with these services in different ways and from different entry points. Tapping on a home screen icon is one of them, but launching via links, or push notifications, or voice may vastly out number that method.

Thus, it is interesting to read the thoughts of Jeremy or Owen or Jason (who has been going very deep!).

As we see the platforms that users use evolve, and as we see more PWAs shipping, it will be fascinating to experiment and see what works well and what doesn’t.

Widening your lens for usability testing

July 6, 2016 Leave a Comment


Do you you start your user testing at the wrong point? I really enjoy watching users interact with my software products. When I get behind the two way mirror I never know what is going to happen, but I do know that I will learn a lot. It turns out that time and time again users think very differently as they try to get tasks done. Seemingly small changes can have large effects, and vice versa.

As you see more and more user tests you also learn the art of how to set them up. One common mistake I have made is seeding the wrong starting point, and thus missing out on a lot of learning.

One recent example revolved around some developer products. I asked a developer to add a feature to an application and set them up with the documentation of said feature. This initial web page anchored them for the entire process. They stayed within the walls of the website to get the job done. There were many learnings to be had on the information architecture of the site and how a lot of great content was locked away from the developer. For example, some of the best content for a particular problem that she ran into was not exposed in the main documentation, but rather in a codelab, and even a Udacity course. The mental model of the developer wasn’t “now I am going to do a codelab”, nor “I will sign up for a course to learn this first”, but rather “I just want the answer to my damn question to keep progressing!”

I have re-run tests like these, and finally I noted my mistake. Why was I anchoring the developer? The next time around the test subject started with a blank slate. What happened next was obvious: they started to google around for the answer. This lead them to solutions all over the web (for good and ill) and gave us a vastly different view of the problem.

We then opened things up once again by changing the request to involve their own systems. Now we got to observe beyond “how to add feature X to a skeleton” and instead see how the feature fits into particular contexts. You quickly uncover the fact that your documentation may not take into account the pain of having to glue together “feature X” with the context of “getting that setup on AWS”.

Getting into the right mindset can be key. This often comes up on the product side when you are making comparisons. When creating a shopping list app you may focus on competing with $OTHER_SHOPPING_LIST_APP when you should be thinking about how to compete with a pen and paper. With payment solutions you are competing with the features intrinsic to credit cards and cash.

There is a time to go narrow and iterate in focused areas, but it is often important to widen the lens.

Next Page »

Primary Sidebar

Twitter

My Tweets

Recent Posts

  • I have scissors all over my house
  • GenAI: Lessons working with LLMs
  • Generative AI: It’s Time to Get Into First Gear
  • Developer Docs + GenAI = ❤️
  • We keep confusing efficacy for effectiveness

Follow

  • LinkedIn
  • Medium
  • RSS
  • Twitter

Tags

3d Touch 2016 Active Recall Adaptive Design Agile Amazon Echo Android Android Development Apple Application Apps Artificial Intelligence Autocorrect blog Bots Brain Calendar Career Advice Cloud Computing Coding Cognitive Bias Commerce Communication Companies Conference Consciousness Cooking Cricket Cross Platform Deadline Delivery Design Desktop Developer Advocacy Developer Experience Developer Platform Developer Productivity Developer Relations Developers Developer Tools Development Distributed Teams Documentation DX Ecosystem Education Energy Engineering Engineering Mangement Entrepreneurship Exercise Family Fitness Founders Future GenAI Gender Equality Google Google Developer Google IO Habits Health HR Integrations JavaScript Jobs Jquery Kids Stories Kotlin Language Leadership Learning Lottery Machine Learning Management Messaging Metrics Micro Learning Microservices Microsoft Mobile Mobile App Development Mobile Apps Mobile Web Moving On NPM Open Source Organization Organization Design Pair Programming Paren Parenting Path Performance Platform Platform Thinking Politics Product Design Product Development Productivity Product Management Product Metrics Programming Progress Progressive Enhancement Progressive Web App Project Management Psychology Push Notifications pwa QA Rails React Reactive Remix Remote Working Resilience Ruby on Rails Screentime Self Improvement Service Worker Sharing Economy Shipping Shopify Short Story Silicon Valley Slack Software Software Development Spaced Repetition Speaking Startup Steve Jobs Study Teaching Team Building Tech Tech Ecosystems Technical Writing Technology Tools Transportation TV Series Twitter Typescript Uber UI Unknown User Experience User Testing UX vitals Voice Walmart Web Web Components Web Development Web Extensions Web Frameworks Web Performance Web Platform WWDC Yarn

Subscribe via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Archives

  • February 2023
  • January 2023
  • September 2022
  • June 2022
  • May 2022
  • April 2022
  • March 2022
  • February 2022
  • November 2021
  • August 2021
  • July 2021
  • February 2021
  • January 2021
  • May 2020
  • April 2020
  • October 2019
  • August 2019
  • July 2019
  • June 2019
  • April 2019
  • March 2019
  • January 2019
  • October 2018
  • August 2018
  • July 2018
  • May 2018
  • February 2018
  • December 2017
  • November 2017
  • September 2017
  • August 2017
  • July 2017
  • May 2017
  • April 2017
  • March 2017
  • February 2017
  • January 2017
  • December 2016
  • November 2016
  • October 2016
  • September 2016
  • August 2016
  • July 2016
  • June 2016
  • May 2016
  • April 2016
  • March 2016
  • February 2016
  • January 2016
  • December 2015
  • November 2015
  • October 2015
  • September 2015
  • August 2015
  • July 2015
  • June 2015
  • May 2015
  • April 2015
  • March 2015
  • February 2015
  • January 2015
  • December 2014
  • November 2014
  • October 2014
  • September 2014
  • August 2014
  • July 2014
  • June 2014
  • May 2014
  • April 2014
  • March 2014
  • February 2014
  • December 2013
  • November 2013
  • October 2013
  • September 2013
  • August 2013
  • July 2013
  • June 2013
  • May 2013
  • April 2013
  • March 2013
  • February 2013
  • December 2012
  • November 2012
  • October 2012
  • September 2012
  • August 2012

Search

Subscribe

RSS feed RSS - Posts

The right thing to do, is the right thing to do.

The right thing to do, is the right thing to do.

Dion Almaer

Copyright © 2023 · Log in

 

Loading Comments...