Software Development

We keep confusing efficacy for effectiveness

September 21, 2022

I often see the same debates occur, especially around web technology. One pattern I see repeated in these debates is how focus is often on the efficacy of a certain technical approach (or tool or library or…) and the practical effectiveness is ignored.

**What the eff are you talking about?**

Before we get into web performance and tech, what’s the actual difference between efficacy and effectiveness?

“Efficacy can be defined as the performance of an intervention under ideal and controlled circumstances, whereas effectiveness refers to its performance under ‘real-world’ conditions.”
A Primer on Effectiveness and Efficacy Trials

I remember when Peter Attia spoke about this in regards to his fish oil consumption. Is It better to take fish oil liquid, or fish oil tablets?

“The fish oil liquid is more potent and more concentrated in a higher dose, and if I take it every single day, I get higher levels in my red blood cell membranes (how we measure levels of EPA and DHA), which is the desired effect. But on average I forget to take it at least 2 times per week because the bottle needs to sit in the fridge and I forget to take it. Conversely, the fish oil capsules are easy for me to put in my pill pack next to my sink, which I never forget to take, but they are not as potent. So, while the liquid is more efficacious, the capsules may be more effective.”
— Peter Attia

Yup, we have all been there. What’s key here is both the fact that Peter actually measures the output (red blood cell membranes) and also the practical input. The details matter. How much is the difference between the two? Is the practical approach so weak as to require doing everything possible to make sure to suck it up and take the liquid? Good to go deeper, but our gut can also tell us that taking something is better than nothing.

You quickly see the efficacy vs. effectiveness pattern everywhere. Let’s bring it into our day jobs and how we debate the usage of technology and the outcomes at the other end.

The reason that benchmarks are so hard to do well, is that they measure a pure path that isn’t actually realistic at all. They are like running traffic directions algorithms without taking into account the other vehicles and traffic patterns on the road, or weather, or the time of the day, etc.

We are debating options while missing many many variables, and ones that really do matter. A subset of them are:

How easy is it to stay on a well lit path?

It is one thing for the creator of the technology to build something they have done before using their creation. But how is it for the average engineer? How variable are the outcomes based on the constraints of the system, the tooling, the documentation, the community, and on and on.

How does your app scale?

The performance of hello world is ~insignificant. How fast if your typical information, and what happens as you add features (code)? I would much develop with a system that allows me to add functionality that balances business value with performance. Compare this to something where the initial scaffolding is faster, but everyone slows over time.

What are you actually building?

The Web in particular is a general framework that has a large spectrum of use cases deployed on top of it (the spectrum mentioned here). The characteristics vary wildly as you look at the relative importance of initial load, amortizing costs for longer app sessions, requirements for collaboration, and more.

With Active Recall, which is an offline-first rich application, data is sync’d in a way that means you are basically always seeing optimistic UI operations. You are logged in by default, and the focus is on very fast actions while you use the application. These requirements land you with a very different architecture where the initial bundle isn’t the focus.

DX vs. UX

So, the next time you see another debate popup pitting DX and UX, remember that they aren’t binary variables at odds with each other. Developer productivity matters, can result in better UX, and make sure that you measuring effectiveness as it relates to your precise context.

And If you work on a platform, or on tooling, get real and do everything you can to help developers where they are, and don’t let perfection become the enemy of progress.

Does mobile documentation matter for developers?

July 9, 2019

Developers code on their laptops, normally docked with their mechanical keyboards and monitors, right?

As such, when it comes to developer documentation, you don’t need to be responsive, and can focus on a desktop-only set of docs.

I have heard this sentiment many times, and at first blush it makes some sense. For all the talk of mobile eating the world, a lot of the business end of things happens on the desktop Web.

Once you have desktop docs going, you look at your stats and see that you were right! A fraction of traffic is coming from mobile! Glad you didn’t waste any time on a great responsive experience! Well, hold your horses. You may have self selected your answers here.

A great developer experience includes a great documentation experience, and that includes mobile. I have seen transitions from desktop-only to responsive, as well as responsive from day one, and both have shown how important it was to go mobile, too.

Take one of the Google developer sites for example, and a recent slice:

mobile: 90+% of share — This slice is from a campaign, so bias, but directionally similar

Why is mobile such a big deal?

All documentation isn’t equal. There are a variety of reasons why someone ends up in your docs. The Web is a viral platform due to the sharing of URLs through all sorts of mediums (via Email, Twitter, Slack, etc). People are increasingly accessing these mediums via mobile, and are tapping in to read from there. I often read a lot of developer-oriented content on mobile, and will save it away and often revisit on desktop.

I also do a lot of memorization through mobile, using space repetition systems. A common task during the day will be saving some developer knowledge into my system.

This is somewhat akin to customers who browse for products on mobile and finish the transaction on desktop.

Beyond documentation

It has been fantastic to see the trend of developer documentation being seen as putting the manual online with some hyperlinks, and instead using the medium to add a lot of interactivity to help you learn.

The best way to learn is to do, and it is important to be able to feel how things work and play with them. We have embedded Glitch into web.dev as an example of bringing you tight playgrounds to explore, and remix for your own experimentation.

How do you want this type of content to be approachable on mobile vs. desktop?

The other important realization for me has been integration. Developers often end up in your docs via search, and they rarely probably go to your root domain and just read read read.

Take time to look in the Search Console to see your analytics to see how users are getting to you, and make sure that you have the content they need. You need a base of reference documentation, and then can walk up the chain to use cases, and finally verticals.

Look for other areas of integration.

Where do your developers live?
Their IDE? And certain plugins?
Do you want to reach them on StackOverflow or Glitch?
Do your error messages link out to documentation to help?
Do your docs work offline? (e.g. if someone is reading on the subway to work, or generally stuck in poor and variable connectivity)
Do you include code from source files that you have tests running against to make sure that a copy paste works?
Have you built a connection between the tools and your documentation so it can be surfaced inline?

There is so much to do, and while “docs” are a first class citizen, it is also important to recognize that good tech writers are still under-rated!

render() your consciousness

February 20, 2018 Leave a Comment

I recently woke up, remembering a dream that tied together the following:

how does our consciousness surface a certain subset of the huge amount of input that is processed in the brain?
how can we best handle all of the events in our computing systems, and render a UI that represents a valid state?
how can we build a UI representation that helps users, with their consciousness, actually get something useful done?

It wasn’t shocking that I had this dream, given that I fell asleep listening to Anil Seth as a guest on Sam Harris’ podcast, which I turned on right after coding.

When we write programs that model the world, one of the areas that we often struggle with is the notion of representing time, and how state changes over time. It is a source of many bugs, and has lead many to look toward immutable state as a way to avoid foot guns. Rich Hickey famously explains his experience on the topic in Are We There Yet?

How our consciousness batches time

When you zoom in with concurrency and time, you get into the notion of something happening at the same time. At first blush this is a simple concept, but when you look deeper at how our consciousness deals with this, it is fascinating. As we get new tools to probe the brain, we are uncovering a lot of new information, such as learning that decisions have been made before you thought you made them.

Simple experiments show the layers of filtering and input changing that occurs. If you touch your finger to your nose, it always “feels” like the touch happens at the same time for both digit and nose, yet your arm is longer than your nose. Inputs from the nerves must reach the brain at slightly different times at therefore at some point they get put together as a batch.

It appears that our component systems are getting masses of input that we are processing, and there is competition for the right level of info to reach our consciousness. We can direct things at times (e.g. we can take control of our breathing, but fortunately don’t have to focus on it to always happen) but mostly we are observing and our consciousness is revealing what has happened.

Back to programming

This shares some similarity with painting the right UI for our users (and vsync). We can gather information from our components, batch together the state, and prioritize what to render. With the appropriate amount of complexity in our state, especially considering the difference between app state and UI state, it may make sense to use a system such as Redux to manage the batching, or maybe you prefer TJ’s state management library https://twitter.com/tjholowaychuk/status/957853652483416064.

Elsewhere, it is common to debounce, sometimes with exponential backoff, to make sure we aren’t wasting a lot of time and resources.

https://github.com/joshwcomeau/redux-vcr

Immutability also has huge side benefits. We can save snapshots that enable improved testing and time travel debugging. Also, any excuse to bring back a VCR is surely a win 😉

Using the past to change the future

Having snapshots of state is one thing, but what about using old states to help you in the now? While we often save past state snapshots, we don’t often use them in rich ways to help change future state, but your brain does just that. Experiment after experiment shows the importance of context.

This video shows how you can be primed to see color in a black and white photo. There are examples of this across our senses. You can listen to some audio that sounds like gibberish, hear key words, and listen to it again and suddenly you understand the gibberish.

Examples of illusions fill books, and are what magicians use to trick our senses. This all makes me wonder if we are going to see richer ML models that help the user based on the past context?

Good UI is magic

A lot of this trickery feels like magic and illusion. This is absolutely true, but great UI is jam packed with this type of illusion. We can use motion and progressive rendering to make it feel like things are happening, keeping the user engaged. We can make sure to prioritize which parts of the UI to update and delay on others.

To do this correctly, often requires you to take the time to map out the SLO of various pieces. For example, when you visit an e-commerce product page, show the core information asap, and wait on data below the fold such as reviews. You should also consider the freshness requirements across these elements too (e.g. the price has to be fresh, but you don’t need to block on that new review!).

I am having a lot of fun learning about how the brain works, and our psychology, and have a feeling like this knowledge is going to be useful for our industry as we get better at delivering UI that truly helps our users consciousness understand the view of the world that matters to them as they get done what they need to get done.

Oh, this sounds like a good excuse to watch Darren Brown explain the tricks of the trade: mentalism, cold reading, neuro-linguistic programming, cognitive illusions, and much more.

Isn’t building software products fun?