Dion Almaer

Software, Development, Products

We keep confusing efficacy for effectiveness

I often see the same debates occur, especially around web technology. One pattern I see repeated in these debates is how focus is often on the efficacy of a certain technical approach (or tool or library or…) and the practical effectiveness is ignored.

What the eff are you talking about?

Before we get into web performance and tech, what’s the actual difference between efficacy and effectiveness?

“Efficacy can be defined as the performance of an intervention under ideal and controlled circumstances, whereas effectiveness refers to its performance under ‘real-world’ conditions.”

A Primer on Effectiveness and Efficacy Trials

I remember when Peter Attia spoke about this in regards to his fish oil consumption. Is It better to take fish oil liquid, or fish oil tablets?

“The fish oil liquid is more potent and more concentrated in a higher dose, and if I take it every single day, I get higher levels in my red blood cell membranes (how we measure levels of EPA and DHA), which is the desired effect. But on average I forget to take it at least 2 times per week because the bottle needs to sit in the fridge and I forget to take it. Conversely, the fish oil capsules are easy for me to put in my pill pack next to my sink, which I never forget to take, but they are not as potent. So, while the liquid is more efficacious, the capsules may be more effective.”

— Peter Attia

Yup, we have all been there. What’s key here is both the fact that Peter actually measures the output (red blood cell membranes) and also the practical input. The details matter. How much is the difference between the two? Is the practical approach so weak as to require doing everything possible to make sure to suck it up and take the liquid? Good to go deeper, but our gut can also tell us that taking something is better than nothing.

You quickly see the efficacy vs. effectiveness pattern everywhere. Let’s bring it into our day jobs and how we debate the usage of technology and the outcomes at the other end.

The reason that benchmarks are so hard to do well, is that they measure a pure path that isn’t actually realistic at all. They are like running traffic directions algorithms without taking into account the other vehicles and traffic patterns on the road, or weather, or the time of the day, etc.

We are debating options while missing many many variables, and ones that really do matter. A subset of them are:

How easy is it to stay on a well lit path?

It is one thing for the creator of the technology to build something they have done before using their creation. But how is it for the average engineer? How variable are the outcomes based on the constraints of the system, the tooling, the documentation, the community, and on and on.

How does your app scale?

The performance of hello world is ~insignificant. How fast if your typical information, and what happens as you add features (code)? I would much develop with a system that allows me to add functionality that balances business value with performance. Compare this to something where the initial scaffolding is faster, but everyone slows over time.

What are you actually building?

The Web in particular is a general framework that has a large spectrum of use cases deployed on top of it (the spectrum mentioned here). The characteristics vary wildly as you look at the relative importance of initial load, amortizing costs for longer app sessions, requirements for collaboration, and more.

With Active Recall, which is an offline-first rich application, data is sync’d in a way that means you are basically always seeing optimistic UI operations. You are logged in by default, and the focus is on very fast actions while you use the application. These requirements land you with a very different architecture where the initial bundle isn’t the focus.

DX vs. UX

So, the next time you see another debate popup pitting DX and UX, remember that they aren’t binary variables at odds with each other. Developer productivity matters, can result in better UX, and make sure that you measuring effectiveness as it relates to your precise context.

And If you work on a platform, or on tooling, get real and do everything you can to help developers where they are, and don’t let perfection become the enemy of progress.

The holy grail of a Web SDK

As I sit here watching WWDC, I get that same feeling. Everything is vertically integrated. The layering of the SDKs, with much that is shared, and then differences for iOS vs. Mac vs. tv, etc. You can’t hope but look at the Web and think “oh man, the paradox of choice over here… with so much evolving and changing to keep up with!”

The old analogy comparing dictatorships vs. democracies continues to ring somewhat true.

I have made the mistake in the past of wanting to chase the holy grail of a Web SDK that has the benefits of a unified stack that takes care of everything. However, I have learned that this is a holy grail in the sense that it can’t actually exist on the Web in the form that you think of.

The quest is wrong. We shouldn’t try to be like Apple, and compete in the same way with a history and ecosystem that isn’t setup to succeed in that form.

Well, what is the right quest?

  • We should embrace the ecosystem, and the players lower in the stack (e.g. browsers shipping the web platform) should embrace the fact that so much evolution happens on top. Plan for this. Be malleable and modular, and allow for shifts and changes. Most of all, be enablers for the stack on top.
  • The web platform can absolutely do better with undergrounding gains that come from the evolution in user space. The Web is a meta-platform that ships with browsers. Anything in user space is fairly costly, but has the trade-off where it can evolve quickly (downloading libraries / updating libraries).
  • It is a fair criticism that the Web has many primitives baked in for documents, but leaves much of app development to user space. If you build something truly app-y, you can end up somewhat frustrated. It’s improving all the time, and there are things that are amazing about building web apps, but there is a ton to improve here.
  • In general, there is value in seeing what primitives are needed to help the various types of experiences that users choose on the Web. It’s so broad, and the needs for content, commerce, apps, and games are different and yet share a lot.

I am seeing a lot of progress in areas that push us forward:

  • WebAssembly as a lot level runtime for client AND server that can change the game, opening up a TON of room for the ecosystem to innovate.
  • Massive CSS improvements with container queries and new helpful syntax bumps, and losing the need for preprocessors for obvious changes (this is all good undergrounding!!)
  • Apple hiring and catching up with app related APIs (e.g. Web Push, making PWA not be somewhat broken). There is still so much that is needed here, both in WebKit catching up and also the entire Web pushing forward. Dogfooding with Active Recall is very humbling!

Let’s lean in. Platform folks work together and build hooks and capabilities that those on top can innovate with. App devs and framework/library folks, share what you need. Together, push on where we want the Web to go and use the fact that it isn’t run by one company.

The rise of the extensible app platforms

I was excited to see Kenneth and his team ship the public beta of the Stripe App platform:

I was excited to dive into it, as I know that the developer experience would be high quality, and I wasn’t proven wrong. Great CLI experience. Great docs. What you have come accustomed too. As I read through the anatomy of what you are able to do with apps, I could see a lot of similarities with Shopify’s app platform.

The body of a Shopify store

Let’s dive into some anatomy. When chatting with one of my favorite new PMs, Ben Sehl, he shared an analogy of how a store is like a body… a metaphor which I will now abuse to kingdom come:

When you create a Shopify store, you get a simple, yet fully functioning experience… like a human with simple attire, let’s call them Shopi. How do you want to take this base and make it your own?

The first step that products and platforms give you is the ability to tweak the look and feel. This is where you start with theming. You can dress up Shopi in many ways, and make them unique. You can make the clothing yourself, or you can find manufacturers (theme developers!) that have pre-built quality goods that you can then customize from there. As the platform matures, so does the supporting ecosystem.

Now, there is a lot that you can do with the look and feel, but you quickly find that you need richer customizations. This is when the app platform comes in (often called plugins/extensions). The platform controls what can be extended. Can you replace limbs? The heart? The brain? How do you do this in a way that doesn’t destroy the body? A great platform allows for layers of extensibility, which is as much art as science. This is the type of work I love doing, and I am excited to see Shopify continue to refine these layers.

Shopify Flow: Helping you automate workflow

Systems such as nervous, circulatory etc can be the glue that makes everything work well. This is the platform doing its job at regulating everything. Connecting things and making sure any one extension doesn’t blow things up. Acting as an engine for connectors and workflows like Shopify Flow.

If you look deeper at Shopify and the Stripe platform, you see a variety of surfaces that offer extension points… e.g. the Stripe Dashboard, or the Shopify Admin. These surfaces have specific extensions with rich context. They offer embeddable UI points, with building blocks to have your extensions fit in (e.g. components that feel consistent for the users of that surface). These extensible app platforms have a variety of surfaces. In Shopify, you can ship an app that enhances the Admin for merchants, or Checkout for users, and more.

NOTE: I was particularly excited to see that Stripe is using the open source remote-ui library that we created and use in Shopify:

Ok. So, at this point, Shopi, is looking great with awesome clothing, a bionic arm, Tony Stark’s heart, and AR glasses. But you may want to customize EVEN more. This is where custom storefronts come in for Shopify. You can tie into the nervous system still via API (GraphQL or REST), but the body can be entirely different… with you building many more layers as you eschew the defaults. It doesn’t even have to be a binary choice… you should be able to weave together custom and the core headful experience.

I look forward to a future where merchants can customize their own Shopi to be exactly what they need, working with the foundations that Shopify gives them, and the creativity and skill of the developer ecosystem to provide apps, themes, and custom storefront work.

I’m also excited to see more extensible app platforms such as Stripe’s, and we share ideas and technology to raise all of the boats.

Extensibility is going to go from changes to the appearance, to bionic improvements, to providing genetic modifications.

The right thing to do, is the right thing to do.