tl;dr LLMs give computers new abilities to be better partners for us humans, and if we build the right systems we can transform how we work together. I have learned some lessons on the building side, but also on how to do more as an augmented human to get the most out of this new world!
A dream stirred me from my sleep. I found myself on the set of ‘Dancing with the Stars,’ but with a twist: my partner was not human, but a robot. As I lay there, half-awake at 3am, I pondered the meaning of this mechanical ballroom dance. Then it clicked… it was a metaphor for the work I’ve been deeply immersed in at the close of 2023: creating computer systems that augment human capabilities, giving developers and their teams superpowers in software delivery.
The Dance
I’ve always believed in the power of combining the best of both worlds: human creativity and computer precision. The best user experiences have always weaved brain and tool, these days including those that are digital.
LLMs have changed the game in that precision has a brand new capability: a new layer of intuition that we can tie to. A way to combine my Systems 1 and 2 brain with a mesh of combined thought. Back in the dream, my subconscious was painting a picture of the ideal partnership where the human mostly leads, and the machine follows in a tightly choreographed back-and-forth. Just like picking up a tool such as PhotoShop, it can still take time to master the steps, and the dance changes as the capabilities change. How can we best use the strengths and weaknesses of each partner so that they work as one?
Crafting the Perfect Partner
I’m currently iterating on a dancer that developers can shape into the best partner possible. Speed and skill are crucial. A slow computer is like a dance partner with two left feet, disrupting the flow and making collaboration frustrating. Skill, on the other hand, is about quality and finesse—leading without stepping on each other’s toes, sharing knowledge to maintain the rhythm.
The Car and the Engine
I was excited to join a Sutter Hill Ventures startup for many reasons, and my expectations have been very much exceeded. Not only do we have a solid financial backing that allows us to really focus on building a game changing product and business, but the support that the Sutter Hill team has is special. I get to work with my favorite UX person there is. The enterprise sales playbook is ready to run. And on and on.
The team itself (founders, CEO, and everyone else!) is not only world class, but there is a strategic bet that I strongly believe in for building the absolutely best product. The heart of the team has AI researches who deeply understand every part of the stack.
It’s one thing to build a car using someone else’s engine; it’s another to be able to fully tinker with that engine or even build your own.
In 2023 we have learned so much as a community. First we had the transformational moment when developers got to poke at what could be done with OpenAI APIs (and then so many more). There was the prompt engineering, RAG’ing, and pushing the boundaries of what’s possible.
Embracing Constant Change
The model tier is just the beginning, and going from demo to a production system requires a world of work to be done around it.
New models and research are popping up on a daily basis, so how do you filter out what could be helpful? How do you determine its utility for your specific needs? How do you ensure your data is accurate and current? Are your evaluations truly reflective of quality, or are you just fitting the last piece of a puzzle?
Metrics
Measuring what matters here is hard. For example, with coding tools, I often see discussion around the amount of codethat is created, or the Completion Acceptance Rate, but when you watch this play out in practice with your users you realize…. wait a minute…
Do we want to always be creating code if it’s adding entropy into the system? If that code is iffy, and if the human can’t tell, then maybe we are adding problems. And, wouldn’t it be nice if we maybe could… delete code and simplify?
For completion acceptance, I can get very different results by changing the system to vary the amount of code that comes back, or the latency, and many of the habits that you build with the developers. The habits have been really fun to watch. Seeing cohorts that start by waiting for the system to do things vs. communicating more and moving quickly.
And when I do side by side comparisons, I see the huge difference where one system can have a hire acceptance rate that ends up with code that doesn’t run. Don’t I really want to be tracking time to running code that is high quality?
Here’s to 2024
We are somewhere in the journey that is akin to constant improvements that we can see with other tools such as Midjourney.
I’m grateful for my team’s collective ability to build everything needed for the ultimate coding dance partner. We are building the platform that enables the building of this partner, to iterate on it, to take in the innovation from open source and our own research, and man I’m having a great time doing it.
I can’t wait to share it with more of you. If you’re a developer who spends most of your day coding, enjoys giving feedback the moulds a product, and are interested in getting early access, I’d love to hear from you.
Happy New Year, and may this become true!
NOTE: Of course, this article was written by both Dion Almaer and the dancer within Type.