Dion Almaer

Software, Development, Products

Developer Docs + GenAI = ❤️

We can do so much to help tech writers scale their efforts and help developers learn!

I know, I know, the current hammer du jour is “GPT for X“.  It behooves us to explore how a promising new technology can help us, and I believe there are two large reasons we should do so:

  • We (industry wide!) need all the help we can get to scale quality developer docs, so using these tools to help anyone writing the docs be productive is hugely valuable.
  • Developers are already looking at products like ChatGPT3 and asking questions of them. Currently the quality is variable depending on what is being used and the corpus that’s been available in the time window. We need to make sure that usage results in useful high quality output.

Helping produce quality developer docs

Every developer product or platform runs into the struggle of keeping high quality documentation up to date and comprehensive. It’s hard enough to execute on reference docs, let alone conceptual docs, tutorials, etc.

There is a resourcing issue at the heart of this:

  • It is very hard to hire great technical writers, because they need qualities such as:
    • Highly technical like an engineer
    • Able to explain and teach concepts
    • Write well.
  • Most (but not all!) engineers aren’t comfortable with taking on the writing aspects of their APIs
    • To scale, engineers, product managers, and developer advocates are needed to take on certain aspects of their docs (just can’t hire enough tech writers!)

GenAI isn’t a silver bullet, and no one would suggest we let the machines generate our documentation, but rather it can be a tool for the humans that do the writing!

  • As you write the docs, you can get various examples from a GPT-like engine that you then refine. ChatGPT3 is able to do a solid job at some of these tasks already, and is only going to get a lot better.
  • To take it to the next level, you can use techniques to refine such as few shot learning, fine tuning, or chaining so the engine can bias towards your view on best practice for whatever you are documenting. E.g. we can train on existing docs, sample code, etc.
    • Imagine including the ecosystem here too. Any great examples of content can be included in the training corpus. Just as some open source projects include the test suites from popular ecosystem libraries to make sure they aren’t broken in CI, you can do the same for content.
  • It will be great to be able to bring in not just knowledge, but also style and tone. It’s fantastic to unblock writer’s block by throwing raw content down and having the tools work for you. There are already tools that let you do things like:
    • Summarize this for me
    • Give me a good heading / title
    • Make this more concise
    • Flush this out with code examples
  • Boilerplate and grunt work… be gone.

These tools can give confidence to those who aren’t professional writers, which is huge!

Beyond docs, towards assistive learning

Helping create fantastic docs is itself a huge win for developers getting the most from your platform in a productive way. I just saw Codesphere sharing some results after they took the time to tune their docs.

But this work goes even beyond improved docs for devs.

You are now able to use the LLMs that have been tuned to your platform in other ways:

  • A chat interface for your platform knowledge
  • Get questions that target the knowledge that developers can use to train themselves
  • Integrate with popular IDEs so the information is there right where they are coding
  • …

Every day we see new examples of this, such as docs search like this:

There are a lot of challenges to go along with the opportunity, but it isn’t hard to imagine a world where you are pair-programming with Shopi and working with the GraphQL APIs is so much easier because of it, or Hydrogen components are being written for you that you can take and tweak, or converting from tech A to B is made trivial, and sooooo much more.

The tech is only getting better, let’s get on board now!

What do the bots think of this idea?!?

It felt appropriate to ask our robot friends for some of their thoughts on the topic, and here are a couple things that emerged:

Challenges of Automation in Technical Writing

As with any automated process, there are certain challenges that come with using GenAI in technical writing. One of the primary challenges is the potential for errors due to the automated nature of the process. While GenAI can help streamline the writing process, it cannot always guarantee accuracy. Additionally, GenAI may not always be able to capture the nuances and complexities of human language, which can lead to misinterpretations and errors.

Another challenge is the potential for bias in the automated process. GenAI relies on the data that is used to train it, and if the data is biased, then the output of the process can be biased as well. This can lead to incorrect or misleading information being generated.

Finally, there is the challenge of cost. GenAI can be expensive to implement and maintain, and it may not always be feasible for organizations to invest in the technology. Additionally, there may be a need for additional personnel to manage the process and ensure accuracy.

Benefits of Using Machine Learning in Technical Writing

Despite the potential challenges, there are several benefits to using GenAI in technical writing. One of the primary benefits is the potential to increase productivity. By automating certain aspects of the writing process, GenAI can help streamline the workflow and reduce the amount of time spent on manual tasks. This can help to free up time for more creative and complex tasks.

Additionally, GenAI can help to improve the accuracy of the writing process. By leveraging the data used to train the system, GenAI can help to ensure that the output is consistent and accurate. This can help to reduce the amount of time spent on manual proofreading and editing.

Finally, GenAI can help to reduce costs. By automating certain aspects of the writing process, organizations can potentially save money on personnel costs. Additionally, GenAI can help to reduce the amount of time spent on manual tasks, which can lead to cost savings in the long run.

Integrating GenAI into Existing Technical Writing Processes

Integrating GenAI into existing technical writing processes can be a great way to increase efficiency and accuracy. By leveraging the power of machine learning, organizations can potentially reduce the time spent on manual tasks and improve the accuracy of their documentation.

However, there are certain challenges that must be addressed when integrating GenAI into existing technical writing processes. One of the primary challenges is ensuring that the data used to train the system is accurate and unbiased. Additionally, organizations must ensure that the system is properly managed and monitored to ensure accuracy and prevent errors.

Finally, organizations must also consider the cost of implementing and maintaining a GenAI system. While the potential benefits are great, the cost of implementing and maintaining the system must be weighed against the potential savings.

Overall, integrating GenAI into existing technical writing processes can be a great way to increase efficiency and accuracy. However, organizations must ensure that they address the challenges associated with the process and consider the cost of implementation and maintenance.

Measuring Success when Using GenAI in Technical Writing

When integrating GenAI into existing technical writing processes, it is important to establish metrics for measuring success. This will help organizations to determine if the process is working and if any changes need to be made.

One of the primary metrics for measuring success is accuracy. Organizations should track the accuracy of the output generated by the GenAI system and compare it to the accuracy of the output generated by manual processes. This will help to determine if the GenAI system is providing more accurate results than manual processes.

Another metric for measuring success is efficiency. Organizations should track the amount of time saved by using the GenAI system and compare it to the amount of time spent on manual processes. This will help to determine if the GenAI system is providing more efficient results than manual processes.

Finally, organizations should also track the cost of implementing and maintaining the GenAI system. This will help to determine if the cost of the system is worth the potential savings.

Overall, measuring success when using GenAI in technical writing is essential for determining if the process is working and if any changes need to be made. By tracking accuracy, efficiency, and cost, organizations can determine if the GenAI system is providing more accurate and efficient results than manual processes.

Developing Best Practices for Using GenAI in Technical Writing

As with any new technology, it is important to develop best practices for using GenAI in technical writing. This will help to ensure that the process is efficient and accurate, and that the output is of the highest quality.

One of the primary best practices is to use data that is accurate and unbiased. This will help to ensure that the output generated by the GenAI system is accurate and unbiased. Additionally, organizations should ensure that the data used to train the system is regularly updated to reflect any changes in the industry.

Another best practice is to ensure that the system is properly managed and monitored. This will help to ensure that the system is functioning properly and that any errors are quickly identified and corrected. Additionally, organizations should ensure that personnel are properly trained in using the system and that any changes to the system are properly documented.

Finally, organizations should also consider the cost of implementing and maintaining the GenAI system. This will help to ensure that the cost of the system is worth the potential savings.

Overall, developing best practices for using GenAI in technical writing is essential for ensuring that the process is efficient and accurate, and that the output is of the highest quality. By following these best practices, organizations can ensure that they are getting the most out of their GenAI system.

And engineer and a robot working together to create better docs.

/fin

We keep confusing efficacy for effectiveness

I often see the same debates occur, especially around web technology. One pattern I see repeated in these debates is how focus is often on the efficacy of a certain technical approach (or tool or library or…) and the practical effectiveness is ignored.

What the eff are you talking about?

Before we get into web performance and tech, what’s the actual difference between efficacy and effectiveness?

“Efficacy can be defined as the performance of an intervention under ideal and controlled circumstances, whereas effectiveness refers to its performance under ‘real-world’ conditions.”

A Primer on Effectiveness and Efficacy Trials

I remember when Peter Attia spoke about this in regards to his fish oil consumption. Is It better to take fish oil liquid, or fish oil tablets?

“The fish oil liquid is more potent and more concentrated in a higher dose, and if I take it every single day, I get higher levels in my red blood cell membranes (how we measure levels of EPA and DHA), which is the desired effect. But on average I forget to take it at least 2 times per week because the bottle needs to sit in the fridge and I forget to take it. Conversely, the fish oil capsules are easy for me to put in my pill pack next to my sink, which I never forget to take, but they are not as potent. So, while the liquid is more efficacious, the capsules may be more effective.”

— Peter Attia

Yup, we have all been there. What’s key here is both the fact that Peter actually measures the output (red blood cell membranes) and also the practical input. The details matter. How much is the difference between the two? Is the practical approach so weak as to require doing everything possible to make sure to suck it up and take the liquid? Good to go deeper, but our gut can also tell us that taking something is better than nothing.

You quickly see the efficacy vs. effectiveness pattern everywhere. Let’s bring it into our day jobs and how we debate the usage of technology and the outcomes at the other end.

The reason that benchmarks are so hard to do well, is that they measure a pure path that isn’t actually realistic at all. They are like running traffic directions algorithms without taking into account the other vehicles and traffic patterns on the road, or weather, or the time of the day, etc.

We are debating options while missing many many variables, and ones that really do matter. A subset of them are:

How easy is it to stay on a well lit path?

It is one thing for the creator of the technology to build something they have done before using their creation. But how is it for the average engineer? How variable are the outcomes based on the constraints of the system, the tooling, the documentation, the community, and on and on.

How does your app scale?

The performance of hello world is ~insignificant. How fast if your typical information, and what happens as you add features (code)? I would much develop with a system that allows me to add functionality that balances business value with performance. Compare this to something where the initial scaffolding is faster, but everyone slows over time.

What are you actually building?

The Web in particular is a general framework that has a large spectrum of use cases deployed on top of it (the spectrum mentioned here). The characteristics vary wildly as you look at the relative importance of initial load, amortizing costs for longer app sessions, requirements for collaboration, and more.

With Active Recall, which is an offline-first rich application, data is sync’d in a way that means you are basically always seeing optimistic UI operations. You are logged in by default, and the focus is on very fast actions while you use the application. These requirements land you with a very different architecture where the initial bundle isn’t the focus.

DX vs. UX

So, the next time you see another debate popup pitting DX and UX, remember that they aren’t binary variables at odds with each other. Developer productivity matters, can result in better UX, and make sure that you measuring effectiveness as it relates to your precise context.

And If you work on a platform, or on tooling, get real and do everything you can to help developers where they are, and don’t let perfection become the enemy of progress.

The holy grail of a Web SDK

As I sit here watching WWDC, I get that same feeling. Everything is vertically integrated. The layering of the SDKs, with much that is shared, and then differences for iOS vs. Mac vs. tv, etc. You can’t hope but look at the Web and think “oh man, the paradox of choice over here… with so much evolving and changing to keep up with!”

The old analogy comparing dictatorships vs. democracies continues to ring somewhat true.

I have made the mistake in the past of wanting to chase the holy grail of a Web SDK that has the benefits of a unified stack that takes care of everything. However, I have learned that this is a holy grail in the sense that it can’t actually exist on the Web in the form that you think of.

The quest is wrong. We shouldn’t try to be like Apple, and compete in the same way with a history and ecosystem that isn’t setup to succeed in that form.

Well, what is the right quest?

  • We should embrace the ecosystem, and the players lower in the stack (e.g. browsers shipping the web platform) should embrace the fact that so much evolution happens on top. Plan for this. Be malleable and modular, and allow for shifts and changes. Most of all, be enablers for the stack on top.
  • The web platform can absolutely do better with undergrounding gains that come from the evolution in user space. The Web is a meta-platform that ships with browsers. Anything in user space is fairly costly, but has the trade-off where it can evolve quickly (downloading libraries / updating libraries).
  • It is a fair criticism that the Web has many primitives baked in for documents, but leaves much of app development to user space. If you build something truly app-y, you can end up somewhat frustrated. It’s improving all the time, and there are things that are amazing about building web apps, but there is a ton to improve here.
  • In general, there is value in seeing what primitives are needed to help the various types of experiences that users choose on the Web. It’s so broad, and the needs for content, commerce, apps, and games are different and yet share a lot.

I am seeing a lot of progress in areas that push us forward:

  • WebAssembly as a lot level runtime for client AND server that can change the game, opening up a TON of room for the ecosystem to innovate.
  • Massive CSS improvements with container queries and new helpful syntax bumps, and losing the need for preprocessors for obvious changes (this is all good undergrounding!!)
  • Apple hiring and catching up with app related APIs (e.g. Web Push, making PWA not be somewhat broken). There is still so much that is needed here, both in WebKit catching up and also the entire Web pushing forward. Dogfooding with Active Recall is very humbling!

Let’s lean in. Platform folks work together and build hooks and capabilities that those on top can innovate with. App devs and framework/library folks, share what you need. Together, push on where we want the Web to go and use the fact that it isn’t run by one company.

The right thing to do, is the right thing to do.