Learning

Active Recall now out of beta and open for all learners!

May 4, 2022

I have been working with a great friend on a learning tool that gives you super powers, and I am very excited for it to launch today. I have been dogfooding it for some time but it’s also just the beginning of the journey to help people succeed through learning and retaining knowledge. The tool is Active Recall, and people are using it to become language proficient, study for exams, retain key knowledge for work, and so much more.

This is my story using it as a lifelong learner.

I wish I understood the power of spaced repetition and active recall as a kid. I believe it would have been life changing, and that it would have resulted in better results in school, life, and work… and all the while saving me time.

We know about the value of compound interest in growing wealth, and the best time to start compounding if you haven’t already, is today.

This is the story I would love to be able to time travel with, to share with a much younger me, and I hope it is helpful as you start your compounding journey.

How I became a lifelong learner

When Walmart acquired my small startup over a decade ago, I remember reconciling the fact that I was brand new to retail, with the knowledge of how Walmart was famously data driven. I knew I needed to up my game as I dove into this new domain, and pondered how.

My wife is a teacher, and we have discussed and debated learning and education the entire time we have known each other. This resulted in a deep dive into the research on how to best study. How can you optimize your time, and get results?

The magic of spaced repetition

The first step on the journey was spaced repetition, and learning how I had gone so wrong with my study habits. Put in the work. Keep repeating. Read and re-read. It turns out that this is wrong-thinking, that is wasteful of time, and doesn’t even work!

Once you start digging, you find that there is a lot of research out there on learning, and books such as Make It Stick bring it all together. It turns out there are two important pieces to the puzzle of long lasting memory. The first piece is to ensure that you are quizzing yourself vs simply reviewing. This forces you to actively recall the material which strengthens your memory.

The second piece is about being quizzed right at a time when you struggle to reach for the answer. This is where the “spaced” in “spaced repetition” comes in. To serve the question up at the right time, the gap in time grows longer when you find the answer, and shortens when you have forgotten it.

After I read this, I admit…. I didn’t quite believe it. Surely constant repetition is better? But it turned out to work wonders in practice. Previously, when I would re-read a passage, I would unwittingly deceive myself, as growing familiarity with the text comes to feel like mastery of the content. Now, when I spaced out the study with larger and larger intervals, my recall was drastically superior.

Thus, the life changing habit was born…

The Active Recall habit

I have a morning routine, and it includes ~five minutes of using Active Recall as I hit “Study All”. It connects me to the knowledge that I want to have available. It gives me confidence that I have access to my knowledge, which will help me though-out my day:

“The very processes that teachers care about most–critical thinking processes such as reasoning and problem solving–are intimately intertwined with factual knowledge that is stored in long-term memory (not just found in the environment).” — Daniel T. Willingham

What shocks me to this day, is that it only takes me a few minutes, thanks to the power of the exponential backoff of the prompting interval.

As I go through the day, I am capturing knowledge as I go. I don’t worry about taking time right then to make the content perfect, as I often don’t have time right away…. so I focus on capturing it and putting it into the Unsorted folder for later gardening.

At a later time, usually when at my laptop, I go through the unsorted content. I clean it up, and I use some tricks to make it more effective for recall and synthesis, such as:

Your brain casts around for answers, it results in generation which creates and firms up associations. Build associations that make sense to you, and your brain can find the path easier.
Much of our brain is tied to visual processing. Lean into that power, and use images in your content to build association. A picture is definitely worth a thousand words.
If you think of an emotional moment in your life, it’s probably quite easy to visualize. Emotions result in strong association, so lean into that too. One example that tells the tale is how I put names to faces. When I met someone named Justin Toupin, I put his information into Active Recall and pictured him being hit in the face with two bowling pins. A lil gruesome I know, but it gets across the point. I never forgot Justin Toupin again!
Create more questions to hit the knowledge from multiple angles. Find the atoms and build from there. An example here is studying the history of World Cup football (soccer for some of you ;). I start simply: For a given year, which country hosted the World Cup? Then I added more knowledge… which country won? Who was the other team in the final? What was the score? Finally I can have compound questions where a given year results in the full score between both teams and includes where it was played.

What do I put in my knowledge base?

After years of using Active Recall and growing my knowledge base, I have seen how it spreads to so many areas. My knowledge base now includes:

Work related. I am in technology, and I have probably the bulk of my cards here. I have general information on the technology I need to understand, engineering, product management, management in general, as well as the lessons I have learned throughout my career. I love how Active Recall is both a general purpose platform (you can put any information into it) but is also purpose built for certain content, such as code!
General knowledge. There is a ton of knowledge that I want to capture and save, on all kinds of topics. When I took a boating exam, I put the information into Active Recall and it is always fresh in my memory. Geography, Science, Health, History, Sport, and on and on. When I read a non-fiction book, I now capture all of the lessons. In the past I would read something and forget 99% of it after the fact… after all that time that I invested in reading it! Now? The important information is always there.
Personal information. Information on people, both my loved ones (What was the name of my uncles first hairdressing salon? What instruments could my Grandad play?), new connections (What was the name of $NEW_PERSONS children?), and… myself (Who was my favorite teacher? How much solar power does our house produce? What is the silent Notepad.exe story?) It is always fun to get a “tickle” when this information pops up. This is why I often put information in that isn’t a Question / Answer…. but just pops up as I enjoy having it back in my consciousness! Quotes from things my kids did and said is a particular pleasure 🙂

Over time, you may not need all of this information top of mind. This is when I suspend that content. Now, it’s available to me via search, but doesn’t come up in studying.

Active Recall gives me what feels like a super power. I am so excited for you to start and share your journey. There are so many other tips and tricks that I am excited to share, but for now… welcome to lifelong learning!

DuoLingo: a (better than!) text book approach to practical spaced repetition

February 1, 2016 Leave a Comment

I am a big believer in spaced repetition as a scientifically proven method to help you learn both effectively and efficiently.

There are several implementations, and one of the most basic and famous ones is the Leitner system, which is a great way to explain the point. Rather than going through a series of flash cards in serial, repeating each of them again and again, why not consider whether you got the question correct or not? When you get the correct answer the question goes to the next level of boxes, which isn’t gone through as frequently. Knowledge that you grok naturally goes further back in the line and you don’t have to be quizzed on it as frequently. The minute you get it wrong however it comes back up.

The most efficient system will know when to ask you so perfectly that you will struggle to answer, but will eventually come up with the right answer. This may not be exactly what you want however, as you may want some knowledge at the tip of your fingers.

There has been a lot of work on various algorithms, and computer systems have been developed to further the cause. The prototypical example of this is the SuperMemo software. Pavel Janeka not only built and iterated on this software over decades, but he published information on each algorithm, and then showed what seemed to work well and what didn’t.

Software such as the original SuperMemo, and the currently popular open source Anki, rely on you to rate how well you knew the answer, to feed the right information back to the system.

Was the answer immediately available in your memory? Did it take a sec? Did you have to really think about it? Once you saw the answer did you think “oh crap, right!” or was it a “huh, really? I had no idea!”. These could have different effects on when to next ask you the question.

A common misunderstanding in learning something is that it is ideal to know the answer immediately. Instead, it has been shown that for long term learning you want to have to think and generate the answer. This is known as the generation effect. Fantastic studies have shown that subtle changes to how you quiz have large effects. For example, simply asking a subject to fill in a word’s missing letters resulted in better memory of the word (crosswords anyone?).

There has been a backlash against standardized testing, but the big mistake is in how we focus our quizzing on assessment rather than using them as a fantastic learning tool.

If you are interested in lifelong learning, or how to help your kids not get stuck in the trap where we think we are learning but in fact we are just binging information that we will forget a week later, check out Make It Stick.

Where DuoLingo Comes In

With all of that context we come back to the product at hand. Once you learn about space repetition you quickly see how DuoLingo offers a user experience that hits the research without the user haven’t to answer how well they knew something (which can be flawed in of itself!).

Let’s look at the different types of question and answers and how the system gathers information.

Choosing Pairs

This type of question offers words in two languages that you need to pair up. If you get one wrong you aren’t finished, but the system can:

Remember which words you may have trouble with (I say “may” because I have fat fingered in the past)
Make sure to quiz you on some of these items again sooner than if you nailed them
Take the time to answer into account. If you are quickly matching that tells you that it is tip of the tongue whereas a more labored approach may mean that you are doing more generation of answers (or you may have a distraction or had to go to the bathroom…)

Translation: Typing or Picking

When asked to translate something into your language of choice there are a few pieces of information available:

Some words have dotted underlines that you can tap on for a hint. The act of doing this shows the system that it wasn’t top of mind
If you are struggling with typing the answer the system can convert to multiple choice to give you a last change to generate the answer. At this point the system also knows that even if you get it “right” you don’t know it as well.

Listening

When listening to the language and then typing in what you hear there is the option to repeat the phrase, and to repeat it slower. If you need to repeat it, the system may know something, but if you repeat it slowly it really knows that you didn’t quite get it.

Grouping of questions

The grouping of questions matters. If you go into a block of questions on the “past tense” then you have content that can make it a lot easier to get the right answer. You get used to the patterns again as you go through. This isn’t all bad, because it allows you to tie together this knowledge and its patterns. However, we always need to remember why we are trying to learn something and get as close to that environment as possible. We probably won’t be stuck in conversations that are limited to these groupings.

“We practice as we want to play, so we can play like we do in practice.”

The Chicago Blackhawks famously made changes to how they practiced ice hockey. They got away from their regular blocks “on Thursday’s at 2pm we do passing like this…” and randomized a lot more. Just when they were getting into the flow they would switch. At first the players hated it and they thought they won’t progressing, but this shows you one of the frustrating tricks… we aren’t good judges on how we are progressing. When we cram we feel confident that we are learning, but this is because it is shallow learning. We are familiar with the content, but this doesn’t mean that we know it.

DuoLingo has another option to allow for this, the “Practice Weak Skills” button that will feed you questions across the board rather than just in one of the groups. It won’t be totally random, and you will still get to fire neurons close to each other.

There is a slew of research on how we learn, and it is criminal how little we put into practice. It is a real pleasure to have seen DuoLingo iterate over the years that I have been using it, and seeing how it applies this research in clear usable ways.

With systems like DuoLingo we can measure large populations learning, in this case languages. We will be able to tweak our learning and prove out hypothesizes. We will be able to better personalize these learnings to you the learner. I can’t wait to see how we progress.

Have you seen other examples of spaced repetition or other practical implementations of learning research?

Going to a conference? Don’t waste your time

September 28, 2015 Leave a Comment

If I could go back in time I would have ample opportunity to fix some mistakes (but I would be scared to do so, else who knows what butterfly side effects there would be!) One general change I would make is how I spend time before, during, and after a conference that I went to.

I wasted a lot of time at conferences. It isn’t that they were a waste of time at all, but I could have gotten a lot more out of them if I handled things differently.

The biggest mistake I would make is how I thought of a “talk”. I would often sit there and take notes. Maybe I would blog about the talk. If you asked me anything about those talks today though, have I retained much? I doubt it.

I think you should look at these talks as inspiration to explore. I would now focus on really listening, and taking the odd note on topics that I want to go deeper on later. Later on I would then elaborate on these notes, create some tasks to explore, and maybe put some questions and answers into my spaced repetition system for anything that I actually want to retain and consolidate from short term to durable long term memory.

As you weed through the content, when there is something of interest expect to really put in some time to dig!

The same can be said for time outside of the presentations. The hallway track has always been an important one for me at conferences as I learn so much for talking to other practitioners about the craft. There is so much value in going deep on a topic and comparing experiences. Once again though, when something potentially fruitful comes about, make sure to note it and revisit later to go deep on.

Present to retain

I have given a few presentations in my time (in the thousands at this point) and I feel a bit bad about them now. Some of the high level “keynote” style ones with Ben were very much in the “inspiration” style. I am proud of some of those, especially one talk that we both knew so well that we had a period of delivery where a randomizer would BEEP, and the person who hadn’t been talking had to instantly take over, even if mid sentence. It kept us on our toes, made each talk different, and I hope was a bit of fun for the crowd?

I am cheesed off about the talks where I went deep and was trying to really get a lot of concepts across. I wish that I would have structured these differently, to help the group be set to retain. For example, I would get more questions going before the content was delivered to get them thinking, and I would make sure that everyone got to walk away with a series of questions. We vilify “tests” these days as they have been so abused as assessment tools, but we shouldn’t throw the baby out with the bath water and ignore how amazing they are as learning tools.

On the coding side I should have not just handed out questions, but also unit tests that are all red allowing you to work through them to turn as many green as possible. I have often dreamed about creating a ton of these for popular open source frameworks. I always enjoy finding unit tests as they can be great examples of how to use an API, since that is what they are often doing!

Beyond

Conferences and presentations are just the beginning. I now think about all of the articles, books, and podcasts that I have consumed but didn’t put the effort in to retain information that I cared too. The research that has been done on learning over the last twenty years makes past truth into myths. Although we are stuck with a paradox where the hard work of real learning doesn’t feel like you are learning as well as the easy work of your short term memory kicking into gear, we have efficient systems that beat rote repetition. I have been excited to see these work, and at least feel good that my kids won’t have to go through brute force time sinks that I did.

As I look at my calendar and see an upcoming conference, I hope to heed my own advice this time around. I will spend some time before hand on what I hope to get out of it: What topics do I want to explore and why? Who do I want to see? And after the fact I will add any knowledge any further investigations into my system to keep building for life long learning.

Have you explored any new learning strategies recently?