• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

Dion Almaer

Software, Development, Products

  • @dalmaer
  • LinkedIn
  • Medium
  • RSS
  • Show Search
Hide Search

Speak and Type; Giving hints to get info to the system faster

February 10, 2014 Leave a Comment

SwiftKey app on my iPhone

My devices can understand me more every day (and not just in a “Her” way!) I want them to listen to me while I communicate with them in other ways. I am really excited when input modalities become AND vs. OR.

What do I mean here? Right now, when I use voice options on my device they are very single tasked. I am typing something in my IM client and can hit a button to say “let me talk to you now” and then I am able to talk to the system, and when “Done” a spinner eventually (hopefully) gets replaced by text that makes some sense. I can type, or I can speak. I jump between the two (e.g. edit something that came back) but this isn’t a nice bidirectional communication between the different input modes.

I want to be able to talk and type *at the same time*. Why would I want to do this? Here are some use cases:

Changing modes so I can tap less to get what I want in the text

E.g. If I don’t want auto-correct on, let me quickly disable it with “auto-correct off”. If I say this before typing “foo” it can stay with that rather than correcting to “for”.

I say “caps” mid sentence and get “year I KNOW” without having to go into caps mode via the keyboard.

Another version of modes is making other selections such as changing the color of a “paintbrush” by saying “red” while your finger keeps tap-painting.

Hinting to get better translation

E.g. I say some words while typing, especially common words that have this annoying feature: two common words, with letter next to each other on the keyboard. If I say “dad” but when typing I hit closer to “sad”, the system can know “I heard the ‘d’ so lets go with Dad!”

Choosing options

With keyboards such as SwiftKey, and Fleksy, I end up typing very differently. It becomes a lot more about making choices based on what they are guessing is coming up. Good stuff, but I find it can also be annoying to be jumping from the letter to the options.

Let’s take SwiftKey and the three options. What if when typing I could make a choice be saying either: the word directly “Hey” or “Left”/“Middle”/“Right” or “1”, “2”, “3”.

Gaming

Then you get into other niches such as gaming. While running around it would be nice to be able to say “change weapon axe”, “switch gear 2nd” and the like.


Will this work?

I am curious to see if this mixed mode would work well. We have seen it happen in other parts of our controls.

In computing we have mice and keyboards. Often we do these in single task mode but think about running around in a first person shooter.

Outside of computing, when you are driving you are doing many things at once, including having your feet on pedals to control speed, while your hands steer.

Speaking of driving, I remember that “expert” driving classes involve you talking through your actions, and that you can get clarity from talking, so I wonder what effects that will have?

Won’t it be socially awkward?

In a vacuum this can work well, but what about when you are out in the real world? I use Siri a lot more when alone than when around other people. Not only is it still strange to talk into your phone, but it also happens to work a lot better when you are in an enclosed space without other noises around to distract the system.

I wonder if this fact will mean that by not always doing the talking, you won’t ingrain the habit, and thus will focus more on getting better with taping vs. mixed mode hints.

Speed matters

I can’t wait to see more and more experiments with mixed mode here. I use a keyboard less and less and through better keyboards I can become pretty efficient, but now and then it is still frustrating to be in a situation where I want to get the info into the system faster than I can.

Maybe we can switch to a direct brain system where we just have to make sense of the craziness of the input?

What do you think?

Share this:

  • Twitter
  • Facebook

Reader Interactions

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Primary Sidebar

Twitter

My Tweets

Recent Posts

  • Generative AI: It’s Time to Get Into First Gear
  • Developer Docs + GenAI = ❤️
  • We keep confusing efficacy for effectiveness
  • The holy grail of a Web SDK
  • The rise of the extensible app platforms

Follow

  • LinkedIn
  • Medium
  • RSS
  • Twitter

Tags

3d Touch 2016 Active Recall Adaptive Design Agile Amazon Echo Android Android Development Apple Application Apps Artificial Intelligence Autocorrect blog Bots Brain Calendar Career Advice Cloud Computing Coding Cognitive Bias Commerce Communication Companies Conference Consciousness Cooking Cricket Cross Platform Deadline Delivery Design Desktop Developer Advocacy Developer Experience Developer Platform Developer Productivity Developer Relations Developers Developer Tools Development Distributed Teams Documentation DX Ecosystem Education Energy Engineering Engineering Mangement Entrepreneurship Exercise Family Fitness Founders Future GenAI Gender Equality Google Google Developer Google IO Habits Health HR JavaScript Jobs Jquery Kids Stories Kotlin Language Leadership Learning Lottery Machine Learning Management Messaging Metrics Micro Learning Microservices Microsoft Mobile Mobile App Development Mobile Apps Mobile Web Moving On NPM Open Source Organization Organization Design Pair Programming Paren Parenting Path Performance Platform Platform Thinking Politics Product Design Product Development Productivity Product Management Product Metrics Programming Progress Progressive Enhancement Progressive Web App Project Management Psychology Push Notifications pwa QA Rails React Reactive Remix Remote Working Resilience Ruby on Rails Screentime Self Improvement Service Worker Sharing Economy Shipping Shopify Short Story Silicon Valley Slack Software Software Development Spaced Repetition Speaking Startup Steve Jobs Study Teaching Team Building Tech Tech Ecosystems Technical Writing Technology Tools Transportation TV Series Twitter Typescript Uber UI Unknown User Experience User Testing UX vitals Voice Walmart Web Web Components Web Development Web Extensions Web Frameworks Web Performance Web Platform WWDC Yarn

Subscribe via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Archives

  • January 2023
  • September 2022
  • June 2022
  • May 2022
  • April 2022
  • March 2022
  • February 2022
  • November 2021
  • August 2021
  • July 2021
  • February 2021
  • January 2021
  • May 2020
  • April 2020
  • October 2019
  • August 2019
  • July 2019
  • June 2019
  • April 2019
  • March 2019
  • January 2019
  • October 2018
  • August 2018
  • July 2018
  • May 2018
  • February 2018
  • December 2017
  • November 2017
  • September 2017
  • August 2017
  • July 2017
  • May 2017
  • April 2017
  • March 2017
  • February 2017
  • January 2017
  • December 2016
  • November 2016
  • October 2016
  • September 2016
  • August 2016
  • July 2016
  • June 2016
  • May 2016
  • April 2016
  • March 2016
  • February 2016
  • January 2016
  • December 2015
  • November 2015
  • October 2015
  • September 2015
  • August 2015
  • July 2015
  • June 2015
  • May 2015
  • April 2015
  • March 2015
  • February 2015
  • January 2015
  • December 2014
  • November 2014
  • October 2014
  • September 2014
  • August 2014
  • July 2014
  • June 2014
  • May 2014
  • April 2014
  • March 2014
  • February 2014
  • December 2013
  • November 2013
  • October 2013
  • September 2013
  • August 2013
  • July 2013
  • June 2013
  • May 2013
  • April 2013
  • March 2013
  • February 2013
  • December 2012
  • November 2012
  • October 2012
  • September 2012
  • August 2012

Search

Subscribe

RSS feed RSS - Posts

The right thing to do, is the right thing to do.

The right thing to do, is the right thing to do.

Dion Almaer

Copyright © 2023 · Log in