• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

Dion Almaer

Software, Development, Products

  • @dalmaer
  • LinkedIn
  • Medium
  • RSS
  • Show Search
Hide Search

Hapi Thanksgiving; It is about systems of nodes scaling, not node scaling

November 27, 2014 Leave a Comment

Not being a native, I still haven’t worked out everything Thanksgiving. It feels like a dry run for the holiday period, and being in retail it became game time.

On the day before our superbowl (Black Friday) I wanted to give my “thanks”. Gratitude is something I can very much get into!

Last year Eran Hammer lead the charge on #nodebf, where he live tweeted how the systems were handling. It was the first year of big node usage, and every year the systems get a larger work out.

https://twitter.com/eranhammer/status/537806060128833536

Fast forward a year and we have multiple teams using node, and some of them are kindly joining the fun and live tweeting at #nodebf again!

We wanted to give a feel for what the systems do, and the planning behind dealing with the traffic, so we have a couple write-ups for you:

Server Side JavaScript Rendering

Traffic ramping up. Solid 10-20% CPU with occasional spike on a cluster (likely monitoring itself) #nodebf pic.twitter.com/JBuDVdSKRA

— Kevin Decker (@kpdecker) November 27, 2014

The mobile website started off as a client side SPA app. However, for the sake of performance, initial load, and SEO reasons we pushed on a server side rendering solution. Now, instead of the client browser, we have a proxy for our customers running node processes to render the application that the client then hijacks. To really push the bar Kevin Decker decided to hack the rendering pipeline to be able to tweak between async and sync on client vs. server. Here’s what Kevin has to say about it all:

“This year marks the first year that we are doing full scale rendering of our SPA application on our mobile.walmart.com Node.js tier, which has provided a number of challenges that are very different from the mostly IO-bound load of our prior #nodebf.

For all intensive purposes, the infrastructure outlined for last year is the same but our Home, Item and a few other pages are prerendered on the server using fruit-loops and hula-hoop to execute an optimized version of our client-side JavaScript and provide a SEO and first-load friendly version of the site.

To support the additional CPU load concerns as peak, which we hope will be unfounded or mitigated by our work, we have also taken a variety of steps to increase cache lifetimes of the pages that are being served in this manner. In order of their impact:”

Follow him, and read more to get the full scoop on the taming of the event loop.

Quimby, our three legged stool

Response times holding at 3ms as traffic increases, except for one mildly erratic service swaying from 3-25ms #nodebf pic.twitter.com/u9QCBtWtor

— Jason Pincin (@jasonpincin) November 27, 2014

We think that the three legged stool that brings together analytics, A/B testing, and client configuration is crucial.

Jason Pimcin talks about the new Quimby service that ties it all together to make life sing:

“Quimby is Walmart’s service layer for mobile clients’ configuration, CMS, a-b testing setup, and a few other sundry related services. It stitches together a constellation of data sources into a concise menu of API calls that mobile clients make to intialize and configure themselves.

Quimby is a REST service layer based upon the Gogo micro-service framework that we in turn built with Node.js, Hapi, Zookeeper, and Redis. Gogo is able to expose an array of web servers as a single host, and offers the ability to isolate tasks into smaller focused processes, emphasizing scalability and failure recovery. For example, a failure in any micro-service will not affect the life cycle of a request. Gogo also offers the additional features required to build distributed services with shared state, such as leader election.”

Scaling systems not languages or platforms

We are fans of node, but we also have a ton of other systems out there powering Black Friday. They range beyond node to Java and Clojure, but there is one thing that ties together the systems…. the people.

While we love to focus on the black and white of technology, and try to make outrageous claims “X is the best way to scale!”, it isn’t about “scaling node”. Our best systems share other more important patterns:

  • Measurement from the beginning: how can you know if you can or can’t scale for a big event without measuring your systems?
  • Your test systems should be as close to production as possible (else the measurements are weak)
  • The architecture must enable scalable performance so there isn’t a cliff due to a dependency that is brittle.
  • Failover is baked in and assumed. Things will go wrong and break at any scale, so you have to assume it and work around it. We have a fair few flags and configuration to reroute manually, and a fair share of automated smarts!

As I settle in to watch the rampant Thanksgiving that is the calm before the Black Friday and Cyber Monday storm, I give thanks to the giants shoulders that we stand on, and a huge thanks to my team. I love these engineers (and I even don’t mind the other folks ;)

Wish us luck!

Share this:

  • Twitter
  • Facebook

Reader Interactions

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Primary Sidebar

Twitter

My Tweets

Recent Posts

  • Generative AI: It’s Time to Get Into First Gear
  • Developer Docs + GenAI = ❤️
  • We keep confusing efficacy for effectiveness
  • The holy grail of a Web SDK
  • The rise of the extensible app platforms

Follow

  • LinkedIn
  • Medium
  • RSS
  • Twitter

Tags

3d Touch 2016 Active Recall Adaptive Design Agile Amazon Echo Android Android Development Apple Application Apps Artificial Intelligence Autocorrect blog Bots Brain Calendar Career Advice Cloud Computing Coding Cognitive Bias Commerce Communication Companies Conference Consciousness Cooking Cricket Cross Platform Deadline Delivery Design Desktop Developer Advocacy Developer Experience Developer Platform Developer Productivity Developer Relations Developers Developer Tools Development Distributed Teams Documentation DX Ecosystem Education Energy Engineering Engineering Mangement Entrepreneurship Exercise Family Fitness Founders Future GenAI Gender Equality Google Google Developer Google IO Habits Health HR JavaScript Jobs Jquery Kids Stories Kotlin Language Leadership Learning Lottery Machine Learning Management Messaging Metrics Micro Learning Microservices Microsoft Mobile Mobile App Development Mobile Apps Mobile Web Moving On NPM Open Source Organization Organization Design Pair Programming Paren Parenting Path Performance Platform Platform Thinking Politics Product Design Product Development Productivity Product Management Product Metrics Programming Progress Progressive Enhancement Progressive Web App Project Management Psychology Push Notifications pwa QA Rails React Reactive Remix Remote Working Resilience Ruby on Rails Screentime Self Improvement Service Worker Sharing Economy Shipping Shopify Short Story Silicon Valley Slack Software Software Development Spaced Repetition Speaking Startup Steve Jobs Study Teaching Team Building Tech Tech Ecosystems Technical Writing Technology Tools Transportation TV Series Twitter Typescript Uber UI Unknown User Experience User Testing UX vitals Voice Walmart Web Web Components Web Development Web Extensions Web Frameworks Web Performance Web Platform WWDC Yarn

Subscribe via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Archives

  • January 2023
  • September 2022
  • June 2022
  • May 2022
  • April 2022
  • March 2022
  • February 2022
  • November 2021
  • August 2021
  • July 2021
  • February 2021
  • January 2021
  • May 2020
  • April 2020
  • October 2019
  • August 2019
  • July 2019
  • June 2019
  • April 2019
  • March 2019
  • January 2019
  • October 2018
  • August 2018
  • July 2018
  • May 2018
  • February 2018
  • December 2017
  • November 2017
  • September 2017
  • August 2017
  • July 2017
  • May 2017
  • April 2017
  • March 2017
  • February 2017
  • January 2017
  • December 2016
  • November 2016
  • October 2016
  • September 2016
  • August 2016
  • July 2016
  • June 2016
  • May 2016
  • April 2016
  • March 2016
  • February 2016
  • January 2016
  • December 2015
  • November 2015
  • October 2015
  • September 2015
  • August 2015
  • July 2015
  • June 2015
  • May 2015
  • April 2015
  • March 2015
  • February 2015
  • January 2015
  • December 2014
  • November 2014
  • October 2014
  • September 2014
  • August 2014
  • July 2014
  • June 2014
  • May 2014
  • April 2014
  • March 2014
  • February 2014
  • December 2013
  • November 2013
  • October 2013
  • September 2013
  • August 2013
  • July 2013
  • June 2013
  • May 2013
  • April 2013
  • March 2013
  • February 2013
  • December 2012
  • November 2012
  • October 2012
  • September 2012
  • August 2012

Search

Subscribe

RSS feed RSS - Posts

The right thing to do, is the right thing to do.

The right thing to do, is the right thing to do.

Dion Almaer

Copyright © 2023 · Log in