Archives for November 2014

Not being a native, I still haven’t worked out everything Thanksgiving. It feels like a dry run for the holiday period, and being in retail it became game time.

On the day before our superbowl (Black Friday) I wanted to give my “thanks”. Gratitude is something I can very much get into!

Last year Eran Hammer lead the charge on #nodebf, where he live tweeted how the systems were handling. It was the first year of big node usage, and every year the systems get a larger work out.

https://twitter.com/eranhammer/status/537806060128833536

Fast forward a year and we have multiple teams using node, and some of them are kindly joining the fun and live tweeting at #nodebf again!

We wanted to give a feel for what the systems do, and the planning behind dealing with the traffic, so we have a couple write-ups for you:

Server Side JavaScript Rendering

Traffic ramping up. Solid 10-20% CPU with occasional spike on a cluster (likely monitoring itself) #nodebf pic.twitter.com/JBuDVdSKRA

— Kevin Decker (@kpdecker) November 27, 2014

The mobile website started off as a client side SPA app. However, for the sake of performance, initial load, and SEO reasons we pushed on a server side rendering solution. Now, instead of the client browser, we have a proxy for our customers running node processes to render the application that the client then hijacks. To really push the bar Kevin Decker decided to hack the rendering pipeline to be able to tweak between async and sync on client vs. server. Here’s what Kevin has to say about it all:

“This year marks the first year that we are doing full scale rendering of our SPA application on our mobile.walmart.com Node.js tier, which has provided a number of challenges that are very different from the mostly IO-bound load of our prior #nodebf.

For all intensive purposes, the infrastructure outlined for last year is the same but our Home, Item and a few other pages are prerendered on the server using fruit-loops and hula-hoop to execute an optimized version of our client-side JavaScript and provide a SEO and first-load friendly version of the site.

To support the additional CPU load concerns as peak, which we hope will be unfounded or mitigated by our work, we have also taken a variety of steps to increase cache lifetimes of the pages that are being served in this manner. In order of their impact:”

Follow him, and read more to get the full scoop on the taming of the event loop.

Quimby, our three legged stool

Response times holding at 3ms as traffic increases, except for one mildly erratic service swaying from 3-25ms #nodebf pic.twitter.com/u9QCBtWtor

— Jason Pincin (@jasonpincin) November 27, 2014

We think that the three legged stool that brings together analytics, A/B testing, and client configuration is crucial.

Jason Pimcin talks about the new Quimby service that ties it all together to make life sing:

“Quimby is Walmart’s service layer for mobile clients’ configuration, CMS, a-b testing setup, and a few other sundry related services. It stitches together a constellation of data sources into a concise menu of API calls that mobile clients make to intialize and configure themselves.

Quimby is a REST service layer based upon the Gogo micro-service framework that we in turn built with Node.js, Hapi, Zookeeper, and Redis. Gogo is able to expose an array of web servers as a single host, and offers the ability to isolate tasks into smaller focused processes, emphasizing scalability and failure recovery. For example, a failure in any micro-service will not affect the life cycle of a request. Gogo also offers the additional features required to build distributed services with shared state, such as leader election.”

Scaling systems not languages or platforms

We are fans of node, but we also have a ton of other systems out there powering Black Friday. They range beyond node to Java and Clojure, but there is one thing that ties together the systems…. the people.

While we love to focus on the black and white of technology, and try to make outrageous claims “X is the best way to scale!”, it isn’t about “scaling node”. Our best systems share other more important patterns:

• Measurement from the beginning: how can you know if you can or can’t scale for a big event without measuring your systems?
• Your test systems should be as close to production as possible (else the measurements are weak)
• The architecture must enable scalable performance so there isn’t a cliff due to a dependency that is brittle.
• Failover is baked in and assumed. Things will go wrong and break at any scale, so you have to assume it and work around it. We have a fair few flags and configuration to reroute manually, and a fair share of automated smarts!

As I settle in to watch the rampant Thanksgiving that is the calm before the Black Friday and Cyber Monday storm, I give thanks to the giants shoulders that we stand on, and a huge thanks to my team. I love these engineers (and I even don’t mind the other folks ;)

Wish us luck!

Measure measure measure. If you aren’t measuring what you are building then you are running blind and foolish. While it is key to measure according to your goals, it is important to keep the soul and purpose at the heart of why you are measuring, and how far you are willing to go.

Bad Baidu

I hate it when companies trick their users. You should respect them, and I don’t like it when companies that show a lack of respect are financially rewarded. I always hope that karma will come around.

It is in this spirit that I was frustrated to see the bad practice that Baidu exhibited in changing the text of their browser to show up higher in the Intent picker list for browsing the Web. Not only did they add a leading space to get to the top (as did CM! The chinese browser wars are truly insane) but they also added “n(Recommended)”. Cheeky buggers.

I imagine some folks at the office giggling and watching a bump in some metric due to this act. It isn’t a new one either. Some of us remember the companies named “AAAAAA Plumber” in the yellow pages. I never used to call them either, and felt sad that they named their company in such a silly manner.

It probably works though. Else, why would people do it? It must at least have the perception of working ☺

This is a somewhat silly microcosm that reminded me why we need regulation. Market forces don’t have a heart. If your only focus is a series of metrics, you are destined to go after the up and to the right tactics. Increasingly, we have the ability to use computing power to run analyzes and change behavior in real-time. This means that it is doubly important to know what you are aiming for as there will be less opportunity for a human to be a filter.

This is why I dislike the notion that we should allow the market to be in charge of offering our healthcare or education services. These are areas that we need to really take a look at.

Think before you metric

If done right, you can choose metrics that are “win win”. You can also structure the “how” that goes with the “what” to make sure that your values aren’t undermined (the “how” can be seen as your own internal regulation of sorts). How far are you willing to go? Is there a way to balance the metrics, so they aren’t all about short term financial impact, but also about long term customer satisfaction.

This is really hard. It is so easy to trick yourself into thinking that you are doing the right thing, and get weaselly.

How Google Works

A good example that comes to mind is from How Google Works. There is a section of the book that talks about how you don’t have to always have a plan on how you are going to make money doing X, and that you can sometimes go with a gut feeling that “we will work it out”.

This somewhat goes against Peter Theil’s views on the importance of “having a plan”, but the devil is in the details. If you have AdWords, you can afford to work out some other things later 😉

One of the examples given was that of Keyhole. Google bought the Boulder company but had no real thoughts on how to monetize it. Google Earth was given away for free, and then the “a ha!” moment happened. They bundled Google Toolbar. Users of toolbar made more money for Google (due to their increased usage) so Earth became a trojan horse.

Now, I don’t know about you, but I am not a huge fan of bundling. When you purchase “A” you often don’t really love seeing “B” show up. We went there with Windows, remember? Ugh. One of the legacies of Steve Jobs is the desire to not sell off the customer experience for some quick money like this.

What was fascinating was that the book spun the bundling as a good thing for customers:

“If our customers were downloading Google Earth they will probably like Google Toolbar”

Oh right! You are helping! So kind of you! Now, I am sure that there were some customers that appreciated this, but come on ☺ I also remember some shady features of toolbar, and then there were the deals with companies to bundle toolbar elsewhere.

If you drink enough kool-aid and think that your company has such a great purpose then you can fall for the “it is a means to an end!” trick.

---

Meditating on purpose

Your purpose is important though (as long as you don’t fall for the trick), and a great purpose can root the path to objectives and then the results that you measure.

Justin Rosenstein just gave a great short talk on this very subject at a recent Web Summit:

He discusses the importance of Clarity, and how the primary function of a leader is to provide clarity to your team.

He then goes on to discuss the Why, How, and Who threw the lens of clarity:

Clarity of Purpose (Why)

It is hard for a sane and happy person to work on something that doesn’t clearly impact the world in a positive way.

Can you answer the following for your company?

• Why does your company exist?
• If it is wildly successful how will the world be different than it otherwise would have been?

Clarity of Plan (How)

What is worse than having no purpose? Having a great one but no plan on how to bring it to the world!

To build the plan at Asana they map a “Master Strategy” for the top level pilars of the company and then map from that plan to various key results (that can be measured), which then are broken down to actionable tasks.

Clarity of Responsibility (Who)

There needs to be a clear owner of objectives and areas, that cascade all the way down. Your job as the DRI isn’t to mandate the how, but to mandate the outcome and trust your people enough to work out good solutions.

If you don’t trust them? You need to replace them, but don’t meddle!

All in all it has me thinking a lot about the yin and the yang of your purpose, and balancing the “how” in a way that keeps you on track, but not in an unethical manner.

I finally got to participate at the spiritual home of the community movement: The November Project in Boston. With the calendar wrapping around to another November I thought it would be a good time to reflect on that experience, as well as a good pondering on what our next project should be.

As I have gone through my own small health renaissance, I have come to see the value of the November Project movement, and how much we need to embrace the principles.

Brogan and Boyan started small, by pushing themselves to get outside and use the city as a (free) gym, and to use each other to make it happen. November is a cold, meh, month in Boston. How do you not get reclusive and instead get out there?

Fast forward to today, and you will find hundreds of people joining B and B, three times a week, to move their bodies, work hard, and hug.

They talk of the tribe and you feel it. They care about each other more than just themselves. The experience is fun. You do silly things. You sweat. You come away with a smile on your face to go with some soreness. What a great way to kick start the day even if you think that you aren’t a morning person.

Aside: I definitely consider myself a night owl and some of my best work and coding has been done between 10pm and 4am. I have fond memories of getting in the flow and looking up to see that the Sun is up.

It is the antithesis of both the treadmill, and even Crossfit.

Treadmill

Running is running. Moving your body is good. Treadmills have the advantage of being able to work well even if the weather sucks outside, and you can watch TV…. So maybe you setup the “I can watch TV, but only if I am working out at the same time!”

But all running isn’t equal at all. Treadmills may work for folks and who am I to disagree, but for me…. I want to use that time to be outside! Breath in the air. Connect with the ground, have your body move to the contours and use your entire body. This leads me to have enjoyed trail running and getting more wild. Less treadmills, and even less roads. Change the up and down, and enjoy it out there. I took the ear buds out and enjoyed being.

The November Project work outs surprise you, inspire you, and have the benefit of community. As we have evolved and become globally connected and have abstractions such as money exchanges, I wonder if we have less reasons to connect locally. Communication used to be so very vital to us all to survival. You need to know if your neighbor has wheat and if you can exchange your products. What was happening in the next town over? Information is still vital, but can more of us both ignore some of it, and also participate in a way that isn’t in person…. And what do we lose there? It feels more and more important to connect in person.

CrossFit

CrossFit has community at its core, and it is one of its real strengths. I think that it has done a lot of good, and lifting weights is such a truly awesome thing to do for your body that it is hard to fault the box too much.

However, it seems like there can be a tendency to push people too hard, and in a way that harms. The epitome of this was when I saw CF fans applauding Carl Yngvar Christensen for his world record squatting. I know that the effort to build that strength and technique is immense, but watch the video and see how he struggles to walk out. Is this what you are striving for? I don’t know about you, but I am looking for resilience, where strength is helpful but not the only marker.

Of course all CF isn’t like this. It is the extremes that really push too far, and have injury go against health. There are great coaches such as Kelly Starrett in San Francisco who really focuses on functional movement. With guides like him and Diane Fu to help you, chances are you will be just fine.

Want to be on the safe side? Body weight exercises can be a great way to get you where you need to be (building your functional base) while minimizing the risk of injury and high level of technical accuracy. There is a long path from a few regular push ups to head stand push ups. Plenty of room to build true strength. You can even look to the unlucky guide of prisons to give you a plan.

Winter means a new year approaches

This is why I resonate with a tribe such as NP. An outdoor community that isn’t a sweaty indoor box.

As November comes around again, and the winter brings in the end of the year, it is good to think about the project of You, and of Us.

What are you going to be?