Scaling with Immutable Data; When Tech meets Retail

I am enjoying my second Black Friday in retail (I work for Walmart Labs Mobile, and we create apps for the global Walmart community). This is game time for us, and along with the nervous stress you get a lot of the joy. You want to be in the big game.

I am new to retail, so it has been a fun learning experience to see what it is all about, and I have been able to meet amazing retailers. I wanted to tell one simple story about thinking through a problem with a retailer.

Today is a run up to Black Friday in the US, and we have been releasing new deals throughout the day, a bunch of them exclusive deals via our apps (iPhone, iPad, Android).

If you are a retailer in the store, how may you role these out? Maybe you would have a special shelf at the front of the store, and for the first run you put out the Furby’s (until they quickly sell out). Then for the second run you put out the TVs, and then the Waffle Irons…. you get the picture. The problem though is that if items don’t sell out, then you may have created a bottleneck in that area of the store. For Black Friday in our stores for example we place the specials all over the place so customers can get some space (which is tough with the number of them in the store!).

Now, let’s move over to the online world. In the eCommerce platform we have a notion of a “shelf” too, which houses product. The natural solution would be to slowly add new products to the magic shelf as the day progresses, and then any items that are not sold off can hang out there. Great!

Well, maybe not so great when it comes to scaling up the solution. We have a crap ton (technical term) of online users hitting us at the same time. Caching is our friend, and the friend of caching is immutable data. It isn’t just for data sets in your code.

To help us out here, we actually setup multiple shelves for each event.

Shelf #1 @ 7am: Furby

Shelf #2 @ 9am: Furby and TVs

Shelf #3 @ 11am: Furby and TVs and Waffle Irons

And so on. Our apps and experiences have pointers to these shelves, so all we need to do is change the pointer, and the apps see the other shelf. The advantage here is that the apps never have to think about reloading a shelf, as the high level product data is the same (there are details within that matter such as product availability but the structure of shelf -> [ products ] is static).

It is a little more work for the merchants to setup as they need to duplicate the product info, and we thank the merchants for doing that work…. but it means that through the immutability we can scale much easier, and the clients can be so much simpler.

The moral of the story for me has been, how can we make as many of our data structures immutable, which often means thinking about the problem differently to how you may do so in a physical store.

After all, as a wise person said:

The basic goal of a database is to maintain the illusion that there is only one copy, only one person changes it at a time, everyone always sees the most current copy, and it is instantly fast. This is impossible to achieve.
This is especially impossible when millions of people are accessing and updating trillions of data elements from all over the world.
The task of database design, therefore, is to come as close to this illusion as possible using hundreds of interlocking algorithmic tricks.