Technology

The Order of the JSON

August 15, 2019

Hey @sendgrid, your team is telling me that we need to send you an *ordered* JSON map object or you will error out. How in the heck do you wind up with a map JSON object that requires *ordered* keys? This so far out of the spec it makes my ankles hurt.
— Brian Fitzpatrick (@therealfitz) August 13, 2019

When I read Fitz’ tweet about ordered JSON my body shuddered, as my brain was flooded with a past experience that taught me a frightening lesson on technology being used by non-technical companies.

It was a moment that had me wonder:

How does world not break due to technology more often
How much time and money is wasted due to some ignorance at some part of the development cycle?

Ok, here goes. My frightening tale around the order of the JSON. I was working on a project for a large Fortune 500 (I will protect the name of the company to protect the innocent!).

The project at hand required integrating a modern front end (mobile apps and web) with a legacy system. This isn’t legacy in a “the engineering team really wants to rewrite it in the new shiny”, but in a “there is COBOL running back there somewhere isn’t there” kinda way.

Much of the system was so old that it was hard to find anyone who knew how it actually worked, and it’s maintenance had been outsourced to some of the typical IT outsourcing companies of the time.

At first all was well. We had created mock services that spoke the protocol and we were building against it. We were dealing with pretty simple REST calls with JSON, so it wasn’t like we were all SOAP-y.

We wanted to integrate with the real systems as soon as they were ready of course, and that’s when it got fun. I remember the first time we spoke to the system an got a terse error message:

ERROR 1000294

That was it. No more info. We asked the service folks for more context, and *two weeks later* we were told “Oh, the JSON payload that you sent us was in the wrong order“.

Huh? Why would you care what order we were sending the name value pairs for this?

We asked if they could fix this and be more flexible. They said they would get back to us.

It turned out their did a “LOE analysis” (level of effort, in case you haven’t had the privilege), and came back to tell us that they would fix it, but it would take 9 people 6 months to do complete. With a straight face. In a way that signalled that this is the cost of doing business and it happens all the time.

We couldn’t wait that long, and we thought about creating a custom emitter that would indeed order the JSON. We wouldn’t want to do it on the client, as what if something changed? We didn’t have faith that this “order” would be set in stone forever.

Ok, let’s handle it on the server then…. and have continuous testing to make sure that if anything changed we would know right away. Not the end of the world I guess.
But it still didn’t feel right. I got on a plane to visit the site where this was all going down.

QA Engineer walks into a bar. Orders a beer. Orders 0 beers. Orders 999999999 beers. Orders a lizard. Orders -1 beers. Orders a sfdeljknesv.
— Bill Sempf (@sempf) September 23, 2014

I mentioned that it was hard to find people who knew how these systems worked. I have previously mentioned how QA engineers are underrated and they came to my aide again here. I hunted down the QA lead and he took me on a tour of how he tests the systems.

I got to learn that we were talking to a service running IBM DataPower Gateway which sat on top of WebSphere which sat on top of the COBOL.

He was showing me the DataPower GUI, and let me play around with it. After awhile of drilling around and going through settings I got to a hidden advanced setting that was a checkbox asking:

Validate order of JSON? [X]

I unchecked it. Ran a client to post the JSON to that instance. It worked just fine.

And, then I sat back and contemplated how this type of event was probably occurring daily, all over the world, and some consulting companies were getting millions of dollars.

Finally, I was so curious about where the root of this came from, and did a search for ordered JSON and ended up on this IBM javadoc.

Of course! I am sure you have never run into scenarios like this in you time in tech, have you?

The Power of the Back Catalog

August 9, 2016 Leave a Comment

I was reading a great book, Console Wars, that details the early gaming console fight for dominance. One period really hit me, and that was when Sega was trying to find room in a world of the dominant Nintendo. The Sega Genesis (or MegaDrive for those in the UK!) got a helping hand when Nintendo decided to ship a new version of their console without the ability to play existing titles. This gave Sega the ability to say “hey, you are buying something new, so try us out!”

That is an example of a mistake hurting adoption, but there are also plenty of examples of the opposite. There are times when a revolution is happening and you need to break with the past (going from vinyl to CDs) but it sure as hell helps if you go through a deprecation phase first.

I was thinking about Apple, and how the Web has helped them succeed with a platform in pretty major ways.

iMac

When Steve came back, decimated the product line, and then showed us the iMac he was able to get into the game by realizing and selling the fact that the things users wanted to do were mainly on the Internet. This gave us the “i” and the push to market those Macs as the best way to access the Internet. At that time the Internet meant many things, but the majority of what people wanted to do was use the Web. They wanted access to the amazing crowd sourcing of information, and ability to find it all via sites such as Google.

Once the Web up leveled from its birth as a document platform to that of a app platform Apple was set. You didn’t need Outlook when you had Yahoo! Mail. Microsoft, by fighting to win the browser war, ended up helping build an app platform that meant their native desktop application dominance no longer mattered too much.

iPhone

By the time the iPhone came along, we had been hearing that the mobile Internet was going to be huge for years. At every JavaOne keynote we heard that developers should be cranking on J2ME to build apps for users. That never came to be, and instead it was all eclipsed by the iPhone.

The iPhone was so much better, but many in the industry poo poo’d it. How could you type without a real keyboard (cried those at Blackberry). It’s user interface was leagues beyond the competition though. Steve showed off the importance of how it brought together functionality such as the cellphone, camera, and iPod all in one. He also showed off Mobile Safari, and I feel like this is often overlooked. We finally had the desktop Web available on a mobile device. Not a crappy WAP/WML experience, but the real Web! Sure, it wasn’t perfect. You had to pinch, zoom, and pan around….. but it played the back catalog. This was vital at a time when there weren’t any third party apps.

Then you fast forward to the era of using the Web to create apps and then native applications on the device. This was critical because you need to go beyond the back catalog. A user is buying the latest device to get great access to all of the affordances and power of the new device.

When creating a new platform, it behooves you to think of the role of the Web. As you create something new it can give you massive useful reach, even if it isn’t perfectly tuned to your platform yet.

When I look at something new, I now look for how they are leveraging the Web. Take VR for example. It is one thing to go after the hard core folk to create WebVR with all of its WebGL goodness. But I am also really excited about work such as CSS VR, because it allows traditional web developers to rather simply (via some CSS vs. learning WebGL/WebVR) join the platform. It’s “pinch to zoom” equivalent may be immersing you in a 360 image while overlaying the important content.

At this point the Web has content at an epic scale. It is content that is indexable, allowing developers to add functionality that crawlers and browsers can understand. It is powerful to enable the loose coupling between all of these agents, compared to setting up APIs for each side to interoperate on. I hope we explore more ways in which the Web can bring functionality to scale in an open manner.

*walks off to play some of the back catalog*

The Principles of Cross Platform Development

September 21, 2015 Leave a Comment

When you said “hybrid” a couple years back, most engineers would assume you were talking about “using a WebView in your native mobile app, maybe with PhoneGap”.

Fast forward to today and you see attention on a whole swath of solutions that try to deal with the fact that our customers want to use our experiences on a variety of platforms, and this is only exploding. Gone are the days where Windows, or The Web Browser, were near-monopoly’s. Now we not only have multiple operating systems, these operating systems come to us from very different types of computing platforms and interfaces: hand held, laptop, desktop, tv, watch, anything-that-talks-to-a-network. That bar for software quality has also been raised, especially on the user experience side, but we have finite resources for product development; hence the desire to get leverage from engineering (as well as design and other areas).

You can quickly start a religious war on the topic, with the two extremes being “everything must be native!” on the one side, and “leverage all the codez!” on the other. In practice there is a huge spectrum of leverage, and you can try to get it in a large variety of ways (e.g. “I will write everything in JavaScript and have it magically run anywhere!”)

I have seen people make choices based too much on bias and looking at the wrong level of abstraction to make the decision, so I have tried to define the principles of cross platform development.

Always Understand The Platforms

There are a legion of Web developers, many of whom would love to keep hacking on Web technology and ignore that iOS and Android stuff. There are managers who see their legions and want to choose solutions that can use them productively. It is smart to get people engaged and productive, but it is wrong to ignore the platform affordances, which will lead you to a destiny of a poor product for your users.

If you are developing a product for iOS, the team should have a deep understanding of the platform and its capabilities. Once you understand the platform, your users, and your products you can marry them all together for the best experience possible.

Don’t build the same experience for mobile and desktop

It isn’t enough to make your desktop web site “responsive”, throw some display:none on your Web app, and walk away for a beer. The mobile experience may be very different than your desktop one. When building out the mobile applications for Walmart we focused much more on the in-store use cases. No one walks into the store with their laptop after all. This meant that the mobile applications were vastly different to the mobile website, which was more focused on eCommerce. This doesn’t mean we weren’t able to get leverage, we were (some parts of the mobile app, the eCommerce parts, mirror nicely!)

The User Experience has to be responsive

If a WebView isn’t up to snuff then I don’t care how many web developers you have, you shouldn’t be using the WebView for the main UI of the application. This normally means that the view layer should render natively, as that is what gives you the right level of responsiveness. This doesn’t mean that you can’t power the view tier in a leveraged manner though. You could have certain logic in JavaScript that powers a native UI. React Native does this very well in the React world, but there are many more ways to do this too.

Your architecture should be flexible and future proof

Where is the escape hatch? What if that one piece of the UI isn’t fast enough, is there a way to get closer to the metal? How hard is it to drop down to that level to make it happen? Putting a native component into a WebView isn’t easy for example.

If iOS X comes out and has some crazy new functionality can you easily incorporate that into your application? Is there a nice abstraction to be able to do this? E.g. <MyComponent> does the right thing, but a very different thing on iOS, Android, and Web.

Be able to test as much as you can

When talking to an ex-Facebooker recently he claimed that the reason why Facebook took off to the races was deltoid, their multi-varient testing platform. This enabled a huge number of experiments to be run concurrently, which has huge changes throughout the product and organization. You can go from hypothesis to data and back again quickly. Rather than argue forever about a feature, just get it out there and see. The Web platform is fantastic at allowing these tests, so what is your strategy for running them on mobile? The app that you ship in the app store is a platform to run your experiences. That platform should have a configuration system, A/B testing system, and a great analytics system that all work together as a three legged stool.

Pragmatic Productivity

Who will be building and maintaining this application?

You can’t blindly do an analysis without looking at the staff, and potential staff, who will be working on the applications. Even if the current shape of the team isn’t the right balance (e.g. a ton of Web talent but less mobile talent) that doesn’t mean that you should fully discount a set of solutions though. You need a plan, which can include outside talent, training, and pairing up resources.

Chances are you will have some people who deeply know the domain at hand, and others that deeply know a platform. If you can get them working well together you can set yourself up for a great future. You don’t want a fantastic native experience that offers a ridiculous subset of features, and you don’t want every feature served up without a thought of the platform.

This also shows that a small team of top notch developers working on a rather small problem may correctly choose a very different engineering approach compared to a hugely complex service with a very large team. There is a good chance that the large team could use a platform that supports multiple ways to integrate with the main app (if there is one).

Tooling: React and RoboVM

Along with the people, you have the tooling and entire development process to look at. Recently I have been looking at two interesting choices to get some leverage. React Native has finally shown up with Android support (still early I know) and a lot of excitement. It is a great choice that trades off Web knowledge with view tier performance. You still need to know how iOS handles layout, and the services and capabilities available to you (as with Android), and that is a good thing! React is also being used as a general view on top of various bindings (such as Quip’s approach on top of shared C++).

RoboVM offers something very differently, but also gives you great leverage without giving up on the platform capabilities. Although you write your code using Java, the view code is split up for iOS and Android. On Android you are just writing a Plain Old Android App, and on iOS you are going through RoboVM. This means that your [Objective kindaRemindsYouOf:C] is Java-like instead (again: you need to know how iOS works!). Some will poo poo this: “I don’t like Java!”, or “Ugh if I am doing iOS I just want to write in Objective-C or Swift!” Well, you can actually do that. You can share the Java business logic code and call into it from your iOS application, so you can choose how far you want to go (escape hatch++). If you have a fantastic Android developer, they can jump and be up to speed pretty quickly (aside from learning iOS which isn’t trivial) and then they can have one IDE open that is working across the entire codebase of both apps and shared code.

Google Inbox uses another strategy where Java offers shared code, using GWT to get to the Web and j2objc to get to iOS. Everyone is hybridizing in different ways in an effort to get some code leverage!

There are trade offs everywhere you look. If you feel like you are getting away with a solution that has no trade offs maybe take another peak. The good news is, just as there isn’t a silver bullet, there also isn’t One True Way. Some times you need to make a decision and charge ahead.

What are the core principles that you look for?