Dion Almaer

Software, Development, Products

Development

The Order of the JSON

When I read Fitz’ tweet about ordered JSON my body shuddered, as my brain was flooded with a past experience that taught me a frightening lesson on technology being used by non-technical companies.

It was a moment that had me wonder:

  • How does world not break due to technology more often
  • How much time and money is wasted due to some ignorance at some part of the development cycle?

Ok, here goes. My frightening tale around the order of the JSON. I was working on a project for a large Fortune 500 (I will protect the name of the company to protect the innocent!).

The project at hand required integrating a modern front end (mobile apps and web) with a legacy system. This isn’t legacy in a “the engineering team really wants to rewrite it in the new shiny”, but in a “there is COBOL running back there somewhere isn’t there” kinda way.

Much of the system was so old that it was hard to find anyone who knew how it actually worked, and it’s maintenance had been outsourced to some of the typical IT outsourcing companies of the time.

At first all was well. We had created mock services that spoke the protocol and we were building against it. We were dealing with pretty simple REST calls with JSON, so it wasn’t like we were all SOAP-y.

We wanted to integrate with the real systems as soon as they were ready of course, and that’s when it got fun. I remember the first time we spoke to the system an got a terse error message:

ERROR 1000294

That was it. No more info. We asked the service folks for more context, and *two weeks later* we were told “Oh, the JSON payload that you sent us was in the wrong order“.

Huh? Why would you care what order we were sending the name value pairs for this?

We asked if they could fix this and be more flexible. They said they would get back to us.

It turned out their did a “LOE analysis” (level of effort, in case you haven’t had the privilege), and came back to tell us that they would fix it, but it would take 9 people 6 months to do complete. With a straight face. In a way that signalled that this is the cost of doing business and it happens all the time.

We couldn’t wait that long, and we thought about creating a custom emitter that would indeed order the JSON. We wouldn’t want to do it on the client, as what if something changed? We didn’t have faith that this “order” would be set in stone forever.

Ok, let’s handle it on the server then…. and have continuous testing to make sure that if anything changed we would know right away. Not the end of the world I guess.
But it still didn’t feel right. I got on a plane to visit the site where this was all going down.

I mentioned that it was hard to find people who knew how these systems worked. I have previously mentioned how QA engineers are underrated and they came to my aide again here. I hunted down the QA lead and he took me on a tour of how he tests the systems.

I got to learn that we were talking to a service running IBM DataPower Gateway which sat on top of WebSphere which sat on top of the COBOL.

He was showing me the DataPower GUI, and let me play around with it. After awhile of drilling around and going through settings I got to a hidden advanced setting that was a checkbox asking:

Validate order of JSON? [X]

I unchecked it. Ran a client to post the JSON to that instance. It worked just fine.

And, then I sat back and contemplated how this type of event was probably occurring daily, all over the world, and some consulting companies were getting millions of dollars.

Finally, I was so curious about where the root of this came from, and did a search for ordered JSON and ended up on this IBM javadoc.

Of course! I am sure you have never run into scenarios like this in you time in tech, have you?

Google I/O 2018: Integration in the Open

Leave a Comment


With Google I/O 2018 kicking into gear, I found myself looking back at my thoughts on last years event and thinking about what is changing and what is continuing with respect to the journey for our developer platforms and ecosystems.

Technology exists within the framework of what is going on in the world. On the one hand, the world gets better for more people every day, but on the other hand we also know how fragile things are, and how much pain people are in as we see growing inequality in areas. Within this context we are seeing technology companies look deeper into themselves, thinking more about what is being created. Are we building experiences that are meaningful for people? That is the question at hand, and the consumer keynote shows some of what Google is thinking about here.

The Integration Story

One of the foundations of integration is componentization, and we are seeing great strides here across our platforms.

Android’s new Jetpack libraries accelerate development

This year builds on last years Kotlin news, and shows our vision for mobile Android development. App bundling and dynamic delivery allows you to split up your app, giving users the ability to download the pieces they need. If you want to reach all of your potential users and deliver the best possible experience to them, you should be focusing on the vitals, and our tooling and platform is working on making life much easier for you here. With Actions and Slices, you can bring pieces of your app to new surfaces on Android to surface where users want them.

Android Studio 3.3 brings a slew of improvements and cuts a huge amount of time out of your development. Did you see Tor demo how quickly the emulator starts now? With Jetpack, we bring together the best of architecture components with the support library, allowing you to develop much faster, reach more users with the results, and all with top notch Kotlin support.

And this year Android really hits new form factors hard with great updates across Wear, TV, and Android Things hitting 1.0.

Spotify shows off their Desktop PWA

On Web, if you follow the Lighthouse (newly updated for I/O!) well-lit path, you will be able to deliver ephemeral experiences that integrate deeply with the host operating system. Performance is still the number one area of focus, but you will see that we are broadly looking to get rid of friction wherever it lay. Web packaging takes fast and makes it instant, and it is now simple to get your users signed up, signed in, and able to quickly pay.

While the first PWAs that drew attention were mobile, with desktop PWA support we get to see how your investment scales. Twitter has been able to take their “lite” mobile app and bring it to the desktop/tablet, Windows store, and with the Chrome desktop PWA affordances you get an amazing experience that marries the your own focused app with browser features such as URL sharing, find in page, casting, etc. This has allowed Spotify to integrate their web experience, for Starbucks to double the usage of their PWA.

Chrome DevTools continue to improve with end to end support for the latest in layout (grid), JavaScript (eager eval, async/await, etc), and even end to end debugging for your node services. With Lighthouse baked in and lighting the way, you can see how far the frameworks are coming along to help deliver the fast experiences the Web needs. Polymer 3 and lit-html allow you to build blazingly fast experiences that are close to the metal, and integrate so well with Redux and friends. Angular 6 joins the custom elements party and the Ivy runtime shows how much of a diet it has been on for great enterprise products. And AMP is here for an opinionated fast path with support for Stories and other new components that you can drop in and use. The Web is here for you, from rich content to complex apps that can tie into new ecosystems with WebAssembly.

Turn on the new Linux support in ChromeOS settings

You may notice more Pixelbooks at I/O this year. I have been using desktop PWAs a lot recently, as I have been living on a Pixelbook. I had tried to make the switch to ChromeOS several times, and this is the first time that it has stuck.

I used to feel constrained inside a browser, but this has flipped since I changed the way I used the system. On my desktop, I normally have a set of pin’d tabs and Ctrl/Command-# to jump to a particular main app (email, calendar, tasks, chat). On ChromeOS I pin those as apps and Alt+# to jump to them.

This frees up my browser. The “email” browser turns out to be my main source of tabs, and I see a new pattern forming. It turns out that a lot of “apps” are sources for browser tabs. I click on links in Gmail AND from Calendar AND Asana AND Chat AND Twitter. This drives an interesting world where the first tab is the main source of other experiences that can be tied to that app instance.

Having the desktop UI (three vertical dots) available brings the best of the browser to the app. The user agent is out of the way, but you have a quick path to find in page, font sizing, casting, autofill, and copying the URL for easy sharing…. and you can imagine so much more.

With the release of secure Linux apps via VM/container goodness, I am able to run my favorite developer tools (Visual Studio Code, Atom, Android Studio) as well as other key Linux apps (including other browsers for testing). It’s still early, but it’s already great to be able to develop here.

Linux and the Web is great, but what about access to the entire Android and Assistant ecosystem? It’s all here, and with the world class security of ChromeOS (and ability to login to a new machine and be right where you left off… which feels like my college days with XWindows!).

Reach the assistant through Web and Android

In fact, the Assistant, and Actions on Google in general, is another perfect example of integration. Web developers, Android developers, and backend developers should all be able to get their experiences in front of Assistant users. With Actions for these platforms, you can get a broad reach, from something quick and direct all the way to a full conversational experience across text and voice, and with RCS you can leverage the texting experience all the way to Android Auto.

ARCore: now with Sceneform, Augmented Images, and Cloud Anchors

AR and VR is yet another example, where the technology is available for you to use the world around you. I can’t wait for a future browser for the world (whether it be Earth or something else immersive), and that world is a lot closer with the new tools that we announced today:

  • Sceneform is a new 3D SDK for Java developers
  • Augmented Images allows you to associate AR content with the real world
  • Cloud Anchors bring a shared collaborative environment to life. Your apps, on iOS or Android, can manipulate space in a way that all of your users can see.

ML SDK across the platforms; from edge to cloud

The other piece of integration is the role of client and server. We continue to live in a disconnected world, with variable levels of compute and network power on our devices married to amazing levels of compute on the server. Developers need the ability to drive as much as possible on the edge where it makes sense, but also use the power of the data center where needed. A strong theme at I/O continues to be the role of machine learning and AI, and not only do we have a lot to share on the side of Google Cloud and TensorFlow, but we keep bringing that to the client with APIs such as ML Kit, able to reach all platforms via Firebase.

In fact, ML Kit is able to infer on device, but also use the Cloud to get more detail (e.g. maybe on device detects a “bridge” in a photo, but the Cloud comes back with a more detailed “Golden Gate Bridge”).

The APIs in the initial kit include:

  • Text recognition
  • Image labeling
  • Barcode scanning
  • Face detection
  • Landmark recognition
  • Smart reply

And you can expect a lot more to come!

Same great design system, themed your way

The last piece is the UI, and we have a major upgrade to the world of Material Design this year. People have noticed our own properties get large upgrades, such as the recent Gmail redesign, and this work is on the back of the new Material Theming approach. Companies have wanted to take the material design system, and really brand it to their needs, and with the new theming system it is much much easier to do just that.

The new system comes with new tools, such as the Theme Editor, which are one of a series of updates where we try to help you build rich experiences with more ease. All of the components are released as open source, for Android, iOS, Web, and now also for Flutter. Flutter announced beta 3 which brings large improvements, and could be a perfect fit for your cross platform needs.

Phew, this is only the tip of the iceberg on what we have in store for I/O this week. The teams have been working hard to build useful things, and share them throughout the week with great talks. We will be rushing to get the content up as soon we can after the livestream.

One last thing. Watching and listening in is great, but there is nothing like touching the technology and seeing where things really are. This is why I always dart over to the codelabs area and try everything out.

Google I/O is truly about the integration of everything that we have to offer across our ecosystems. In that spirit, it’s most fun to see how these things work together and build bridges.

https://mobile.twitter.com/kelseyhightower/status/993846267976470528

I have always been drawn to open ecosystems, so I love that openness is baked into our DNA. I love looking at the history of Android, Chromium, TensorFlow, Kubernetes, Flutter, Polymer, Angular, AMP, gVisor, and the myriad of over initiatives that have open source at the core.

“platforms integrate
together in the open
all for us humans” — Steven Colbert #io18

Republicans, The Lottery, and Apps

Leave a Comment

Without attempting to be too political, I have always been a touch confused at the working class folk who are strong supporters of the current Republican ideology. It seems pretty clear that the fiscal policy favors the rich, as is evidenced by the current discussion on income inequality (although you can argue that the democrats don’t also shoulder the blame!).

A simple view could be one of:

a) “I hate taxes. The government wastes that money so I shouldn’t have to give them anything to waste!”

b) “One day the American Dream ™ will come my way, and then it will be in my favor!”

I won’t bother to discuss a), although I am a fan of the fact that we have shared infrastructure. The attitude shared in b) is the one that relates to the rest of this discussion.

It also ties into the lottery system, and how we manage to fund programs by using the law of large numbers. If you get a lot of people to pay a couple bucks that adds up big with enough of a population but in theory doesn’t have a large effect on them. Huh, that isn’t all bad. What if we took the change from every purchase and also donated it. That scale could do so much good! However, isn’t that what taxes is for? Isn’t that what people seem to hate so much? What have taxes ever done for me!!!! (Other than pave society?)

The lottery sells hope. It is actually the lack of the american dream that paradoxically calls to many. Deep down you know that you are fighting an uphill battle to earn these riches, so you reach for luck instead. It is easy to poke fun and say that the lottery is a tax for those who are bad at Math, but it isn’t our place to judge. When you plan a vacation, you often get more joy from the build up and anticipation, so maybe some of this helps here.

Bring in the Apps

Building an app, getting it into the app store, and hoping for the customers to come raining in sure feels like a lottery these days doesn’t it? When a new platform comes along there is the rush to the gold. You can often create something new and get lucky, or you can copy history and make money selling tools and infrastructure to the miners.

At this point there is already an app for that, and if there isn’t you need to fight for mindshare in a store with masses of choices.

When creating your lovely application it sits proudly on your homescreen. This is where you envision it on a large number of users phones. This is the pride of place for the applications that need repeat engagement and access. Does your application really warrant that?

I was working with a friend who has a fantastic new group event service. It was a great example of something that it useful, but an app may not be needed. The event organizer may want the access and the rich functionality. The group they run may be an important part of their lives. But for the other members of the group? It may not be as important.

That doesn’t mean that they wouldn’t want to get notified when something important changed for their group. It doesn’t mean that if they enter the experience it should only be able to tell them basic info if they have a good internet connection.

This is why I am excited for progressive web apps. I picture a world where this application can get a broad reach and also get the engagement capabilities it needs.

You always need some luck when you come up with a new business or service. To give yourself a chance for it to grow and flourish it may be wise to not have to hit the homerun of the homescreen. Allow your business to work through frictionless links, A/B test without large delays, and don’t waste time and resource porting an experience multiple times.

The right thing to do, is the right thing to do.