If you’re going to take one thing away from a discussion about DevOps and Android, I hope it’s this: Your Android MDM is the only reason updating your device fleet is any more difficult than updating your smartphone. With a vibrant hardware and software ecosystem, it’s never been more apparent than now that the world of dedicated devices — unitaskers like kiosks, payment terminals, and even many consumer electronics — is ready for a DevOps revolution with Android as its standard bearer.

Why Android for DevOps?

While Android found its global fortune in the smartphone, when the OS was conceived back in the late 2000s, it was meant to be used on professional SLR cameras. That vision never materialized (and the smartphone has driven much of Android’s evolution to date), but a much greater one did: that of the Android ecosystem and its many colorful incarnations.

Today, the Android Open Source Project (known as “AOSP”) underpins more devices in more scenarios than it’s likely reasonable to list in a single place. The smartphone still comprises Android’s white-hot molten core, acting as the center of gravitational force. But travel back up to the surface of that ecosystem, and you’ll see a whole lot of biodiversity. What’s more, tons of those devices aren’t even running an operating system Google itself would call “Android,” instead utilizing the open source code Google publishes as part of AOSP to assemble a fully functioning (sometimes, not even fully) OS. Google has little control over these AOSP devices (especially since most don’t receive OS upgrades), and because they lack Google’s own mobile services and applications, they often too lack the necessary cloud APIs and support infrastructure to enable firmware over the air (FOTA) and even may struggle to deploy basic application updates in a reliable, repeatable way.

Perhaps this shouldn’t have happened, and perhaps Google would have done more to support these AOSP projects with the benefit of hindsight, but a new market reality has set in: Android is the choice for dedicated devices, and not simply because it is “free” (and let’s drop the pretense: No software is truly free if you’re the one who has to implement it). Android offers a huge library of APIs and supporting documentation, a ready-made UI, a reasonable library of stock applications, cross-platform development tools, and a hardware partner platform second to none. Android was also designed for the future: wireless, always on, and always connected (often without a constant power source). Android being “free” surely helps, but that’s no longer the driving force behind its adoption. Frequently, Android is just better than anything you could ever hope to build yourself.

Designed to receive FOTA upgrades in place (with minimal disruption using features like A/B system partitioning), Android presents a generally unrealized opportunity in a massive number of industries: devices that can meaningfully improve and iterate in the field with software and firmware innovations, using code-driven automation to deploy the two intelligently.

FOTA control is the gateway to Android DevOps

Being able to upgrade your fleet’s firmware over the air is a huge technical advantage. Thankfully, much of the Android FOTA infrastructure problem has been solved … for Android smartphones. High-end handsets from Samsung and Google receive monthly firmware updates with security fixes, quarterly maintenance releases, and annual OS upgrades. Applications update seamlessly in the background, with the user rarely the wiser. Achieving similar results on a dedicated Android system like a stationary kiosk, though, presents major challenges.

Android as a “complete” operating system is available to build using AOSP. You can even flash that OS straight to hardware using an Android Generic System Image (GSI), erasing whatever out of the box software experience the OEM placed on the device. But this is in an idealized scenario: A(n unlockable!) consumer smartphone, running consumer smartphone applications, with a user who expects a consumer smartphone experience. It’s also not a scalable solution.

If you want to update the firmware on a retail store kiosk, you’ll need a distribution platform and infrastructure to manage it (read: a cloud). And what happens when something inevitably goes wrong? If your device’s manufacturer upgrades your firmware and it breaks a mission critical application, who do you turn to? Frequently, the answer is “no one.” Most device OEMs aren’t interested in bespoke firmware support (not unless you’re ordering in some truly huge quantities), and even if they were, don’t have tools designed to be used by third parties to manage these processes (let alone manage them precisely). Many won’t even allow you to opt out of upgrades you don’t want.

Owning your firmware image and managing FOTA deployment is step one on the path to implementing true Android DevOps for devices. With Esper, you can leverage our distribution of Android to keep your fleet up to date with the most recent security patches, platform features, and enable unique functionality unavailable on the typical Android consumer experience (or, just as importantly, shelter your devices from unwanted updates). To learn more about our solution, check out this page.

Breaking the Android MDM box

Once you have control over your AOSP firmware and FOTA strategy, an entire world of new possibilities opens up. Your devices become fully manageable objects, allowing an incredible degree of control and customization. There are new challenges, too, but frankly, they’re the kind you want to have (when you have the tools to address them).

Most companies using AOSP to build firmware for their devices never have a DevOps realization “moment.” Instead, they’re frequently locked into the MDM waterfall software deployment cycle, in which a very fragile, very user-unfriendly, but very cheap management interface will allow you to deploy applications and set policy from the cloud well enough to reach your devices at a theoretically massive scale.*

(*Eventually. And probably not actually all of them. And you won’t actually know which ones received that update, or why they didn’t get it, or whether there’s something you could have even done to prevent a botched rollout in the first place… unless you decide to build all this telemetry yourself.)

We think MDMs have their place in the enterprise, but when it comes to testing updates precisely, deploying at scale aggressively, and continuously monitoring rollouts — the agile DNA of DevOps — existing MDMs offer you little more than a batch script with PR.

If you’re disappointed by Android MDM offerings (and we don’t blame you), you’ve likely investigated how easy it would be to just become GMS certified and roll with Google’s Play ecosystem and tools. For some companies, this is the best way to get there. If your timeline for certification isn’t tight, and if the Play Store is going to provide a major content value add to your device, Google has a path for you. But if your device is unusual in some way, or otherwise not cut in the cloth of a typical Android smartphone, you may start hitting some pretty big snags. While we don’t have time to dive deep into the particulars here, this post provides a thorough overview of how to think about building a GMS versus non-GMS Android device. You can also check out Google’s CDD (Compatibility Definition Document) if you’re looking for a relaxing evening read. Consider it the ever-evolving sacred stone tablet that defines just what a “true” Android device is and will be, often forecasting changes years into the future. (Hey, nobody said building a multi-billion device ecosystem wouldn’t result in some seriously hefty documentation.)

This is in large part because Google’s owned Android ecosystem was built by and for massive smartphone manufacturers, with a lot of rules and restrictions to match. Most MDMs, as a result, have shaped their products and tools in the image of those smartphone manufacturers — a “one size fits all” approach designed for the billions. That’s a topic for another day (and Google’s rules are well meant, often protecting consumers), but what you need to know from a DevOps perspective is this: If you can’t use Play, achieving truly agile, precise, and observable software deployment with Android isn’t going to be like flipping a light switch. But Android is the ideal platform on which to build these processes because it’s already architected out of the box with the appropriate connection points to enable them. See where we’re going?

Esper is DevOps for Android devices

If you have landed between this Android MDM-shaped rock and a GMS hard place, you’re asking the kinds of questions we did when Esper was formed.

If you want to continuously improve and update your devices in the field, step one is a way to deploy updated software to those machines. But we all know it’s not “just” deploying: It’s doing so in a way that’s more sophisticated than an MDM’s limited console and more process-driven and replicable than a one-off script. This is where things understandably get tricky: Just how complex do you want this tool to be? What sort of workflows and deployment structure are you going to need to build for not just today, but three or four big hardware launches down the road? That’s why robust application and firmware deployment was mission one at Esper, and it continues to be the core piece that differentiates us. We’ve scaled for customers deploying software on 5 devices, and we’ve scaled for customers at 30,000.

Crucially, Esper can provide a framework to go from 5 devices to 300,000 (or more!) so that your deployment strategy and automation can grow with you. At the scale of six or seven figures, managing fleets is no longer a problem for a console to solve (even if we do love ours and think it is very pretty) — you need automation. And not only do you need that scalable framework to enable said automation, you need code to feed it, and you need that code to be tailored to your specific deployment strategy and fleet composition.

You could build all this, but if there are ready-made tools, why would you?

Believe it or not, there’s an answer to that question in the affirmative: If you don’t know you’re trying to solve a DevOps problem, you won’t be looking for a DevOps solution. Looked at from the lens of traditional Android MDM fleet management, app updates and FOTA are levers to be pulled, with the resulting fallout of failed end deployments to be managed and triaged — a sea of nails just begging for IT hammers, like said one-off script (or, worse, a human being who has to go on-site and troubleshoot a botched device update directly). Dealt with ad hoc, the problems of deployment can be mitigated and managed with MDM tools to a degree, but you’ll never truly escape the inherently crude, indiscriminate nature of those daunting deployment levers.

Once you’ve accepted that “necessary evil,” it’s easy to brush off other solutions as more trouble and cost than a migration would be worth. But if you added up all the hours you spent trying to make that square MDM peg fit into a triangular DevOps hole (and all the time your dev team could be spending on iterating software instead of software deployment), we strongly believe most orgs would see the math differently. Practicing proper Android DevOps would not only save them money, they’d create better products for their customers — and be able to do so faster and with far greater flexibility of scale.

If you’re ready to take the Android DevOps plunge, come talk to us.