Nikhil Punathil

February 04, 2021

In today’s rapidly changing technology landscape, operating system updates have become an essential part of keeping your smart devices secure and up to date. Regular updates to both the feature set as well as the security implementation are how a modern operating system can stay relevant. For an open source operating system such as Android, regular security patch updates are crucial to ensure devices are patched against an ever growing list of Common Vulnerabilities and Exposures (CVEs). Fortunately for us, Android by design has made this process as easy as possible. Ever since the first public release of Android, it has included the ability to download and install OS updates Over-the-Air (OTA), without requiring a physical connection to a computer, and the system has only gotten more robust since. 

In this article, we will go over some of the common reasons why you should consider including a Firmware-Over-the-Air (FOTA, used interchangeably with OTA) update system in your dedicated device deployment plan, how FOTA works on Android and how you can use the Esper platform to simplify this process. 

WTF? (Why the FOTA?) 

There are many reasons you may want to keep your dedicated Android devices up to date.

Security: Every day more and more vulnerabilities are being discovered, both in Android as well as the underlying Linux kernel, that a potential bad actor could utilize to compromise your devices. Every month, Google releases a list of these vulnerabilities that Android is affected by through their Android Security Bulletins. Google recommends that all OEMs update their devices every month with the relevant security patches, though most consumer OEMs tend to prefer quarterly updates. For dedicated Android devices on a network edge, it is essential that the firmware is kept up to date with regular security updates. It is also recommended that the Linux kernel used in the device is kept up to date with the upstream branches, to ensure maximum security coverage. In most cases, it is not feasible to manually install updates to devices, especially in large fleets, which is why a FOTA update system can be incredibly helpful.

Feature Updates: While this may be relevant mostly for consumer devices, it is still an important reason why you may want to consider a FOTA system. Improvements to Android’s feature set can come in two ways: through app updates and through firmware updates. Usually, features that involve hardware require changes to the Android kernel and/or the system. Some features, such as adding a toggle for the navbar, while seemingly software-only at first glance, require changes to the Android framework, by virtue of Android’s architecture and API implementation. A FOTA system can help streamline this process by not requiring an onsite technician visit every time a minor feature is added. 

Android Version Upgrades: The mantra “if it ain’t broke, don’t fix it” is not quite relevant to Android. Most dedicated devices may not benefit from the consumer focused feature upgrades every new version of Android offers. However, even if your devices are running fine on an older version of Android, it is important to remember that Google only supports a particular version of Android with security patches for four years. If you intend on using your devices for a relatively long time, and you want them to stay secure, you will need to plan for at least one major Android upgrade. Android version upgrades involve major changes to many parts of the system firmware, making the process more complex than just a regular update. An automated update over the air can simplify this process significantly. 

How FOTA Updates Work on Android

Android has included support for FOTA updates since at least Android 1.6 Donut. While the initial implementation was somewhat crude, the design has since been updated significantly to the point where today, updates are seamless and barely noticeable. Before we dive into the different kinds of update architectures on Android, it is important to understand the basics of how an Android device’s OS files are structured. Android consists of some essential partitions that house specific files:

/bootloader: The bootloader is a small program that is the first to load when a device is turned on. Its role is to verify the integrity of various partitions and load the kernel onto the RAM. The code for the bootloader resides in the /bootloader partition. 

/boot: The /boot partition consists of the Android Linux kernel, as well as the ramdisk, which is a set of scripts that determine the order by which processes are started by the system. 

/data: This is where all the user data, such as installed apps, app data, media (such as photos, downloads, etc.) resides. This partition is wiped when the device is factory reset. 

/recovery: The recovery is a mini operating system whose role is to act as an interface to update other partitions while they are inactive. In traditional devices, FOTA updates are applied by the recovery.

/system: The system partition contains most of the Android system files, such as system apps, media (including the bootanimation, ringtones, etc.), low level C libraries, the Android Run Time (ART) libraries and the Android framework. 

/vendor: The vendor partition is similar to the /system partition, and it houses device specific system files, such as userspace drivers (extensions from the kernel drivers) and HALs. The vendor partition was separated from the system with Project Treble.

In addition to these, there may be other partitions as well, depending on the OEM’s device architecture. A FOTA update may update one or all of these partitions. With that in mind, there are two kinds of OTA implementations on Android:

A/B (Seamless) Updates

A/B updates are arguably the best option for OTA updates. In order to completely streamline the update process, the device has two copies of all essential partitions, labelled _a and _b (ex: system_a and system_b). Each set of partitions is referred to as a slot. At any given time, a single slot is considered active. When an OTA update is deployed to the device, an OTA update manager app on the device downloads the update file and installs the update to the inactive slot. The next time the device is rebooted, it reboots into the updated inactive slot, thus completing the update seamlessly. 

There are advantages and disadvantages to this design: 

  • Updates are quick and seamless with minimal downtime.
  • In case the update is buggy/broken or if the update failed, you have the ability to boot into the older, working slot and avoid the device being offline. 
  • The system files would use twice as much space as they need to, since each partition has a redundant counterpart, resulting in a slightly higher BOM cost. 
  • The lack of a recovery partition may make it challenging to manually install OTA packages.

Traditional (non-A/B or A-only) Updates

The outdated, yet more common traditional system update is less impressive when compared to A/B updates. However, if you believe in simplicity or have storage constraints, this may yet be a good option to offer basic FOTA support on your devices. Devices set up for traditional updates have only one set of partitions, unlike A/B devices. The firmware also includes a recovery, a small program/OS that is required to install updates to the main system partition (since it can’t be updated while it’s active). There are yet again, advantages and disadvantages to this implementation: 

  • Uses half the space for the firmware as a similar A/B device, reducing BOM cost compared to A/B.
  • The recovery allows for easier access for engineers to manually update devices.
  • Updates are not seamless, and the device may be offline for 15-30 mins, depending on the update size. 
  • Since there are no redundant partitions, a broken update would require a manual rollback. 

Cloud Infrastructure

Most OEMs of consumer Android devices provide a FOTA service to push out updates. In that case you are dependent on the device maker for these updates and have limited control. Android does provide some limited capabilities for when available firmware updates can be installed on target devices (typically a 30 day delay is the best you can get). 

OEMs are also prone to deprioritizing updates, particularly major version upgrades, since the focus is on selling their latest offerings, which may be fine for consumer devices, but is a major challenge in enterprise scenarios. This is best evidenced by the current Android version distribution chart, where only 8.2% of devices are running last year’s Android 10 release and less than 0.1%, the minimum bar to make it to this chart, are running the 6 month old Android 11 release. If you are dealing with a custom Android device, it can be much more involved.

In that case, in addition to preparing your Android device for OTA updates, you would also need to set up cloud infrastructure that would host your OTA update packages and deploy them to your devices. The process becomes more complicated when there are additional requirements, such as the ability to selectively rollout the update to a group of staging devices, before it is rolled out to the entire fleet.

How Esper Can Help Set-up and Manage Your FOTA Updates

Evidently the process to set up FOTA for a dedicated device is much harder than it would seem for consumer Android devices. However, for customers using Esper Enhanced Android (EEA) through the Esper platform, you can abstract away all the technical jargon and simply utilize our user-friendly cloud interface to deploy FOTA updates to any and all EEA devices in your fleets on your schedule

We are extending our recently added Pipelines feature to include FOTA so you can set up multiple staging groups of devices to verify and validate an OTA update before it is rolled out completely. You will also be able to set specific times when updates are to be installed, to minimize the impact of a potential downtime. 

Esper handles all the cloud infrastructure setup required so that your update rollouts are worry-free. Devices running Esper Enhanced Android can support both traditional and A/B updates (depending on the device architecture) out of the box, which means no work needs to be done on the device end either. EEA FOTA is planned to roll out in the first half of 2021 as an additional service available for our customers to purchase.

Esper Can Also Help With FOTA for Custom Android Devices

It’s also possible to apply Esper’s FOTA infrastructure to non-EEA custom Android devices. We’d be happy to discuss your particular implementation and see if Esper can provide the FOTA infrastructure, just reach out to us and we’ll see what we can do.

Thanks to Google’s Android Security Bulletin, regular security updates for a supported device are straightforward to implement. However, Android version upgrades via FOTA are significantly more challenging, primarily because they are dependent heavily on the hardware and the OEM/ODM. 

With every new Android version, there are many under the hood changes and improvements, which require extensive work within the lower levels of the device’s source code. These changes are very hard to implement without the help of the device maker and their internal device documentation. However, there are exceptions to this rule, such as the Android for x86 ecosystem, which relies on the x86 device makers’ support for Linux, Android’s “parent”, for the drivers and device binaries. Evidently, version updates are specific to each use case and require due diligence and investigation to figure out . 

Conclusion

Hopefully this article was able to teach you something new about FOTA updates on Android. Keeping a device up to date can be the difference between a security incident that could compromise your entire company, or being a secure company that blackhat hackers hate. 

We’ve just scratched the surface of FOTA, and we plan to dig into additional areas around FOTA including:

  • The typical process that updates are created and distributed by different categories of Android devices (for example consumer devices versus custom devices).
  • How FOTA differs for carrier specific consumer devices compared to open carrier devices.
  • Ensuring FOTA is discussed and contractually agreed to when setting up purchase of a custom device from a device manufacturer.
  • How to achieve long-term support on Android for dedicated device deployments, and recognize situations where it will not be possible to achieve.

If you would like to learn more about how Esper can help with your specific FOTA deployment needs, contact us at sales@esper.io and we will be happy to get in touch with you.