15 Release Train Principles for Android DevOps

DevOps is a dynamic practice that spans people and culture, technologies, and processes. DevOps isn’t dogma. It's a set of tools and rules that lays the foundation for DevOps culture. This means Android DevOps teams have the freedom to pull automation practices and working methods from complementary disciplines that benefit both the development team and the operations team.

Esper’s Android DevOps team uses Release Trains to enhance our release management process, which is a concept born within the Agile methodology — continuous integration and continuous delivery. According to Scaled Agile Framework, “the Agile Release Train (ART) is a long-lived team of Agile teams, which — develops, delivers, and operates, one or more solutions in a value stream.”

When put into practice, release trains are defined by a fixed schedule for grouping new features, bug fixes, and improvements into frequent software development releases. At Esper, our release trains depart on a weekly basis. Here’s how we put the principles of DevOps into practice at our Android DevOps startup and how it’s created an advantage.

DevOps for Device Management

15 Release Train Principles for Android Developers and Operations

All release train workflow practices unite cross-functional teams around a single set of shared principles, including a unified value stream or DevOps pipeline. The goal is to create optimization and synchronization across teams and collaborations among team members for a predictable, fast rate of improvement.

There’s no single formula for implementing a release train. The 15 principles we lay out below blend Esper’s continuous deployment approach with the core principles outlined by Scaled Agile Framework. 

The right formula for DevOps engineers to plan the release management at your organization will also likely vary. The specifics should vary depending on an organization’s growth stage, industry, size, and other variables. But, here’s what’s worked for Esper’s DevOps team.

Release Trains are Compatible with DevOps Sprints 

We still execute biweekly sprints and releases, even though we’ve now adopted weekly release trains.

Similarly, there are no changes to our production deployment schedule. We use both release trains and twice-monthly DevOps sprints since it creates unity and speed across teams.

Biweekly QA Testing

Similarly, the adoption of release trains hasn’t impacted our overarching approach to QA testing before biweekly sprints. Generally, we spend 5 focused days on QA testing during alternating weeks. But, this isn’t absolute. 

Sometimes sprint velocity and feature change availability necessitate a different approach to testing. QA testing can start earlier during some sprints.

Appoint a Release Train Engineer

We have a dedicated Release Train Engineer, who helps us consistently execute on this release management practice. Our RTE is focused on removing blockers and managing both risks and dependencies. They also work closely with product management, engineers, and leadership to create customer value streams to ensure stable software delivery.

Plan Dev Testing at the Onset of Each Sprint

Each story or bug within a train or sprint has an associated sub-task for developer testing. The subtasks must include planned developer testing scenarios to be part of a biweekly sprint or a candidate to board a weekly release train. 

Designate a Dedicated Environment for Developer Testing

Developer testing transparency is a core part of Esper’s approach to successful bi-weekly release trains. In addition to planning developer testing scenarios, we plan a designated environment for developer testing activities.

Without a designated environment and approach to testing, there’s a loss of transparency – especially when you’re coordinating a team of teams that specialize in hardware, apps, ops, and more. Poor transparency is a blocker to moving work across silos or teams which slows down the value stream. 

Dedicated environments need to create transparency across the DevOps organization to avoid blockers and delays. This supports the DevOps mission of continuous development, continuous improvement, and continuous testing.

Set Criteria to Board a Train

Esper’s DevOps team has identified four distinct criteria for a feature, bug, or improvement to board a weekly release train.

• All changes across applications involved in a feature or change must be complete from a code standpoint.
• Integration and unit testing also must be complete, with results available to the Release Train Engineer.
• A pull request to the QA testing environment also must be ready, with detailed notes.
• And finally, the story or bug and related tasks must be listed in “resolved” status in Jira and assigned to the QA team.

Set a Deadline for Pull Requests to Develop

We opted to create a deadline for merging pull requests to the QA tester environment, which is among the last stages a feature or bug has to pass before it can board a weekly train. At Esper, our deadline for pull requests is the close of business on Fridays.

Avoid Including Late Pull Requests

Once we hit Friday's close of business, there are officially no more pull requests to develop allowed to board the train the following week – with the exception of bug fixes for planned changes. Near-misses are inevitable, but making too many exceptions to late pull requests can delay the entire train from delivering a value stream on schedule.

Address Testing Blockers

Any issues that block testing in the testing phase need to be addressed immediately. The release manager should work with product managers, engineers, and others to keep the train in forward motion. Daily Bug-Review meetings with QA and other teams help in this regard. Balancing frequent releases with quality results means careful testing and making sure that developer testing doesn’t fall behind schedule.

Isolate Release Items That Cause Downtime

If a change causes more than one day of downtime, the entire change-set needs to be rolled back for continued testing in the developer environment. In other words, a release item that’s causing significant downtime will miss the current train unless it can board again without derailing the schedule.

Downtime resulting from failed bug fixes is relatively rare, but it’s something we account for in our approach to release trains. Sometimes, bug fixes have to disembark the weekly train unless the downtime issue is resolved in 24 hours or fewer.

QA Testing Schedule

Considering the 5-day QA testing schedule explained previously, Esper’s team opts for a 3 day-2 day split. We spend 3 days testing in the QA  environment and 2 days testing in our pre-production environment.

Pre-Production Environment Testing

QA leverages a dedicated environment that mirrors production for the final round of testing. This is essential to avoid any unwelcome surprises after or during the production deployment. This testing phase ensures that critical functionality is intact and the new code works well in a production-like environment.

Backwards Compatibility Always Matters

Much like developer testing, QA testing must include backward compatibility scenarios as well to ensure continuous feedback loops across individuals and teams. All individuals who contribute to the value stream are responsible for working with the RTE and Product Leads to remove blockers from the weekly release train.

Accommodate Minor Release

Finally, we’ve used the release train concept to accommodate the occasional Minor Release in the development process. These are mini-trains that are limited to bug fixes or quick, high-priority customer requests. The release schedule still happens on the same day of the week as a full release train, but the difference lies in our approach to high-quality testing.

Release & Deployment Checklist

We’ve established a detailed checklist and readiness process that’s executed throughout the lifecycle of a release and deployment to production. In addition to quickly identifying potential risks that might derail a release train, this process also helps us identify and drive improvements in a continuous manner.

Advantages of Release Trains for Android DevOps

Ultimately, the release train concept allows Esper’s Engineering team to maintain a rapid pace of innovation without too much risk of failure. Also, it creates a clear separation between development issues and release concerns.

Quality isn’t an optional part of the release train model. Release quality is tied to customer value in this release management model and a healthy Android DevOps practice. It’s okay to spend longer testing larger features before they’re allowed to board the release train. In fact, that’s much better for the Engineering team and customers when the alternative is slowing down an entire train of other features and fixes to accommodate a laggard.  Finally, the release train approach benefits Esper’s Android DevOps customers beyond value streams since it’s a commitment to consistency. Our customers can rely on a predictable schedule for releases and sprints each month.

Free eBook: Applying DevOps to Device Management