Zen and the art of platform engineering

Achieving and maintaining any sense of Zen requires a commitment to keeping everything in balance. Too much of any one thing inevitably creates levels of friction that, over time, become unsustainable. Within a DevOps context, the current manifestation of Zen is organizations embracing platform engineering methodologies that enable them to standardize around a common set of tools and practices, all while empowering – without burdening – their developers.

Platform engineering, at its core, is a methodology for enabling DevOps at scale without sacrificing innovation. In this digital era, as the number of applications continues to increase, there is a critical need to rethink the building and deployment of applications so that it becomes more efficient for all involved parties, developers, cybersecurity, compliance, and IT operations teams alike.

DevOps’ goal has always been to increase the drumbeat at which organizations get to continuously deliver business value through software, by getting all IT functions to smoothly collaborate. But as the adoption of DevOps grew in organizations, so did the complexity of the world they live in: from highly powerful but complex cloud infrastructure to increasingly sophisticated security attacks, and constantly evolving and cumulative compliance requirements. In many organizations, this has started to feel like an untenable situation.

Now is the time to step back and define a unified approach that strikes a delicate balance between efficiency and the freedom to innovate that so many developers prize, all while making sure organizations are able to standardize their tools and processes, as well as trust their overall security and compliance posture. As the protagonist in the novel “Zen and the Art of Motorcycle Maintenance” discovers, not every issue can be rationally addressed when people who invariably value and cherish their own insights and ideas are involved.

Platform engineering: A brief history

Leaders intuitively understand that, for better or worse, every action creates an equal and opposite reaction. As DevOps adoption grew in organizations, it became very clear that one of its most central constructs, continuous integration and continuous delivery, that is, pipelines that fully automate the lifecycle of software development and delivery, led to huge gains in development velocity: developers were able to focus on creating business value through their code, and, for any new piece of code contributed, a pipeline would promptly execute and give them quasi-instant feedback if anything wasn’t OK. Witnessing this success, an increasing number of new “missions” of other functions were forced into those pipelines, from security to compliance and a lot more, shifting left towards developers the burden to instantly react when things go wrong – without them necessarily equipped to understand the reason for the failures.

In parallel, the world developers lived in increasingly got more complex, with powerful but complex cloud native infrastructure, open-source projects, and practices constantly emerging and being adopted. This led to an overflow of cognitive load, and, ultimately, an inability for developers to truly deliver on their initial mission: coding business value! A new balance had to be redefined, one that allowed organizations to standardize their tools and practices, keep a strong security and compliance posture, leverage modern and constantly evolving cloud technologies, all while giving back velocity (and happiness!) to developers.

Platform engineering arose to address this exact problem. Platform engineering, when done right, creates a unique opportunity to improve the DevOps experience by abstracting away for developers – as much as possible – this ever-increasing complexity, allowing them to focus on their work, all while enabling them to look “under the covers” and modify and expand it as they see fit. This shifts the dynamic from a message of control, to one of empowerment, where development teams are helped and kept focused by default, but trusted to selectively go deep and make changes if they think there is value in doing so.

A matter of trust

Ultimately, successful adoption of platform engineering requires a level of trust to be firmly established. The issue is that trust, as always, is hard to win and is easily lost. IT leaders who embrace platform engineering will need to place a special emphasis on transparency. Development teams are likely to appreciate the potential of platform engineering, but psychologically they are just as likely to view any change to existing DevOps workflows with suspicion: will the pipelines that they brought to the organization later be taken hostage? Developers need to be able to adopt new tools in ways that platform engineering teams will support rather than reject. Otherwise, developers will simply find ways to write code in the shadows, regardless of what policies might be violated.

Establishing trust may require IT leaders to move slower than they would prefer, but when it comes to DevOps, the single most important asset any organization has is a culture that values software engineering best practices infused with a high degree of empathy.

Summary

Today, being able to consistently deliver high-quality software is one of the most important competitive differentiators an organization can have. The challenge is achieving that goal in the most fiscally responsible and efficient way possible. Any organization given infinite resources can build and deploy software. Very few organizations, however, enjoy that privilege, so optimizing DevOps workflows quickly becomes a high priority as the amount of code being managed only continues to grow.

Each organization will need to decide for themselves whether a shift to platform engineering will require new or additional DevOps tools and platforms to unify workflows, but one thing that is certain: Some level of reengineering of existing workflows will be required to create a more symbiotic relationship between developers and IT operations teams. Organizational change, of course, never comes easy, so the most important thing IT leaders can do is establish the goal and then over-communicate how each change moves the organization closer to achieving that goal. Don’t take anything for granted.

At the same time, IT leaders need to make sure that the rest of the management team is on board with those changes. There inevitably will be platform engineering miscues that might slow down delivery of one project or another, but there’s little doubt that platform engineering will lead to more software being delivered faster at a lower cost and, most importantly, with less stress for all concerned.