For IT specialists on the bleeding edge, the recent All Day DevOps Event was a blissful 15 straight hours of high-quality learning. Honestly, it was probably more useful than many live all-weekend conferences I’ve attended. I won’t attempt to point out everything I learned (and am still learning, thanks to the videos that are free for everyone to check out on their own time – which I highly recommend my colleagues in this business to do). Still, I did want to put just a few observations out there about what I see happening. I’d love feedback.
Agile development is happening and DevOps is being applied in sluggish organizations
What are some of the most stringently-regulated organizations on the planet? Hospitals and health organizations. Health Insurance Portability and Accountability Act (HIPPA) compliance alone is a notoriously challenging thing for administrators and tech professionals to manage. But beyond that, there are other regulations around patient safety. After all, people’s lives actually depend on the software they’re running on diagnostic and therapeutic machines. They need to be extremely cautious. As such, I expected these kinds of organizations to be the last ones to get on board with DevOps and Agile development.
I was a bit off on my assumption, apparently. That much was clear from Mykel Alvis’ presentation, which shows how these organizations are moving forward. Alvis works with these kinds of organizations. They might be expected to introduce automation in a sparing way to avoid risk. Any time you introduce software, integrating it with countless pieces of hardware and administrative systems, you’re increasing complexity and the risk of a breakdown somewhere along the line.
Alvis flips that premise: Every single system that supports the code is automated, and therefore everything is code. Therefore all defects are defects of code, and can be tested and addressed as code defects. Address every problematic part of an organization’s infrastructure, from systems and processes to data through a paradigm of code; deal with systems only through automation. Test everything, constantly. “Infrastructure, data, everything – we treat it like code… In our space, for our purposes, data is just another artifact that we deal with… Everything is handled by code… We treat code like it’s central to our lives, because it is.”
How does that work in practice? For instance, let’s say you have a security officer who comes along and says they thinks a systems is insecure and vulnerable. Well prove it, go write tests that show the insecurities, duplicate that component in a sandbox and write all the tests you want. Now the burden of proof is on the security officer to write the test that’s going to prove it is insecure.
The aim is to deploy robust systems that are all automated, to make all operations more efficient through continuous deployment.
Promoting DevOps culture in transition
Some organizations are still catching up on the DevOps trend. Those that do will benefit from others who have prepared the ground. Ed Ruiz took up this theme from the perspective of his organization, ASPPH, representing education and research organizations across the USA. Their 10-person team recently transitioned to a process of continuous development.
As Ruiz acknowledged, the keys to success may not be new, but they are relevant. The leadership has to articulate a vision of what they’re trying to achieve (typically, efficiencies that match up to business goals). The organization should have a culture of sharing, where documentation is a living process and knowledge doesn’t get put into personnel silos or hidden desktop folders.
Moving to a continuous development model can be disruptive to personnel who are used to doing things in a certain way and have perhaps built up their own fiefdoms. If that resistance continues, it is important to nip it in the bud and find the technical experts who are more interested in the benefit to the organization than to preserving their old job description.
Finally, leadership buy-in is essential. This applies to any IT shakeup. Still, with DevOps, the idea is that the IT manager should be willing to go to the CEO or COO and trade a little political capital. In Ed’s situation, that investment paid off – at this point, they’ve stabilized the cost of continuous development processes, reduced budget allocation down from its peak and reduced their organization’s overall technology spend from before the transition started.
Getting results with DevOps
While Ruiz’ presentation offered some relatively modest metrics for success, Fannie Mae DevOps Manager Barry Snyder had an eye-opening take on what DevOps can do for a large-scale organization (Fannie Mae is the famous government-sponsored mortgage loan company in the USA). They now have 155 teams using scrum and SAFe (and growing).
For IT departments that are often used to incremental improvements over long periods of time, DevOps seems to offer a genuinely disruptive alternative. Some results seen since their transition: 90 percent reduced development time (ie. where they were previously doing 300 to 400 builds a year, they’re now at 1,200). Their release cycle went down from a range of 9 months to a year and a half down to a much faster monthly cycle.
It used to take them from two to four months to provision, develop and test servers. Now it takes minutes.
Their number of builds has gone up 10 times, from a peak of 400 per month to 4,000. Deployments have gone up by a similar value.
Snyder included a large number of other meaningful metrics of success (and I urge anyone wondering about the ROI of DevOps to go through the complete presentation linked above). But the overall picture was clear: DevOps enables large organizations to do more, with greater quality, with less resources.
The all-day DevOps event was a huge learning opportunity overall. One of the biggest lessons: DevOps has been around long enough that the jury can no longer possibly be out on the benefits for organizations that already invest in IT infrastructure. The business case for companies to make the transition is there.
It’s now up to IT managers and CTOs to make the case to the company about moving forward, and they’ll have plenty of ammunition to do so for when they’re ready to pull that trigger.
