Most companies say they want “the DevOps” but they aren’t willing to do what’s necessary to get it. By now, you must have seen the reports, statistics and graphs showing all of the benefits of delivering software using DevOps-style methodologies. The data is clear. Companies that adopt DevOps have proven that changing a company IT culture can have a significant impact on business metrics including market position and profitability. It can be done. But does your company have the people to do it?
Technical Skills
The biggest change an IT organization endures in adopting a DevOps culture is automation. It’s about scale. It’s about taking existing tedious manual tasks and turning them into automated processes. The first barrier are technical skills.
Software engineers have technical skills. They can program. However in a traditional organization, not everyone in the software delivery side of the house can program. You have business analysts (BA), who are really good at taking business processes and documenting them into functional requirements and use cases for the backlog. Then there is a quality assurance (QA) team who’s responsibility is to ensure the software works according to both the functional and non-functional (performance, etc) requirements. There is also the system administrators (SA) who handle a ton of the foundational work needed, such as operating system management and building storage solutions for the applications to run on. For any company wanting the DevOps transformation, these groups will absolutely need to upgrade their technical skills.
For example, let’s take an existing QA team with little or no technical skills. Their work is very cyclical and can be very volatile. Typically for a piece of software, they have a set of processes documented (user scripts) that encompass everything that needs to be tested upon a new release of the software. When new requirements come in, they should expect their scripts to be changed to accomodate the new requirement. When a new release is made available by the development team, they go into their testing cycle and the QA team begins to execute all of the user scripts by hand and validating the results they expect. I’ve seen QA cycles take anywhere from 2 days to 2 weeks. For larger projects, it can take 2 months of back and forth to get things signed off.
The reality is that these user scripts and manual processes are largely unnecessary and can be automated. For example, the script that tells the tester to open a screen, enter some information and then validate the information is saved is completely automatable using tools like Selenium. But building automated tests require programming skills so adoption of automation tools means your QA teams need to learn how to program. There will always be a place for non-technical people to test software in the form of exploratory testing, however that form of testing can easily be offloaded to the BAs or business folks.
Similarly, the traditional system administrators will suffer from the same issues with adoption of DevOps without upgrading their skillsets. Scale is the new norm and the ratio of system administrator to machine is growing exponentially. The days of upgrading operating systems one-by-one are going the way of the dinosaur. System admins now need to think in scale and idempotency – how do I write a single set of processes that will upgrade all of my servers, in an orderly fashion and in a way that doesn’t impact the business? You can’t do that at the command line and you damn sure can’t do that clicking through wizards.
These processes can be automated. Using tools like Ansible can do things like upgrade yum packages across large sets of machines. Or better yet, you can build and maintain a set of images and just roll out fresh new machines every time (if your CICD process works correctly). Again, there will always be a place for a system administrator’s non-technical skills as humans are really good at novel problem solving such as investigating outages but everything else can, and should, be automated away. The role should fundamentally change from “maintain the datacenter” to “maintain the automation”.
Finding the Right People
The first misnomer of automation is staff reduction. People get scared when the functions that make up the core of their value to the organization start to get automated. Changing to a DevOps culture will always create some levels of job insecurity as those who are technically skilled will be valued more than those without technical skills.
It’s very important to give your existing staff the opportunity to grow as the culture changes. The culture change is slow. It takes time. Years, in fact. That time can be used to provide some space to those without technical skillsets needed to obtain them. The mantra should be similar to the “shift left” mentality, meaning that we want people to move from non-technical roles to technical roles. The goal should be to get people to automate low value tasks and use that new found bandwidth to start doing higher value tasks.
Using the QA team as an example, it’s going to take some time for them to adjust and change to the new technologies. They are not only going to have to learn how to program, but they will be terrible at it to start. The first few tests will be riddled with gaps and will need to be rewritten at least twice. They should start with the very basic tests and then slowly build their technical skills. This slow build has to be OK with the organization and its leadership.
As they automate away the tedious manual process, it frees up more time to build more complex tests. Perhaps the new class of tests can not only do a regression of the user interfaces, but maybe the QA team members become full team members of the development team and start not only building the tests, but also start fixing the bugs they find. The important thing to note is that the process feeds on itself. More automation creates more opportunities to automate more processes. While the initial efforts may be “extra work” – they have to continue to do their day job – the work needs to focus on freeing up time associated with the manual tasks and using that time to create more automation.
However, there are going to be some tough conversations as well. Some people don’t want to learn programming. Some people learn it and they simply don’t like doing it. It’s going to happen and that’s OK. For those people, it may be an opportunity for them to move to a more business focused team. For some of them, it may be time for them to move to a company that better aligns with their skillsets.
The recruiting pipeline also needs to change. Hiring is going to change. Interviews will change. They will all be more technical. The new hires will also be more expensive. Technical skills are in high demand and the people you can attract will require higher total compensation. However, it should be considered as an investment. Paying an additional $20k to a person with these more advanced technical skills should in aggregate reduce the overall costs of total operations.
Additionally, roles may change as well. What used to be a 5 person QA team and a 10 person development team may turn into a 12 person development team and a 3 person QA team. Some may characterize the full DevOps transformation resulting in no QA team at all, but rather a larger development team who is now completely responsible for all development and testing.
It Starts with Leadership
“If you always do what you’ve always done, you’ll always get what you’ve always got.”
– Henry Ford
One of the tougher discussion points is regarding leadership. DevOps and automation has been around for a while now. The first question that should be asked is whether or not the current IT leadership is willing or able to adopt DevOps. If you are currently in a company with an IT culture entrenched in manual process, slow delivery, constant outages or constant excuses of why they can’t change, it’s usually a reflection of leadership. If they wanted to adopt DevOps, why haven’t they?
Additionally, the mandate for adopting DevOps must come from senior leadership, preferably the C-level. The answer is accountability. If the team leads and directors don’t make automation a priority, then the developers, system administrators and QA teams won’t either. You need leaders who draw the line in the sand and will hold the line. “We won’t deliver software anymore without a fully automated regression test” is easy to say, but when a deadline is approaching that directly affects a C-level’s yearly bonus, it’s going to get really difficult.
Why can’t it be a grassroots effort? Because it’s always a “crabs in the bucket” issue. There is nothing more frustrating for an individual contributor on a team to put forth the effort of automation and building things the DevOps way to then have everyone around them, including their management, ignore their efforts and continue with the standard operations. Not to mention, there can be negative feedback to individual contributors who attempt it. It can be viewed as a political risk as to shine a light on the team’s inadequacies. Your teams are willing to change but after a while, if they are not rewarded for their efforts or even penalized for them, they will stop trying. And they will start to look for a new company with a better cultural fit. It’s almost like The Matrix. The ones who wake up and that have the technical skills to do so will almost always “unplug” from your organization and move on.