DevOps — scale up quickly, effectively and efficiently and avoid pitfalls
You’ve set up your DevOps teams and they’re already adding value to the developing, building, deploying and running of innovative applications in the cloud.
Now what? How can you develop this model to empower other development teams across the organisation without losing control of them? How do you coordinate while giving them space to innovate?
The opportunities to become more agile and productive are substantial — but only if you manage your teams and systems properly to ensure you’re exploiting the full potential of the DevOps approach.
Scaling up DevOps — the most common pitfalls:
1. Waiting too late to re-evaluate your DevOps approach
Ironically this can occur when DevOps become too successful. Teams that are delivering well are often given additional projects to manage and because of this growth in workload they can end up being divided into new separate scrum teams or hiring new engineers, both Developers and DevOps. But often the increase in headcount isn’t sufficient to match the extra work that is coming into the team. This is especially true when you have DevOps engineers within teams. The natural reaction is for teams to work harder and faster. Frequently though, the result here is greater stress and lower productivity.
Instead, DevOps engineers and managers need to empower development teams, so they can manage more of their own DevOps tasks.
Instead of trying to take on more work and thereby becoming the bottle neck in the production line, DevOps engineers need to work with their development teams to ensure that they are able to maintain their DevOps function self-sufficiently. To do this they need to provide tools and guidance to avoid becoming a dependency on the teams themselves.
2. Mismanaging silos
Traditionally when scaling DevOps companies have chosen to create a new silo to work alongside development teams. This approach of a centralised DevOps team does offer some advantages. For example, it makes it possible to balance levels of experience across the workforce by mixing senior and junior DevOps in a single team with constant dialogue between them. Other benefits include economies of scale and greater consistency for technical solutions across teams. This approach also brings with it the longstanding problems associated with Operations teams. It creates a division between DevOps teams and Development teams — the obstacle that DevOps is intended to remove.
The other option is to embed a DevOps engineer within every development team. This means teams don’t have to compete with other teams for support from a shared DevOps resource. The danger here though is that you soon lose consistency and coordination between teams. In addition to this, DevOps are also harder to recruit than developers.
Every development team gets their own resource
This might sound sensible but in reality, it’s very unlikely that you’ll be able to find a consistently good DevOps team for every project. It’s especially difficult to create teams that share the same ideas about best practice and implementation methods.
Although companies have avoided the problems created by silos with this method, they often experience a lack of coordination and integration. The danger is that teams go rogue, with teams beginning to adopt different approaches. A lack of consistency and quality control will quickly develop.
How to avoid these and other pitfalls:
1. Handling DevOps silos
Rather than fall into a silo culture, the solution is to mix teams of varying experience and authority and making that knowledge freely available by establishing a DevOps tooling team. This team should be a cross functional team which focuses on the creation of tools for activities such as build, deployment and monitoring to enable development teams to self-service their DevOps requirements. Essentially you create a team whose primary role is to do the heavy lifting for development teams. It should champion and promote DevOps to other teams. However, it must not become a silo itself by actually doing the work for them or it will find itself overwhelmed and ending up creating a new bottle neck.
Rather than demanding development teams adhere to their rules, this DevOps tooling team should seek to convince them of the merits of the DevOps structures and methods with constant dialogue.
Think “Guardrails” not “Gates.” In other words, allow your development teams to do what they need to do using guardrails to keep them going in the right direction and on track, rather than slowing them down with gates.
2. The Spotify Squad framework
This system of squads, tribes and guilds is familiar to many as a method of scaling Agile. Creating, for example, a guild where people from across the whole company can share ideas and knowledge in a specific area is a great way to scale up DevOps. This approach is particularly useful where DevOps is considered to be less of a dedicated role and more of a group function for all teams.
Establishing guilds or chapters is a good way to build teams and to share knowledge and best practice.
Initially it’s often a good idea to assemble a guild to solve a particular problem. Once the idea has been established, guilds might offer advice and support on issues such as web technology, test automation or agile coaching. They can allow engineers to pass on information which they have but which is outside their own scope of operations. Each guild often has a co-coordinator.
3. Working effectively beyond the pilot phase
Teams will often be granted an exception when they’re trying out DevOps or Agile for the first time, so they don’t have to tick all the usual change management boxes. But then if problems arise and there is resistance they have to return to more normal controls and a highly structured methodology such as Information Technology Infrastructure Library (ITIL).
There need not be a conflict here — ITIL can be adapted to fit with the working methods of DevOps and Agile. Rather than arguing the case for more control and permissions in order to deploy faster and increase automation it’s often better to examine the governance being used. Developer and DevOps teams need to make the effort to meet the governance requirements that Ops teams currently provide. It’s also important for Ops teams to accept that there must be a change in the process.
4. Light touch change management
Change management can slow and discourage continuous deployment. The trick here is to demonstrate that changes can go into production seamlessly with minimal chances of them causing problems. You also need to reassure Ops teams that you can easily revert to the previous state. This involves accurately tracking versions, changes and automate every step of the deployment process.
5. Centralising monitoring is fine but put the development team in charge of their own metrics
It’s important that teams take ownership of their applications from development through to support. They need to be aware that if something goes wrong with their application they will be the ones to get the call for help in the middle of the night. This principle can then be complemented with more generalised tools that provide automatic and continuous discovery, code-level diagnostics and real-time analytics. Work with centralised monitoring teams or create your own monitoring, but make sure your metrics reflect the needs of your application.
6. Make security integral from the start
Before anything goes live, in addition to using your own internal security teams, bring in a third party for an extra layer of cover.
Security is not an add on — it should be front of mind for every engineer in the tech stack.
The goal of every engineer should be to ensure that these security reports are as short as possible. And every point raised should be assessed and action taken to resolve it.
Key points:
- Move early — anticipate your scaling needs in advance and prepare for them.
- Make scale a culture not a role — establish a tooling team that can empower others by providing them with the right tools rather than doing the work for them.
- Guardrails not gates — allow your development teams to do what they need to do using guardrails to keep them going in the right direction rather than slowing them down with gates.
- Work with your Ops teams not against them — in order to scale ensure that your Ops teams are fully onboard and are seen as colleagues and not competitors.
