Transitioning to DevOps

In my previous article I pondered a couple of issues that make creating a DevOps team a challenging endeavor. Today I want to discuss the following question:

If you have n development teams and one operations team, how can you create m DevOps teams?

In my experience, this is the situation faced by most teams when they think about integrating operations into their agile software development practices. But before discussing this question, let's look at the pros and cons of the situation. Let's start with the pros.

Why do we want DevOps teams?

I read an apt quote by Liker & Morgan in their book on the Toyota Product Development System:

But there was one big problem with [the approach of organizing people on behalf of their function] -- functional specialists tended to bond and become more wedded to their function and profession than to the company and its products and customers. Their measure of success was how well the functional department performed and how big a budget it garnered. They believed their profession could be the savior of the company; if they ran things, the company would be successful beyond belief. As a result, no function did a particularly good job of working in coordination with the other functions.

In my experience, this translates directly to operations and software development in traditional (pre DevOps) companies. Collaboration is at the heart of every agile methodology and if you don't have real collaboration (aka "working together to achieve a goal") because two departments seek different goals (e.g. uptime vs. number of features released) you won't be able to work effectively.

Another big factor is how teams communicate with one another. Many operations and development teams communicate only through tickets. For modern software development this is not enough, because the environment in which the freshly developed software runs is constantly changing on its own(new versions of application containers like Tomcat or databases like MySQL; changes in system configuration; different performance patterns due to updated hardware; etc.) For this reason the input from operations is needed throughout the whole development process; especially during planning where such changes can be addressed proactively.

A third factor is knowledge. Having someone with a different skill-set in the planning session can bring fresh ideas and different perspectives to the table ("Switching to NoSQL is great, but we have to factor in the cost of adapting our backup- and monitoring-systems.")

Last but not least, we'd like to develop a holistic, value-oriented view on our software development efforts. We don't want to focus on tasks, but on the value generated by those tasks. And because services need to be operational in order to generate value (in contrast to software that gets shipped to the customer), this means we have to have someone in the team who cares about the operational aspects.

Why wouldn't we want DevOps teams?

First of all, isn't it better to have specialists working together in a team? They can exchange ideas, talk in their own language and together get better at their craft. When we move to DevOps we loose that, because an operations specialist now works together with software developers and not with other people from his "profession".

Another aspect is the big structural impact such a change could have in a large organization. In some organizations operations and software development are in completely different branches of the org-chart: i.e. the "lowest" common line manager of an operations team and a software development team, is the CEO of the company! If you want to create DevOps teams in such a structure, there are some tough nuts to crack.

Flexibility is also a valid concern. If you have DevOps teams with one team member per team skilled in "operations", what happens when this person is on vacation?

When talking about a way to move from the scenario posted at the beginning of the article to a structure of DevOps teams, we have to address these concerns.

Moving to DevOps

So, how do we go from n development teams and one operations team to m DevOps teams?

Let's say we have 7 development teams and an operations team comprised of 7 specialists. This is mathematically simple: we create 7 DevOps teams by moving each member of the operations team to the 7 software development teams. This will allow us to reap the benefits discussed before. But what about the concerns raised above?

A standard practice in scaled agile environments (and that's what we're dealing with when we have 7 teams), is to have communities of practice (see e.g. Succeeding with Agile by Mike Cohn) ). A Community of Practice (CoP) is a group of people of the same "profession": you can have Java CoPs, Web-UI CoPs, Testing CoPs, etc. Goals for these groups are to define and install "best practices", to solve difficult problems together and to generally foster learning. CoPs are typically self-organizing (just like the agile teams): they decide for themselves how they want to run their respective community.

To give a concrete example of how such CoP could look like, let's consider a Java CoP in our example setting. Let's say the Java community consists of ~25 developers and they all meet every other Friday for two hours. On one of these Fridays they discuss upgrading the JVM to the latest version and agree that they want to test the new version for two full sprints on one of the continuous integrations servers, before rolling it out to production. Next, they talk about aligning on the currently used version of the Apache Commons library, as someone discovered that 5 different versions are currently deployed throughout the project. It is agreed that the all continuous integrations shall be configured in a way that the build breaks if multiple versions of the same library are detected. Tom picks up this task. (Later on, he will put this on the product backlog of his team.) Last but not least, one of the developers gives a short presentation on a new library she learned about at a conference last week and invites all the teams to give it a try in one of their next sprints.

Coming back to our 7 DevOps teams, what we would have in addition to them is (at least) an operations CoP. This group is comprised of all operations specialists from the DevOps teams. They define rules and practices that transcend the individual team. They could, e.g., define a monitoring interface for all teams to use, so that the company can utilize a unified monitoring system. It's noteworthy, that these CoPs not only influence the individual agile teams, but that the teams also influence the CoPs, because every team is represented. This is very different to the traditional approach of a (system) architecture team designing something like a unified monitoring system and then "forcing" the teams to use that. The CoP approach maintains the autonomy of the agile team.

The idea of an operations CoP addresses the first concern discussed earlier: How do operations specialists collaborate if they're distributed across agile teams. It also addresses the concern of the big organizational impact, because a CoP is not an organizational unit. It does not appear in an org-chart. 

We now still have the open concern about "flexibility": what happens when one of the operations guys (or gals!) is on vacation? In the original scenario (7 software development teams and one operations team) the remaining 6 people just take on the extra load when one member of the operations team is on holiday (hopefully they had some slack before). Or they push back to some of their customers (the 7 development teams) due to their reduced manpower.

For DevOps the solution is unfortunately not that straight forward, but the concern is still addressable (IMHO). Basically there're two options: 1. a developer from the team takes over the tasks of the operations specialists with guidance from the CoP (where he or she becomes a temporary, honorary member ;-)) or 2. an operations specialist from another DevOps team takes over. The second approach is a bit more messy, as the two DevOps teams in question now have to plan together, because they effectively share a team member. In addition to being easier, the first approach has the benefit of further spreading operational skill and knowledge which increases flexibility in the long run.

Even if we could not completely address all concerns (but 2.5 out of 3 is not too bad), I think that the stated benefits of going the DevOps route and "resolving" an operations team into the (former) software development teams, outweigh by far the impediments that can surface.

Introduction to DevOps: Agile Development and Operations Hand in Hand

The concept of DevOps is on the rise: there's a conference, ThoughtWorks developed a tool and has a well-received book out on the subject and there's a Wikipedia page.

So, what is DevOps, anyway? From my perspective it's the agile idea of a cross-functional team taken to the next level.

The idea of a cross-functional team has always been to include all people that are needed to generate value (see e.g. the first Scrum paper by Schwaber & Sutherland from 1995). That means, if you need to operate your product in order to generate value from it (e.g. because your product is a web application), a cross-functional team must also include the people who deploy, operate and manage it.

This is not a a very new idea. Facebook has been operating with this mindset from the very beginning (as far as I can tell). And as an example for a big enterprise "doing it",  we have the famous 2-pizza teams from Amazon: their engineering force is comprised of truly cross-functional teams in the above sense, with each team being small enough so it can be fed with two pizzas. 

Now, there're two aspects I want to talk about. Why is this idea becoming relevant now (as opposed to five years ago) and if it's such a good idea, why isn't everybody just doing it?

Why is this becoming relevant now?

In my opinion, there are a couple of factors contributing to the fact that many companies feel the pain of not having integrated software development and operations on a team level.

The cloud

More and more applications are becoming web applications (or web services), resulting in a software development lifecycle which suddenly doesn't end anymore with shipping the software to the customer. These companies now have to start providing a service instead of a product to their customers and thereby become "operators" of their own software (more on this here).

The rise of agile

It may appear a bit paradoxical, but the rise of agile methods in software development "solved" for many teams the communication problem between product management or the "business" side and software development. Oftentimes however, an implementation of an agile method, say Scrum, is limited to these two areas. The Scrum "Development Team" includes software developers and testers, but no one from operations. The Scrum Product Owner deals with features and customers and doesn't care much about the operational side of things ("It just has to work").

When a Scrum team set up in this fashion starts to perform, it stresses the interface between development and operations, by continuously releasing new software versions. At this point, the interface becomes the new bottleneck and these teams will have to start thinking about solutions.

The increasing use of off-the-shelve components

When you develop web software, there are a number of components that you have to rely on in order to get anything done: web servers, application containers, databases, etc. Such components share an interesting property: they are not integrated into your software like libraries or frameworks. Instead they provide an environment which enables your software to run in the first place. The difference between a component like Tomcat and a framework like Spring, is that Tomcat needs a non-trivial amount of deployment, configuration and maintenance of itself. The skills needed to do this properly are not skills a typical software developer has. Take the aspect of networking as an example: not only do you need to know the basics of networking (say, the difference between broadcast and multicast), but also the current physical setup of your infrastructure (say, if there's more than one class D network).

So, as products include more and more components that need to be operated by "specialists", the question of how to integrate these specialists with the development teams becomes more and more relevant.

Why isn't everybody just doing it, then?

The idea of DevOps at its core is quite simple (see above). The execution is not. DevOps has many facets which you all need to get working in order to become successful. There're the technical aspects of automated deployment & configuration management. There's also the aspect of the "technical process", i.e. how to use version control, build tools and ticket systems effectively (see e.g. Agile ALM). And then there's the aspect of organizational and cultural integration…

One Team

It's quite easy to say "we need one cross-functional team that has all the skills and competencies necessary to deliver customer/company value". It's quite hard to implement. Just to give an idea, here's a list of things that make implementing DevOps hard from an organizational and cultural perspective:

• Merging a development and operations team simply into one creates a team that is usually too large to be effective
• If you have n development teams and one operations team, how can you create m DevOps teams?
• How to deal with (unpredictable) maintenance tasks? Do you put them on the backlog?
• How to deal with firefighting?
• How do you integrate on-call duty or 24/7 on-site presence into agile frameworks like Scrum?
• How do you deal with regulatory requirements, like restricted access to certain systems?
• On the cultural side: developers value "new stuff" while sysadmins value stability (very generally speaking).
• ...

As I don't want to end this article with a bunch of open questions or "problems", let's discuss the first one.

Merging Dev and Ops

Let's create a concrete, albeit textbook, example: You have a Scrum Development Team consisting of 7 developers. In addition you have a 5 person operations team. If you just form a DevOps team from these two, you end up with 12 people, which is too large for effective collaboration (7±2 is the magic number). That means you need to form two (DevOps!) teams from 12 people. It also means you're entering the realm of Scrum at scale, i.e. Scrum with more than one team (see e.g. Scaling Lean and Agile Development).

Feature Teams

One common idea for scaling agile development are feature teams. A feature team is end to end responsible for a set of features, which typically involves all software layers/components. (The traditional counterpart of a feature team is the component team, which is responsible for a layer or component.) 

The main benefit of feature teams is that there're fewer inter-team dependencies. If you have component teams, (nearly) every new feature that needs to be implemented needs collaboration from all teams. With feature teams you still need collaboration, but there will be a significant amount of features that can be developed by one team on it's own (to learn know more about feature vs. component teams, see e.g. Agile Software Requirements).

This of course, fits very well into the idea of DevOps. If you regard "infrastructure" as simply the lowest layer of your software, this layer should be included in every feature team. And there you are: a DevOps team!

Applying this to our example means that we have to identify (at least) two feature-areas and build the teams accordingly. Let's say you develop a software that allows small business owners to create their own webshops. One team could work on all features related to the visitors of the individual shops and the other team on features available for the shop creators. Or one team could deal with everything that is related to "social" (like Facebook, Twitter integration, etc.) while the other team works on everything else.

Ok. Now we have an idea for one piece of the puzzle. But, as we're agile, one next step is all we need... Let's get to work!