10 October, 2011

Introduction to DevOps: Agile Development and Operations Hand in Hand

The concept of DevOps is on the rise: there's a conference, ThoughtWorks developed a tool and has a well-received book out on the subject and there's a Wikipedia page.

So, what is DevOps, anyway? From my perspective it's the agile idea of a cross-functional team taken to the next level.
The idea of a cross-functional team has always been to include all people that are needed to generate value (see e.g. the first Scrum paper by Schwaber & Sutherland from 1995). That means, if you need to operate your product in order to generate value from it (e.g. because your product is a web application), a cross-functional team must also include the people who deploy, operate and manage it.


This is not a a very new idea. Facebook has been operating with this mindset from the very beginning (as far as I can tell). And as an example for a big enterprise "doing it",  we have the famous 2-pizza teams from Amazon: their engineering force is comprised of truly cross-functional teams in the above sense, with each team being small enough so it can be fed with two pizzas. 

Now, there're two aspects I want to talk about. Why is this idea becoming relevant now (as opposed to five years ago) and if it's such a good idea, why isn't everybody just doing it?

Why is this becoming relevant now?
In my opinion, there are a couple of factors contributing to the fact that many companies feel the pain of not having integrated software development and operations on a team level.

The cloud
More and more applications are becoming web applications (or web services), resulting in a software development lifecycle which suddenly doesn't end anymore with shipping the software to the customer. These companies now have to start providing a service instead of a product to their customers and thereby become "operators" of their own software (more on this here).

The rise of agile
It may appear a bit paradoxical, but the rise of agile methods in software development "solved" for many teams the communication problem between product management or the "business" side and software development. Oftentimes however, an implementation of an agile method, say Scrum, is limited to these two areas. The Scrum "Development Team" includes software developers and testers, but no one from operations. The Scrum Product Owner deals with features and customers and doesn't care much about the operational side of things ("It just has to work").
When a Scrum team set up in this fashion starts to perform, it stresses the interface between development and operations, by continuously releasing new software versions. At this point, the interface becomes the new bottleneck and these teams will have to start thinking about solutions.

The increasing use of off-the-shelve components
When you develop web software, there are a number of components that you have to rely on in order to get anything done: web servers, application containers, databases, etc. Such components share an interesting property: they are not integrated into your software like libraries or frameworks. Instead they provide an environment which enables your software to run in the first place. The difference between a component like Tomcat and a framework like Spring, is that Tomcat needs a non-trivial amount of deployment, configuration and maintenance of itself. The skills needed to do this properly are not skills a typical software developer has. Take the aspect of networking as an example: not only do you need to know the basics of networking (say, the difference between broadcast and multicast), but also the current physical setup of your infrastructure (say, if there's more than one class D network).
So, as products include more and more components that need to be operated by "specialists", the question of how to integrate these specialists with the development teams becomes more and more relevant.

Why isn't everybody just doing it, then?
The idea of DevOps at its core is quite simple (see above). The execution is not. DevOps has many facets which you all need to get working in order to become successful. There're the technical aspects of automated deployment & configuration management. There's also the aspect of the "technical process", i.e. how to use version control, build tools and ticket systems effectively (see e.g. Agile ALM). And then there's the aspect of organizational and cultural integration…

One Team
It's quite easy to say "we need one cross-functional team that has all the skills and competencies necessary to deliver customer/company value". It's quite hard to implement. Just to give an idea, here's a list of things that make implementing DevOps hard from an organizational and cultural perspective:
  • Merging a development and operations team simply into one creates a team that is usually too large to be effective
  • If you have n development teams and one operations team, how can you create m DevOps teams?
  • How to deal with (unpredictable) maintenance tasks? Do you put them on the backlog?
  • How to deal with firefighting?
  • How do you integrate on-call duty or 24/7 on-site presence into agile frameworks like Scrum?
  • How do you deal with regulatory requirements, like restricted access to certain systems?
  • On the cultural side: developers value "new stuff" while sysadmins value stability (very generally speaking).
  • ...

As I don't want to end this article with a bunch of open questions or "problems", let's discuss the first one.

Merging Dev and Ops
Let's create a concrete, albeit textbook, example: You have a Scrum Development Team consisting of 7 developers. In addition you have a 5 person operations team. If you just form a DevOps team from these two, you end up with 12 people, which is too large for effective collaboration (7±2 is the magic number). That means you need to form two (DevOps!) teams from 12 people. It also means you're entering the realm of Scrum at scale, i.e. Scrum with more than one team (see e.g. Scaling Lean and Agile Development).

Feature Teams
One common idea for scaling agile development are feature teams. A feature team is end to end responsible for a set of features, which typically involves all software layers/components. (The traditional counterpart of a feature team is the component team, which is responsible for a layer or component.) 
The main benefit of feature teams is that there're fewer inter-team dependencies. If you have component teams, (nearly) every new feature that needs to be implemented needs collaboration from all teams. With feature teams you still need collaboration, but there will be a significant amount of features that can be developed by one team on it's own (to learn know more about feature vs. component teams, see e.g. Agile Software Requirements).
This of course, fits very well into the idea of DevOps. If you regard "infrastructure" as simply the lowest layer of your software, this layer should be included in every feature team. And there you are: a DevOps team!
Applying this to our example means that we have to identify (at least) two feature-areas and build the teams accordingly. Let's say you develop a software that allows small business owners to create their own webshops. One team could work on all features related to the visitors of the individual shops and the other team on features available for the shop creators. Or one team could deal with everything that is related to "social" (like Facebook, Twitter integration, etc.) while the other team works on everything else.

Ok. Now we have an idea for one piece of the puzzle. But, as we're agile, one next step is all we need... Let's get to work!

4 comments:

  1. Good writing.
    I think one important factor why devops becomes envougein the Java Community is that Java EE requires closer collaboration between developers and operators. This is true 'cause in Java EE 6 the majority of api is provided by the container. In programming models outside Java EE the framework experts reside in the development team (say Spring users that deploy the stuff into the webinf/lib folder). In an Java EE model however the operators gain more responsibility 'cause its the JEE container that ships with the framework integrated. Also, migrating business apps to newer server versions requires close collaboration between development and operations ...
    Cheers, Niklas

    ReplyDelete
  2. Thx, Niklas. That's an interesting observation. I must confess I haven't' looked at JEE for a while, because I was mainly involved in JSE projects (plain Servlets, Spring, ...). I will take a closer look now. Thx, again!

    ReplyDelete
  3. On passing the buck

    The cultural gap between developers and operators you mentioned forms a major obstacle to build cross-functional teams. Because of the self-contained nature of the two groups there is a very unfortunate tradition of passing the buck: Developers just want their code executed, even testing is an impertinence. What is the problem of just running their code?

    On the other hand, ops teams often blame developers to act unthoughtful on resources and overall infrastructure. They think of developers as smaller brothers with a free pass and of themselves as the older, more serious ones that are left holding the bag.

    To overcome this gap is the crucial challenge while forming a devops team. Unfortunately this fails if you try to implement formal procedures such as Scrum with all its religious overhead instead of implementing the right spirit itself.

    ReplyDelete
  4. Hi, nice article, I really like the idea of closer collaboration between dev and ops.

    I wrote about something similar recently, in relation to support work and the agile framework:
    http://paddycorry.wordpress.com/2011/10/22/agile-the-swat-paradox/

    ReplyDelete