General, Software Engineering

The dream of DevOps

By now, most folks have heard of the DevOps movement. This isn’t meant to be a post about what DevOps is or is not, however it is meant as an opinionated piece on building inclusive teams. There’s plenty of articles out there, some phenomenal books, and lots of great talks on what DevOps is. I must define how I see DevOps, and that is as the integrated approach to development, where all stakeholders of a software project work together in a coordinated effort to iterate on delivering a project. A full-stack delivery team if you will. Most importantly, what it is not is a bunch of siloed teams interacting through infrequent narrow scopes of interaction in hopes of delivering on a project.

Early in my career, when I was a software design engineer in test, the predominant style of software deliver was one where there would be many weeks of development time, followed by a shorter amount of time for testing and stabilization. This amounted to a milestone. For large software projects like Windows was (where I started my career), there would be 3 or more of these 3-4 month milestones, followed by a stabilization milestone to iron out all the issues of integrating all the moving pieces. From the perspective of software testing, this put a lot of pressure on software test teams to run fast and find bugs in those testing milestones. By default, this created an “us versus them” culture. You’d hear things like the classic “works on my machine” or the blame game of “why didn’t you find so and so bug” or “this isn’t implemented like the specification stated” (when specifications were available). Testing integration points between teams became even harder, as these were typically different organizations with even fewer communication points. If things worked at first glance, the types of bugs that exist were ones leading to performance issues as the interaction models between features were not well understood earlier. Additionally due to this silo creation, integrating code changes between teams was relatively infrequent, perhaps a handful of times per milestone.

One of the things that I, along with many other test leads, worked on was to partner with our development teams early in the development process. This meant that software testers were seen as equals at the table during the software development process. This led to many interesting improvements. For instance, testers were now helping in designing features. Some patent ideas in teams and related organizations that I worked in were generated by testers! Testers had a unique perspective as they had seen issues come in from customers via forums, conferences, and bug reporting systems. This voice was now heard. They also helped mold features so that features become more testable. Many testers also have a knack for breaking or exploiting software. This type of collaboration between development and test teams led to software that was designed better and was overall more reliable and secure.

In effect, this type of change was an early foray into Agile. Designs weren’t created in a fell swoop, they were iterated on within this team of developers and testers.

Jump to today. Largely software testing teams have been absorbed into development teams. In teams that I’ve led recently, every developer on the team is responsible for testing. There may be a few individuals within the team who have more of a knack for software testing than others, so they may pick up the lion share of test framework and coordination. However, these folks are in the same team, either reporting directly to the same manager or matrixed in. (Organization design will be an upcoming blog post.)

With the increase in focus on delivering software more quickly, DevOps became a hot topic. Often this is seen as paired with shifting from shipping “boxed” software to software as a service. When software made this transition to being provided as a service, it was clear that how we shipped had to change. Shipping software once per year wasn’t going to cut it. The software had to continually get better. Now that software was a service, infrastructure (e.g. how the software was run and operated) came to the forefront as well.

“DevOps Teams” with DevOps Engineers were born. These were folks who are responsible for building the ability to deliver software quickly. Cloud Native Engineers and Software Reliability Engineers/Systems Development Engineers were born out of these “DevOps Teams” to run and operate the software. To those paying attention to what the DevOps movement is, this isn’t supposed to be a siloed team. However, in the recent organizational understanding since this was a new function of engineering it had to be a team. Start seeing the parallels to treating software testing as a separate silo?

With DevOps and infrastructure, being treated as a silo, many of the same side effects that existed with test teams being a silo start to become evident. Take infrastructure that modern software runs on as an example. With infrastructure being looked at as an afterthought, just as testing was, how the software is run and operated takes a hit. In some cases, even the security and performance take a hit due to the workarounds needed to be put in place to operate the software per the original design. Looking at delivering software also takes a hit, though these are sometimes harder to spot. Issues here often come at the cost of build or deployment times, for instance not relying on pre-built artifacts or pulling in too many dependencies, or worse, thinking a monolith is a microservice.

How do we get out of these problems? We apply what we’re learned as software test teams have evolved. To build truly great software, that can be built and deployed quickly, run and be operated smoothly, and is keeping up with what the audience is looking for, we need integrated teams. These are teams where there is expertise in each of these areas. Adding in DevOps and Cloud Native engineers into software development teams ensures that building and deploying quickly as well as reliable infrastructure aren’t an afterthought. These require design alongside the software that’s being built. This is the DevOps dream realized.

In coming blog posts, I’ll cover some ideas on how to integrate these ideas into existing software organizations and how to better incorporate these concepts in the case of reorganizations.

Additional Resources


It’s been a while…

8 years to be exact! A lot has happened in those 8 years. While the journey has been fun, with various twists and turns, it’s been worthwhile. The purpose of this post isn’t to recap those 8 years. What I’d like to do is start up blogging again, with the main focus around writing about the things that I’m really interested in, reflecting back at some experiences and learnings, and thinking out loud.

I started this blog with a focus on simplicity having grown it from a nearly strict .net/Windows/WIC blog in the days where I blogged on MSDN. Simplicity is something that I still find critical, and is expressed in so many different ways. In the past two years, operational excellence of running distributed systems has been an area that I’ve been focusing quite a bit on. This will definitely be a topic that I’ll examine further; looking at how operationally excellent thinking is a form of simplicity.

While reviving this blog, I found all the drafts of posts that I had started 8 or so years ago. Many of the posts were going to be around software testing and electronics-based projects. I may still focus on some of these, but I don’t expect to revive those old drafts. There’s too many new topics and interest areas to jump into.

Here’s a list of things that I’ll jump into at some point in the blog:

  • Python projects
  • Operational Excellence
  • Monitoring and Alarming
  • neo4j and Graph Databases
  • Building tools for internal engineering efficiency
  • Running engineering teams

While listing those is good, this time through I would like to keep this blog going and not stop after a few months or take 8 year breaks between writing. So, onto continuing this journey…