It’s a start to a new year and a new decade! It’s a time for resolutions and beginning new habits. Instead of resolutions this year, I’m setting a few goals for myself and areas to be mindful of throughout the year.
Blogging. Late in 2019, I picked up blogging after eight years. I’m going to keep the conversation going and aim to put up new posts throughout the year. Topics will include software development, engineering management, tutorials, and possibly other areas as I find new areas to explore. If there are areas of particular interest, please reach out to me via a comment on this post, LinkedIn, or email.
Mentoring. Also late in 2019, I started mentoring at Galvanize in Seattle as one of their startup mentors. I’ve engaged with two startups, helping review the cloud infrastructure and provide technical guidance as they scale. Earlier in 2019, I helped out in PuPPy with the interview mentoring as well. Sharing my experience and helping out the broader community has been very fulfilling. I’m going to continue those mentoring avenues. Also, I am looking to mentor one or two individuals outside of work.
Learning. I enjoy picking up new technologies and diving into them. I’m looking forward to working on some tutorials as blog content, and also learn some new areas in technology this year. Some areas may include data pipelines, machine learning, VR or AR, and service meshes.
Photography. Several years ago, I enjoyed various types of photography. While I don’t currently have any more professional gear, I do want to spend some more time with photography. I hope that by the later part of the year to jump into some serious photography.
Music, particularly Trance. I’ve been into electronic music for as long as I can remember. This past year I became re-energized. Expect to see a lot more music, club nights, festivals, and concerts throughout the year.
In the previous post, I focused on some common ways to organize with DevOps folks in mind. A quick recap… commonly organizations will stick with hierarchical approaches, but this creates silos. Some organizations choose to use a hybrid approach, where DevOps engineers are still centralized with the hope that engineers that are in a matrix organization will learn the DevOps ways, but this creates problems around having DevOps engineers feeling part of the software delivery team. A common approach is a full-on matrix organization, where engineers with different skill sets and reporting to different managers work together on software delivery teams, leading to well-balanced teams with a diversity of views, opinions, and approaches.
A Stepping Stone…
How are true matrix organizations and software delivery teams part of the organizational journey? The reasons may be subtle, ultimately boiling down to organizing so that autonomy, accountability, and purpose can flourish. Daniel Pink goes into depth on the subject in his acclaimed book Drive. In brief, he describes that these are the main attributes that motivate us, allowing us to be delighted in what we do and be fully engaged.
Self-contained software delivery teams are, by definition, autonomous. They include engineers with varied experience and specialties allowing these teams to operate as a full-stack team. These teams are empowered to deliver on a single purpose in an end-to-end fashion. Lastly, they are held accountable for delivering on the mission and purpose of the team.
This model has been highlighted extensively by Spotify. The essence that organization are these software delivery teams, which they define as squads. Engineers with the same specialties form chapters, and engineers with an interest in similar topics form guilds. Those concepts allow for knowledge sharing, best practices, and common solutions to be shared between squads. Ultimately, though, the squads are autonomous, accountable, and share a single purpose.
Despite Spotify growing, using this foundation of squads has allowed them to remain nimble. It’s a good example for other organizations to take away key learnings from. Specifically around empowering autonomous teams, focusing them, and holding them accountable for delivery in their area.
Going Further… Laloux Model
Is the matrix organization the “be all, end all” of organizations? No way! There is a continuity of organizational models. In the book by Frederic Laloux, Reinventing Organizations, he outlines what that continuum looks like.
According to the model, there are 5 main categories of organization: red, amber, orange, green, and teal. A red organization is one where power and fear are in control. An orange organization is one where structure, hierarchy, and process reign supreme. An amber organization is a meritocracy with innovation and accountability, typically in a hierarchical structure. Amber is where most companies are today. The last two are where the concepts of autonomy, accountability, and purpose shine. In a green organization, individuals are empowered, there is a shared purpose and values. Lastly, in a teal organization, power is shared and authority is decentralized, leading to autonomous teams that wax and wane as the needs of the organization change.
Teal organizations, in particular, reflect the principles of Agile organizations. These organizations adapt to needs. Employees in these organizations are encouraged to be authentic. These unique characteristics align well with Daniel Pink’s research as well. The whole organization becomes autonomous and self-regulating, where the purpose of the organization resonates with each person in it.
If this topic is of interest to you, I highly encourage you to research it further. I’ve linked several lengthier summaries in the references section, and ultimately I encourage you to read the book itself. The book covers many case studies of different organizations, which is tremendously useful in understanding the journey these organizations have been on and how shifting towards teal has allowed for those organizations to grow.
Achieving the DevOps Dream
The original post in this short series was around achieving the DevOps dream. With a look into matrix organizations, and what other types of organizational models exist, allowing a DevOps culture thrive is achievable. Organizations can shift from creating isolated “DevOps Teams” to integrating DevOps and infrastructure within teams and allowing for shared responsibility of development and operations. This is a step in the right direction. As the organization grows, it will come to realize that teams also do not need to remain static as well. They need to change as the focus of the organization changes. In time, the benefits of building interdisciplinary teams that change over time become the norm.
With these types of organizations comes happier employees that have clear purpose and autonomy, and are held accountable. The organization also ends up delivering customer experiences that delight and are responsive to feedback.
In conclusion… I hope that you enjoyed this series and you’re able to take away some information to add to the organizational conversations within your organizations.
In my previous post about the Dream of DevOps, I discussed my vision for how we achieve the true vision of DevOps. Namely how having an inclusive culture around software development where not only developers work in a silo, but rather include disciplines such as software testing, infrastructure, and build/deployment.
What DevOps is not…
Let me first go into what DevOps is not meant to be. It’s not meant to be a separate team. Unfortunately, I see many organizations falling into the trap of saying, “We need a DevOps team.” If the organization read The Phoenix Project or any number of other articles on the topic, they would quickly realize the fallacy of building a “DevOps team”. The whole point is that you want integrated project teams that solve customer problems.
How did we get to thinking we need “DevOps teams”? As eluded to in the prior blog post, I feel that this is because many organizations see this is a new engineering function. Since this new engineering function doesn’t fit within the context of the pre-existing functions (such as frontend or backend engineers), there’s a feeling that this needs to be a separate team, operating alongside other software teams.
Instead of treating DevOps as a new team, an organization needs to come to the understanding that this is a function of a team.
Some organizational styles
While many software engineering organizations may start off with the need for a new “DevOps team”, engineering leaders should begin to see the need to have this function be part of software delivery teams.
How does an organization begin to transform with respect to DevOps? An early way to begin making changes is creating a matrix organization. This is an organization where a software delivery team is comprised of engineers from a variety of different teams managed by multiple managers.
The matrix organization
A good example of this is a team delivering something like a web storefront. This application involves a web frontend for users to interact with, some backend for product listings, shopping cart and checkout, some infrastructure that the store will run on, some build and deployment pipelines, and perhaps even a mobile application. In this fictitious organization, let’s start with the frontend engineers having one manager, backend having another manager, mobile having yet another, and let’s say that the build/deployment and infrastructure are managed by someone else. In a true matrix organization, the project scope would be roughly identified and some seed of folks from each of these teams would begin to work together as a software delivery team. They would wholly understand the needs of the customer and work to build a solution meeting those needs across each of their areas of experience. This software delivery team would likely have a team lead, but ultimately not be managed by any of the managers of the individuals working on the team. This type of organization is shown in figure 1.
The hierarchical organization
Contrast this to a classic hierarchical organization where software delivery teams do not exist. Rather projects are defined and are worked on in silos. The backend team may be told that they need to deliver a way to list products and provide a shopping cart and checkout. The frontend team would be told that they can query the backend for a list of products. So on and so forth. In this manner, each team only gets a small glimpse of what they need to deliver without understanding the bigger picture. Designs of one team may be incompatible with the other’s leading to rework later. This type of organization is detailed in figure 2.
The hybrid organization
A third way to organize is a hybrid of the two styles above. Let’s call it the hybrid matrix organization. In this organization, it generally follows a matrix organization. The key difference is that certain functions — typically “DevOps”, infrastructure, or testing — will remain under a manager with either dotted-line reporting into a project or working with an engineer in a project team that has some interest in taking on those skills. The idea of this is to allow this specialized team to remain intact while providing input into software delivery teams and grow interested engineers on those teams to take on the work the specialized team would have done in the past. Some teams may not see a need for input from this specialized team, in which case there would not by any liaison. Additionally, there may be engineers who do not work with any software delivery team and instead focus on building some shared tooling or have some specialized role that otherwise doesn’t lend itself to working within a software delivery team. This organizational model is outlined in figure 3.
Contrasting these organizations
Some may say that how we organize doesn’t matter since software eventually gets delivered in either organization. Some software will indeed be delivered in any of these organization styles. However, what will be impacted is how teams interact.
In a hierarchical organization, teams may play the blame game when dependencies between managers aren’t delivered in a timely and high-quality manner. Requests of one team on another for features or addressing defects may be delayed as this requires injecting the requested work into a different schedule that has different delivery priorities. Requests to infrastructure or test teams often come late and these teams are generally not consulted on the software implementation.
In a matrix organization, software delivery teams are empowered to work together on delivering a set of functionality on the same schedule. Feature requests and defects are a normal course of business and prioritized according to the priorities for that software delivery team. There are no requests for infrastructure or test teams as they are fully incorporated into the fold of the software delivery team.
A hybrid organization has some benefits of a matrix organization expect when it comes to that specialized team. If there is a liaison established, the partner in the software delivery team may be keen to take on the work of that specialized team, but they may also be overwhelmed with their normal responsibilities or have been selected without having any interest in being the liaison. This poses a new problem unique to this organization type. In the case where no liaison is established upfront, but the need arises later, similar problems to that of the hierarchical organization emerge. Additionally, there become questions about what other work that a specialized team is delivering.
Of the organizational models presented here, a true matrix organization offers the ability for software delivery teams to deliver software holistically.
While this is relevant to the previous post on the Dream of DevOps, this is the first of two parts on organizing for collaboration. This part focused on three common ways to incorporate DevOps, infrastructure, and test engineers into software delivery teams. The next part will look at some of the foundations of this including diving into organizational research and Agile management practices.
A teaser… what if teams were able to self-organize as the needs of the organization evolved?
By now, most folks have heard of the DevOps movement. This isn’t meant to be a post about what DevOps is or is not, however it is meant as an opinionated piece on building inclusive teams. There’s plenty of articles out there, some phenomenal books, and lots of great talks on what DevOps is. I must define how I see DevOps, and that is as the integrated approach to development, where all stakeholders of a software project work together in a coordinated effort to iterate on delivering a project. A full-stack delivery team if you will. Most importantly, what it is not is a bunch of siloed teams interacting through infrequent narrow scopes of interaction in hopes of delivering on a project.
Early in my career, when I was a software design engineer in test, the predominant style of software deliver was one where there would be many weeks of development time, followed by a shorter amount of time for testing and stabilization. This amounted to a milestone. For large software projects like Windows was (where I started my career), there would be 3 or more of these 3-4 month milestones, followed by a stabilization milestone to iron out all the issues of integrating all the moving pieces. From the perspective of software testing, this put a lot of pressure on software test teams to run fast and find bugs in those testing milestones. By default, this created an “us versus them” culture. You’d hear things like the classic “works on my machine” or the blame game of “why didn’t you find so and so bug” or “this isn’t implemented like the specification stated” (when specifications were available). Testing integration points between teams became even harder, as these were typically different organizations with even fewer communication points. If things worked at first glance, the types of bugs that exist were ones leading to performance issues as the interaction models between features were not well understood earlier. Additionally due to this silo creation, integrating code changes between teams was relatively infrequent, perhaps a handful of times per milestone.
One of the things that I, along with many other test leads, worked on was to partner with our development teams early in the development process. This meant that software testers were seen as equals at the table during the software development process. This led to many interesting improvements. For instance, testers were now helping in designing features. Some patent ideas in teams and related organizations that I worked in were generated by testers! Testers had a unique perspective as they had seen issues come in from customers via forums, conferences, and bug reporting systems. This voice was now heard. They also helped mold features so that features become more testable. Many testers also have a knack for breaking or exploiting software. This type of collaboration between development and test teams led to software that was designed better and was overall more reliable and secure.
In effect, this type of change was an early foray into Agile. Designs weren’t created in a fell swoop, they were iterated on within this team of developers and testers.
Jump to today. Largely software testing teams have been absorbed into development teams. In teams that I’ve led recently, every developer on the team is responsible for testing. There may be a few individuals within the team who have more of a knack for software testing than others, so they may pick up the lion share of test framework and coordination. However, these folks are in the same team, either reporting directly to the same manager or matrixed in. (Organization design will be an upcoming blog post.)
With the increase in focus on delivering software more quickly, DevOps became a hot topic. Often this is seen as paired with shifting from shipping “boxed” software to software as a service. When software made this transition to being provided as a service, it was clear that how we shipped had to change. Shipping software once per year wasn’t going to cut it. The software had to continually get better. Now that software was a service, infrastructure (e.g. how the software was run and operated) came to the forefront as well.
“DevOps Teams” with DevOps Engineers were born. These were folks who are responsible for building the ability to deliver software quickly. Cloud Native Engineers and Software Reliability Engineers/Systems Development Engineers were born out of these “DevOps Teams” to run and operate the software. To those paying attention to what the DevOps movement is, this isn’t supposed to be a siloed team. However, in the recent organizational understanding since this was a new function of engineering it had to be a team. Start seeing the parallels to treating software testing as a separate silo?
With DevOps and infrastructure, being treated as a silo, many of the same side effects that existed with test teams being a silo start to become evident. Take infrastructure that modern software runs on as an example. With infrastructure being looked at as an afterthought, just as testing was, how the software is run and operated takes a hit. In some cases, even the security and performance take a hit due to the workarounds needed to be put in place to operate the software per the original design. Looking at delivering software also takes a hit, though these are sometimes harder to spot. Issues here often come at the cost of build or deployment times, for instance not relying on pre-built artifacts or pulling in too many dependencies, or worse, thinking a monolith is a microservice.
How do we get out of these problems? We apply what we’re learned as software test teams have evolved. To build truly great software, that can be built and deployed quickly, run and be operated smoothly, and is keeping up with what the audience is looking for, we need integrated teams. These are teams where there is expertise in each of these areas. Adding in DevOps and Cloud Native engineers into software development teams ensures that building and deploying quickly as well as reliable infrastructure aren’t an afterthought. These require design alongside the software that’s being built. This is the DevOps dream realized.
In coming blog posts, I’ll cover some ideas on how to integrate these ideas into existing software organizations and how to better incorporate these concepts in the case of reorganizations.
8 years to be exact! A lot has happened in those 8 years. While the journey has been fun, with various twists and turns, it’s been worthwhile. The purpose of this post isn’t to recap those 8 years. What I’d like to do is start up blogging again, with the main focus around writing about the things that I’m really interested in, reflecting back at some experiences and learnings, and thinking out loud.
I started this blog with a focus on simplicity having grown it from a nearly strict .net/Windows/WIC blog in the days where I blogged on MSDN. Simplicity is something that I still find critical, and is expressed in so many different ways. In the past two years, operational excellence of running distributed systems has been an area that I’ve been focusing quite a bit on. This will definitely be a topic that I’ll examine further; looking at how operationally excellent thinking is a form of simplicity.
While reviving this blog, I found all the drafts of posts that I had started 8 or so years ago. Many of the posts were going to be around software testing and electronics-based projects. I may still focus on some of these, but I don’t expect to revive those old drafts. There’s too many new topics and interest areas to jump into.
Here’s a list of things that I’ll jump into at some point in the blog:
Monitoring and Alarming
neo4j and Graph Databases
Building tools for internal engineering efficiency
Running engineering teams
While listing those is good, this time through I would like to keep this blog going and not stop after a few months or take 8 year breaks between writing. So, onto continuing this journey…
This year was the first time that I attended Maker Faire! It certainly did not disappoint! While Maker Faire wasn’t last weekend, it’s still pretty recent, so I figured I have to post something while it’s still relatively fresh on people’s minds. Each year prior, I always remember reading in amazement at the various posts coming out of Maker Faire. The work highlighted at the event has always amazed me. In person, the Faire is really quite something to experience. There’s a lot to see, so here’s a brief recap of Maker Faire weekend.
Flying in on Friday I was greeted with some great weather that stayed with all the attendees through Sunday. Hey even upon approach to SFO, I got to see the Golden Gate Bridge from my seat. Since I stayed right by SFO, Saturday morning I figured that I’d over to San Mateo bright and early never having been to that part of the Bay Area before. Arriving in San Mateo, I found a Starbucks very quickly and found the free parking beneath the Whole Foods. Still at this point it was before 8am, so I grabbed some coffee and breakfast and stuck around Starbucks. Never having been to Maker Faire before, I didn’t know what to expect in terms of lines. So after enjoying breakfast, I walked over to the fairgrounds around 8:45. By 9 I was standing in line waiting for the gates to open at 10. When I got there, there were only a few folks in line ahead of me, which was great. Since they had been to Maker Faires in the past, they talked about how the Faire has changed over time and gave some handy tips (like immediately getting the “get to do cool things” armband… this lets you participate in the Faire events, for example soldering in booths or entering the Tech Shop).
Let’s just say Saturday was a really long day. I took my time to explore and get my bearings of the fairgrounds. There was good WiFi and AT&T 3G coverage, so using the online map of the fairgrounds and schedule was good, but there was also a paper version. My first stop was over to the Tech Shop. Since I was one of the first people in the gates, the whole fairgrounds were pretty empty to start off with… actually great to kinds of see what’s what.
I got to see some of the famous Maker Faire cupcakes driving around as well as some hand crafted olde-style cars. At the Tech Shop, I was able to talk to some of the vendors and actually get a nameplate inscribed for my son (see the video posted below) with a real laser printer. 🙂 Very cool stuff. From there I moved to the outdoor Google booth which was staged in a few freight-liner style shipping containers.
They were just demoing off the Arduino <-> Android work they had announced a few weeks prior at Google I/O. Unfortunately I didn’t get a free ADK from them. They demoed off some Android controlled robots with embedded Arduinos and other similar projects.
Moving along, I headed into the Fiesta Hall where later in the day they had the Tesla Coil concerts on the Tesla Stage. There were a bunch of other exhibits here including demos of the POV’s on bicycle wheels, the MakerBot, talks on incorporating LEDs in clothing that happened later in the day, and Neon Sharks.
Initially I did not spend much time in this hall, but I ended up exploring more here later in the day and on Sunday. I ended up finding some cool other areas in there, such as the section for Hackerspaces as well as some art exhibits towards one side of the hall.
The main event for me surrounded the huge Expo Hall. This is like a garage tinkerer’s paradise. There’s everything from electronics to Legos to building blocks to kinesthetic art to crafty stuff all jam packed into one hall. Really great stuff. Over the two days I managed to solder together a MiniPOV and a Drawdio at the element14 booth.
SparkFun had another huge soldering station setup, but I didn’t end up doing any soldering there since the lines were generally pretty long. None-the-less, they had a vending machine for various part that they sell and ended up getting a Barometric Sensor for a project that I have in mind. Among my favorite booths here was the SeeedStudio booth which also housed Ian from Dangerous Prototypes and the B-Squares Project. They happened to also be across the way from the Make: Live stage, which was handy for some of the talks that I attended. Roaming through this hall takes a ton of time. There’s lots to see and stuff that’s easy to miss, even the 10th time through. Some of the big highlights here included a huge Lego setup as well as a building blocks section where kids built very tall towers.
The outdoor areas were jam packed as well. The South Lot had tons of outdoor sculptures, and I learned only on Sunday that there were metal working, glass blowing, and other classes that you could sign up for.
This was also home to the Whiskeydrome as well as the Life Sized Mousetrap game. The West Green was a great place to hang out on the grass and drink some beer or eat some food. The West Lot had a Homegrown village where you could learn about eating locally grown foods. There were some food vendors there as well as the Coke Zero+Mentos Demos. I have to say, other than eating lunch at Big Bubba’s (hey with a grill that big, you kinds have to try their BBQ!) I really didn’t spend much time in this section.
I spent some time in the Maker Shed, which is basically a store where you could get Arduinos, electronics projects and kits, books, T-shirts, etc. Some of the stuff you see online and wonder if it’s worth ordering, so it was actually nice to get to look at these items. Another pavilion, which was a bit underwhelming, was the Health 2.0 exhibit. Don’t get me wrong, the stuff there was cool, like doing microscopy with DSLRs and the GE Whole Body Analysis, but it was really light compared to the rest of the Faire. I hope in future years this area expands more.
All-in-all, it was a busy two days! On Saturday I didn’t leave until 7pm and Sunday I was first in line at 8:15, and left around 1:30 to head to SFO to catch my flight back to Seattle. From a logistical standpoint, showing up early is a great idea… I beat the crowds and line getting in. It actually seemed that the folks who travelled from out of the Bay Area were eager to get into the Faire. Parking at Whole Foods next to the aforementioned Starbucks was great. It’s only about a 15 minute walk, and with nice weather this was not bad. I wish I had attended more talks, for instance I missed Massimo Banzi and Adam Savage talk. 😦 So next year I definitely need to plan these into my day. I didn’t see the battleship area since the lines were long and I just didn’t plan around the show times. Also there were sections, especially outside, that I don’t feel like I spent enough time at. All things to take into account for next year!
Below you’ll find a pieced together video of all the short clips I took at the Faire, as well as a selection of photos.
There has been much talk about simplicity (here, here, here). Simplicity of design. Simplicity of use. Simplicity! What does it all mean?
Having spent some time thinking about this, I wanted to share my thoughts on the subject. My thoughts are on the general subject of simplicity and I’ll be using the phrasing of “product” and “consumer” to refer to the abstract concept of a product and target audience. Additionally there are some specific examples for illustrative purposes.
Are simplicity and complexity related? The real difference between simplicity and complexity is the way in which complexity is presented to consumer. Products that have been designed with simplicity in mind may still be complex in their functionality or topic, but not so in their usage. To illustrate these points, I point to The Elements for iPad. When taking chemistry in high school, students learn about the periodic table of elements, but it’s a pretty dry subject causing them to have to memorize types of elements, number of free electrons, and mnemonics such as BrINClHOF. However, the iPad app takes the periodic table of elements to the next level, taking a complex topic and making it simple to understand. It does so by not only presenting the standard information about each element, but showing photos of products made from each element and explaining other usages for them. Essentially they were able to bring objects from the everyday world we live in and tie it in with the elements that they are comprised of.
But don’t consumers want products with many features and if everything is boiled down won’t they move to other products? I’d like to present the argument that one of the main reasons we don’t see products embracing simplicity is because it is easier to design a product that is complex, and much harder to design a product that embodies simplicity. Examples of this can be seen in many places, perhaps the stereotypical example being that of the Apple iPhone. Both the phone and OS embody the essence of simplicity of design. The phone has one main button on it (other than the power, volume, mute), contrast this to Android phones having 4 or more buttons that encapsulate much of the same functionality of the iPhone’s single button. Another example of the growing movement towards simplicity is Amazon’s quest to answer the demand from consumers to make packaging on product less frustrating. Why is it that Amazon had to spearhead this effort in the first place? Applying my earlier argument here, it’s easier for the manufacturers to make packaging that hard for the consumer to open since it actually takes more effort to create packaging that is easily opened.
The article at PresentationZen really highlights the essence of this argument. The way they put it there is that:
Simplicity doesn’t necessarily mean removing the complex; it means removing the superfluous.
PresententationZen’s summary plays nicely along with the argument set forth here. Spend the time to create simple solutions instead of taking the easy route of just putting a product out there. It’s not easy to spend the time to remove the superfluous to come up with a design that can remain as full functioned as its complicated counterpart, but in a simpler design.