ShadowOps is Not Just Bad DevOps

September 5, 2016

Some alarmists or buzzword-feeders like to say “DevOps is dead.” But this statement either betrays a grossly inaccurate understanding, or it’s a deliberate mischaracterization of the nature of DevOps (i.e., what it is, what its limitations are, and what problems may surround it).

The goal of this post is to provide an accurate understanding of DevOps and to explain how this can help to prevent the emergence of ShadowOps in your organization.

Automation and Alignment of Incentives Across Business Units

While many blogs speak to how or why “DevOps” was invented, “DevOps”, or “DevSecOps”, or “SecDevOps”, or “BizDevOps”, or “DevBizOps”, or “BizDevSecOps” — or any of the other hybrid, mashed up variations — are just terms that are intended to describe the automation and alignment of incentives across business units.

The “DevOps[Foo]” worlds can probably be best described as “post-waterfall” software development environments where departments are not neatly lined up from inception to delivery, where each stakeholder blocks or adds requirements to push to the next stage of development, thereby “requiring” complex feedback “models” to describe consensus-building activities between groups that do not directly interact on the directional graph of product development.

“Lean”, “process review”, “kanban”, “bimodal IT”, and all of these software development and operations process discussions — as well as the placement of “DevOps” in the constellation of software processes — are relative to the terms we use and how we discuss them. For all the talk of anti-patterns, the problem often comes down to a lack of basic training, heavy responsibility siloing, and inflexible processes. And when talking about this, the people inside doing the work can come up with infinite metaphors, breakdowns, and perspectives to describe the unique changes each organization would need in order to move from where they are to a place where incentives are better aligned across the organization’s business units.

So to clarify… “DevOps” and its derivatives are terms that are used to describe the alignment of the incentives between disparate groups in an organization. “DevOps” speaks to the alignment of incentives of developers and operations. This usually results in improvements in development and delivery processes. “DevSecOps” describes the alignment of development, security, and operations. This usually produces improvements that increase delivery velocities while making the process more secure — and hopefully makes security processes easier to implement and execute. “BizDevOps” is similar for business, development, and operations units to align their incentives. All of these scenarios assume that incentives are most closely aligned with the delivery goals of the organization. Including security in the picture would mean that development, delivery, and operations want to have all the improvements described above, but would also want to be secure about them, and not simply be experimental.

Systems Thinking vs. Perverse Incentives

To permanently move past these issues, I propose the organizational adoption of two mature conceptual tools. These are foundational organizational perspectives, so implementing them would not necessarily be easy, but they would cause a paradigm shift allowing for a complex, but slower and longer-term positive change. These conceptual tools are:

Systems Thinking (nurture this at all costs): Systems thinking is a conceptual orientation of considering systems as a whole and focusing on the interaction of the parts of the system and how the people, actors, structures, processes, and outcomes of those systems interact to give rise to higher order behaviors which cannot be defined by any single component of the system. (See https://en.wikipedia.org/wiki/Systems_thinking)

Perverse Incentives (avoid/minimize/obliterate at all costs): A perverse incentive is an incentive that has an unintended and undesirable result which is contrary to the interests of the incentive makers, not dissimilar to a “cobra effect.” (See https://en.wikipedia.org/wiki/Perverse_incentive)

I have simply defined these terms, but they are complex topics, and you should consider the Wikipedia articles I’ve referenced as jumping off points for a more complete understanding. I highly suggest reading the Wikipedia pages and many of the original sources that are cited.

The Emergence of ShadowOps

So… since you have now taken at least a moment to form these concepts in your mind…

Given the fluid nature of what “DevOps” or team collaboration means to different organizations….

The collision of perverse incentives and DevOps gives rise to…

ShadowOps… which I define as the systems created in response to perverse organizational incentives in order to solve a problem in an individual business unit, but which actually lead to increased risk and decreased overall alignment within the organization. This WILL dramatically affect both the long-term stability of development pipelines and the operational activities of traditional or cloud operations professionals such as disaster recovery, compliance, bug triage/repair, security, planning, and service availability.

ShadowOps is an emergent property (read: symptom), and can become a pattern, or chronic condition of a system consisting of misaligned human behaviors, often referred to as a perverse result in social science.

Because of its nature, ShadowOps is not discovered until numerous disparate organizational patterns and behaviors become regularly executed and their systemic outcomes begin to interact. Often due to an absence of systems thinking regarding the interaction of groups in increasingly complex organizations, individuals and groups in organizations will make decisions in order to achieve short-term solutions, and often these will be made in isolation or in ignorance of the greater system that their team and decisions affect.

Here’s an example for your consideration. An Operations team and their Development team are not aligned on incentives. A developer is tasked with releasing a new system by next Wednesday to meet a sales requirement or roadmap goal. If the developer completes the task on time, they get a cookie from their manager. If they do not complete the task on time, they are fired (to be dramatic). The operations person is incentivized to keep the production systems running with 5-9s availability and to maintain compliance in the delivery systems. When the developer asks the operations person to open a new port so a service can communicate with a new third-party service, the operations person, concerned with compliance may say “No, we do not connect to any third-party services. It says so in our compliance manual.” The operations professional has done their job and receives their cookie. At this point the developer has two choices, but only one where they will be rewarded with a cookie.This could lead the developer to engage in ShadowOps by launching the system on a node outside of the operations-managed systems, so they can meet their own deadline. In situations like this it is also very common for developers to simply launch or integrate a third-party service, leading to potential IP or financial issues.

This happens because the operations and developers I describe above do not share the same goals about progressing their company. They are only responsible for executing a small part of their organization’s overall tasking. This is the result of perverse, or at least divergent, incentives.

In addition to increased risk, the perverse incentives that cause ShadowOps to emerge are what hackers directly exploit for the vast majority of their most successful patterns and methods, specifically social engineering. When perverse incentives are created in any system, these imbalances can easily create vulnerable targets for potential hackers to install malware, acquire credentials, exfiltrate data, or socially engineer throughout a hacker-targeted system. For example, implementing a strong password security policy with a very high rate of rotation (say once a week or month), may theoretically lead to better security, but in actuality, it can lead to insecure password practices. The security team has now won their cookie for the day by implementing the password policy. But if the new policy is not accompanied with funding for tools (e.g., a high quality password manager for end users) and time for training, this could lead to the leaking of said new secure passwords. Specifically, in the case of passwords that rotate often, people who do not use secure password managers will write these passwords down on paper, if they do not also receive the training that explains how this would completely defeat the purpose of rotating passwords frequently. Those passwords may be inherently secure, but credentials could easily be determined by someone who simply walks by a visible password note.

Eliminating ShadowOps: Achieving Alignment Across Business Units

Here are a couple of practical solutions specifically oriented towards eliminating ShadowOps and achieving alignment across business units:

Provide consistent tooling so developers and operations are working from the same tool set, which will prevent either team from working past each other to accomplish separate goals. Provide the same set of tooling, testing, and deployment available for your developers, to your security team, and get their security deliveries in the same pipeline. If you run your production systems on Ubuntu on AWS, developers, security teams, product management, and operations should all be using the same tools to launch an instance to test, develop, secure, or scale. This removes a large number of barriers to communication by providing a common tool set to enact change on a platform. This will get developers past the problem of “But it works on my laptop!”, and security teams will no longer simply be throwing changes over a wall to operations, since they can now contribute DIRECTLY to the code used to configure systems.
Emphasize cross-team communication when solving issues, especially security issues to ensure or at least mitigate the creation of perverse or misaligned incentives.
Automate security processes the same way you automate operations processes. In many ways, security is simply a specialization of operations that has achieved a divergent status through a lack of tooling in the past and the siloing of operations tooling in more “traditional” organizations. A very simple example of how to automate security processes is by having operations automate the installation of security or “unattended” updates to mitigate out-of-date software packages instead of simply producing compliance reports on a regular basis. If security teams can easily implement and test a fix to a system, it will reduce the overall workload of communicating that work to a new team to complete giving you simple cost reduction.

A Final Word . . .

Applying systems thinking is a powerful way to directly address inter-unit complexities in organizations of any size. Only by thinking through the interactions and outcomes of your organization and working to explicitly align your teams, will you be able to reduce or eliminate problems such as ShadowOps and the outcomes of perverse incentives. If you treat DevOps as a way to improve cross-unit communication, it should be very easy to achieve many improvements in efficiency, security, and the bottom line.

Business & Finance Articles on Business 2 Community