post

Fundamentals for Scaling DevOps

In my last article, I promised you I’d dive deeper into two things – Our FlowStates DevOps Reference Model, and the secret scaling sauce of DevOps.

Since that first article, there have been exciting revelations from the 2017 State of DevOps report that corroborate our approach (published on June 6th by Puppet Labs and DORA (DevOps Research Assessment), led by Dr. Nicole Forsgren, Gene Kim and Jez Humble).

For the first time, Transformational Leadership is included in the survey and found to be a significant factor in differentiating between High and Low performing teams. The report is in its 6th year of publishing, and reflects approximately 27,000 total survey responses.

Get it here – 2017 State of DevOps Report

I’ll break this up into several articles to help introduce our concepts and how we got to our model. This article will address our observations, beliefs and experiences with scaled transformations that prompted us to think more deeply about CALMS and how to tie it back to success. In following articles, I’ll discuss our model more deeply, with a heavy emphasis on culture and leadership, and tie it back to the new 2017 State of DevOps Report by Puppet and DORA.

Today, let’s talk about our observations with the CALMS model in scaled transformations and our adaptations based on experience.

The foundation of our model is well established in the DevOps community. The most important DevOps elements are represented by a CALMS mnemonic. Culture, Automation, Lean (process & interaction), Measures (Metrics) and Sharing (collaboration). Our model is derived from CALMS, but we’ve extended it.

In our view, CALMS is a great foundation, but since it is a high-level mnemonic, it doesn’t capture the complexities of transformations in larger, older organizations. First, in practice, it is IT Team focused without enough detail on business and culture, essentially the holistic systems view. Second, the cultural component (which includes leadership) is much broader than a bullet point can capture. Third, we’ve seen that dev teams naturally gravitate towards CALMS, but operations (any post-production activity) is often left behind. For this reason, we include an Operations dimension to call attention to post-production teams. I know, it’s supposed to be DevOps but we think most organizations still need a nudge here.

With that in mind, let’s examine CALMS from our perspective.

In General:

Every company is different. There are principles that everyone can follow, but different skills, size, corporate age, maturity levels, and cultures make the implementation look a little different for each company.

In a large company, we often see the equivalent of many smaller “companies” or teams which are tied to organizational boundaries. They have their own rules, subcultures, and customer / product characteristics.

Most teams / companies only focus on automation and tooling. They achieve some level of Continuous Integration (CI) and sometimes a bit of Continuous Delivery (CD). They may do some agile and talk a little about the sharing aspects of culture, but they never really unite cross-functional teams behind the effort and create common incentives.

This will work for a while, but it begins to fall apart when they have a need to scale or cross organizational silos. This is where having a strong leader who can transcend organizational barriers is crucial to success.

On Automation:

Everyone in IT starts with tooling. We are technologists after all, and we feel safe working with things we know – tools. Unfortunately, most DevOps initiative also end here. Is tooling or CICD a form of DevOps? Yes. If all you do is automate, do you generate durable services with the best ROI possible? Probably not, and definitely not if you need to do DevOps at Scale.

Technologists often get tangled up in tools. The goal of DevOps isn’t tooling, it’s results.

If a DevOps transformation is struggling, 9 times out of 10 there is a tooling argument occurring. This is just the symptom. The root cause is more likely your cultural component: turf wars, lack of leadership or lack of Systems Thinking. Systems Thinking is taking into account the entire delivery process from Business to Customer. It is Gene Kim’s “First Way” in his book “The Phoenix Project”, and is what Eli Goldratt, author of “The Goal” talks about in his Theory of Constraints.

Don’t get me wrong – tooling is critically important and you need it as the foundation of all other efforts, but it only gets you so far. You can’t be successful with your Agile, Lean or DevOps transformation without good automation, but you need more than automation to anchor and scale.

To borrow another phrase from Eli Goldratt, Automation is “necessary but not sufficient” for scaled transformations. When tooling is localized without thinking of high-level Lean Process or Cultural aspects, you end up with pockets of unconnected excellence and no executive champions. Your ‘product’ has no flow. If you don’t anchor DevOps behaviors in your culture, it will eventually regress to an ineffective state.

On Lean:

In many DevOps efforts, there is no effort to lean out existing processes during automation. I once had an executive tell me he wanted a “one day onboard” and then later a “one hour onboard”. What he meant was take an existing manual build and deploy process and wire it up in the DevOps tool of the day.

There is some small incremental value in porting your existing manual POM or Solutions file into Jenkins, but the real exponential value is if you examine the entire delivery process to eliminate steps, scripts and handoffs. You also begin to consider architecture or delivery decisions such as how to handle configuration files and parameters more effectively. You should become familiar with Value Stream Mapping for your more complex delivery processes.

The process engineering, or Lean Value Stream, is not a one hour or one day exercise. In a larger organization, it typically spans 3-4 teams, 3-4 tools and 3-4 processes. Our fastest onboard time of 4 hours was a simple .Net app with a homogeneous pattern. We averaged 2-3 weeks for an onboard, and we were working around the client team delivery schedules. Our longest was many months as we unraveled years of bad process wrapped in even worse process. An “Automation Onboard” is really a Process Re-Engineering Onboard, and often a Behavior and Culture Re-Engineering effort.

On Measures (or Metrics):

Many teams aren’t measuring anything, or if they are, it is done poorly. We often work with Build / Release Engineering teams who have no real handle on their metrics. Even if their tools tracked it, they aren’t watching the metrics or trending. When you start an initiative, the baseline is important. You can’t know how far you came if you don’t know where you started.

I tell my teams that from the beginning, build metrics into what you do. If you don’t have metrics, you can’t defend your services or ask for more funding. What gets measured gets done, and you can’t improve what you don’t measure. There will never be enough funding for everything you want to do, so good metrics let you know where to focus your efforts for quick, meaningful wins. The DevOps world is an 80/20 world – explore and exploit.

If you’re scaling services, you should also know your gearing ratios, meaning how many applications can each of your automation engineers support, or your ops teams support. If groups have the data and are honest about it, about 4:1 is common, meaning 4 apps per engineer. A high performing team ratio is 40:1 or better.

On Culture:

One thing is clear – the bigger you are, the more culture will matter for success or failure of your transformation. This translates into needing higher quality leadership as your transformation ambitions grow.

Here’s an eye-opening chart from a 2016 Continuous Delivery Survey by DZone.

 

DZone_CDSurvey

Interestingly, the point at which culture becomes the biggest barrier is Dunbar’s Number, ~150. (Robin Dunbar, Anthropologist). Here’s Wikipedia on Dunbar’s Number: “Dunbar’s number is a suggested cognitive limit to the number of people with whom one can maintain stable social relationships”

It isn’t surprising that you need talented leadership at many levels to pull off a successful transformation. Most studies show that upwards of 70% of business, digital and IT transformations fail or do not achieve their stated outcomes. From our DevOps perspective, we attribute 100% of the failed or substandard transformations we’ve seen to culture and leadership.

Think of it this way: companies of every size have roughly the same access to the same DevOps tools. There are great open source tools in almost every category, and if you want commercial products, you can easily get what you need for under $1MM. So, tools aren’t the issue. How about Lean process? Everyone has the same access to YouTube, blogs, webinars and books. The knowledge is freely available. Knowing about lean isn’t the problem. What differentiates you from your competition is your people, and by extension, your culture.

Here’s a bit of bad news. Unless you’re a very small company or the CEO, you really can’t change overall company Culture directly. Now for the good news: you can change it locally and you do this through behaviors, goal alignment and proper incentivization. You can then broaden your sphere of influence as your DevOps success grows, very much like ripples in a pond. Think of it like sphere of control and sphere of influence. You can grow it and influence the broader company culture, but it takes time, and it takes some transformational leaders willing to take the risks of going against the grain almost every day.

Here’s how it works. Get clear on the results you’re after. These should be business goals. Then decide what behaviors you need to achieve those results. Put Process (Lean) and Technology (Automation) guardrails in place to reinforce the behavior you want. Now your behaviors influence culture by delivering results against business objectives, which further anchors desired behavior in your culture.

Lead with behavior, not with tools. In our model, we include behavior in the Culture component.

With all this in mind, the DevOps idea of Culture is big and we’ve made it even bigger. Once you make it past your initial tooling automation, the exponential results come from cultural dynamics. We’ll need to do an entire article series on Culture & Leadership, but here’s our quick view. We lump “Sharing (collaboration)” into culture since it is part of a healthy DevOps environment. We also expand it with discussions of Behavior, Trust, Empathy, Ownership, Accountability, Systems Thinking, and answering “Why”. We add Leadership in various forms: Transformational, Servant, and Intent-Based.

Here’s what we’ve done to CALMS:

  • We start and end with Culture
  • We broadened how we approach culture by adding leadership, ownership and trust

David Marquet on Intent-Based Leadership

Navy Seals Jocko Willink & Leif Babin on Extreme Ownership

Stephen M. R. Covey on Trust and Culture

  • We folded Sharing & Collaboration into Culture
  • We elevate Operations as an explicit component, and not just buried somewhere in the “L” or “S” of CALMS
  • The beginning of our culture conversation includes the “Why”

Simon Sinek’s TED talk on Why

I’ll leave you with this as a teaser into our next discussion. This is our view of CALMS. Notice that Culture (Leadership) wraps everything, and we include Operations as a dimension. The outer Culture Wheel isn’t really a linear progression, but it does approximate the awareness and experience growth that emerges as individuals begin to truly operate as trusting, empathetic, results-focused teams.

Culture is the driver for high-octane DevOps and organizational results. It is also the most difficult to change.

Most teams start in the green and stay there. They give up the exponential returns available to those who fuse behaviors and teamwork into their culture. It is hard, but worth it.

The path to scale runs through culture.

 

FlowStates_DevOps_wheel

post

What is DevOps? It Depends…

People typically have the same three questions about DevOps:

  1. What is it?
  2. How do I start?
  3. How do I grow or scale?

Today let’s talk about that first question – what is DevOps?

 

I think this is the most underrated question about DevOps. No one wants to ask it because they don’t want to appear uninformed. The truth is that it depends. The answer is not simple and it depends on you, your goals, your organization, your situation and your culture. A DevOps presentation might be anything from AWS Lambda to SAFe4.0 to cultural trust surveys. If you do some thoughtful automation and you’re getting impactful business results, you’re probably doing DevOps right for your situation.

 

The DevOps community reinforces this frustrating ambiguity by purposely not defining it or providing something like a DevOps manifesto. We prefer to talk about it in terms of principles, a movement or a philosophy. I think of it as a continuum ranging from simple automation in a small team to hardcore full-stack at enterprise scale with full cultural buy-in and aligned incentives.

 

If you think you’re looking at the IT equivalent of a Special Forces team, you’re looking at DevOps. It’s like that famous phrase – “I’ll know it when I see it”. I know I’m looking at DevOps when I see a business value chain, inter-disciplinary teamwork, empathy, and ownership.

 

Here’s an example.

In a past role at a big bank, my team had automated over 800 applications from source to prod. This included individually analyzing and re-engineering every current build and deploy process to lean it out. We ran a highly responsive platform service team for our internal clients. I recall sitting down with an enterprise architect, who quickly told me “That’s not DevOps”. I thought that was a curious statement, given the huge value and time to market reduction we had created in our division.

As I thought about it, I realized his definition of DevOps was probably full-stack, meaning all infrastructure provisioned along with the applications. We had contemplated doing full-stack at the beginning of the project, but we made a deliberate choice to focus on the things we could control at the time that also had the most impact – the application stack. Like I said – it depends. Results are better than philosophy any day of the week.

Execs – don’t be afraid to ask the question. You might be surprised in the variety of responses (or lack of responses) from your different teams, consultants and vendors. In fact, if the person you’re talking to doesn’t begin their answer with “tell me more about your goals and situation”, you probably want to find another advisor.

 

Where did DevOps come from?

In case you haven’t figured it out, DevOps is a portmanteau of Development + Operations. Andrew “Clay” Shafer and Patrick Debois discussed agile infrastructure at the Agile 2008 conference. From those discussions, they created a series of DevOpsDays conferences in 2009, which launched our term DevOps. A major originating point of DevOps was a desire to use agile development concepts in the infrastructure world.

Many seasoned IT folks will tell you we’ve used DevOps concepts for years before the term was coined. My team and I were automating full application delivery back in 2005 for a hundred apps in a bank that no longer exists (wasn’t our fault…).

 

No Manifesto

DevOps is constantly evolving, and this is where the fun starts. A word of caution – if you’re the type of person who likes definitions, structure and manifestos, DevOps will feel uncomfortable. DevOps is intentionally fluid. The founding fathers of DevOps, never wanted anything like the Agile Manifesto. Here is a perfect explanation by Christopher Little (https://devops.com/no-devops-manifesto):

“If you have a manifesto for something, then all conversation in the domain will, rather quickly, go back to the manifesto for resolution instead of everybody working together. That’s not handy in an ever-fluid, often on-fire, scale-up fast, resilience-oriented, rapidly changing and evolving IT situation.”

He nailed it. Solve together, don’t just check boxes.

We do agree on one thing. Good DevOps has a core of collaboration, empathy and culture. It is about breaking down silos between teams to deliver more value to the customer more quickly. It is about people.

But does a lack of consortium structure mean we’re somewhere between chaos and cowboy? Most definitely not. It simply means we’re self-organizing, picking ideas out of any tool box that help us deliver on our unique business goals. It means we need to be more disciplined, more trusting and more collaborative. These are the really hard parts of Big League DevOps, the Secret Scaling Sauce, and I’ll expand on it in a future article.

The term DevOps is a bit limiting after only 8 years, and this is a tribute to the evolutionary nature of DevOps and our exponentially increasing technology landscape. In 2009, the focus was on helping Dev and Ops work together, and sharing best practices. Today, an accepted scenario for DevOps is to include not just Dev and Ops, but also Security, QA, Audit, Risk, Architecture and the Business throughout the delivery cycle. Let’s throw cloud, microservices, containers and serverless architecture in while we’re at it.

I don’t think the term is going away any time soon, so we’ll use it to mean the entire business value chain of delivery. We definitely take the big picture view. In Gene Kim’s awesome book, “The Phoenix Project”, he describes this as The First Way. It is Systems Thinking, where the system includes Business and Customer.

 

Let’s recap:

• There is no formal, one-size-fits-all definition of DevOps, and we don’t want one.
• The term was coined in 2009 but many of the concepts are best practices and principles that have existed for years in high-performing teams.
• DevOps is bigger than just Dev and Ops.
• DevOps is about trust, teams, collaboration and removing silos, not just tools.
• DevOps grows and adapts to new problems in culture, process and technologies.

The Final Lap – A DevOps Reference Model

“The code is more what you’d call ‘guidelines’ than actual rules.”

– Hector Barbossa, Pirates of the Caribbean.

pirate

 

In keeping with the spirit of “no manifesto”, I’m going to outline our reference model of DevOps, or perhaps our overarching guidelines. At FlowStates, we use an acronym which is widely known in the DevOps circles – CALMS. Culture, Automation, Lean, Measures, Sharing. There are other variants such as CALMSS or CAMS. This is a great way to make sure you’re considering the most important dimensions. We’ve added depth to each element based on our experiences. Remember, we view DevOps as a continuum, it is adaptive, and the usable form depends on the client.

I’ll briefly summarize our Kingsmen CALMS and dive deeper into it in the next article.

C – Culture – The most important differentiating aspect, but also the hardest to directly control. Teams can’t often move organizational culture, so we focus on changing behavior to influence culture. We expand this to include concepts like behavior, trust, organization, incentives, ownership, and leadership. Most importantly, we include the ‘Why” of what you do. You’re going to pick up a little social psychology along the way.

A – Automation – The technical foundation of your delivery process and services, also called Continuous Integration & Continuous Delivery (CICD). This is a big bucket which could include technical strategy (cloud anyone?), QA, RM, Security, Risk, and Architecture.

L – Lean – Lean Process and Interaction. It can be either in technical automated processes, soft process interaction (Change Management for example), or interactions between people and technical or soft processes. This refers to Value Stream Maps, replacing manual steps with automation, pruning legacy processes, and minimizing delays between steps

M – Measures – You can’t improve what you don’t measure. Meaningful metrics from the beginning. What is our baseline and how do we know what success will look like? How do we know if we’re helping the business? Use Objectives and Key Results (OKRs).

S – Sharing – Share and collaborate everything. Lessons learned, back each other up, Empathy, common incentives, align activity to business unit goals and objectives. Blameless post-mortems. There is no more “Us and They”. There is only “We”. (h/t David Marquet).

Daunting? Don’t let the breadth overwhelm you. DevOps is definitely a journey and not a destination. It is possible to create a roadmap that starts with simple, clear actions while you grow into a DevOps Jedi Master. We’ll discuss that in future articles. That’s probably enough for today. Feel free to ping us at FlowStates or leave comments below if you want to discuss or comment on anything we’ve presented.

post

What is “FlowStates”?

Welcome to FlowStates DevOps Consulting Services.

We’ve spent over 20 years working in large financial institutions.  Creating effective delivery services was a daunting task.   Significant regulatory constraints, heterogeneous technologies, and sprawling, outdated infrastructure are the norm.  Culture and behavior are deeply embedded.  Numerous mergers create competing cultures and sub-cultures.

Yet in the face of this, we were able to create truly effective software delivery services.

We came to realize that “Flow” epitomizes why we’ve been successful.  We created and enabled states of flow.

We break down barriers between teams, between ideas, between behaviors, between cultures, between risk and governance bodies, and between technologies.  We’ve been able to massively scale automated delivery services for hundreds of applications.

Ultimately, in each case, it can simply be expressed as “we enabled flow”.

To become leaders in their industries, our clients must learn to remove as much friction as possible from every direction.  There are multiple dimensions of “flow” to create and friction to remove.

So, what is FlowStates?  It’s being in the zone.  It is frictionless teamwork.  It is harnessing automation properly.  It is creating conditions which allow teams to continuously collaborate with a single focus – optimizing the business value chain.  Move the right ideas as quickly as possible from concept to ROI.

Here at FlowStates we can help you find and harness your delivery flow.  Together, we can optimize the way you quickly move business value from idea to customer, across the entire delivery value chain, and dominate your industry.

post

Cost of Delay

If you quantify only one thing, quantify the Cost of Delay.  – Don Reinertsen Read More