You can’t do "DevSecOps" without doing "DevOps"
The knowledge shared in this post is derived from my experience building the DevSecOps program at Thermo Fisher Scientific - a global Fortune 100 laboratory sciences company with over 130,000 employees and $39 Billion in annual revenue.
Full Disclosure up-front: I am employed as a Principal Security Specialist at GitHub at the time of publishing this blog post.
The unfortunate side effect of marketing buzzwords are that they often trick security practitioners into believing we can accomplish different outcomes by continuing to do the same things we’ve always done. This, my friends, is the definition of insanity - which is why it’s important we acknowledge that security teams can’t just do “DevSecOps” without first embracing “DevOps”. Security’s failure on this front is probably why so many companies continue using rebranded “sparkling AppSec” technologies that have failed developers for the last two decades.
And no - by embracing DevOps I don’t mean that security practitioners suddenly need to start writing Dockerfiles and Helm charts for artisinally crafted, home-grown, API-driven microservice security applications. What I’m referring to here are the words of Jez Humble (one of The DevOps Handbook authors), who once stated that DevOps is “just communication, coordination, and collaboration”.
If we are to succeed at securing companies and meeting the regulatory requirements we’re held to, security professionals must start by embracing DevOps through communicating, coordinating, and collaborating with our colleagues who write software.
Communication
Before security teams buy - or build- anything for our colleagues in development, we should be having regular conversations with the software engineers responsible for building the company’s most profitable technologies. Most security teams fail to deliver meaningful changes that provide impact at scale because we skip this step in the process. I mean seriously - if you want to know which security tools these teams will actually use, just ask them.
And before anyone makes a predictably snarky comment - you don’t have to give developers the option of “none” here. The right way to approach buying (or building) security technologies for our colleagues is to understand their work environment. What technologies do they interface with regularly? Do they enjoy that experience - or is there something out there they’d prefer to be using? Can we find a way to procure that technology and bundle security into it? We can only learn these things from ongoing discussions with software development teams.
I’m sure at least one person reading this is probably thinking: “but I can’t talk with all of my development teams. We have a thousand developers for every member of our security team”. And the truth is that you don’t need to scale that far; You only need to develop relationships with the 20% of development teams that represent 80% of the risk to the business. More on that later.
Coordination
If you’ve ever talked with development teams, you’ll probably learn of at least one technology they absolutely hate working with. It could be anything from a workflow management system, the version control software, or a ticketing system required as part of the change review process. It might even be that security solution you’re making them use. Whatever the technology is - this is where you’ll find opportunities to make their job easier while implementing security technologies.
For development teams at Thermo Fisher Scientific, this was the version control system. Without going into too much detail here (as I’ve written an entire blog post on the subject), the development teams were tasked with maintaining their own archaic system; this required the attention of at-least three developers to keep running. When the security organization offered to buy a solution that we would manage, development teams rejoiced - and their leaders seized onto an opportunity to reallocate budgets (and development talent) toward other meaningful pursuits.
This was a win-win situation for the business, and we were able to roll it out quickly at scale due to mutual agreement on pursuing this change which helped both organizations be more productive. We were only able to accomplish this through building meaningful relationships with the development organization - and then coordinating on efforts to maximize positive outcomes for everyone involved.
Collaboration
As relationships between security and development teams grow, you start finding there’s a multitude of win-win opportunities waiting to be unleashed. This especially happens when teams share things like their Objectives and Key Results (OKR’s) - allowing teams to collaborate on delivery timelines that are mutually beneficial and reduce risk to the business.
In my experience, the mutual trust between teams turned into receiving feedback from developers about technical debts that posed a risk to the business. They wanted to address these debts, but couldn’t because project managers were not giving them the time needed to do this work. By helping these developers raise awareness of the issue, they agreed to also address security-related tasks as part of paying down the technical debt. Win-win.
Simply put, there needs to be a sense of shared goals and strong partnership between development and security teams to achieve such outcomes. That is only accomplished by building relationships through communication, and then strengthening those relationships through coordination and win-win opportunities. This eventually leads to ongoing collaboration that both reduces risk to the business - while also making development teams more efficient.
How to support this content: If you find this post useful, enjoyable, or influential (and have the coin to spare) - you can support the content I create via Patreon.️ Thank you to those who already support this blog! 😊 And now, back to that content you were enjoying! 🎉
Build or Buy?
In the words of former Target CIO Mike McNamara: “Build what will provide a competitive advantage, buy the rest”.
And let’s face it: unless you’re Microsoft, Apple, Google, or Amazon - you probably don’t have the finances necessary to build and scale teams that develop technologies to make your company competitive through security.
Moreover, the reason these companies can successfully build in-house technologies that allow them to compete on security is because they can scale teams to a point of resilience against attrition. They’re also attracting some of the world’s best talent who can build these technologies because they can afford it. Unless your company is willing to invest a substantial amount of money to compete through security, you’re almost always better off buying a technology and scaling it.
Success at Scale
As I mentioned earlier - you don’t need to build relationships with the entire development organization to achieve success at scale. You only need to engage with the 20% of teams that represent 80% of the risk to the business. This is known as the Pareto Principle - and it shows up all over the place.
By now I’m sure someone is probably yelling “but my auditor(s) / regulator(s) / leadership demand that I protect 100% of the business!!!” - and you will. But first, by focusing on the 20% of the business that represents 80% of the risk, you generate the necessary outcomes which allow you to address the remaining business risks.
First, by showing you can successfully work with 20% of teams that generate 80% of the risks, you gain the trust of your leadership and development colleagues. This allows you to make further inroads into the business to uncover new risks. It also proves that you can deliver on your objectives and key results.
Secondly, the security program gains momentum as you prove to leadership that you can have an impact at scale. This earns your team a reputation for getting things done - which usually leads to further investments and more opportunities to make an impact. It’s very difficult for others to slowdown progress when you move quickly.
And finally - you gain instant credibility when you meet new leaders and teams. Through leveraging the relationships you’ve built with 20% of the development organization, you open doors to otherwise reluctant leaders and developers; Share what you’ve accomplished with these other teams. This information presents both a “what’s your excuse?” challenge, and unlocks a natural inclination toward competition between business groups.
Doing DevSecOps
Eventually you’ll find yourself deeply embedded in the business. Through communication, coordination, and collaboration you will have built a security program that meets developers where they are - all while helping the business make progress toward reducing risk and transforming how they work in ways that are more efficient and productive. This is what it means to do “DevSecOps”.
We can’t win together if we don’t work together.
As always, thanks again for stopping by to read my pseudo-random musing 😊 While I draft my next blog post, you can git checkout other (usually off-topic) content I’m reading over at Instapaper - or review my DevSecOps Essentials series for a detailed walkthrough of how my team delivered security at scale for Thermo Fisher Scientific.
Until next time, remember to git commit && stay classy!
Cheers,
Keith // securingdev
If you found this post useful or interesting, I invite you to support my content through Patreon 😊 and thanks once again to those who already support this content!