The DevOps movement has changed how we integrate and publish our work. It has taken us from slow, sometimes yearly, release cycles to daily (or even hourly, in some cases) releases. We are capable of writing code and seeing our changes in production almost instantly. While that can give our customers and us a warm and fuzzy feeling, it can also provide an opening for malicious attackers.
DevOps was an amazing first step to break down walls and support fast responses to market changes and customer demands, but there is still an important wall we need to break, one important group we need to bring into the fold: security operations (SecOps).
To include this crucial group in the continuous integration and deployment (CI/CD) of changes to production, we are reimagining DevOps as DevSecOps. The challenges we face with this effort are the same ones we faced in bringing development and operations together: developers want to move fast and change often while operations wants stability and infrequent change. Security teams tend to favor stability and infrequent change, because change may mean repeating security testing and certifying the environment all over again.
When we are moving at the speed of DevOps, how can we expect these teams to redo their work every day or week?
Security in layers
Before diving into this question, we should talk about a key security practice: security in layers or defense in depth. Security in layers is the practice of applying multiple security measures, with each layer overlapping the previous and the next to create a net of security controls that work together to secure technology systems.
In a layered security approach, companies work to mitigate intrusion into their technology systems by employing access controls like WAN gateway firewalls, on-premises key-card entry, and data-at-rest encryption. The list of controls is extensive, but the point is, no single control can adequately protect the technology system. The same approach applies to performing security analysis on our applications.
Reach out to your company's application security team and ask them what scans and tools they use to ensure the applications you write are secure. Chances are, they will not reply with one tool because no single tool does it all. Instead, they will probably provide you a list of tools or types of tools they utilize or expect the development team to utilize.
That brings us back to our previous question: How can we expect to maintain a continuous deployment cycle while doing all of these scans and using all these tools? This is a daunting task; some of these scans and tools take hours, days, or longer.
Inline scanning
While some security tools and scanners do take a long time to run, there are some faster tools we can and should utilize earlier in our development lifecycle to form our first layer of DevSecOps. This is the idea behind shift-left: move processes from the end (or right) side of our development lifecycle to the beginning or middle, i.e., further left.
This first layer should include tools and scanners that take seconds (or maybe a few minutes) to run. Some common examples are code linters, unit tests, static code analyzers like SonarQube, third-party dependency vulnerability checking like OWASP Dependency Checker, and possibly a subset of integration tests.
You may be asking, "How do linting code and running unit tests fit into DevSecOps?" Bugs in software can provide the perfect opening adversaries are looking for. For example, in its past two critical web app security reports (2013 and 2017), OWASP listed code injections as the number one vulnerability. Linters, unit tests, and static code analysis can help catch some of our mistakes and possibly help prevent security vulnerabilities in our code. Remember, it is all about layers, and this is just the first.
Since these tools and scans take so little time, it's best to push them as far left in the development lifecycle as possible. When developers push code to our Git repositories and open pull requests, these tools and scanners run to make sure the code passes before it is merged. In addition to ensuring our trunk branch stays in a buildable state, having these tools so early in the development lifecycle provides feedback to developers early and often.
Pre-deployment scanning
The second layer of DevSecOps includes tools that run inline with our deployment pipeline and take minutes to maybe an hour to complete. This might include a more in-depth third-party vulnerability scan, Docker image scan, and malware scan.
One key to this layer is that the scanners and tools operate after the build artifacts are generated and before they are stored anywhere like Artifactory or Amazon Elastic Container Registry. More importantly, any failures in this layer should immediately stop the current deployment and provide feedback to the development teams.
Another key is parallelization in this—and all future—layers. Developers want their changes deployed as fast as possible, and running multiple scans serially that can take up to an hour each will slow the deployment cycle needlessly. By running these tools in parallel, the decrease in deployment speed is equal to the longest-running scan.
Post-deployment scanning
The next layer of DevSecOps comprises the tools and scanners we can and should employ after we deploy our code to a pre-production environment. These tools might include performance and integration testing and application scanners like OWASP Zap. We should strive for this layer to run quickly, hopefully in an hour or less, to provide fast feedback to developers and limit the impact on our CD process.
To ensure we do not mistakenly deploy vulnerable code to production, this layer should run as part of the CD pipeline with the goal being to delete the artifacts, destroy the environment, or roll back the environment in the event any of the scanners find vulnerabilities or otherwise fail.
Depending on industry, security, and regulatory requirements, we can automate deployment to production after this layer completes successfully. There should have already been enough automated scanning and testing in the pipeline to reasonably attest to the application's security and ruggedness.
Continuous scanning
Most of the scanners and tools we have discussed have been embedded in the CI/CD pipeline. Our goal has been to provide a reasonable assurance of the application's security while balancing the effect these tools have on our CI/CD pipeline's timeline.
The last layer of DevSecOps is continuous scanning or continuous security (CS). Just as continuous integration, testing, and deployment are synonymous with DevOps, continuous security is synonymous with and the cornerstone of DevSecOps. This layer includes tools like Nessus, Qualys, IBM App Scan, and other infrastructure, application, and network scanning tools.
Rather than being embedded in the CI/CD pipeline, CS surrounds it as an asynchronous, ongoing process and provides continual feedback to developers. How developers receive and respond to that feedback needs to be discussed and agreed upon. One possibility is to have findings filed as tickets to the development team with a severity classification. Developers would work any Sev1 as soon as it is received and tackle Sev2 and Sev3 under a longer turnaround time.
How these tools and scanners start and how often they run is another aspect of CS that the stakeholders should agree upon. Tools that have an API could be kicked off through the CI/CD pipeline after the deployment is complete. Others may need to be done as needed or based on some time cadence. No matter how it is done, the important thing is that these tools and scanners do not run just once—or even just once or twice a year. Instead, these tools and scanners should run as often as possible and as often as it makes sense for the application.
Conclusion
Just as we cannot ensure the security of our technology systems through the use of one or two tools or security principles, the security of our applications cannot depend on just one or two tools or types of scanners. It takes a layered approach of applying different tools and scanners to reasonably ensure the security of our applications and the infrastructure they run on.
By separating our approach to DevSecOps into layers, we can more easily strike that balance between security, industry, and regulatory requirements and the desire to move fast and deploy often. After all, one of the benefits of being capable of deploying numerous times a day is that we do not need to wait until the next release cycle, three or six months down the road, to push a fix for that Sev1 security vulnerability.
2 Comments