Static Code Analysis: Accuracy, Speed, or Completeness
TL;DR / Summary at the end of the post. Full Disclosure up-front: I am employed as a Code Scanning Architect at GitHub at the time I published this article.
In Part 1 of this series I discussed the technological differences between Static Code Analysis (SCA), Binary Static Analysis (BSA), and Comprehensive Static Analysis (CSA). In these next posts I will dive further into why these forms of analysis produce wildly different results. I will likewise discuss what implementation looks like for a given technology, and what that means for your AppSec / DevSecOps program as a result.
As you might expect with Static Code Analysis (SCA) - sometimes known as Semantic Code Analysis - the benefits of performing regular expressions against a codebase (while performing shallow dataflow mapping) is that you are able to quickly produce findings. Likewise, the process of integrating SCA tools into your development process is relatively simple given how such tools operate. Beyond that, language coverage for SCA tools tends to be very broad, with a quick turnaround for coverage of new languages, libraries or frameworks.
So what are the drawbacks to SCA tools, you ask? Well, there are several. Firstly, while performing scans with SCA tools can be a relatively simple prospect - the volume of false positives coming out of such tools is generally very high. This often results in a great deal of friction between Development and Security teams, as developers are loathe to waste time on security findings that aren’t generally exploitable (such as findings generated from local sources - more on that in Part 3 of this series).
How to support this content: If you find this post useful, enjoyable, or influential (and have the coin to spare) - you can support the content I create via Patreon.️ Thank you to those who already support this blog! 😊 And now, back to that content you were enjoying! 🎉
Secondly, because of the sheer volume of findings produced, project managers and development teams push back on allocating time to remediating supposed vulnerabilities. This happens partly because of the high false positive rate mentioned above, and partly because the amount of effort required to remediate such a large volume of findings is unrealistic; It’s difficult for developers to know where to get started when a mountain of findings are presented to them. Remember, companies exist to sell a product - they do not exist to sell an absolutely secure (and probably unusable) product.
What this means for Application Security teams is that you will need to continuously invest time and effort beyond implementation in order to tune the tool to reduce developer friction. Likewise you will spend a lot of your political capital toward convincing project managers and developers to remediate findings that may-or-may-not be real. In my experience, Static Code Analysis tools require a high degree of investment beyond implementation that can lead to burn-out within your AppSec team, as well as increased friction with your development organization.
So how do Static Code Analysis (SCA) tools rank in comparison to other forms of analysis?
- Accuracy: Low [ – ]
- Speed: High [ + ]
- Completeness: Medium [ = ]
- Impl. Complexity: Low [ + ]
- Developer Friction: High [ – ]
TL;DR / Summary
While Static Code Analysis tools are easy for application security teams to setup, they tend to produce a large volume of false positives - thus creating a lot of friction between security and development teams. The ongoing investments for Static Code Analysis tools will be focused on tuning scans and reviewing findings to eliminate false positives. While the licensing costs for such tools are generally “cheap”, the ongoing investments required to successfully implement such tools (read: drive remediation of findings) can become very expensive in terms of people and time.
Stay tuned for my next Accuracy / Speed / Completeness review for Binary Static Analysis, and in the interim remember to git commit && stay classy!
Cheers,
Keith // securingdev
If you found this post useful or interesting, I invite you to support my content through Patreon 😊 and thanks once again to those who already support this content!