Free/Open Source SAST Scanner Comparison (ASP.NET Core) - November 2019
Published on: 2019-11-18
Introduction
I recently did a comparison of the free and/or open source scanners because of a lack of quality comparisons between available options. I thought we'd do a similar comparison for a .NET Core scanner comparison because I couldn't find any available. So, I've decided to do a comparison of the results of the source code scanners using a slightly modified version of the website I used in the DAST comparison.
Methodology
Running scans using the more stereotypical SAST scanners (SonarQube and VisualCodeGrepper) was straightforward — I followed the instructions of the software vendors to scan my site. Three of the five "scanners" we used (FxCop/Roslyn Analyzers, Puma Scan, and Security Code Scan) aren't SAST scanners in the traditional sense — they're intended to give you feedback directly in Visual Studio instead. To make it easier for me to pull results and make comparisons, I pulled the results into a database much like I would if I were running these as a traditional scanner.
In all cases, after running the scans, I filtered out the results that weren't relevant to security, then compared the remaining findings to the list of vulnerabilities I expected (or hoped) the scanners would find. I noted any anomalies, and in one case, added a vulnerability to my list.
Vulnerabilities
The website has a wide variety of vulnerabilities, each marked with a difficulty of Very Low to Very High. A complete list is available at the end of the report. Here's a breakdown of what I mean by each difficulty:
- Very Low: This is a simple, straightforward problem that a scanner should easily find.
- Low: Most scanners should find most of these.
- Medium: A good scanner probably should be able to find this, but there is some twist to the vulnerability that makes it harder to find.
- High: I believe that it is possible for a scanner to find this, but I'll be surprised if they do.
- Very High: This is difficult or impossible for an automated scanner to exploit this. I will include it in our test in case I'm wrong.
I excluded several vulnerabilities that existed in the website because I thought that there was no possibility that the scanners would find them (and wasn't pleasantly surprised). I also listed the vulnerabilities in the place I thought were most likely the scanners would find them, not necessarily in the place that the vulnerability would need to be fixed. This became important with the SQL injection attacks in particular, where the SQL was parsed unsafely in one class but the database call occurred in another. This separation confused all but one of the scanners.
Overall Findings
As with the DAST comparison, the overall performance for each scanner was disappointing. A full 41% of the findings were missed by each scanner, and no scanner found more than 34% (39/116), and even that number was inflated by the high number of CSRF tokens my site was missing. So, it is important to keep that in mind when reading these results for two reasons:
- Results will be low for each scanner, but that may still mean that it did well in comparison to the others in the group
- If you're going to stick with free scanners, you should very strongly consider running multiple code scanners in addition to DAST/Active scans and manual penetration tests
FxCop/Roslyn Analyzers
FxCop or Roslyn Analyzers is an analyzer that is intended to work directly within Visual Studio to help you find code issues, including (but not limited to) security problems. It is not intended to be run as a separate scanner, but can be if you know how to pull the individual rules directly from the source and apply them to your code.
Overall Results
FxCop generally did well finding the SQL injection issues — finding all obvious vulnerabilities, and also was the only one of these scanners that flagged the vulnerable indirect database calls as a possible issue. (Though it did so in such a way that leads me to believe that it would have a large number of SQL injection-related false positives as well.) It also flagged all of my questionable or bad choices of encryption and hashing algorithms, though it missed all other cryptography issues.
The scanner missed all other issues, including each of the web-specific issues.
Overall, FxCop found 12 out of the 116 items, making it below average of these scanners.
Puma Scan
Puma Scan has several versions available on their website, including a free "community" version. You can also download the source code and run your own scans. The Puma source is based on the same Roslyn source as FxCop, so if you write a wrapper to run FxCop scans, you can run Puma scans with next to no additional effort.
Overall Results
Puma Scan did a good job in flagging all of my POSTs that didn't have anti-CSRF attributes applied, found all of the obvious and verifiable SQL injection vulnerabilities, flagged all of my obviously bad choices of cryptography algorithms, and flagged my use of ECB as my cipher mode.
Puma Scan missed my use of RC2 encryption (which is generally safe with a long enough key, but I didn't have that), all logic errors, and all XSS issues. There were no issues that only Puma Scan found.
Though I didn't track this because of my use of custom authentication, Puma Scan also flagged all of my methods as missing Authorization checks. This may be good or bad for you, depending on whether you want a large number of false positives in order to more easily find a true positive.
Overall, Puma Scan found 36 of the 116 issues, making it above average of these scanners.
Security Code Scan
Like FxCop and Puma Scan, Security Code Scan is a Roslyn-based Visual Studio extension that looks for vulnerabilities right in Visual Studio. Because it is a Roslyn-based scanner, if you choose to run scans outside of Visual Studio, you can reuse code that calls FxCop or Puma Scan to run Security Code Scan.
Overall Results
Like Puma Scan, Security Code Scan found all of the missing CSRF attributes and misconfigured Cipher Modes. Like FxCop, Security Code Scan flagged the RC2 encryption. Unlike either, though, Security Code Scan found the unsafe file names and flagged my lack of SSL and header-only protections on the cookies I created.
Unfortunately, Security Code Scan missed all remaining issues, including most of the SQL injection vulnerabilities, XSS vulnerabilities, and faulty business logic.
Overall, Security Code Scan found 39 of 116 issues, making it above average of these scanners.
SonarQube
Like Puma Scan, SonarQube has both a commercial version and a free "community" version that is open source. SonarQube is not a Roslyn-based scanner, so I chose to run it via the UI. Unlike the other scanners so far, though, it does offer support for languages outside of the .NET stack. I have only evaluated its ability to scan the ASP.NET Core site here.
Overall Results
SonarQube found three links on the home page that did not include an attribute called rel='noopener noreferrer', which helps prevent pages opened in a new tab from controlling the opening page. Not only was this not caught by any of the other scanners in this comparison, it was not caught by any of the DAST scanners.
SonarQube also recommended that I not use either DES or RC2, which most other scanners also recommended.
Unfortunately, SonarQube missed everything else. It is worth noting that SonarQube supports running a Roslyn-based scanner and including those findings in a SonarQube report, which would significantly increase the number of security findings in a report. If you do use SonarQube, I very strongly recommend that you do so.
Overall, SonarQube found only 7 of 116 issues, making it below average compared to the other scanners.
VisualCodeGrepper
VisualCodeGrepper is a scanner that supports multiple languages (though not as many as SonarQube), and is available as a download from SourceForge or source code on Github. For my evaluation, we chose to use the version on SourceForge for ease of use and installation.
Overall Results
VisualCodeGrepper was the only scanner that found any XSS issues at all, albeit found them in the cshtml.g.cs versions of the views created in the obj folder (so you need to compile code to get all results — you can't just pull source code and scan) and the scan found many XSS false positives. VisualCodeGrepper was also the only scanner to find the hard-coded password found on the home page and highlighted my XML usage as a possible XXE risk (it is).
Oddly, this scanner flagged my use of the ECB cipher mode, but ignored the fact that I'm using DES. My opinion is that using ECB is a much lesser security risk than using DES, so this was an unexpected finding.
Unfortunately, VisualCodeGrepper didn't find anything else. Most notably, it didn't find any of the SQL injection vulnerabilities.
Overall, VisualCodeGrepper found 24 of 116 issues, putting it about average compared to other scanners.
Summary
It's tough to call these results anything other than "disappointing". There was not a single finding that was discovered by all 5 scanners, and there were several obvious items, such as hard-coding the Key and IV for encryption or having an unvalidated redirect after logging in, should have been found and flagged by at least one of the scanners, but they weren't.
It is my experience that commercial scanners are much more effective at finding items, and I'm planning to show this next year. However, these scanners are quite expensive. If you're looking to do SAST scanning on a budget, as mentioned earlier, you should very strongly consider running multiple code scanners in addition to DAST/Active scans and manual penetration tests.
Appendix: All Results
| Page | Vulnerability | Difficulty | FxCop | Puma Scan | Security Code Scan | Sonar Qube | Visual Code Grepper |
|---|---|---|---|---|---|---|---|
| / | Password field has autocomplete turned on | Medium | |||||
| / | Information leakage - incorrect username has different message than incorrect password | Very High | |||||
| / | Page has IgnoreAntiForgeryToken in place | Low | |||||
| / | Unvalidated redirect after user logs in | Very Low | |||||
| / | Confirm Password field has autocomplete turned on | Medium | |||||
| / | Password field has autocomplete turned on | Medium | |||||
| / | Decryption service uses DES, which is not safe to use | Very Low | Y (2) | Y | Y | Y | |
| / | Decryption service uses ECB mode, which is not safe to use | Very Low | Y | Y | Y | ||
| / | Method uses an decryption service with a hard-coded IV | Low | |||||
| / | Method uses an decryption service with a hard-coded Key | Low | |||||
| / | Encryption service uses DES, which is not safe to use | Very Low | Y (2) | Y | Y | Y | |
| / | Encryption service uses ECB mode, which is not safe to use | Very Low | Y | Y | Y | ||
| / | Method uses an encryption service with a hard-coded IV | Low | |||||
| / | Method uses an encryption service with a hard-coded Key | Low | |||||
| / | "Method calls FromSql and includes user input unsafely, making the method vulnerable to SQL injection attacks" | Low | Y | Y | |||
| / | Method adds a misconfigured cookie to the response | Very Low | Y | ||||
| / | An "admin" section should be protected via authentication | High | |||||
| / | Pages do not check user role before showing information | Very High | |||||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | Page uses user-supplied data in unprotected redirections | Medium | Y | ||||
| / | Method is vulnerable to SQL Injection attacks | Low | |||||
| / | Encryption service uses ECB mode, which is not safe to use | Very Low | Y | Y | Y | ||
| / | IV is hard-coded and is not randomized | Very Low | |||||
| / | Key is hard-coded and is not randomized | Very Low | |||||
| / | RC2 method uses a ridiculously short key, which is not safe to use | Medium | Y | Y | Y | ||
| / | Method enumerates all files within a particular folder and shows them on various pages | High | |||||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | Page calculates price based on values passed in hidden fields that can be changed by hackers | Very High | |||||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | File save method uses the original file name without checking for invalid characters | Very Low | Y | ||||
| / | Page saves files from file upload without checking contents/format, etc. | High | |||||
| / | Page uploads are stored within the file root | Medium | |||||
| / | Page creates a new user with a hard-coded password | Medium | Y | ||||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | Method unsafely uses user input to create a filepath and write contents to the screen | Low | Y | ||||
| / | Page adds misconfigured cookie to the page | Low | |||||
| / | Page creates a hash with the unsafe MD5 algorithm | Very Low | Y | Y | |||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | Page is vulnerable to value shadowing | High | |||||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | The default XmlUrlResolver is used, which makes the method vulnerable to XXE attacks | Very Low | ? | ||||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | Page runs a calculation on the query string value | High | |||||
| / | A misconfigured cookie is added to the response | Medium | Y | ||||
| / | A misconfigured cookie is added to the response | Medium | Y | ||||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | Method unsafely concatenates a string that is used in a SQL query | Very Low | Y | Y | |||
| / | Method unsafely concatenates a string that is indirectly used in a SQL query | Very Low | Y | Y | |||
| / | Method unsafely uses user input in an unsafe SQL query via a string.Format | Low | Y | Y | |||
| / | Method unsafely uses user input indirectly in an unsafe SQL query via a string.Format | Low | Y | Y | |||
| / | Method unsafely adds user input in an interpolated string that is used in a SQL query | Low | Y | Y | |||
| / | Method unsafely adds user input in an interpolated string that is indirectly used in a SQL query | Low | Y | Y | |||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | Method looks in several places for a value, making it vulnerable to value shadowing | Very High | |||||
| / | User input is partially encoded and is unsafely added to an input attribute | High | |||||
| / | User input is unsafely added to an input attribute | High | |||||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | Controller method lacks anti-CSRF token | Very Low | Y | Y | |||
| / | ExecSQL does not clean queries before executing them against the database | High | Y | ||||
| / | Decryption service uses ECB mode, which is not safe to use | Very Low | Y | Y | Y | ||
| / | IV is hard-coded and is not randomized | Very Low | |||||
| / | Key is hard-coded and is not randomized | Very Low | |||||
| / | RC2 method uses a ridiculously short key, which is not safe to use | Medium | Y | Y | Y | ||
| / | HTML creation is not actually a problem here, but we'd like a good SAST scanner to flag it | High | |||||
| / | Page shows username and passwords for all users in the system | Medium | |||||
| / | Page shows credit card numbers in plain text | Medium | |||||
| / | All values on the page are editable via hidden fields by hackers | High | |||||
| / | Page shows sensitive information (credit card number) | Medium | |||||
| / | Page sends sensitive information via query string to the next page | Low | |||||
| / | Page uses Html.Raw(), which is vulnerable to XSS attacks if not handled properly | Medium | Y | ||||
| / | Page uses Html.Raw(), which is vulnerable to XSS attacks if not handled properly | Medium | Y | ||||
| / | Page includes link to beefproject.com and is missing noopener and noreferrer "rel" items | Low | Y | ||||
| / | Page includes link to docs.google.com and is missing noopener and noreferrer "rel" items | Low | Y | ||||
| / | Page includes link to norbergconsultinggroup.com and is missing noopener and noreferrer "rel" items | Low | Y | ||||
| / | Page includes text within an AngularJS controller | Very High | |||||
| / | A connection string (with password) is included in the comments behind the page | Low | |||||
| / | Page unsafely includes form data within CSS on the page | Very High | |||||
| / | Page unsafely includes query string text within script on the page | High | Y | ||||
| / | Cross-Domain Script Include | Low | |||||
| / | Page uses Html.Raw(), which is vulnerable to XSS attacks if not handled properly | Medium | Y | ||||
| / | Site uses a vulnerable version of bootstrap | Low | |||||
| / | Site uses a vulnerable version of jquery | Low | |||||
| / | Page uses Html.Span(), which is a custom method that's vulnerable to XSS attacks | Very High | |||||
| / | Page uses Html.Italic(), which is a custom method that's vulnerable to XSS attacks | Very High | |||||
| / | Page uses Html.Raw(), which is vulnerable to XSS attacks if not handled properly | Medium | Y | ||||
| / | Page uses Html.Bold(), which is a custom method that's vulnerable to XSS attacks | Very High | |||||
| / | Page uses Html.Raw(), which is vulnerable to XSS attacks if not handled properly | Medium | Y | ||||
| / | Page uses Html.Raw(), which is vulnerable to XSS attacks if not handled properly | Medium | Y | ||||
| / | Page uses Html.Raw(), which is vulnerable to XSS attacks if not handled properly | Medium | Y | ||||
| / | Page uses Html.Raw(), which is vulnerable to XSS attacks if not handled properly | Medium | Y | ||||
| / | Page uses Html.Raw(), which is vulnerable to XSS attacks if not handled properly | Medium | Y | ||||
| / | Page uses Html.Raw(), which is vulnerable to XSS attacks if not handled properly | Medium | Y | ||||
| / | Page uses Html.Raw(), which is vulnerable to XSS attacks if not handled properly | Medium | Y | ||||
| / | Page uses Html.EncodeDoubleQuotes(), which is a custom method that's vulnerable to XSS attacks | Very High | |||||
| / | Page uses Html.Raw(), which is vulnerable to XSS attacks if not handled properly | Medium | Y | ||||
| / | Page uses Html.Raw(), which is vulnerable to XSS attacks if not handled properly | Medium | Y | ||||
| / | Page adds text as HTML to an element via JQuery, which is vulnerable to XSS attacks | High | |||||
| / | Page uses Html.Raw(), which is vulnerable to XSS attacks if not handled properly | Medium | Y | ||||
| / | Page uses Html.Raw(), which is vulnerable to XSS attacks if not handled properly | Medium | Y | ||||
| / | Page uses Html.Raw(), which is vulnerable to XSS attacks if not handled properly | Medium | Y | ||||
| / | Page uses Html.EncodeSingleQuotes(), which is a custom method that's vulnerable to XSS attacks | Very High | |||||
| / | Page uses Html.Raw(), which is vulnerable to XSS attacks if not handled properly | Medium | Y | ||||
| / | Site uses a vulnerable version of AngularJS | Medium | |||||
| Startup.cs | Error page is used for both development and production | Low |
