Code analyzers like FindBugs, PMD, and Sonar provide a nice safety net for catching potential quality issues early, during code and build.

By default, PMD over-emphasizes style, leading to a lower signal-to-noise ratio, but FindBugs is quite good at catching real issues that slip past standard, compiler, lint, and IDE checks. Surprisingly good at times, given that there’s only so much you can learn from static analysis.  So I was skeptical that any commercial static analyzer could “take it to the next level” and provide qualitative improvements over open source tools.

That is, until I tried Coverity.

I recently got the chance to try Coverity’s Advisors hands-on, and met with two of their sales engineers to learn more. Coverity is not cheap, but its cost can often easily be justified for commercial code based on time savings and quality improvements.

I ran Coverity against three projects: two server apps with UIs and web services, and an Android app. In all three cases, I was impressed with the findings, particularly the High and Medium Impact ones like resource leaks and the various null dereference checks. These are the value add checks that you don’t get from its embedded FindBugs or from other open source tools. Of course, the UI presentations like code paths (in both Eclipse and the web interface) were also very helpful.

There were quite a few false positives, but not enough to be a problem. Many of the issues were deliberate fail-fasts and mocks in JUnit test classes, and could be marked as ignores. Some of the more interesting finds were null dereferences of results from calls to third party libraries – where I expected a method to always throw an exception on error, but in some situations it could return null instead. This demonstrates how the tool “looks” into underlying JARs to find undocumented behaviors.

Coverity also provides a broad range of security advisor and webapp-security checks, looking for vulnerabilities to SQL injection, XSS, and other exploits in the OWASP and CWE lists. Coverity can receive feeds from other complementary analyzers such as Android Lint, clang, PMD, etc., and include them on its dashboards. This is a nice feature to manage all issues from one common database.

Full runs with most checks enabled took time, often over 30 minutes, so these wouldn’t be done on every continuous integration build. If Coverity ever adds incremental analysis, that would help there.

It’s impossible for developers to catch all potential weaknesses through manual code inspections alone, so when it comes to static quality analyzers, good tools are important. Coverity is top-notch, and this evaluation demonstrated to me the value of a purchased tool.