Actionable Intelligence from Running JVMs

Your JVMs are Pretty Smart – You Should Listen to Them

People don’t watch or listen to their JVMs, which is a shame because they have a lot to teach.

Background: Developer Productivity

Censuswide recently performed a survey of developers and asked them to divide up how they spend their time in aggregate. The mean results appear below:

Columns 2, 4, and 5 are interesting for our discussion here and they tell us that the surveyed developers spend over 40% of their time on average on code maintenance, testing & quality assurance, and vulnerability management activities. This article discusses how deriving runtime intelligence from your JVMs can help you shrink columns 2 and 5 (and also take column 4 down a notch) so that you can spend more time in column 1. 🙂

What Can Your Running Production JVMs Tell You?

One very important thing that your production JVMs can tell you about is: all of the code they run. For our purposes here, what’s important about that is that you can use this to tell you about all of the code that does not run. Let’s say your development team started out as a small shop with informal coding standards and a get-it-done-right-now-before-our-funding-runs-out mentality. Many software development organizations start out like this – and then never change their practices. Fast forward 5 or 10 years and you might find yourself living in a world with:

• bloated global libraries full of platform code that need to get included with every build
• forgotten knowledge of whether older features are ever used anymore
• uncertainty about how much of that code even runs anymore

Even if this doesn’t match your circumstances, chances are that over that 10 years you’ve needed to update the version of Java you’re using (maybe more than once), or update a major framework on which your applications depend (looking at you, Spring Boot…), or completely refactored things to move to a microservices architecture, or added new features that deprecated old ones. Any or all of these things are likely to result in some code getting left for dead.

Note that we are not just discussing unreachable code. Unreachable code can’t run because there is no control flow path to it from the rest of the program. This is only a subset of the problem – the rest of the problem is made up of perfectly reachable code that could run, but never does – that is, public classes and methods that simply aren’t being used. These two issues combine to produce a bunch of code that is unused and dead – it just lies there, adding technical debt.

Unused and Dead Code: The Silent Killer of Developer Productivity

A recent evaluation that took place in a large Java software shop resulted in the following:

• Nearly 50% of the code in a set of tested applications was simply never used

• As many as 200,000 specific maintenance activities per year was taking place against this unused code
• The actionable intelligence provided could save over 25 FTEs annually, or over 55000 hours of wall clock time.

Of course, your mileage may vary but even if these numbers are cut in half, what software development organization doesn’t want to shrink column 2 in the developer productivity graphic by 27,000 hours of time every year?

There are a lot of ways to try to find unused and dead code, such as IDEs, static analysis tools, code coverage tools, and even manual inspection (Eeeek!). Some of these approaches tend to be incomplete (i.e. finding unreachable code but not unused reachable code) or provide false readings (i.e. code that would never actually run in production is exercised by testing tools). A code coverage tool such as JaCoCo is a good option in development/test environments but it tends to generate results based on specific test cycles. JaCoCo might be telling you about a bunch of tests for code that you dont actually need if that code doesn’t ever run in production. What you need to know about is code that never runs in production – that is what will enable you to speed up your pipelines and remove both unnecessary tests and unnecessary code.

Inventorying All of your Running Code

The undisputable truth about the code that actually runs in production comes from the things doing the running: your production JVMs themselves.

So, what if you did this:

1. In a production environment, when a Java program starts, perform a quick initial scan of all of the code that could run – you can do this by examining the classpath after all of the class loaders have run and including any extra jars that the program might use.
2. Record all of the code that runs, and when it last ran. Of course you’ll want to observe for a while to make sure all of the production behavior of your application has a chance to be invoked, but considering the minimal effort involved, why not just sit back, let your apps run, and use the results as a verification – or refutation – of the results you get from the tools mentioned above?
3. Perform some simple arithmetic:

(All of the Production Code that Could Run) – (All of the Production Code that Does Run) = (All of the Production Code that Never Runs)

Once you’ve identified chunks of code that may not ever run, what’s next? The first thing you could do is mark individual methods or classes as @Deprecated. That will start immediate labor savings for you, and if you’re using Java 9 or later you could add the since and forRemoval attributes as well. You then have a process in place for timing the actual removal of that code. Ultimately, the benefit here will be reclaiming software development capacity by reducing technical debt.

Detecting All of your Running Vulnerable Code

A side effect of the above is that seeing all of the code that actually runs means seeing which vulnerable code actually runs (it’s a simple matter to compare the code that has run to a curated Java CVE knowledgebase). That means you can give a huge assist to software composition analysis and other scanning tools by prioritizing vulnerability triage based on whether or not you are actually running vulnerable code:

It’s almost cliche to talk about the friction between security and development and the issues that can result but it’s a measurable effect, and increasing the efficiency of Java CVE triage will increase the efficiency of your SDLC, and shrink the size of column 5 in the developer productivity graphic.

This use case also applies to code that you didn’t write. Your outsourced or vendor-developed apps might well find themselves the targets of attack, so understanding their intrinsic attack surface based on running code allows you to have substantive, productive conversations with your vendors.

Conclusion

Many software development organizations suffer from the problem of dead code which accumulates over time as codebases grow and age. Anti-patterns such as boat anchors and kitchen sink libraries contribute to this problem. In addition, many software development teams encounter friction with security teams – their mutual goals ought to be aligned towards securing the business while maximizing developer speed and productivity. The “single source of truth” for what code you actually need – and what code actually makes you vulnerable – is your running production JVMs. They provide a treasure trove of useful information that can help you reclaim engineering capacity, reduce technical debt, and shrink your Java attack surfaces while aligning your application security and software development teams and practices.