Fixing the Billion-Dollar Mistake

There are mistakes that are just costly. Then there is the null reference. A language feature that was added in 1965 “simply because it was so easy to implement”. Since then, null references have caused countless bugs, system crashes and security problems – and cost billions of dollars in maintenance and troubleshooting. It has been eliminated in many new programming languages and this article will show the practical ways to eliminate this language mistake even in Java.

Null references: A historic mistake with dramatic consequences

Sir Tony Hoare is best known for developing quicksort, the Hoare logic (a set of logical rules for reasoning rigorously about the correctness of computer programs) and formulated the dining philosophers problem along with Edsger Dijkstra. I apologise if my summary of his immense impact on software engineering is too truncated. He is also a Turing Award winner. He received the award for his “fundamental contributions to the definition and design of programming languages”.

One of the programming languages he worked on in 1965 was ALGOL W and in particular its comprehensive type system for references in an object-oriented language. This language did check every reference at compile-time to make sure every pointer points to something that exists and is of the appropriate type. This eliminated a whole set of errors that cannot occur.

Case closed, null pointer completely avoided. If not for the fact that Tony worked for a company that sold ALGOL computers and most potential customers did not have ALGOL but Fortran programs. To run these programs on the ALGOL machines, a transpiler was written from Fortran to ALGOL. After about a year of work the transpiler was done but no customer wanted to use it. The transpiled Fortran programs all produced subscript errors caused by pointers that pointed to null and/or had the wrong type. Great, you have found a whole set of errors that could occur.

The customers wanted their programs to run though. They did not want to fix all the problems first. And Tony’s company wanted to sell computers. So, he added the null reference. It was not the only solution to the problem. Even back then Sir Tony Hoare knew of another solution with disjoint unions. But null was easier to implement. Now you have to either check every reference at runtime or risk disaster. But at least you could run Fortran on the ALGOL machines.

The problem was that ALGOL inspired many other programming languages such as C or Simula. Simula was billed as ALGOL 60 with classes and considered the first object-oriented programming language. It also inspired Bjarne Stroustrup, creator of C++, and James Gosling, creator of Java. This all leads to Sir Tony Hoare apologising in 2009 for inventing the billion-dollar mistake, humorously I might add.

The actual cost

The null reference is a problem prevalent in most modern programming languages. Perhaps the only problem more widespread is the Year 2k Bug which did cost an estimated $300 billion. In relation the null pointer reference with its countless bugs, system crashes and security problems might actually have cost billions of dollars in maintenance and troubleshooting. It is after all a problem that has been around for 60 years.

That the null pointer is widespread is supported by data from Harness (formerly OverOps) which in 2020 crunched anonymized stats from over a thousand Java applications monitored by them. They found that 97% of logged errors are caused by only 10 unique errors. The number one error being the NullPointerException. It took the top spot in 70% of their production environments.

Language-agnostic is the CWE Top 25 Most Dangerous Software Weaknesses List. The list demonstrates the currently most common and impactful software weaknesses. This list changes year by year as the list reflects the 30.000 reported Common Vulnerabilities and Exposures (CVE) of that year. The NULL Pointer Dereference has been in spot 21 (2024), 12 (2023), 11 (2022), 15 (2021) and 13 (2020).

That list however “just” looks at security issues. How much time is spent on maintenance and troubleshooting for all the bugs and system crashes is not included.

Next, we will dig into why the maintenance and troubleshooting is so expensive. While reading the next chapters, keep in mind that developers have been fixing these problems for the last 60 years. In total, the mistake might actually have cost around a billion over the years.

The error is not where you found it

Take a look at the following code. Can you guess where the error is?

public String findNotebookMaker(EmployeeId id) {   
    var employee = company.getById(id);
    if (employee == null)
        return "Employee does not exist";
    /* ... */
    return employee.notebook().maker();
}

The problem is the following:

    return employee.notebook().maker();
                   ^^^^^^^^^^^
    // java.lang.NullPointerException: 
    // Cannot invoke "Notebook.maker()" because the return value of "Employee.notebook()" is null

Which is impossible.

An employee always has a notebook! We designed it that way!

Bewildered developers

The thing is, under certain conditions (f.ex. notebook is being repaired), an employee does not have a notebook. The bewildered developers only knew the state before this feature was added. To them an employee always had a notebook. Or maybe they just forgot implementing it. In the end it does not matter. The exception flew anyway, and it is not the problem. The actual problem is here:

public void startNotebookRepair(EmployeeId id) {   
    var employee = company.getById(id);
    if (employee == null)
        return;

    /* ... */
    company.put(id, employee.withNotebook(null));
    //                the actual problem  ^^^^
}

The problem is not where the exception occurs but where the null value is set. In production code there can be a lot of conditions that lead to null being set. First, we need to find the condition that leads to our null pointer infindNotebookMaker.

Then we need to decide if setting null in startNotebookRepair was correct or a bug. If it was a bug, we can only hope that no other condition will also set the notebook to null, because that will cause the next exception. If it was correct, we need to decide how findNotebookMaker should react to a null notebook. If we just return null; then we have just delegated the decision making again.

Clearly stack traces are useless for null pointer exceptions. They do not actually show what caused the problem. We have to search for the actual point of origin and then change the design accordingly. Which is what makes null pointers so expensive.

The wrong way to address this problem is to always check every reference for null before access and always write mitigation strategies. The better way is to find a way to mark what can be null and for the compiler to remind us whenever that changes.

The trend towards safety in programming languages

We have already seen that ALGOL W provided a comprehensive type system; that unfortunately did allow the null reference. A feature that most popular programming languages kept ever since. But starting around the year 2000 languages were created and subsequently have become popular that saw things differently.

2005 saw the release of F#, a new functional programming language for .NET. It is billed as empowering “everyone to write succinct, robust and performant code”. F# has the null keyword for interop with C# or VB.NET but it is not permitted as a regular value. Instead, F# has the option type.

In 2010 we got Rust, “a language empowering everyone to build reliable and efficient software.”. In Rust there are no null references. Instead, Rust has optional pointers.

Kotlin arrived in 2011, offering a “modern, concise and safe programming language”. It has null but you can only assign it to nullable types.

Robust, reliable, safe. There is a certain trend here. Newer programming languages are designed with the lessons from the past. But what about older programming languages?

Everyone tries to deal with null

According to the surveys by Redmonk, TIOBE and PYPL, the top 5 programming languages year over year include Python and Java, most often C# as well. These languages are also 20 to 31 years old. Thankfully they have not rested on their laurels but continued to adopt to the changing needs, specifically when it comes to null handling.

Python tries to deal with null

Python, perhaps best known for making everything simple and fun (see xkcd 353 for “proof”) uses dynamic type checking. So it came as a surprise when they added type hints in 2015. With Python 3.5 it was possible to describe what the expected types and return types where (def greeting(name: str) -> str:). It was also possible to declare that something was Optional[T] and could be None. These hints will inform you that something you try to access might be None (Pythons version of null). There is still no type checking happening at runtime though.

C# tries to deal with null

C#, Java’s “cousin”, has not been shy to evolve. In 2005 the language already added nullable value types, i.e. the ability to declare even primitives as nullable (valid: int? number = null;). It wasn’t before 2019 that they added nullable reference types. These allow classes to be declared as non-nullable (not allowed: User user = null;) or nullable (allowed: User? user = null;). The compiler enforces that a non-nullable type has to be initialised to non-null and cannot be set to null ever. The compiler also enforces that before accessing a nullable reference, the developer always has to check for null, which is where null-propagators ?. or null-coalescing operators ?? come into play.

In C# finding the Notebook maker could look like the following:

public string? FindNotebookMaker(EmployeeId id) {   
    /* ... */
    return employee.notebook?.maker();
    //                        ^ 
    //              returns null, if notebook is null
}

or it could look like:

public string FindNotebookMaker(EmployeeId id) {   
    /* ... */
    return employee.notebook?.maker() ?? "No maker";
    //                                ^^ 
    //     returns left side if non-null otherwise the "No maker" 
}

The first example delegates the null-handling to whatever is calling the method. The second example decides that a default value is the right choice. Neither is better and other choices are possible, but we have to chose.

Nullable types force us to make the choice ahead of time. We avoid the run-time bug because we get to make a decision while we design the feature and thus while we have the most knowledge about the problem.

Nullable types make C# designs better since 2019. Not all designs though, because enabling it for all projects would mean so many compile-time errors that no one would upgrade to C#8. The same problem that the Fortran to ALGOL transpiler had. Instead, the nullable types are only enabled in a nullable aware context which can gradually be enabled per module and enabled/disabled for individual files.

Java tries to deal with null

Java, known for going the extra mile when it comes to backward compatibility, should have the hardest time fixing the billion-dollar mistake. null is so baked into the language that tackling the problem would require “six Ph.D theses, knotted together.”.

As luck would have it one JDK project has been doing these Phd theses for the past 10 years and their work will reach preview in the coming years. It would be great though if we could use something right now.

Java has OptionalReturn

Something that has been available in the JDK since 2014 is Optional<T>. This type is great to make APIs fluent but it does not actually eliminate null pointer exceptions. It is not suitable for optional fields or parameters either. To see why we have to go back to the original design intention.

Optional should have actually been called OptionalReturn<T>, as in optional method return. It was designed together with the Java 8 Streams API. See the following code:

String employeeNameById(EmployeeId id){
    return employees.stream()
                    .search(it -> it.id() == id)
                    .name();
    //              ^^^^^^^
    // non-obvious potential NullPointerException
}

search() returns the query result or nothing, when nothing matches. This example where “no result” is represented by null is fictional however. In the actual Stream Api they added a limited mechanism to represent “no result” and keep the Api fluent (see also Optional – the mother of all bikesheds):

String employeeNameById(EmployeeId id){
    return employees.stream()
                    .filter(it -> it.id() == id)
                    .findFirst()
                    // returns an Optional
                    .map(Employee::name)
                    .orElse(“UNKNOWN”);
}

Methods that might return “no result” can instead return Optional<T>. Then the caller has the option how to react to that. What it is not good for is the other direction, passing it as a parameter:

myMethod(a, Optional.ofNullable(b), c, Optional.ofNullable(d));

While not horrible this method call is now quite noisy and widespread use of Optional.of makes our code base very cluttered. The clutter becomes even worse when we have to interact with optional fields:

if(sth.a().isEmpty()
    && sth.b().isPresent()
    && sth.b().get().bb().isPresent()){

    aMethod(sth.b().get(), sth.c().get());
}

Rather than being helpful, Optional<T> has become a crutch that clutters the codebase. Additionally it does not actually eliminate NullPointerExceptions because we can still write the following

record Card(String name, Optional<Integer> value){}
void play(Optional<Card> card){ }

void doSomething(){
    var card = new Card("Ace", null);
    //                         ^^^^
    //               perfectly valid
    play(null);
    //   ^^^^
    // same for this
}

It would be much better if could keep the nullness information but without the clutter and with actual protection from null pointers. This is where Nullaway and JSpecify come into play.

Add compile-time null checks with Nullaway and JSpecify

NullAway is a compile-time nullness checker. It was written by Uber because they wanted a tool to eliminate NullPointerExceptions that can be deployed large-scale (see their 2019 paper). In their experience the build-time overhead is less than 10%, so they can run it on every build. It is a plugin for error prone, a static code analyser from Google. To use it you have to add it as a plugin to your gradle or maven file (full example):

// add processor to pom.xml (or .gradle):
// abbreviated for readability, full example in Github
<build><plugins>
<plugin>
    <artifactId>maven-compiler-plugin</>
    <configuration>
        <compilerArgs>
            <arg>
                -Xplugin:ErrorProne
                -Xep:NullAway:ERROR
                -XepOpt:NullAway:AnnotatedPackages=de.richargh.module-a
            </>
        </>
        <annotationProcessorPaths>
            <path>
                <groupId>com.google.errorprone</>
                <artifactId>error_prone_core</artifactId>
                <version>${error-prone.version}</version>
            </>
            <path>
                <groupId>com.uber.nullaway</>
                <artifactId>nullaway</artifactId>
                <version>${nullaway.version}</version>
            </>

NullAway will now assume that all variables in the packages de.richargh.module-x are non-null by default. Any line where we pass null to a non-nullable field will be greeted with a compile-time error like this: [NullAway] passing @Nullable parameter 'null' where @NonNull is required

Next, we can start marking fields that are actually nullable as such. Up until 2024 we would have used the quasi-standard JSR-305 for that. It was a set of annotations proposed by the static analyser FindBugs. Although widely adopted (maven central lists packages from Google, Atlassian, Eclipse and many more) the JSR has been dormant since 2012 and was never finalised, the annotations never completely defined. Until JSpecify took up the work.

JSpecify is a set of annotations, specifications and documentation for static analysis. It is defined by Google, Oracle, Uber (NullAway), Broadcom (Spring), JetBrains, PMD, Sonar, and many more. Version 1.0.0 was released 2024 and includes four nullness annotations that are generally useful in the following pairings:

• To declare types in package, class, method, or constructor as non-null by default, use @NullMarked. Anything that can be null needs to be explicitly marked as @Nullable.
• To declare types in package, class, method, or constructor as null by default, use @NullUnmarked. Anything that can never be null needs to be explicitly marked as @NonNull.

To use these annotations with NullAway you simply have to depend on them:

// abbreviated for readability
<dependencies>
    <dependency>
      <groupId>org.jspecify</groupId>
      <artifactId>jspecify</artifactId>
      <version>1.0.0</version>
    </dependency>
</>

No additional configuration is necessary. Uber, who developed NullAway, is after all a member of JSpecify. So is Broadcom, the developers of Spring. The next Spring Framework is currently shipping the first milestone releases where their own JSR305 annotations are completely replaced with JSpecify. They also use NullAway to enforce those. Whether Spring Boot 4 will also have all the new annotations is still not decided as of time of writing.

Fixing the Billion-Dollar Problem with the Compiler

Once you have the annotations you simply have to use them to mark something as @Nullable. Our notebook repair example becomes:

/// Shop.java v2
public void startNotebookRepair(EmployeeId id) {   
    var employee = company.getById(id);
    if (employee == null)
        return;

    /* ... */
    company.put(id, employee.withoutNotebook());
}

/// Employee.java v2
public record Employee(
    EmployeeId id,
    String name,
    @Nullable Notebook notebook) {
//  ^^^^^^^^
// new JSpecify annotation

    /* withoutNotebook() … */
}

Which means the following throws a compile-time exception:

// v1
public String findNotebookMaker(EmployeeId id) {   
    var employee = company.getById(id);
    if (employee == null)
        return "Employee does not exist";
    /* ... */
    return employee.notebook().maker();
    //                        ^^^^^^^^
    // [NullAway] dereferenced expression employee.notebook() is @Nullable
}

And we have to design a solution:

// v2
public String findNotebookMaker(EmployeeId id) {   
    var employee = company.getById(id);
    if (employee == null)
        return "Employee does not exist";
    /* ... */
    return employee.notebook() != null
                ? employee.notebook().maker()
                : EMPLOYEE_DOES_NOT_HAVE_A_NOTEBOOK;
}

The compiler helped us immensely. It forced us to fix the root cause, not a symptom. We had to specify whether setting null was allowed in the first place.

This compile-time check is the same approach that Rust, F#, as well as Kotlin have picked and the one that C# migrated toward. Granted the other programming languages picked a different syntax but the idea remains the same. In fact, the idea remains the same as the one that Sir Tony Hoare had in 1965.

Gradual Migration

Adding NullAway to a code base is amazing. You will catch so many bugs hiding in plain sight. Every green field project should do so. But adding it to the entirety of existing projects with thousands or millions of code lines would be a nightmare. This is where the configurations options of NullAway come into play. They are all prefixed with -XepOpt:NullAway.

The simplest way to start is with :AnnotatedPackages=de.richargh.module-a,de.richargh.module-b, which takes comma-separated list of packages to check. Start with one module or even just a small part of a module, then gradually add more. If you just want to experience NullAway, then start with a simple module. Generally it is better to start in a business-critical module. Even if the migration takes longer, the value of migration is just that much higher and easier to pitch. Once you have migrated enough modules you might decide to flip the inclusion. Include all and strip those you have not migrated yet via :UnannotatedSubPackages=de.richargh.module-y,de.richargh.module-z :AnnotatedPackages=de.richargh.

The alternative to AnnotatedPackages is to mark packages that NullAway should check directly via JSpecify annotations. To do so replace :AnnotatedPackages= with :OnlyNullMarked=true. Then null-mark all desired modules in their respective package-info.java:

@NullMarked 
package de.richargh.module-a; 

import org.jspecify.annotations.NullMarked;

// place this package-info.java into the root of the module you want to mark

Eventually you will also want to flip to exclusions here, null-mark your whole project and exclude modules:

@NullUnmarked 
package de.richargh.module-z; 

import org.jspecify.annotations.NullUnmarked;

Whatever approach you pick, you need to be careful of objects that come from unmarked modules. NullAway has no information about the nullness of objects from beyond the edge of a module. It will let you access them without checks.

It is thus in your best interest to quickly migrate modules that are often exchanging data with each other. A prime example is a module that is used by multiple modules. Until this shared module is checked with NullAway, you cannot trust the output in any other module, even if that module is already checked with NullAway. Otherwise your null-safety will always be on shaky ground.

6+1 Tipps to get the most out of null-checking

(0) Don’t trust beyond the edge.
Anything not checked by NullAway is beyond the edge. As previously discussed, that includes your own unchecked code. It also includes most third-party code. Spring Framework being a rare, annotated exception. Particularly dangerous is deserialised json however. Mapping-Frameworks like Jackson see null-checks as validation. And Jackson does not do validation:

/// RenterDto.java
public record RenterDto(
    String id,
    String name){
}

/// somewhere else, perhaps in a Controller
var json = """{ "name": :"Alex" }""";
var result = objectMapper.readValue(json, RenterDto.class);
//                                  ^^^^
//                            does not throw an exception
// result = RenterDto[id=null, name=Alex]
result.id() == null; // true…

So be sure you know what happens at the edge. For Jackson in particular you might want to add validation checks. You could write those with Hibernate Validator and write your own ConstraintValidator. Here is a custom DefaultIsNonNullableValidator to get you started.

(1) Make non-null your default
NullAway assumes this by default. It is still worth repeating. Something being optional is the exception, not the rule.

(2) Model Optional Fields with @Nullable
Do not use Optional<T> but stick to annotations:

public class <coolClass> {
    private @Nullable Address address;
}

public record <coolRecord> (
    @Nullable Address address){
}

(3) Never pass null, consider explicit methods instead

// if you have this method:
var p1 = notebookPriceFor(
    employeeId, notebookType);

// ❌ then do not do this:
var p2 = notebookPriceFor(
    null, notebookType);
//  ^^^^
// What does passing null mean semantically? 

// ✅ create an explicit method that indicates that a query without an employee, is just an estimate
var p2 = estimateNotebookPrice(
    notebookType);

(4a) Mark nullable method returns

@Nullable myMethod(A a, B b) {
	/*…*/
}

(4b) Occasionally consider Optional for method returns, which gives choices to caller
Suppose you have the following repository method:

// Optional is quite useful for repositories
Optional<Address> getAddressById(employeeId id) {
	return Optional.ofNullable(addresses.get(id));
}

Then the caller has all the choices:

// Choice A: caller says null is not allowed
var account = accountsfindAccountById(id).orElseThrow();

// Choice B: caller says default is possible
var account = accounts.findAccountById(id).orElse(Account.none());

// Choice C: caller uses fluent chain
var budget = accounts.findAccountById(id)
					 .map(Account::budgetId)
	                 .flatMap(budgets::getById)

(5) Return empty collections or arrays, not nulls

// ❌ not like this
// just makes the API more difficult to use for all callers
@Nullable List<Address> recentAddresses(){
	return employee.isTrackingAllowed()
					? recentAddresses
					: null;
}

// ✅ like this
List<Address> recentAddresses(){
	return employee.isTrackingAllowed()
					? recentAddresses
					: Collections.emptyList();
}

(6) Use JDK 16 records for domain types

/// Address.java
public record Address(
    String city,
    @Nullable String street
) {
}

/// somewhere-else.java
var address1 = new Address(
    “Brühl”, "Berggeiststraße 31-41");
var address2 = new Address(
    “Brühl”);
//         ^^
// Compile-time error because parameter is missing
// null is allowed, we just have to be explicit about it

When working with null-checks, you want to gradually reduce ambiguity. A typical source of ambiguity are partially instantiated domain objects. Records do not allow such cases and require all parameters to be declared, even if they are null. Records are also very compact and reduce a lot of boilerplate. If your records have a lot of states in which they can start out, then consider using static factory methods to instantiate them.

The Future – Six Ph.D. theses, knotted together

Last I talked about fixing the billion-dollar mistake (JavaLand 2023), I was hoping we would get JDK null-safety sometime around 2029.

Roughly one year later, Brian Goetz, steward of the Java language, held a talk about Java’s Epic Refactor. In this talk he gave major updates on Project Valhalla, aka “six Ph.D theses, knotted together.”. Their goal is to bring value classes into Java that “code like a class, work like an int”, a goal which is very different from any null-marking support. Brian announced that as part of that work they would also bring Null-Restricted and Nullable Types in Java.

Once this enhancement lands in the JDK, we would be able to do the same thing that NullAway with JSpecify gives us today, but without NullAway or JSpecify and with a more compact syntax. Instead of @Nullable we would postfix types with ?, instead of @NonNull we would postfix types with !. Instead of a compiler plugin, all would be checked by the compiler. Our employee would now look like this:

public record Employee(
    EmployeeId! id,
    String! name,
    Notebook? notebook) {
}
// very similar to how employee would be modelled in C# or Kotlin. 

The ! are currently a requirement to mark something as null-restricted.. Without those markers the code stays backward compatible and the nullness of a type is unspecified. No spec has been finalised though.

It would be great to have something like @NullMarked in Java as well. Something that we can set, so we only have to mark nullable types with ? and that is indeed part of suggested future work. Before this simplification can happen, the actual Null-Restricted and Nullable Types have to be implemented, and that is the crux of the matter. Restricted and nullable types are not the goal of Project Valhalla. They are something the project needs to make value types extra awesome. I think we will get them, but take that with a grain of salt. I am not a Valhalla developer. In the meantime, NullAway is a great choice to finally fix the billion-dollar mistake.

References

• NullAway has good documentation if you want to get started
• My full NullAway+maven example can be found here
• My previous presentation slides and recordings for the topic can be found here