Reverse Engineering Java: Backtracking Native Crashes

“what abstraction conceals is exposed at the fault site.”

Introduction

Java applications run on an abstracted, memory safe platform. But when JVM crashes, things collapse at the bottom native layer, and the root cause analysis becomes very hard. This is because the abstractions that lie between the actual crash context and the program context are too wide and deep, and the knowledge about the pathways between these abstractions are neither well documented nor well known to the developers.

This article walks through debugging a Java crash from the ground up: starting with raw native instructions and moving through JVM internals, calling conventions, object layouts, and the runtime’s translation of high-level semantics into low-level behavior. This guide equips engineers with the foundational skills to backtrack a failure, interpret crash artifacts with confidence and isolate root causes independently.

Why Are Crashes Complex?

Java, as a platform-independent language, and the JVM, as a platform-independent execution engine, deliberately hide the underlying platform details. This abstraction is a great feature and works perfectly fine at the develop, build and execution phases. However when the JVM crashes, the failure surfaces in the native layer in a raw and context-less manner, and completely detached from the abstracted semantics Java developers are familiar with.

To understand such failures, we must peel back those layers of abstraction multiple times. That means we need to reconstruct the layers between native instructions, JVM internals, the Java bytecode and the original Java code. Only by de-abstracting each layer we can meaningfully connect the fault site to the actual fault source.

For example a crash that manifests like below:

0x7ff2345bc  mov rdx, [rax, 0xa0]

would mean nothing in the Java source. This needs to be de-abstracted to for example:

x = obj.y;

form, for a developer to relate to, and make some reasoning about.

Why a Custom Debugger Is Needed

General-purpose debuggers like gdb, lldb, dbx and windbg only see JVM as a native (C++) application and show artifacts (stack trace, register context, raw memory etc.) from the point of view of native state. However, a JVM crash encompasses both: the native state where the real crash occurred, and the managed runtime and application state whose semantics led to that native state. Both the states are critically important in problem determination. No single off-the-shelf tool understands this combined picture.

A custom debugger translates states in both these layers: it helps to decode JVM frame layouts, interpret metadata, reconstruct Java stacks from native frames, understand calling conventions generated by the JIT and map raw addresses back to bytecodes, methods or objects. The custom debugger makes the JVM’s internal structures legible, correlates native artifacts with Java-level behavior and offers the cross-layer visibility required to backtrack a crash. Without these capabilities, a normal debugger shows only meaningless addresses and apparently corrupted frames, leaving the real root cause invisible.

Setup

While the methods and principles discussed here apply across JVM implementations, the diagnostic tooling, command syntax, internal VM structures and generated machine code shown are specific to the Semeru JVM and are used for illustration.

Environment Setup: Ensure that the JVM is running on a supported operating system and hardware architecture. Refer to the official IBM documentation for supported platforms.

Diagnostic configuration setup: For the JVM to generate meaningful diagnostic data during a failure, specific diagnostic settings must be configured at OS level to avoid truncated coredumps before starting the application. Mustgather for Runtimes Java Technology   document provides detailed guidance on must-gather setup/gather with respect to platforms and problem scenarios.

Tools: Unlike other production anomalies, crash debugging usually requires usage of one or more tools depending on the nature of the crash, the crashing context and the execution environment (platform, architecture etc.) Below listed are the most commonly used tools for crash debugging.

Custom (runtime aware) debugger:  For OpenJ9 system dumps, the primary tool is jdmpview that is bundled within the JDK. jdmpview can interpret JVM internal structures, display Java and native stacks, reconstruct mixed call stacks (native + interpreted + JIT), inspect heap objects and VM metadata. More details can be found in the jdmpview documentation.

Diagnostic data: when your Java application crashes, there are a number of diagnostic data files generated by default: javacore, snaptrace, jitdump and system(core) dump. The system dump (core file) is the most critical artifact in the crash context. It contains a snapshot of the entire process address space at the time of the crash, including registers, native stack, Java stack, heap memory and JVM internal structures.

Problem Determination

Launch

Alright, let us start our debugging journey! First, launch the core file on to the debugger.

jdmpview -core <corefile>

Extract Crash Context (jdmpview)

Next, let us start by examining the crash context. This includes information on the failing module, failing thread, and its entire register context including the failing instruction, and the signal which the processor raised at the time of the crash.

Extract Failing Thread (jdmpview)

Clearly, this is the lowest level context with little or no clues about the application context. So let us de-abstract the thread and map it into the application context:

So the failing thread is “remote-data-transport-thread”. Though this information alone is not going to be enough, we have a foot hold in the wild!

“method of walking through unclear terrains:

step back. breathe and assess.
hold your ground. gather the data.

then look for a wedge. a narrow opening.

open it up. cut through it. widen it.
turn the wedge into a corridor.
move through it. make progress.
gather new insights in the process. repeat.
make new pathways. progress again!

that is how we create order from chaos.”

Extract Java Stack (jdmpview)

Now that we know the failing thread, let us see what is its “Java” state. This is well known as the Java stack.

Extract Native Stack (gdb)

Now that we have a strong sense of “where” the crash is happening through de-abstracting from the native to the Java level, lets now repeat the same for “what” the crash is, how did it manifest, what it means in the immediate context and what it means in the application context.

Java being managed runtime, the faults are contained and preprocessed for several dump functions to capture various fault contexts. So frame 0 to frame 14 belong to the JVM that handles the dump processing. This means the actual failure occurred in frame 15, jvmtiGetClassSignature method of JVM’s JVMTI module. If we look beneath that, we see it is triggered from the native function Java_org_transporter_remote_Remote_xport0, which needed methods and classes to be resolved as part of the transport function.

A rough theory we can make at this point is that some bad input was passed to the JVMTI function from one of its callers. But no evidence, only a hypothesis! Let us prove or disprove it.

Extract Failing Instruction (gdb)

Next, get the failing instruction.

Then, dump the signal context (from the gpinfo above):

The most important information here is the signal number and the signal code.
Signal number: 0xb (11) is the infamous SIGSEGV (segmentation violation)
Signal code: 0x1 (1) means the reason for SIGSEGV: SEGV_MAPERR (address not mapped to object)

Now, dump few instructions in and around that address to get a little more context. Which instruction caused this? What it was trying to do? Which address is not mapped?

This means load the value stored at memory address %rbp + %rax + 4 into r10.
Clearly, the source location is holding the bad memory address.

Lets compute that manually:

So clearly, rbp (-1) is pointing to a bad memory location.

So the immediate root cause of the crash is that rbp holds invalid content.

Backtrack (gdb)

How did rbp get that value? Let us backtrack a bit.

Wait a minute. What is this rbp, and where it has come from? Let us trace it back till the beginning of the method.

At this point, we also want to learn a little bit about the x86-64 ABI summary to understand the transition between functions.

The first six arguments are passed in registers rdi, rsi, rdx, rcx, r8, and r9 in that order, with the return value in rax.

• Registers are divided into two types: caller saved (volatile) and callee saved (non-volatile).
• Caller saved registers (rax, rcx, rdx, rsi, rdi, r8, r9, r10 and r11) can be freely mutated by the callee.
• Callee saved registers (rbx, rbp, r12, r13, r14 and r15) must be preserved by the callee before writing.
• rbp was traditionally used as a base pointer, but it is not required as per the ABI.
• The stack grows downward, the tip pointed to by rsp. local variables are typically allocated by subtracting from rsp.

With that lets look at the preamble of our method:

So the badness in RBP originates in RSI. and that is our second parameter. What is the second parameter? Lets look at the code:

Root cause: Xport::resolve0() passed a corrupted kclass pointer to jvmtiGetClassSignature().

The function expects kclass to reference a valid J9Object (the class object). Instead, the value was 0xffffffffffffffff, which is not a valid heap address. Subsequent dereferencing of this invalid pointer caused the failure.

Where Do We Go from Here?

We could repeat the same descent: step into Xport::resolve0() and trace how the corrupted klass value was produced and propagated. The methodology would not change. It is the same disciplined backtracking: validate inputs and follow data flow.

We stop here, because the purpose of this article is not to fully resolve this one problem, but to illustrate the discipline and tooling required to de-abstract a JVM crash and trace it to its true source. While the problem will change from story to story, the method does not!

Summary

Java promises memory safety, portability and abstraction from the underlying platform. But when it fails, the crash surfaces at the lowest native layer: with raw instructions, registers, memory addresses and faulting pointers. The semantics we are comfortable with (objects, methods, classes) are nowhere in sight.

In this article, we demonstrated the process of reconstructing the bridge between the original Java intent and the low level fault site. We began at the bottom, at the exact native instruction that failed. From there, we interpreted control flow and data flow with the help of the application binary interface, explained JVM internals where necessary, reducing abstraction at each step, bringing more and more clarity.

Once we understand how arguments flow through registers, how objects are represented in memory and how the runtime actually operates, a crash dump stops being scary, and becomes a rich source of information and signals. It reveals rich insights that no trace tool can fully capture!

This article is part of the JAVAPRO magazine issue:

From Coder To System Designer

Understand what it means to move from coding to designing systems in the age of AI.
Take a closer look at modern Java platforms, architectural thinking, and the responsibilities that come with shaping complex software systems.

Discover the edition →