Connecting Java Reinforcement Learning to Python Gymnasium

This three-part series builds a Java Reinforcement Learning Python Gymnasium pipeline from scratch. Part 1, this article, explains why the project java-rl-dqn-to-rainbow¹ starts with Gymnasium integration. The main reason is testability. CARLA² is a very large and complex environment, so the algorithm needs simple environments such as CartPole³ or Mountain Car⁴first, allowing shorter testing cycles. The implementation starts with DQN⁵, one of the simplest algorithms combining Reinforcement Learning and Deep Neural Networks. Each subsequent step refines and upgrades the algorithm until the second article delivers the complete Rainbow DQN implementation. The last article integrates CARLA and trains the agent with Rainbow DQN⁶.

What’s Reinforcement Learning? A Simple Introduction

Reinforcement learning algorithms require environments to interact with. In practical terms, an environment in Gymnasium is a deterministic or stochastic simulator. It exposes a state, accepts an action, and returns a reward and the next state, as shown in the diagram below:

A 4×4 grid, where each state marked as S_index starting with 0 and finishing at 15, has holes with a -1 reward (negative) and the present has a +1 reward (positive). In the original implementation, the rules just terminate the game. The image only illustrates the concept.

Originally published: This article first appeared on Substack⁷as part of a three-article series on building a Java-based Rainbow DQN agent that connects Python Gymnasium and CARLA.

Gymnasium in the Java Reinforcement Learning Ecosystem

The Java ecosystem lacks native training environments for reinforcement learning. Python fills this gap with Isaac Lab⁸ (Robotics mainly), the now-unmaintained OpenAI Gym⁹ and the fork Gymnasium¹⁰, which is an ecosystem on its own with many environments under the Farama¹¹ umbrella.

Example of the two environments that are covered in Video Games and Reinforcement Learning¹² and How to Never Forget Deep Q-Networks: Memory Palaces Meet Reinforcement Learning¹³ articles:

_{Trained agent running in Frozen-Lake environment}

_{Trained agent running in Pong environment}

Evaluating Java Reinforcement Learning Integration Options

Two integration candidates stood out: Py4J¹⁴ (used in PySpark¹⁵) and ZeroMQ¹⁶. ZeroMQ proved to be the better choice, as it is a brokerless messaging library with zero-copy optimizations. Py4J, by contrast, uses a gateway (broker) that translates calls between the languages.

The initial trial started a Python process from Java and used the Request and Response pattern for communication. Synchronizing the two platforms proved problematic; any ordering issue left the socket in an invalid state.

For example, sending two consecutive Java requests breaks the socket. The same occurs if Python sends a response twice. The following order must always be respected: req→resp→req … and so on¹⁷.

Even restarting the socket requires restarting the server or client as well. The project then switched the protocol to the Pirate Patterns to make the communication reliable¹⁸.

Even so, the goal was a seamless integration between Java and Python. The evaluation covered two options: the FFM API¹⁹ (Foreign Function and Memory API) and JavaCPP²⁰. Both offer efficient integration with native code.

I chose JavaCPP for two reasons: it includes CPython²¹ integration and works easily with C++ code²², useful for future CARLA integration. The FFM API requires creating a C-API first, then generating bindings with jextract²³ for any library not written in pure C.

JavaCPP

JavaCPP calls native Python functions by implementing most of the CPython C-API²⁴, with some macro exceptions due to parser limitations²⁵.

The javacpp-presets²⁶ already includes the Gym 0.26.2²⁷ library and ALE 0.8.0²⁸ (Arcade Learning Environment), and the gym module worked successfully. Inside the library, each platform bundles an embedded Python version within the jar to execute the gym code. Example:

// omitted code
public class gym {
    private static File packageFile;

    public static synchronized File cachePackage() throws IOException {
        if (packageFile != null) {
            return packageFile;
        } else {
            // This is the location of the jar that will be loaded
            packageFile = Loader.cacheResource("/org/bytedeco/gym/python/");
            return packageFile;
        }
    }
// omitted code

The physical location²⁹:

Similarly, this can be observed with CPython³⁰, but separated by platform:

Setting Up the Python Environment

To control the version and libraries, the project creates a gymnasium folder in the root with the following content:

README.md
requirements.txt

README.md contains the instructions to install uv³¹ and the correct Python version, along with the environment variables that Java exports for its use:

// omitted
pip install uv
uv python install cpython-3.12.1-windows-x86_64-none
uv venv --python 3.12.1
.venv/Scripts/activate
uv pip install -r requirements.txt
// omitted

- `export JAVA_RL_SITE_PACKAGES=/path/to/java-rl-dqn-to-rainbow/gymnasium/.venv/include/site/python3.12`

The requirements.txt contains the libraries to be installed:

numpy==2.3.5
pygame==2.6.1
matplotlib==3.10.8
gymnasium[classic-control,box2d]==1.2.2

Java Reinforcement Learning Python Gymnasium: Integration Roadmap

With the integration options evaluated, the implementation planning can proceed:

1. Python starts as a fully managed embedded process.
2. Code executes inside this process and the output returns to Java.
3. Java creates the object inside Python and invokes its methods.
4. Java reads Python arrays with zero-copy or memory copy depending on the scenario.
5. env.render() displays the environment output in Java using JFrame.
6. The environment executes actions using env.step(action) and returns the method output.
7. The step method output converts to DJL³².
8. The project creates multiple tests for different environments to validate usefulness.

During development, many segfaults occurred, as the lifecycle of Python requires manual management. For an alternative approach to Java and Python integration, see Bridging Java and Python for AI/ML in Production.

Embed Python Lifecycle

Calling Py_Initialize() and Py_FinalizeEx() more than once does not guarantee correct behavior due to external modules, as is the case of this project.

That is why the project avoids Py_FinalizeEx(), as Gymnasium depends mainly on NumPy and PyGame³³; calling it can trigger a segmentation fault.

To work safely with Python in a multi-thread environment where Java has its own thread state, the integration requires PyGILState_Ensure() and PyGILState_Release(), e.g.:

// insideGil() wraps this block, accepting a Runnable or Function
var gstate = PyGILState_Ensure();
try {
    result = callSomeFunction();
} finally {
    PyGILState_Release(gstate);	
}

Without this, race conditions can leave objects in an invalid state and cause a segmentation fault³⁴, crashing the JVM. For example, multiple threads decrementing the reference count variable simultaneously can produce a reference count of 0 instead of 1, causing a double-free³⁵.

However, if only the thread that called Py_Initialize() is used, in most cases the code block above is not needed. The method insideGil() encapsulates this block to avoid the boilerplate.

To understand why this matters, it helps to know about the GIL (Global Interpreter Lock), a Python mechanism that controls thread access to the interpreter. Py_Initialize() shares it across all threads by default, which is a limiting factor.

NumPy mitigates this by releasing the GIL during intensive C operations, which is why it remains fast.

To circumvent this limitation and better utilize the machine’s multiprocessing, the project can adopt the vectorization³⁶ feature from the Gymnasium library, which runs environments in parallel and returns multiple values at once. If necessary, Gymnasium4j will include these features in the future.

Multiple Python Interpreters

An alternative is Py_NewInterpreterFromConfig(), which allows its own GIL per interpreter, introduced in Python 3.12³⁷.

Another useful method is PyGILState_Check(), which returns 1 if the thread holds the GIL; useful for debugging the application³⁸.

Memory management³⁹ outside Python also requires attention. Since the project embeds Python in Java, it must manage reference counting manually. Below is a list of the available methods:

Returns a new reference (you own it; you must decrement it):

• PyObject_CallObject()
• PyObject_Call()
• PyObject_GetAttrString()
• PyImport_AddModule()
  • The project usually creates and forgets the new module, as the Python process releases it on finish. This is the case here, used in initPython(), explained later.
• PyUnicode_FromString()
• PyLong_FromLong()
• PyFloat_FromDouble()
• PyBool_FromLong()
• PyBytes_FromStringAndSize()
• PyByteArray_FromStringAndSize()
• PyDict_New()
• PyTuple_New()
• PyList_New()
• PyObject_Str()
• PyUnicode_AsUTF8String()
• PyDict_Items()

Returns a borrowed reference (you don’t own it, do not decrement reference):

• PyTuple_GetItem()
• PyList_GetItem()
• PyDict_GetItem()
• PyModule_GetDict()
• PyErr_Occurred()

Steals a reference (takes ownership from you):

• PyTuple_SetItem()
• PyList_SetItem()

PythonRuntime: Java Reinforcement Learning Python Bridge

Proceeding to the PythonRuntime, the main methods are initPython(), exec(), and eval():

The initPython() method is called only once when the Java process starts. It is synchronized to ensure the code is initialized only once. cachePackages() returns the location of the javacpp CPython and custom libraries installed using uv (the environment/property JAVA_RL_SITE_PACKAGES must be filled).

The globals variable holds all variables created in Python by the integration:

    @SneakyThrows
    public static synchronized void initPython() {
        if (initialized) return;
        if (Boolean.getBoolean("python.initialized")) {
            initialized = true;
            return;
        }

        initialized = Py_Initialize(cachePackages());
        globals = PyModule_GetDict(PyImport_AddModule("__main__"));

        if (!initialized) {
            throw new IllegalArgumentException("PythonRuntime is not initialized!");
        }

        System.setProperty("python.initialized", "true");
    }

The exec() method is where Python functions are executed in the interpreter:

    public static void exec(String code) {
        PyErr_Clear();
        try (var _ = PyRun_StringFlags(
                code,
                Py_file_input,
                globals,
                globals,
                null
        )) {
            checkError();
        }
    }

Every variable is saved inside globals. For isolated executions, where variables do not live outside the scope, execIsolated() can be used as an alternative. Below an example of its usage:

// Code inside unit test
        exec("""
        import numpy as np
        arr_little = np.array([1, 2, 3], dtype='<f4')  # explicit Little-endian
        arr_big = np.array([1, 2, 3], dtype='>f4')     # explicit Big-endian
        """);

The eval() is very similar to exec(), the difference lies in the input parameter:

Py_eval_input expects an expression like (1 + 1) or NewInstanceExample(), while Py_file_input executes any code without returning a value:

    public static PyObject eval(String expression) {
        PyErr_Clear();
        try (var result = PyRun_StringFlags(
                expression,
                Py_eval_input,
                globals,
                globals,
                null
        )) {
            checkError();
            return result;
        }
    }

The PyErr_Clear() clears any non-fatal error that was not cleared and could interfere with PyErr_Occurred(). The checkError() method checks if any error occurred after execution, using PyErr_Occurred(). The pattern is to clear possible old errors and check again at the end.

The other utility classes are PythonDataStructures and PythonTypeChecks, explained as needed, but the names are intuitive.

Java Reinforcement Learning Python Gymnasium: NumPy Integration

The NumPyByteBuffer class is used to make an efficient transfer between the ndarray data structure in Python and Java’s ByteBuffer (or other languages that support the Buffer Protocol).

NumPy arrays implement the Buffer Protocol⁴⁰, This allows zero-copy communication because their memory is contiguous (a single block, like a primitive array)⁴¹.

The static initializer is responsible for ensuring endianness alignment (the way bits are ordered in the hardware)⁴² between Java and NumPy. It is implemented this way to perform the check only once:

    static {
        initPython();

        BYTE_ORDER = insideGil(() -> {
            exec("import numpy as np");
            exec("_test_arr = np.array([1], dtype=np.float32)");

            try (var testArr = eval("_test_arr")) {
                String byteOrder = attrStr(attr(testArr, "dtype"), "byteorder");
                return switch (byteOrder) {
                    case ">" -> ByteOrder.BIG_ENDIAN;
                    case "<" -> ByteOrder.LITTLE_ENDIAN;
                    default -> ByteOrder.nativeOrder();
                };
            } finally {
                exec("del _test_arr");
            }
        });
    }

The fillFromNumpy() method handles this communication by making a copy of the Python ndarray. If receives a native array, it reuses it, avoiding array copying. However, since no efficient cycle management is yet implemented in this project (such as an Object Pool pattern), a temporary copy is made, as incorrect deallocation can also cause segmentation faults.

    public static void fillFromNumpy(PyObject ndarray, ByteBuffer buffer) {
        buffer.clear();
        try (var view = new NumPyBufferView(ndarray)) {
            int size = view.capacity();
            if (size > buffer.capacity()) {
                throw new IllegalArgumentException(
                        "Buffer too small: capacity=" + buffer.capacity() + ", required=" + size
                );
            }

            buffer.put(view.buffer());
        }

        buffer.flip();
    }

The NumPyBufferView uses the native Python C-API to bridge this communication. @Delegate from Lombok⁴³ is used to expose the exact same public API of ByteBuffer without requiring a manual implementation, establishing a standard to use the Python object as closely as possible to a ByteBuffer. 

AutoCloseable is implemented to ensure the view is deallocated by the caller using the correct C-API:

public class NumPyBufferView implements AutoCloseable {
    private final Py_buffer view;
    @Delegate
    private final ByteBuffer buffer;

    public NumPyBufferView(@NonNull PyObject ndarray) {
        this.view = new Py_buffer();
        int rc = PyObject_GetBuffer(ndarray, view, PyBUF_SIMPLE);
        if (rc != 0) {
            throw new IllegalStateException("PyObject_GetBuffer failed (array not contiguous?), return code: " + rc);
        }

        long size = view.len();
        this.buffer = view.buf().capacity(size).asByteBuffer();
    }

    /**
     * Used internally!
     */
    ByteBuffer buffer() {
        return buffer;
    }

    @Override
    public void close() {
        PyBuffer_Release(view);
    }
}

The method PyBuffer_Release() deallocates only the view. The original PyObject must be deallocated separately.

When the ByteBuffer is modified, the changes are reflected in the Python object:

    exec("import numpy as np; arr = np.array([1.0, 2.0, 3.0], dtype=np.float32)");
    try (var pyArr = eval("arr");
         var view = new NumPyBufferView(pyArr)) {
        var buffer = view.asFloatBuffer();
        assertEquals(1.0f, buffer.get(0), 0.0001f);
        assertEquals(2.0f, buffer.get(1), 0.0001f);
        assertEquals(3.0f, buffer.get(2), 0.0001f);
        buffer.put(0, 4f);
        buffer.put(1, 5f);
        buffer.put(2, 6f);
        assertArrayEquals(new double[]{4., 5., 6.}, toDoubleArray(pyArr), 0.0001);
        IO.println(toStr(pyArr)); // Output: [4. 5. 6.]
    }

Gym Class: Building Gymnasium Environments in Java

To manage the environments, the Gym class is provided, making reference to the gymnasium as gym common import standard, and the Env class, the actual result of invoking the make() method, inspired by the original Python method:

@Slf4j
public class Gym {

    @SneakyThrows
    public static IEnv make(String name,
                            NDManager ndManager) {
        // omitted code
    }

    public static final class EnvBuilder {
        // omitted code

        String generatePyEnvScript() {
            var makeCall = generateMakeCall();
            var importLibsPy = generateImportLibsPy();
            if (wrappers.isEmpty()) {
                return """
                %s
                env_%s = %s
                """.formatted(importLibsPy, varEnvCode, makeCall);
            }

            var importPy = generateImportFromPy();
            var wrappedEnvPy = generateWrappedEnvPy();

            return """
            %s
            %s
            env_%s = %s
            %s
            """.formatted(importLibsPy, importPy, varEnvCode, makeCall, wrappedEnvPy);
        }

        public EnvBuilder add(@NonNull IWrapper wrapper) {
            // omitted code
        }

        public Env build() {
            return new Env(varEnvCode, envName, generatePyEnvScript(), ndManager);
        }

        // other methods omitted
    }

    public static final class PyMap {
        // omitted code

        public String toPyDict() {
            return params.entrySet()
                    .stream()
                    .map(entry -> "'" + entry.getKey() + "': " + entry.getValue())
                    .collect(Collectors.joining(", ", "{", "}"));
        }

        // omitted code
    }
}

The EnvBuilder is responsible for customizing the gymnasium environment, such as using ALE and other libraries. 

Wrappers can be applied to modify the environment’s step and observation space, such as skipping frames, converting to grayscale, resizing and concatenating (also known as stacking) the states: MaxAndSkipObservation, GrayscaleObservation, ResizeObservation and FrameStackObservation⁴⁴, which are the classes passed to the add method from EnvBuilder.

The generatePyEnvScript() method from EnvBuilder is the bridge to build the Python code, returning the configuration as a String. PyMap is a parameter configuration dictionary, equivalent to a HashMap in Java. Below are some examples of usage presented in the unit tests using JUnit 6:

// omitted code

class GymTest {
    @Test
    void shouldTestGeneratedWrappers() {
        var envId = "CarRacing-v3";
        var script = Gym.builder()
                .envName(envId)
                .importLib("ale_py")
                .params(Gym.builderMap()
                        .put("domain_randomize", true)
                        .put("continuous", true))
                .add(new DelayObservation(1),
                     new GrayscaleObservation(false),
                     new NormalizeObservation(),
                     new MaxAndSkipObservation(4),
                     new FrameStackObservation(4),
                     new ReshapeObservation(new int[] {1, 84, 84}),
                     new ResizeObservation(new int[] {50, 50, 1}))
                .generatePyEnvScript();
        // omitted code
  }    

    @Nested
    @DisplayName("PyMap Tests")
    class PyMapTest {
        // omitted code
        @Test
        void shouldConvertToPyDictFormat() {
            var pyMap = Gym.builderMap()
                    .put("seed", 42)
                    .put("render_fps", 60);

            assertEquals("{'seed': 42, 'render_fps': 60}", pyMap.toPyDict());
        }

        // omitted code
    }
}

Possible output of generatePyEnvScript():

import gymnasium as gym, ale_py
from gymnasium.wrappers import DelayObservation, GrayscaleObservation, NormalizeObservation, MaxAndSkipObservation, FrameStackObservation, ReshapeObservation, ResizeObservation
env_2931ac287e0f4fdfae8a4ed7b75347dc = gym.make('CarRacing-v3', render_mode='rgb_array', domain_randomize=True, continuous=True)
env_2931ac287e0f4fdfae8a4ed7b75347dc = DelayObservation(env_2931ac287e0f4fdfae8a4ed7b75347dc, delay=1)
# omitted code

Env Class: Managing Gymnasium in Java

The render_mode='rgb_array' is a default value, as the main purpose is to display the environment visually when env.render() is called.

Below the main methods of the IEnv interface, implemented by the Env class:

public interface IEnv extends AutoCloseable {

    boolean closed();

    boolean scalarObservation();

    ActionSpaceType actionSpaceType();

    String actionSpaceStr();

    String observationSpaceStr();

    ActionSpaceType.ActionResult actionSpaceSample();

    Pair<NDArray, Map<Object, Object>> reset();

    EnvStepResult step(ActionSpaceType.ActionResult action);

    EnvStepResult step(ActionSpaceType.ActionResult action, NDManager manager);

    BufferedImage render();

    NDManager manager();

    @Override
    void close();
}

The Env constructor contains the native PyObject wrapper, used to call methods in Python. NDManager is used in DJL to create the architecture, tensors and numerous Deep Learning utilities. manager.newSubManager() is called to ensure that when the environment is closed on the Java side, the objects allocated inside DJL are deallocated.

ActionSpaceType is a type-safe wrapper of Python’s ActionSpace, providing an encapsulated execution of the main environments. varEnvCode is a mechanism to avoid overwriting Python’s global variables (e.g.: env_cf59c3da7e24499fa9f1d6860a534cd7), as the Python interpreter uses the globals⁴⁵ for each process.

Another reason is that if execIsolated were used, the same variable would need to be recreated every time, requiring a scoped design that would differ significantly from the original API.

public final class Env implements IEnv {

    // omitted code

    private final NDManager manager;
    @Getter
    private final String varEnvCode;
    @Getter
    private final String envName;
    private final PyObject pyEnv;
    private final PyObject pyActionSpace;
    private final PyObject pyObservationSpace;
    private final PyObject pyRender;
    private final PyObject pyStep;
    private final PyObject pyReset;
    private final ActionSpaceType actionSpaceType;

    // omitted code

    Env(@NonNull String varEnvCode,
        @NonNull String envName,
        @NonNull String generatedScript,
        @NonNull NDManager manager) {
        initPython();
        this.varEnvCode = varEnvCode;
        this.envName = envName;
        this.manager = manager.newSubManager();
        exec(generatedScript);

        this.pyEnv = eval("env_" + varEnvCode);
        this.pyActionSpace = attr(pyEnv, "action_space");
        this.actionSpaceType = detectActionSpaceType(pyActionSpace);
        this.pyObservationSpace = attr(pyEnv, "observation_space");
        this.pyRender = attr(pyEnv, "render");
        this.pyStep = attr(pyEnv, "step");
        this.pyReset = attr(pyEnv, "reset");
    }

    // omitted code
}

The methods eval and attr are from the PythonRuntime class. eval evaluates and executes Python’s interpreter, returning a PyObject, an object that contains a pointer to the native Python object, without passing through any gateway. attr follows the same idea, but for accessing attributes and methods, since in Python both are objects and can be retrieved by name from any PyObject.

// PythonRuntime.class

// omitted code
    public static PyObject attr(PyObject obj, String attr) {
        var result = PyObject_GetAttrString(obj, attr);
        if (isPyNull(result)) {
            PyErr_Print();
            throw new IllegalArgumentException("Attribute not found: " + attr);
        }
        return result;
    }
// omitted code

An equivalent example in Python, for the Env’s constructor, would be:

pyActionSpace = env_cf59c3da7e24499fa9f1d6860a534cd7.action_space
pyRender = env_cf59c3da7e24499fa9f1d6860a534cd7.render
# omitted code

Env.render(): Visualizing Gymnasium in Java

The Env.render() method is used to get the visual state of the environment, enabling the possibility to debug and verify each step taken as the state changes, from s0 to s1, s1 to s2 and so on. The PythonRuntime.callFunction() executes the render method from the Python runtime’s env variable. The fillFromNumpy() method transfers bytes to the native ByteBuffer, which as shown above, is created only once since it holds an image.

// Env.java
    public BufferedImage render() {
        try (var ndarray = callFunction(pyRender)) {
            if (renderMetadata == null) {
                renderMetadata = EnvRenderMetadata.fromNumpy(ndarray);
                imageBuffer = ByteBuffer
                        .allocateDirect(renderMetadata.size())
                        .order(ByteOrder.nativeOrder());
            }

            fillFromNumpy(ndarray, imageBuffer);

            return ImageFromByteBuffer.byteBufferToImage(
                    imageBuffer,
                    renderMetadata.width(),
                    renderMetadata.height(),
                    renderMetadata.channels() == 4
            );
        }
    }

_{CartPole execution inside Java, ignoring the done() method.}

EnvRenderMetadata.fromNumpy() handles the conversion of the NumPy array to EnvRenderMetadata, applying the appropriate treatment to produce the correct type and shape for DJL’s NDArray.

public class EnvRenderMetadata extends EnvStateMetadata {

    // omitted code
    static EnvRenderMetadata fromNumpy(PyObject arr) {
        var base = EnvStateMetadata.fromNumpy(arr);
        return new EnvRenderMetadata(
                base.shape,
                base.dtype,
                base.djlShape,
                base.djlType,
                base.size
        );
    }
    // omitted code
}

Env.reset(): Starting a Gymnasium Episode

Returning to the Env class, reset() must be called before invoking render() and step(), and is required after each finished episode⁴⁶. The code handles scalar and array observations, as each environment has its own rules, covering most cases.

// Env.java
    @Override
    public Pair<NDArray, Map<Object, Object>> reset() {
        this.stateMetadata = null;
        this.stateBuffer = null;

        try (var result = callFunction(pyReset)) {

            var pyState = getItem(result, 0);
            var infoMap = getItemMap(result, 1);

            if (!hasAttr(pyState, "shape")) {
                this.scalarObservation = true;
                long observationValue = toLong(pyState);
                var state = manager.create(observationValue);
                log.debug("Discrete observation: {}", observationValue);
                return new Pair<>(state, infoMap);
            }

            this.scalarObservation = false;
            this.stateMetadata = EnvStateMetadata.fromNumpy(pyState);
            this.stateBuffer = onHeapBufferNumpy(stateMetadata.size());

            fillFromNumpy(pyState, stateBuffer);

            var state = manager.create(
                    stateBuffer,
                    stateMetadata.djlShape,
                    stateMetadata.djlType
            );

            return new Pair<>(state, infoMap);
        }
    }

Env.step(): Executing Actions in Gymnasium

The Env.step() is responsible for advancing the state (e.g.: s₁ to s₂) and executing the action in the environment. Note that each value must be retrieved separately from the result using the getItem* methods, as the result is a Python tuple:

// Env.java
    @Override
    public EnvStepResult step(ActionResult action, NDManager manager) {
        try (var result = callFunction(pyStep, action.pyObj)) {
            NDArray state;

            if (scalarObservation) {
                var pyState = getItem(result, 0);
                long observationValue = toLong(pyState);
                state = manager.create(observationValue);

                log.debug("Discrete observation after step: {}", observationValue);
            } else {
                if (stateBuffer == null) {
                    throw new IllegalStateException("You should call reset() first!");
                }

                fillFromNumpy(getItem(result, 0), stateBuffer);
                state = manager.create(
                        stateBuffer,
                        stateMetadata.djlShape,
                        stateMetadata.djlType
                );
            }

            double reward = getItemDouble(result, 1);
            boolean terminated = getItemBool(result, 2);
            boolean truncated = getItemBool(result, 3);
            var infoMap = getItemMap(result, 4);

            return new EnvStepResult(reward, terminated, truncated, infoMap)
                    .state(state);
        }
    }

The following unit tests demonstrate that Env can handle multiple types of environments:

// GymActionSpaceTest.java
        @Test
        @DisplayName("MountainCar-v0 should have Discrete(3) action space")
        void testMountainCarActionSpace() {
            try (var env = Gym.make("MountainCar-v0", ndManager)) {
                // omitted code
                try (var action = env.actionSpaceSample()) {
                    assertEquals(DISCRETE, action.spaceType());
                    var result = env.step(action);
                    assertNotNull(result);
                    assertNotNull(result.state());
                    assertFalse(result.state().isReleased());
                    // omitted code
                }
            }
        }

        @Test
        @DisplayName("LunarLanderContinuous-v3 should have Box(2,) action space")
        void testLunarLanderContinuousActionSpace() {
            try (var env = Gym.make("LunarLanderContinuous-v3", ndManager)) {
                // omitted code
                try (var action = env.actionSpaceSample()) {
                    assertEquals(BOX, action.spaceType());
                    var result = env.step(action);
                    assertNotNull(result);
                    assertNotNull(result.state());
                    assertFalse(result.state().isReleased());
                    // omitted code
                }
                // omitted code
            }
        }

        @Test
        @DisplayName("FrozenLake-v1 should have Discrete(4) action space")
        void testFrozenLakeActionSpace() {
            try (var env = Gym.make("FrozenLake-v1", ndManager)) {
                // omitted code
                try (var action = env.actionSpaceSample()) {
                    assertEquals(DISCRETE, action.spaceType());
                    var result = env.step(action);
                    assertNotNull(result);
                    assertNotNull(result.state());
                    assertFalse(result.state().isReleased());
                    // omitted code
                }
                // omitted code
            }
        }

With the Gymnasium integration established and validated, the Rainbow DQN algorithm will be implemented in the next article, using these simple environments as a stepping stone before training the agent in the CARLA environment. The complete Java Reinforcement Learning Python Gymnasium integration code is available at java-rl-dqn-to-rainbow.

1. Link to the project. ↩︎
2. CARLA is an open-source simulator to train autonomous driving agents: link. ↩︎
3. CartPole environment: link. ↩︎
4. Mountain Car environment: link. ↩︎
5. DQN is Deep Q-Network. ↩︎
6. Improvements in the Deep Q-Network algorithm: link. ↩︎
7. Substack link. ↩︎
8. Isaac Lab: link. ↩︎
9. OpenAI Gym: link. ↩︎
10. Gymnasium: link. ↩︎
11. Umbrella Farama projects: link. ↩︎
12. Article link. ↩︎
13. Article link. ↩︎
14. Py4J link. ↩︎
15. PySpark link. ↩︎
16. ZeroMQ link. ↩︎
17. ZeroMQ is excellent, but you have to consider the trade-off between each scenario. ↩︎
18. Reliability pattern for ZeroMQ that handles reconnection and message retries, found at: link. ↩︎
19. FFM API link. ↩︎
20. JavaCPP link. ↩︎
21. JavaCPP’s CPython binding link. ↩︎
22. The FFM API tends to be faster because it doesn’t have the overhead of JNI, usually not relevant, but depends on your needs. ↩︎
23. jextract documentation link. ↩︎
24. CPython C-API documentation link. ↩︎
25. You can check the source code at link and limitations at link. ↩︎
26. javacpp-presets link. ↩︎
27. JavaCPP Gym binding link. ↩︎
28. JavaCPP ALE biding link. ↩︎
29. The gym library is platform independent, unlike CPython which is distributed as OS-dependent binaries. ↩︎
30. CPython github link, not JavaCPP’s bindings. ↩︎
31. The uv Python’s package manager’s link. ↩︎
32. DJL, acronym of Deep Java Library, a framework used in Deep Learning, with PyTorch, TensorFlow and JAX bindings, found at: link. ↩︎
33. NumPy is a numerical computing library and PyGame is for game development, both for Python. ↩︎
34. Segmentation fault: error raised when a program accesses invalid or unauthorized memory. ↩︎
35. Double free: error where a program frees the same allocated resource more than once, causing undefined behavior. ↩︎
36. Vector API: interface for running multiple independent environments in parallel and processing batched observations, actions, and rewards. ↩︎
37. CPython commit introducing a per-interpreter GIL via Py_NewInterpreterFromConfig() as part of PEP 684. ↩︎
38. More details at: link. ↩︎
39. The Python docs explain this in more detail: link. ↩︎
40. Buffer Protocol detailed explanation: link. ↩︎
41. Explanation of what’s contiguous memory: link. ↩︎
42. Endianness explanation: link. ↩︎
43. Lombok can be found at: link. ↩︎
44. These wrappers can be found at the original documentation: link. ↩︎
45. Further evidence can be found at the links from official documentation and real-python site. ↩︎
46. An episode is the period from when the environment starts until a terminated or truncated state returns. After that, call reset() to restart from the beginning. ↩︎

This article is part of the JAVAPRO magazine issue:

Autonomous Java

Explore how modern Java is evolving beyond frameworks and APIs toward intelligent, highly automated software systems. Discover why sustainable engineering, maintainable architectures, automation, semantic search, AI agents, and runtime optimization are becoming the foundation for the next generation of enterprise applications.

Discover the edition →