Data-Driven Projections with Kotlin Notebooks

Data-Driven projections help developers who organise work, estimate features and want a smarter way to see into the future. All the while staying inside their favourite IDE.

Why We Need Better Projections

In the before-fore times I was working in a team that stared at the same old backlog of work items, asking “Will we able to finish by next quarter?”.
We’d say “If everything goes well… maybe?” And then, surprise surprise: things didn’t go well. We’d have to drop scope while spending even more time in re-estimation meetings. New risks emerged. The cycle repeated.

This felt like a vicious cycle caused by “guesstimation”: vague dates -> faint hopes -> surprise late deliveries -> fatigue -> even fuzzier future dates.
Today, I know there’s a way out: we capture the data we already have, run simulations to see likely outcomes, visualise what’s actually going on, and build team confidence from what we know rather than what we hope.

Enter the realm of Monte Carlo simulations and Aging Charts.
These aren’t exotic maths for people with PhDs, they’re simply tools you can start using today, instead of doing estimation meetings.
With minimal overhead, we can turn “Will we finish on time?” into “There’s an 80% likelihood of finishing this batch of work by this date”.

In the rest of this article, I’ll share code snippets from a hands-on Kotlin Notebook, included in the accompanying GitHub repo, that reads in-production dates (when a work item was moved to done), runs simulations, and produces aging charts. We’ll walk through the why, the how, and the results, with ready to use Kotlin Notebooks to adapt to your situation.

Overview

We’ll start with explaining MonteCarlo simulations, which will help you replace estimation meetings.
After concluding with the Kotlin Notebook example for those simulations, we’ll explain Aging Charts and how you can use those.
Again concluding with a Kotlin Notebook example for generating Aging Charts.
Then we’ll go over some tips ‘n tricks from my personal experience, and a conclusion that sets you on the path to better predictability for you and your team.

Strap yourselves in!

Monte Carlo: You Only Need In-Production Data to Get Started

You don’t need perfect estimates or crystal-ball planning tools. What you do have, typically, is a history of when work items actually got done. If your team tracks work items (say, in Jira, YouTrack, Miro, or whatever) and records the “done” date, then you already have raw data for a forecast model.

By feeding that historical data into simulations that produce a probability distribution (a Monte Carlo simulation), you can project future outcomes for batches of work. The only “input” you need is this: for each work item its done date, and optionally its start date (giving you the cycle time). If you have that, you can ask: “Given our historical throughput, what’s the likely date by which the next N work items will be done?”.
It’s important to realise that it’s about the next batch of work items, not just an individual work item.

For many teams this is huge. You stop relying on gut-feels and optimistic guesses, and instead work with distributions and chance levels: “We’re 85% likely to finish these 20 work items by May 14th”. That gives you the language of risk, not hope.
Best thing is that you don’t need to do estimation meetings anymore that keep interrupting your productivity and feel like ritualistic and meaningless ceremonies.

What Monte Carlo Is Good For

Let’s pause for a moment on Monte Carlo. If you’ve heard of it, you might relate it to “mathy”, “finance” or “physics” territory. But at its heart, it’s quite simple: you model many possible futures by sampling from your historical distribution of outcomes. Then you summarise what those futures look like.

For example: if your team historically finishes 5 to 10 work items per week (average 7, but sometimes 5, sometimes 10), a Monte Carlo simulation will run many “weeks” sampled from that distribution, and say: in 80% of the runs you’ll finish 20 work items by X date.

In effect: You define your “unit” (could be work items per week, days per work item). You sample many runs (say 1,000 or 10,000) of how long it takes to finish the backlog you want to have a prediction for. You pick a percentile (e.g. 85% or 90%) as your safe forecast.

Quick note: the unit you choose will define the granularity of your forecast.
In the example notebook and in my current team, we’re using number of work items finished per day.
Most of the days we’ll have 0 work items finished, but we’d rather answer “when will it be done” than “how many weeks will it take to be done”.

In the repo you’ll find a Kotlin notebook that does exactly this. Here’s the data-cell that defines the MonteCarlo simulation:

import kotlin.random.Random

data class Trial(val totalDays: Int)

fun monteCarloFromThroughput(
    storiesInNextMilestone: Int = 13,
    trials: Int = 10_000,
): DataFrame<Trial> {
    val values = historicalThroughput.storiesDelivered.values().toList()

    val runs = List(trials) {
        var done = 0
        var days = 0
        while (done < storiesInNextMilestone) {
            done += values[Random.nextInt(values.size)] //some days we had delivered 2 stories, other days 1, most days 0
            days += 1
        }
        Trial(days)
    }
    return runs.toDataFrame()
}

You’ll find more polished code in the notebook: filtering out zero‐throughput days, selecting more relevant time windows, etc.

//The data-cell in the notebook that generates the graph below:
import org.jetbrains.kotlinx.statistics.distribution.NormalDistribution
import org.jetbrains.letsPlot.Stat
import org.jetbrains.letsPlot.core.spec.plotson.BinStatOptions

val distributionBarChart = plot(frequencyPerTotal) {
    layout { size = 2000 to 900 }
    bars {
        x(totalDays)
        y(frequency)
        alpha = 0.6
    }
    // vertical markers
    vLine { 
        xIntercept.constant(p50)
        type = LineType.DASHED
    }
    vLine { 
        xIntercept.constant(p85)
        type = LineType.DASHED
    }
}
distributionBarChart

Why this Matters

You get probabilistic forecasts, not a single deterministic “We’ll be done on June 1”. Which enables you to visualise risk (e.g. at 50% you finish by May 1; at 90% by June 15). Based on data! Yay!

Teams that use this language stop committing to “we’ll finish by X” and start saying “We are likely to finish by X unless something changes”. This changes behaviour, expectations and trust.

Furthermore, it’s very cheap to get going: just your historical done‐dates, Kotlin notebook, maybe a CSV import. You don’t have to deal with “getting experienced in doing estimation meetings”, or phrases like “can you re-estimate this because I’ll have a hard time selling this to our customer”, or some other similar nonsense.

Common Questions

Q: Don’t you include weekends and holidays in your simulations?
A: Good observation. That is indeed the point. We want realistic simulations. In reality, we also don’t work weekends or during holidays, right?

Q: We’re only starting and have a very limited amount of data. Will this be useful to us?
A: Without going into too much detail, you can get meaningful predictions from data of having completed 5 work items. Read “When will it be done” for proper info.

Q: We just hired 2 new junior members. Won’t this impact our predictability?
A: It sure will! Good thing you can easily select data that is relevant to your situation. 😉

Q: Do all our work items need to have the same size?
A: No, but I will say that your prediction accuracy gets a boost when they are similar in size.

Actually, this last part could use a bit more of an explanation.

Goodhart’s Law

“When a measure becomes a target, it ceases to be a good measure”

Charles Goodhart

Applied to work items having the same size that law could look like this:
We know this work item will take longer than the others, so we’ll split it up into 2 work items. That way, “we’ll get more work done and our metrics will look better”.

If you split up the bigger work item into 2, and those 2 work items still manage to deliver value individually, then this is great! You’ve now managed to focus more on finishing WIP and delivering value more quickly.

If they don’t deliver value individually, then it’s indeed more likely Goodhart’s Law in practice.
“Deliver value” is a pretty broad spectrum, however:

• Learning about technical issues you face: value++
• Learning about how your users interact with an experimental feature: value+++
• A new partial feature already drives up sales: value++++

So, try to initially define work items that are similar in size. It’s ok if they’re not, and it’s better if you split them up when they end up delivering value individually.

Pro-tip: keep track of how many times work items were split up, and use this data in your MonteCarlo model to get even better predictions!

Practice Time!

If you’re following along with the accompanying repository, now is a good time to check out and play around with 01_DataExploration.ipynb, 02_MonteCarlo.ipynb and 03_When_Will_It_Be_Done.ipynb.

There’s a tricky bit you “just need to know” when working with Kotlin Notebooks:

Type inference on dataframes happens when a data-cell has run. This is why it’ll be important to put your transformations into one data-cell and then continue working with them in a next data-cell.
That’s also the reason why sometimes it’s better to interrupt chaining calls on your Dataframe. The classic example is filtering out null values, and still having to deal with the type being a nullable type in the next chain call.

Come back to this article when you’re done playing around to learn about another important component of working with your data: getting fast feedback to adjust your process using Aging Charts!

Aging Charts: What They Are & How to Build Them

If Monte Carlo gives you “when will we finish X amount of work items”, then Aging Charts tell you how old the finished work typically gets.
They answer: “Given the current Work In Progress, which items are getting old?”.
It’s the equivalent to burn-down charts when you’re working in sprints: “Do we need to do something to finish work quicker?”.

In practice, an aging chart is a scatter plot of the age of each active work item versus time. Over time, items that are not closing will drift up (age increases). If you overlay your typical cycle time, you can see which items are “stuck”.

To be able to run useful Aging Charts, in addition to the “done date” you needed for MonteCarlo’s, you will now also need the “start date”.

If you want an even more detailed overview you can add all the dates of “the value stream”: basically all the steps a work item goes through in your team. For example when we pull in a work item in our backlog, it’ll go through analysis (where we know when this started and finished), development, testing, when it was deployed to acceptance and when to production, or if it was abandoned (no longer relevant, we chose a different option, …).

In the repo you’ll find a Kotlin notebook that loads your work items, computes cycle times, and plots the distribution over time showing outliers more easily. The relevant snippet that calculates the Cycle Times looks like this:

val offsetWhenFinishedSameDay = 1

val valueStream = cleaned
    .add("AnalysisTime") {
        `In Analysis`.date.daysUntil(Analyzed.date) + 
                offsetWhenFinishedSameDay
    }
    .add("CycleTime") {
        `In Analysis`.date.daysUntil(`In Production`.date) + 
                offsetWhenFinishedSameDay
    }
    .select("StoryId", "AnalysisTime","CycleTime")

Just like the MonteCarlo notebooks, you’ll see it uses DataFrame to do data exploration, filtering and transformation, along with Kandy, a simple chart library to visualise what the mean (50 percentile) and 85th percentiles are for work items that are being analysed (actively in analysis) and work items that are being developed.

//The datacell that generates the scatterplot below:
val agingChart = plot(valueStream) {
    layout { size = 2000 to 900 }
    points {
        y(CycleTime)
        x(StoryId)
    }
    // vertical markers
    hLine {
        yIntercept.constant(percentiles["p50CycleTime"])
        type = LineType.DASHED
    }
    hLine {
        yIntercept.constant(percentiles["p85CycleTime"])
        type = LineType.DASHED
    }
}
agingChart

Why this Matters

It’s the tool that gives you the earliest possible feedback for on-going work

More specifically it provides early warning for “drift”: It’ll become obvious which work items need more attention to progress on.

It helps you wake the team up: when you show “Today most of our open work items are 45 days old, up from 30 last quarter”, it hits different.

It ties nicely into Monte Carlo: if you have a bunch of very old items, your future projections should probably need to account for that latent work (a.k.a. drag; work in progress that significantly impacts todays projections).

More Practice Time!

Time to explore! This time you should figure out 04_AreWeGoodBusy.ipynb.

Bringing It Together: From Data to Insight to Action

Putting it all in your toolkit means you can run a short “forecasting ritual” each iteration, backlog refinement meetings, or at roadmap reviews. It might go like this:

Export your list of done-dates for the last N weeks (e.g., last 12–24 weeks).

Load it into your Kotlin notebook that’s supposed to live in your projects’ code repository.

Run a Monte Carlo simulation for “X work items remaining” in order to get finish date distribution.

Run aging charts to visualise age distribution, so you can highlight “old” items in your work item dashboard (e.g. “danger zone”, “about to retire”, …).

Interpret results in team meeting: “If everything remains the same, we have an 80% chance of finishing by June 10.”. Or during daily stand-ups: “We have 1 item that’s nearing the 50th percentile. What options can we come up with to do something about it?”

As a team, decide action: maybe stop starting new work items, focus on clearing old ones (by swarming on it), adjust expectations with stakeholders, update roadmap, …

Assign a different team member to follow up these metrics for next meetings, so that knowledge spreads inside the team.

Here’s what a simple Kanban board could look like in Miro, where the aging of work items is displayed on the black post-its. If you use these in combination with your Aging Charts, you’ll be able to act more easily on items that are getting “too old”.

Why Kotlin Notebooks?

Many teams use spreadsheets, or maybe built-in work-tracking tools for forecasting. So why use Kotlin notebooks?

The existing work-tracking tools typically don’t offer transparency on how they got these charts and predictions. If you build it yourself, you also own the process yourself, because you understand the why and how.

The language is familiar to you as a developer. No new platform to learn.

Kotlin notebooks let you mix code, comments, charts in one place. The narrative stays with the code.

Since it’s code, it’s easier for developers to start caring about process. 😉

You can version-control your notebook with your git repo. The commits will tell a story about the evolution of your codebase AND your development process.

You can customise: connect to your own backlog system, fetch data automatically (with minimal API usage), then run the notebook.

It supports re-use: once set up, each time you need a new prediction you just click a few cells, refresh your data, regenerate charts. Minimal. Backend. Cost. You’re welcome.

Kotlin notebooks are readable in your remote repository (i.e. GitHub or GitLab), allowing for integration into slides or management dashboards.

Tips & Pitfalls from Experience

Read “When will it be done” by Daniel Vacanti to learn much more about data-driven projections than the surface I barely scratched in this article.

Little’s Law indicates the only thing you have control over is Work In Progress, so minimise it as much as you can!

That means don’t pick up new work when you already have too much WIP; improve your skills instead.

Data hygiene matters: remove entries that aren’t “production done” (e.g., closed across QA but not released). Use consistent definitions.

Exclude outliers: if you had one 120-day work item, it will skew your distribution too much without being relevant. Consider filtering or grouping by item size.

Use stable units: Pick throughput units that make sense (work items per week or per day). Be consistent!

Beware “scope creep” if you picked work items per week as the unit: If you keep adding work mid‐sprint, your historical data may under‐represent actual throughput. Monte Carlo’s usefulness requires that the “same process” continues.

Visualise the tail: Often the 90-95th percentile is what matters for risk communication (“we finish by X only after two standard deviations”).

Communicate well: Replace “we’ll finish by June 1” with “we’re 80% likely to finish by June 1, 90% by June 20”. Stakeholders appreciate transparency. If you have to dumb it down, just give ranges: “We will likely be done between June 1st and June 20th”.

Label your work items with how close they are to the 50th and 85th Aging percentiles, so that it’s super obvious in your visualisation of your work (i.e. a KanBan or Scrum-board) which work items need attention.

Make it a habit: Treat the notebook as part of your iteration start or backlog-triage ritual. Don’t let it sit idle.

And finally a Kotlin Notebooks caveat: kotlinx:dataframe is still in development at the time of writing, but the development team is nearing a stable 1.0.0 release. And much like every library dependency, you might want to lock down the current version, or keep on running on the bleeding edge (and report bugs to the dev team).

Conclusion

If you’ve ever felt the pain of over-optimistic planning, surprise delays, and frustrated stakeholders, you know how easily estimation can become a vicious cycle.
By adopting a simple data-driven approach (using historical done-dates), running Monte Carlo simulations in a Kotlin notebook, and visualising backlog drift with Aging Charts, you can switch to a virtuous cycle in which you measure, visualise, decide, act and improve.

The repository “data-driven-projections” gives you a low-cost, high-leverage toolkit. Grab it, adapt it, run it.

The future doesn’t have to be guesswork, it can be probabilistic insight. And for a feature team, that kind of insight is gold.

Resources

The Data-Driven Projections GitHub Repository, containing the Kotlin notebook examples
When Will It Be Done – Daniel Vacanti
FlowMetrics and Little’s Law
Kandy Docs
DataFrame Docs

Getting help

The #notebooks channel on the kotlinlang slack has very helpful people (including the Notebooks development team) who will gladly help you along on your journey!

This article is part of the JAVAPRO magazine issue:

Engineering Intelligence – Building Systems in the AI Age

Explore what engineering means in the AI age — beyond code generation and automation.
Gain insights into performance-critical Java systems, evolving architectures, and the responsibilities that come with building intelligent software.

Discover the edition →