From Prototype to Production: Building Safe and Reliable AI Agents with Spring AI and MCP

The AI agent demo works perfectly in the development environment. Your prototype handles natural language queries, executes tool calls, and generates sensible responses. But when stakeholders ask ‘Can we deploy this?’, reality sets in. That question exposes the massive gap between a functional prototype and a Spring AI production system ready for real traffic, unexpected failures, and business-critical operations.

That question reveals the massive gap between a working prototype and a production system. The demo doesn’t handle failures gracefully. It has no guardrails against harmful outputs. There’s no visibility into what the agent is actually doing. Cost per request? Unknown. Error rates? No idea. Recovery strategy when the LLM provider goes down? Haven’t thought about it.

This article walks through the journey from prototype to production-ready AI agent using Spring AI and the Model Context Protocol (MCP). I’ll cover architecture decisions, safety mechanisms, failure recovery patterns, and observability requirements that separate demos from deployable systems. If you’re building AI features in a Java shop, this is the practical guide you need.

Spring AI Production Requirements: What Changes

Production AI agents face constraints that prototypes ignore. Here’s what changes:

• Reliability requirements jump from “mostly works” to “five nines”. When your agent is booking actual flights or processing refunds, failure isn’t an option. You need retry logic, circuit breakers, and fallback strategies. The LLM provider will have outages, therefore your system needs to handle them gracefully.

• Cost monitoring becomes essential and you need to track token usage and set per-user budgets to avoid surprises

• Safety isn’t optional. Your agent will receive adversarial inputs like prompt injection attempts, requests to leak system prompts, attempts to bypass policies. Users will try to extract training data or generate harmful content. Input validation and output filtering become critical infrastructure, not nice-to-haves.

• Observability shifts from debugging tool to operational necessity. When something goes wrong at 2 AM, you need to know what the agent was trying to do, which tools it called, what context it had, and where it failed. Logs, metrics, and traces aren’t optional at this time, they’re how you keep the system running.

• Compliance and auditability matter. In regulated industries, you need to demonstrate what data the agent accessed, what decisions it made, and why. That requires structured logging, audit trails, and the ability to replay interactions.

Moving to Spring AI production means addressing concerns that prototypes ignore: reliability, cost, safety, and observability. These aren’t optional features, they’re the foundation of any system that needs to run reliably under real-world conditions.

Foundation for Spring AI Production: Why Spring AI and MCP

Why Spring AI for Production Agents

Spring AI hit 1.0 GA in May 2025, marking its transition from experimental project to production-ready framework. If you’re already running Spring Boot applications, Spring AI gives you AI capabilities with familiar patterns.

The framework provides provider-agnostic abstractions. Your code talks to ChatClient, not “OpenAI’s specific API” or “Anthropic’s particular format”. When you need to switch from OpenAI to Claude, or add a fallback to Ollama for cost optimization, you’re changing configuration, not refactoring core logic. This matters when you’re operating under real-world constraints where provider choice depends on cost, latency, and availability.

Observability comes built-in through Micrometer integration. Token usage, latency, error rates, these all flow into your existing monitoring stack. If you’re already using Spring Boot Actuator with Prometheus and Grafana, your AI components report alongside everything else. No separate observability platform needed.

The Advisor pattern lets you inject behavior around every LLM call. If you want to log all prompts, then you need to add an advisor. If you need to count tokens before and after, then you need another advisor. Implementing a simple cache? Advisor again. This composable middleware approach means adding production concerns doesn’t require touching your core agent logic.

The Model Context Protocol (MCP): Standardizing Tool Integration

The Model Context Protocol, introduced by Anthropic and now an open standard, solves a real problem in AI development: every application was rebuilding the same integration patterns. Need to give your agent file system access? Build custom functions. Database queries? More custom code. Web search? Yet another integration.

MCP standardizes this. It defines how AI applications discover and use external tools, resources, and data sources. Instead of every agent implementing its own file system access, you configure an MCP server that exposes filesystem operations. That same MCP server works with Claude Desktop, VS Code extensions, and your Spring AI agent. Write the integration once, use it everywhere.

The protocol uses a client-server architecture similar to Language Server Protocol. MCP servers expose capabilities like tools, resources, and prompts. MCP clients discover these capabilities and make them available to LLMs. The communication happens over stdio (for local processes) or SSE/HTTP (for remote services).

Spring AI’s MCP support, added in recent milestone releases, provides both client and server implementations. On the client side, you configure connections to MCP servers and Spring AI automatically exposes their tools to your chat model. On the server side, you can expose your own business logic as MCP tools using simple annotations. This bidirectional support means you can both consume existing MCP servers and create new ones for your domain-specific needs.

Building the Foundation: Basic Agent Setup

Let’s start with a working agent, then progressively add production concerns. I’ll build a customer support agent that can look up orders, check shipping status, and process returns.

Project Setup

Head to start.spring.io and configure:

• Spring Boot 3.5.1
• Java 21+ (virtual threads help with concurrent tool calls)
• Dependencies: Spring Web, Spring AI OpenAI, Spring Boot Actuator
• Add MCP Client starter: spring-ai-starter-mcp-client

Your pom.xml should include:

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-openai-spring-boot-starter</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-mcp-client</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

Add Spring AI milestone repository:

<repositories>
    <repository>
        <id>spring-milestones</id>
        <name>Spring Milestones</name>
        <url>https://repo.spring.io/milestone</url>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
    </repository>
</repositories>

Basic Agent Implementation

Start with the simplest possible agent, one that can answer questions using an LLM:

@Service
public class SupportAgent {
    private final ChatClient chatClient;

    public SupportAgent(ChatClient.Builder builder) {
        this.chatClient = builder
                .defaultSystem("You are a helpful customer support agent.")
                .build();
    }

    public String handleQuery(String userMessage) {
        return chatClient.prompt()
                .user(userMessage)
                .call()
                .content();
    }
}

This works, but it’s useless for real support tasks. The agent can chat but can’t look up orders, check inventory, or take actions. It requires additional tools to achieve all of these.

Adding Tools: MCP Integration

Tools are how agents interact with external systems. Spring AI supports two approaches: defining tools directly in code, or connecting to MCP servers that expose tools. We’ll use both.

Creating Custom Tools

For internal business logic, define tools as plain methods:

@Component
public class OrderTools {
    
    @McpTool(description = "Look up order details by order ID")
    public OrderDetails getOrder(
        @McpToolParam(description = "Order ID to look up") 
        String orderId
    ) {
        return orderRepository.findById(orderId)
            .orElseThrow(() -> new OrderNotFoundException(orderId));
    }
    
    @McpTool(description = "Get shipping status for an order")
    public ShippingStatus getShippingStatus(
        @McpToolParam(description = "Order ID") 
        String orderId
    ) {
        Order order = getOrder(orderId);
        return shippingService.getStatus(order.getTrackingNumber());
    }
}

The @McpTool annotation marks methods that should be exposed to the agent. Spring AI automatically generates function schemas and handles parameter marshaling. The LLM sees these as callable functions with typed parameters.

Wire these into your agent:

@Service
public class SupportAgent {
    private final ChatClient chatClient;
    
    public SupportAgent(
        ChatClient.Builder builder,
        ToolCallbackProvider toolProvider
    ) {
        this.chatClient = builder
            .defaultSystem("You are a customer support agent. " +
                          "Use available tools to help customers.")
            .defaultTools(toolProvider)
            .build();
    }
    
    public String handleQuery(String userMessage) {
        return chatClient.prompt()
            .user(userMessage)
            .call()
            .content();
    }
}

Now the agent can call your order lookup functions. When a user asks “What’s the status of order ABC123?”, the LLM will recognize it needs the getShippingStatus tool, extract the order ID from the query, and call your method.

Connecting to External MCP Servers

For capabilities like web search or file system access, use existing MCP servers. Configure them in application.yml:

spring:
  application:
    name: ai-agent-demo
  ai:
    openai:
      api-key: ${OPENAI_API_KEY}
      chat:
        options:
          model: gpt-4
          temperature: 0.7
    mcp:
      client:
        stdio:
          connections:
            brave-search:
              command: npx
              args:
                - "-y"
                - "@modelcontextprotocol/server-brave-search"
              env:
                BRAVE_API_KEY: ${BRAVE_API_KEY}

Spring AI launches the MCP server as a subprocess, connects over stdio, discovers available tools, and automatically makes them available to your agent. No additional code needed because the configuration is enough.

Production Safety: Implementing Guardrails for Spring AI Agents

Your agent now has capabilities, which means it has ways to cause damage. Guardrails aren’t about limiting functionality, they’re about preventing harm while preserving utility. Think of them as defensive programming for AI systems.

Input Validation: Stopping Bad Requests Early

The first line of defense happens before any LLM call. Validate inputs to prevent prompt injection, detect out-of-scope requests, and catch obvious bad inputs:

@Component
public class InputGuardrail implements RequestAdvisor {
    
    private final Set<String> blockedPatterns = Set.of(
        "ignore previous instructions",
        "system prompt",
        "you are now",
        "disregard all prior"
    );
    
    @Override
    public AdvisedRequest adviseRequest(
        AdvisedRequest request, 
        Map<String, Object> context
    ) {
        String userMessage = request.userText();
        
        // Check for prompt injection attempts
        for (String pattern : blockedPatterns) {
            if (userMessage.toLowerCase().contains(pattern)) {
                throw new PromptInjectionException(
                    "Request blocked: potential prompt injection"
                );
            }
        }
        
        // Validate scope
        if (!isInScope(userMessage)) {
            throw new OutOfScopeException(
                "I can only help with order and shipping questions"
            );
        }
        
        return request;
    }
    
    private boolean isInScope(String message) {
        // Custom implementation for scope validation
        return true;
    }
}

This runs before every LLM call. Simple pattern matching catches most amateur prompt injection attempts. Sophisticated attackers will bypass this, but it stops casual probing and keeps your logs cleaner.

Output Filtering: Preventing Harmful Responses

Even with good prompts, LLMs sometimes generate problematic content. Filter outputs before returning them, like in the example below:

@Component
public class OutputGuardrail implements ResponseAdvisor {
    
    private final ChatClient moderationClient;
    
    // Regex patterns example for PII detection
    private static final Pattern EMAIL_PATTERN = 
        Pattern.compile("[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}");
    private static final Pattern PHONE_PATTERN = 
        Pattern.compile("\\b\\d{3}[-.]?\\d{3}[-.]?\\d{4}\\b");
    private static final Pattern SSN_PATTERN = 
        Pattern.compile("\\b\\d{3}-\\d{2}-\\d{4}\\b");
    
    public OutputGuardrail(ChatClient.Builder builder) {
        this.moderationClient = builder.build();
    }
    
    @Override
    public ChatResponse adviseResponse(
        ChatResponse response,
        Map<String, Object> context
    ) {
        String content = response.getResult()
            .getOutput().getContent();
        
        // Check for PII leakage
        if (containsPII(content)) {
            return sanitizePII(response);
        }
        
        // Use LLM-based moderation for complex cases
        if (needsModeration(content)) {
            // Simplified example
            if (!isSafeContent(content)) {
                return createFallbackResponse();
            }
        }
        
        return response;
    }
    
    private boolean containsPII(String content) {
        return EMAIL_PATTERN.matcher(content).find() ||
               PHONE_PATTERN.matcher(content).find() ||
               SSN_PATTERN.matcher(content).find();
    }
    
    private ChatResponse sanitizePII(ChatResponse response) {
        String content = response.getResult().getOutput().getContent();
        
        // Redact PII
        content = EMAIL_PATTERN.matcher(content).replaceAll("[EMAIL REDACTED]");
        content = PHONE_PATTERN.matcher(content).replaceAll("[PHONE REDACTED]");
        content = SSN_PATTERN.matcher(content).replaceAll("[SSN REDACTED]");
        
        // Create new response with sanitized content
        Generation sanitized = new Generation(new AssistantMessage(content));
        return new ChatResponse(List.of(sanitized), response.getMetadata());
    }
    
    private boolean needsModeration(String content) {
        // Check if content might need moderation
        // Simple heuristic: check for potentially sensitive keywords
        String lower = content.toLowerCase();
        return lower.contains("password") || 
               lower.contains("credit card") ||
               lower.contains("secret");
    }
    
    private boolean isSafeContent(String content) {
        // Custom implementation for scope validation
        return true;
    }
    
    private ChatResponse createFallbackResponse() {
        String fallbackMessage = 
            "I cannot provide that information. " +
            "Please contact our support team directly for assistance.";
        
        Generation fallback = new Generation(new AssistantMessage(fallbackMessage));
        return new ChatResponse(List.of(fallback));
    }
}

Output filtering is a chance to catch problems. You can use regex for obvious patterns (credit card numbers, SSNs), but consider LLM-based moderation for nuanced cases. OpenAI’s moderation endpoint is fast and free.

Failure Recovery in Spring AI Production Systems

LLM providers have outages, APIs timeout, and tools throw exceptions. Your agent needs to handle failure at every level. The goal isn’t perfection thought, it’s graceful degradation. When something breaks, the system should degrade to a simpler, but still functional state rather than failing completely.

Retry Logic with Exponential Backoff

Most LLM failures are transient. Rate limits, temporary network issues, brief provider outages, all of these resolve quickly. One option is to implement smart retries and below you can find an example:

@Configuration
public class ResilienceConfig {
    
    @Bean
    public RetryTemplate aiRetryTemplate() {
        RetryTemplate template = new RetryTemplate();
        
        ExponentialBackOffPolicy backoff = 
            new ExponentialBackOffPolicy();
        backoff.setInitialInterval(1000);  // 1 second
        backoff.setMultiplier(2.0);
        backoff.setMaxInterval(10000);     // 10 seconds
        
        SimpleRetryPolicy retryPolicy = 
            new SimpleRetryPolicy();
        retryPolicy.setMaxAttempts(3);
        
        template.setBackOffPolicy(backoff);
        template.setRetryPolicy(retryPolicy);
        
        return template;
    }
}

Wrap your chat client calls:

public String handleQuery(String message) {
    return retryTemplate.execute(context -> {
        return chatClient.prompt()
            .user(message)
            .call()
            .content();
    });
}

Three retries with exponential backoff handles most transient failures. You don’t need to retry forever, if after three attempts, for example, the failure is persistent, then you need a different strategy.

Circuit Breakers: Failing Fast

When a provider is consistently failing, stop hammering it with requests. Use Resilience4j circuit breakers:

@Configuration
public class CircuitBreakerConfiguration {
    
    @Bean
    public CircuitBreaker aiCircuitBreaker() {
        CircuitBreakerConfig config = CircuitBreakerConfig.custom()
            .failureRateThreshold(50)
            .waitDurationInOpenState(Duration.ofSeconds(30))
            .slidingWindowSize(10)
            .build();
        
        return CircuitBreaker.of("ai-provider", config);
    }
}

@Service
public class SupportAgent {
    
    private final ChatClient chatClient;
    private final RetryTemplate retryTemplate;
    private final CircuitBreaker circuitBreaker;
    
    public SupportAgent(
        ChatClient.Builder builder,
        ToolCallbackProvider toolProvider,
        RetryTemplate retryTemplate,
        CircuitBreaker circuitBreaker
    ) {
        this.chatClient = builder
            .defaultSystem("You are a customer support agent. " +
                          "Use available tools to help customers.")
            .defaultTools(toolProvider)
            .build();
        this.retryTemplate = retryTemplate;
        this.circuitBreaker = circuitBreaker;
    }
    
    public String handleQuery(String message) {
        return circuitBreaker.executeSupplier(() -> {
            return retryTemplate.execute(context -> {
                return chatClient.prompt()
                    .user(message)
                    .call()
                    .content();
            });
        });
    }
}

After 50% failure rate over 10 requests, the circuit opens. New requests fail immediately for 30 seconds, giving the provider time to recover. This prevents cascading failures and reduces wasted resources.

Spring AI Production Observability: Metrics and Logging

Observability separates production systems from prototypes. When your agent fails or behaves unexpectedly, you need to understand why. This requires metrics to measure system health over time, structured logs to capture what happened during specific interactions, and distributed traces to follow requests through your entire stack.

Metrics: Quantitative Health

Spring AI integrates with Micrometer, so metrics flow into your existing monitoring. Enable observability in application.yml:

spring:
  ai:
    chat:
      client:
        observations:
          include-prompt: true
          include-completion: true

management:
  endpoints:
    web:
      exposure:
        include: health,metrics,prometheus
  metrics:
    export:
      prometheus:
        enabled: true

Key metrics to track:

• Token usage (input and output separately) – directly maps to cost
• Latency percentiles (p50, p95, p99) – user experience indicator
• Error rates by type (timeout, rate limit, validation failure)
• Tool invocation counts – which capabilities get used
• Circuit breaker state changes – early warning system

Set up Grafana dashboards to visualize these metrics in real time. Configure alerts that trigger when error rates spike above acceptable thresholds, token usage approaches your daily budget limits, or request latency degrades beyond your SLA targets. These metrics provide early warning signals that let you identify and fix problems before they impact users or generate complaints.

Structured Logging

Logs explain what happened during specific interactions. Use structured logging with context fields like conversation ID, user ID, and request metadata. This contextual information makes debugging dramatically easier because instead of searching through unstructured text, you can filter logs by specific conversations or users to understand exactly what the agent was doing when problems occurred.

It is important to implement logging at key points in the request lifecycle: before calling the LLM (to capture the prompt and available tools), after receiving the response (to record token usage and tool calls), and during any error conditions.

You can use Spring’s MDC (Mapped Diagnostic Context) to attach conversation-level context to all log entries within a request, then clear it when the request completes. This approach ensures every log line carries enough context to be useful during incident investigation.

Distributed Tracing

Traces show the complete request path through your system. Spring AI’s Micrometer integration creates spans automatically. Enable OpenTelemetry:

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>

Configure in application.yml:

management:
  tracing:
    sampling:
      probability: 1.0  
  otlp:
    tracing:
      endpoint: http://localhost:4318/v1/traces

Distributed tracing reveals the complete flow of each request through your system, making performance bottlenecks immediately visible. Spring AI’s Micrometer integration automatically creates spans for LLM calls and tool invocations, showing you where time is actually spent. You’ll often discover that 90% of latency comes from slow tool calls rather than LLM inference, or that your retry logic is making problems worse by repeatedly attempting operations that are failing permanently. Traces transform invisible performance issues into concrete, measurable problems you can actually fix.

Lessons Learned and Key Takeaways

Building production AI agents is less about the AI itself and more about the infrastructure around it. The LLM is actually the easy part especially as the models improve every month, APIs are straightforward, and frameworks like Spring AI handle most of the complexity. The real challenge lies in making your agent reliable, safe, and observable under real-world conditions.

The most important lesson from production deployments is to start with safety, not features. Before adding capabilities, implement guardrails. Your first advisor should validate inputs and filter outputs. Add monitoring before expanding what the agent can do. This inverted approach, to building safety infrastructure first, then layering intelligence on top, prevents the painful rewrites that happen when security becomes an afterthought.

Treat LLM calls exactly like external API calls, because that’s what they are. They timeout under load, hit rate limits during traffic spikes, and fail when providers have outages. Implement retries with exponential backoff, use circuit breakers to fail fast during extended outages, and design fallback strategies for when primary providers are unavailable. Architectural patterns for distributed systems apply directly to AI agents.

The Model Context Protocol changes how we think about tool integration. Instead of writing custom code for every capability your agent needs (filesystem access, database queries, web search) you configure MCP servers and let Spring AI discover and expose their tools automatically. This standardization reduces maintenance burden and makes testing straightforward. Write the integration once as an MCP server, and it works across every MCP compatible application.

Remember that AI agents are still software, and Spring Boot’s core strength is building reliable, maintainable systems at scale. Apply the same engineering discipline to your AI features that you apply to your REST APIs and microservices. The technology might be new, but the requirements haven’t changed. Production systems need observability, graceful failure handling, and defense in depth. Focus on these fundamentals, and your AI agent will scale alongside the rest of your application.

References and Further Reading

Spring AI Official Documentation – https://docs.spring.io/spring-ai/reference/

Model Context Protocol Specification – https://modelcontextprotocol.io/docs/getting-started/intro

Spring AI MCP Guide – https://www.baeldung.com/spring-ai-model-context-protocol-mcp

Spring AI Observability Deep Dive – https://bootcamptoprod.com/spring-ai-chat-client-metrics-guide/

AI Guardrails in Production – https://www.wiz.io/academy/ai-security/ai-guardrails

Building MCP Servers with Spring AI – https://www.danvega.dev/blog/2025/09/24/cyc-mcp-server-spring-ai

Spring AI Advisors Explained – https://medium.com/@nikil-kumar/understanding-spring-ai-advisors-the-secret-layer-behind-smarter-llm-applications-717a1b429530

This article is part of the JAVAPRO magazine issue:

From AI as a Feature tu AI as Infrastructure

Move beyond AI experimentation and into AI engineering.
Explore the architectures, platforms, and operational practices required to build trustworthy AI systems at scale. From governance and observability to modern Java infrastructure, this edition examines the foundations of production-ready AI.

Discover the edition →