Spring Boot: Stream AI Responses with SSE to Deliver Progressive Chat UIs
Learn how Spring Boot can stream AI responses with SSE, unifying Ollama and Claude streams into a single contract to render progressive chat UIs and improve UX.
Spring Boot applications that call LLM providers often default to a synchronous request/response model: submit a prompt, wait for the model to finish, return a JSON payload. That approach works, but it obscures the model’s incremental output and creates a sluggish user experience. This article shows how to implement streaming AI responses in a Spring Boot backend, expose a purpose-built endpoint that forwards provider tokens as they arrive, and keep the service layer provider-agnostic so front ends can render text progressively. The pattern improves perceived performance and simplifies client integrations while keeping code readable and maintainable.
Why progressive streaming matters for AI-driven apps
Perceived latency is one of the first things users notice when interacting with conversational AI. Modern chat services show output as it is generated, which makes interactions feel responsive even when total generation time remains unchanged. For applications built on Spring Boot, treating streaming as an API-level feature — not merely a front-end trick — is essential. If the backend buffers the whole response, the client has no way to surface partial text or tokens. Exposing streaming at the API layer enables true progressive rendering, better accessibility for slow networks, and finer-grained UX patterns such as token-level typing indicators, partial synthesis, and intermediate semantic processing.
Designing a provider-agnostic streaming contract
One of the hardest engineering decisions when supporting multiple LLM vendors is preventing provider-specific wire formats from leaking into application code. Different providers stream differently: some emit newline-delimited JSON records, others relay upstream Server-Sent Events (SSE) or custom chunked formats. Instead of propagating those differences through controllers and services, define a simple contract at the client boundary: a synchronous chat method for non-streaming use, and a streaming variant that accepts a callback invoked for each text chunk. The rest of the application treats the stream as a sequence of text fragments, oblivious to how they were delivered. That approach reduces conditional code, simplifies tests, and keeps the controller focused on delivery semantics rather than parsing logic.
Adding a Spring Boot streaming endpoint
Expose a dedicated endpoint for streaming so existing JSON endpoints remain unchanged for clients that need whole responses. For streaming, use the text/event-stream media type and a server-side emitter that pushes events as they arrive. In Spring Boot, SseEmitter provides a lightweight abstraction for SSE; create an instance with an appropriate timeout and start the streaming work in a background thread. The controller should accept the validated prompt, create the emitter, invoke the service’s stream method with a callback that forwards chunks to the emitter, and complete or completeWithError once the provider stream ends or an exception occurs. The controller only handles SseEmitter events — the service and provider clients implement the text extraction, keeping concerns separated.
Parsing Ollama’s newline-delimited JSON stream
Some providers, like Ollama, stream one JSON object per line where each object contains a delta or response field. The client implementation should enable streaming in the request payload, read the response body line by line, parse each line as JSON, and extract the relevant text field. For each non-empty text fragment, call the onChunk consumer. This line-oriented parsing maps naturally to the callback model and can be implemented with a small utility that iterates through InputStream lines and emits parsed objects. By isolating this logic in the provider client, you avoid scattering JSON parsing and streaming state across the service layer.
Parsing Claude’s upstream SSE stream
Other providers use SSE to stream structured events that bundle event names and payloads. In that case the client needs to detect event boundaries, filter on the event name of interest, and decode any content deltas embedded in the event payload. The goal at the application level is still the same: extract successive text fragments and forward them via the shared onChunk callback. Implementing an SSE reader that understands event delimiters and aggregates multi-line data blocks lets the Claude client focus on mapping semantic event types to text, while the common contract ensures the downstream code only sees plain strings.
Shared plumbing without overengineering
It’s tempting to build an elaborate framework to support every conceivable streaming format, but often a small shared helper that abstracts iteration over JSON lines and iteration over SSE frames is enough. Implement utilities that provide an iterator-like interface over incoming events or lines and expose methods to register handlers. With these helpers in place, provider clients remain compact: they construct the upstream request, feed the raw stream to the helper, and map parsed events into text chunks. This minimizes duplication, reduces the cognitive load when inspecting clients, and makes it obvious where provider-specific parsing lives.
Frontend integration: why EventSource isn’t always the right choice
A common pitfall is attempting to use EventSource for POST-based streaming endpoints. EventSource is designed for persistent GET streams and does not support sending a request body. For POST /api/chat/stream endpoints, the correct front-end approach is to use fetch(), then read the response.body as a ReadableStream. Parse SSE frames from the stream and append each chunk to the view as they arrive. This grants full control over request payloads and headers while enabling efficient incremental rendering on the client. The pattern also works well with existing UI frameworks and progressive hydration strategies.
What the streaming endpoint should emit and how clients should consume it
At the application level, each streaming fragment should be a simple, well-typed event such as "chunk" with a text payload. That keeps the client-side parser trivial: extract the chunk’s text and append it to the transcript. Avoid bundling extraneous metadata in the streaming path; metadata like message IDs, usage tokens, or debug diagnostics can be emitted as distinct events or omitted from the live stream to prioritize smooth parsing and low latency. The client should also handle out-of-band control events (errors, completion signals) to present graceful fallback UI states like retry or partial result indicators.
Validation, request shapes, and API ergonomics
Replace unstructured request maps with clear request DTOs or records so Spring Boot can validate input automatically and generate accurate API documentation. A typed record for the prompt enforces constraints such as non-blank text via standard validation annotations and keeps controller code tidy. Use @Valid on controller parameters so invalid inputs return consistent ProblemDetail responses via a global exception handler. The global handler should map common exceptions—validation failures, downstream timeouts, and unexpected server errors—to user-friendly error payloads. These refinements make the API easier to adopt and reduce front-end edge cases.
Error handling and resilience in streaming paths
Streaming introduces different failure modes than synchronous APIs. The provider may terminate the connection mid-stream, or an intermediate network layer may drop packets. Implement timeouts and backpressure-aware readers in the client, and surface meaningful error events via the SSE channel so the front end can show partial content and offer a retry. For critical production systems, consider retry semantics for transient upstream failures, idempotency tokens for repeated prompts, and circuit breakers to avoid cascading faults. Instrument the streaming pipeline with metrics: number of chunks, average chunk size, stream duration, and failure rates. Those observability signals help tune provider selection and capacity planning.
Testing strategies for streaming features
Unit testing provider clients can use mocked InputStreams that simulate newline JSON or SSE frames to verify parsing behavior and chunk emission. End-to-end tests can start a local test server that emits controlled streams so the controller and emitter behavior are validated under realistic conditions. For load tests, simulate many concurrent streaming sessions and measure memory usage and thread model effects; SseEmitter and reactive stacks behave differently under high concurrency, so testing should inform choices like thread pools or using a reactive WebFlux alternative if necessary.
Performance and deployment considerations
Streaming reduces perceived latency but can increase connection lifetimes and resource usage because each client keeps an open stream. Ensure the server is configured to handle many concurrent long-lived connections: tune connection timeouts, request thread pools, and keepalives. Consider the tradeoffs between the Servlet-based SseEmitter approach and reactive implementations using WebFlux, which can be more efficient for very high connection counts. Also monitor provider costs because some vendors bill by tokens or time; progressive streaming does not necessarily increase total token count, but it can encourage longer sessions and more iterative prompts, affecting cost models.
Integration with related tools and ecosystems
Streaming is relevant across the AI toolchain: front-end frameworks (React, Vue) adopt streaming-friendly rendering patterns; automation platforms and CRMs can show intermediate responses in workflows; developer tools and observability platforms capture streaming telemetry to diagnose stalls. Make sure your OpenAPI documentation and developer portal describe both synchronous and streaming endpoints so integrators can choose the appropriate integration path. Streaming also plays well with server-side orchestration such as queuing partial results into downstream processors, or piping token deltas to real-time analytics and moderation services.
Broader implications for teams and product decisions
Treating streaming as an API-level capability influences product decisions beyond technical implementation. UX designers can prototype conversational interfaces that feel immediate, and product managers can explore features that consume partial outputs (e.g., live summarization, progressive translation). For engineering teams, a provider-agnostic contract protects business logic from vendor-specific wire formats and reduces future migration costs. It also encourages clearer SLAs around response latency and stream durability, and helps security teams reason about streaming-specific threats like injection across incremental payloads or event replay.
When to choose this pattern and who benefits most
This streaming architecture is most valuable when your application emphasizes interactive text generation: chat interfaces, coding assistants, dynamic content editors, or real-time summarizers. Teams building server-rendered apps with many small short-lived requests might not see the same benefit. Developer tooling, content platforms, and customer support chat solutions typically benefit the most. Any team that supports multiple LLM providers will gain the most from a unified streaming contract because it reduces integration surface area and allows switching providers without changing controller or service logic.
Operational checklist before enabling streaming in production
- Define clear request contracts and use validation annotations for incoming prompts.
- Add a global exception mapper that returns ProblemDetail for predictable client behavior.
- Implement shared stream parsing utilities to avoid duplicated parsing code.
- Instrument stream metrics (duration, chunk count, error rate).
- Harden the client with timeouts, retries, and circuit breakers.
- Choose a server model (Servlet+SseEmitter vs WebFlux) based on expected concurrency.
- Update API docs and developer guides to include streaming examples for fetch() consumption.
- Add end-to-end streaming tests that simulate provider behavior.
Streaming is both a UX and an API design decision; treating it as the latter pays dividends in clarity, maintainability, and developer ergonomics.
Forward-looking: as language models and provider ecosystems evolve, expect more standardization around streaming semantics and richer event types (structured deltas, semantic markers, and partial metadata). That will make provider-agnostic streaming contracts even more valuable, enabling server-side orchestration, moderation, and offline processing while preserving a snappy, token-by-token user experience in chat UIs. Continued investment in observability, graceful degradation, and clear API contracts will keep teams ready to adopt new providers and capabilities without rewriting core application logic.
