mq-bridge: Build a portable Rust remote job system that starts with a file and scales to NATS
Build a remote job system with mq-bridge: define Rust job structs once, start with a file backend for development, then switch to NATS via configuration.
A practical pattern for remote jobs with mq-bridge
Every backend will eventually need to hand work off to another process — parsing files, sending emails, generating reports — jobs that must not block a main service and may run on different machines. mq-bridge offers a transport-focused approach for Rust applications that keeps your business logic in plain Rust types while letting you swap the delivery mechanism by changing configuration instead of rewriting handlers. This article walks through a working pattern from local development with a file-backed queue to a production-ready NATS JetStream deployment, explains what mq-bridge provides out of the box, and shows how to test handlers without external infrastructure.
Defining jobs as plain Rust structs
The foundation of this approach is to represent each background task as a simple Rust struct. Instead of coupling logic to a specific queue API, you model the job payload with Serde-serializable types — for example, a SendEmail struct with fields for recipient and subject, and a GenerateReport struct with a user_id. Each struct also exposes a static identifier string (a kind) that the messaging layer uses to route and decode messages.
Keeping jobs as plain data types separates concerns: the job definitions and handler logic stay in your codebase, fully typed and compiled, while the transport — file, memory, NATS, or a database-backed queue — is treated as a replaceable detail.
Registering typed handlers with mq-bridge
mq-bridge maps incoming messages to handlers by type. In worker code, you register a TypeHandler and add handler functions keyed by the job kind. Each handler receives the deserialized job struct and returns an acknowledgement indicating success. In the example, a SendEmail handler logs the recipient and simulates latency with a short sleep before returning an Ack result; a GenerateReport handler logs the user ID and similarly acknowledges completion.
That handler registration is the only place business logic lives. Because the handlers consume strongly typed structs, you avoid ad-hoc parsing inside workers and keep the behavior consistent when the transport changes.
Two binaries: worker and submit
A small, practical project layout separates the consumer and producer into two binaries: a worker that subscribes and processes tasks, and a submit tool that publishes a single job. This pattern mirrors real deployments where one or many workers process a stream of messages while publishers live in application code or auxiliary tooling.
In development you can run the worker via cargo run –bin worker and submit jobs with cargo run –bin submit. The worker waits for incoming tasks; the submit binary connects to the configured input and publishes a single message carrying the serialized job and its kind metadata.
Start simple: file backend for local development
One of mq-bridge’s core ideas is that you should be able to start without Docker or a broker. For fast iteration, you can use a file endpoint: the publisher appends new JSON lines to jobs.jsonl and the worker watches that file. The example uses a File consumer mode that can either delete consumed lines (Consume { delete: true }) or keep them for replay (delete: false). There is also a GroupSubscribe mode that tracks offsets in a separate .offset file to avoid re-delivery without deleting lines.
This file-backed workflow works entirely offline and is great for debugging handlers, demonstrating the API, and running local CI where bringing up external services would be cumbersome.
Move infrastructure into configuration (JSON config)
To decouple infrastructure from code, mq-bridge supports loading routes and endpoints from a JSON configuration file. Instead of constructing Route and Endpoint objects in Rust, you serialize the same structure to a config.json and use serde_json::from_str(include_str!("config.json")) to load it at startup. The worker then calls route.with_handler(jobs) to attach the typed handlers to the configured input.
Because both worker and submit binaries read the same configuration, you can maintain a single source of truth for the endpoints and change transport details without touching handler code. The example shows a config that describes a file input with path, delete flag, and mode, and a null output.
Enable production transport: switch to NATS with a single config change
When you’re ready to run workers on separate machines, the transport should move away from a file to a broker. In the example, switching from the file endpoint to NATS JetStream requires only two changes: enabling the mq-bridge "nats" feature in Cargo.toml and updating the JSON config to describe a NATS input with URL, subject, and stream names. The handler code remains unchanged.
The tutorial includes practical instructions for starting a NATS server with JetStream on macOS (brew install nats-server; nats-server -js), via a Debian/Ubuntu .deb package, or running the official Docker image (the example uses docker run -p 4222:4222 nats:2.12.2 -js). After that, restarting the worker and submit binaries makes both talk to NATS while business logic continues to live in the same Rust handlers.
Middleware, retries, and a dead-letter queue in configuration
Switching to a featureful broker like NATS unlocks middleware capabilities that are configured rather than coded. The example JSON config demonstrates adding a retry middleware (with max_attempts, initial interval, multiplier, and maximum interval values) and a dead-letter queue (DLQ) middleware that writes failed messages to an error.log file when retry attempts are exhausted.
Because these features live in the runtime configuration, you can evolve operational behavior — retry policies, DLQ destinations, and other processing stages — without changing the handler code. This makes it possible to tune delivery semantics and failure handling in staging or production environments through config changes alone.
Multiple backends available through configuration
mq-bridge is transport-agnostic in the sense that a range of backends are supported and can be selected by configuration. The example notes that MongoDB, MySQL, MariaDB, and PostgreSQL can act as queue backends with a config change. It also mentions that core messaging operations — receive and publish — share the same API across endpoints (Http, gRPC, MongoDB, Kafka, RabbitMQ, and NATS), and all use a common CanonicalMessage struct for transport.
This uniform interface lets you pick a backend that matches operational constraints — ease of deployment, persistence guarantees, or ecosystem fit — while keeping handler code stable.
Testing handlers without external services using the memory endpoint
A valuable testing pattern shown in the example is swapping the transport for an in-memory endpoint during tests. mq-bridge provides a MemoryConsumer and memory endpoints that implement the same traits as other transports. The submit logic can publish into an in-memory publisher and a local MemoryConsumer can receive the message for assertions.
The tutorial includes a tokio::test that constructs a local MemoryConsumer, instantiates a Publisher backed by an Endpoint::new_memory topic, invokes the same send_mail helper used by the submit binary, and asserts that the serialized payload and metadata match expectations. This lets you test the exact TypeHandler behavior used in production without files, brokers, or test containers — only the transport is swapped.
What mq-bridge provides out of the box
By focusing on transport and a consistent message shape, mq-bridge gives several practical benefits in the example:
- A single, typed handler surface: handlers operate on deserialized Rust structs keyed by a KIND identifier.
- Config-driven endpoints and routes: change a JSON config to go from file to NATS to other backends.
- Shared Publisher/Consumer API: publish and receive use the same methods and message struct across endpoints.
- Middlewares via config: retries and DLQ handling can be added declaratively.
- Lightweight local iteration: file and memory endpoints enable offline development and tests without spinning up services.
The tutorial also mentions a convenience app, mq-bridge-app (installable via cargo install mq-bridge-app), which can forward messages between endpoints or provide a UI to create JSON configs — useful for operations teams that want a simple way to manage routing and deployments.
Limitations and what not to expect
The author of the example is explicit about scope. mq-bridge is transport-centric and does not attempt to be a full-fledged event framework or opinionated guidance layer. Some broker features may be emulated and others might not yet be implemented; configuration is validated at runtime rather than fully at compile time. Event sourcing, aggregate management, and other high-level patterns are out of scope for the library as presented; documentation is still evolving and the example is positioned as a practical first step.
Treat mq-bridge as a pragmatic tool to unify message shapes and swap transports easily, not as a comprehensive alternative to larger frameworks such as Watermill (Go) or Java Spring for complex domain-driven event handling.
Broader implications for developers and operations
This pattern — keep business logic in typed domain objects and treat transport as a pluggable detail — has implications beyond the specific library:
- Developer velocity: Teams can iterate handler logic quickly using a file backend and exhaustively test behavior in memory before introducing networked brokers. That reduces friction in onboarding and local debugging.
- Operational agility: Moving transport concerns into config files reduces deployment risk; operators can switch to a different broker or adjust retry and DLQ policies without redeploying application code.
- Testability: A consistent messaging API and in-memory endpoint make it far easier to write deterministic unit and integration tests for background processing, improving CI reliability and reducing reliance on ephemeral test infrastructure.
- Ecosystem fit: Because mq-bridge supports multiple backends (databases, Kafka, RabbitMQ, NATS, HTTP/gRPC), organizations can adopt it incrementally and integrate with existing systems rather than forcing a wholesale platform migration.
- Clear responsibility separation: By decoupling transport from handler code, teams avoid leaking operational concerns into business logic, which simplifies reasoning about fault handling and scaling strategies.
These benefits are most relevant for teams that value simplicity in the messaging layer and prefer to maintain control over failure semantics via configuration. They also align with broader trends toward infrastructure-as-config and composable platform building blocks.
Operational recipes and practical commands
The tutorial includes concrete, reproducible steps you can follow:
- Initialize the project and add dependencies with cargo init and cargo add mq-bridge serde tokio tracing tracing-subscriber.
- Create two binaries, worker and submit, to separate consuming and publishing roles.
- Use a file endpoint (jobs.jsonl) for local development; append lines or run the submit binary to publish jobs.
- Convert route and endpoint declarations to a JSON config and load it with serde_json::from_str(include_str!("config.json")) so both submit and worker use identical endpoint definitions.
- Enable the mq-bridge nats feature in Cargo.toml to build NATS support, and start a NATS server with JetStream using brew, a Debian/Ubuntu package, or docker run -p 4222:4222 nats:2.12.2 -js.
- Add middleware entries in the config for retry and dlq behavior, pointing the DLQ to a file like error.log if desired.
- Use memory endpoints and the provided MemoryConsumer in tests to validate publishing and receiving without external services.
Following these steps produces a small, maintainable codebase where the business logic does not need to change when the underlying transport evolves.
Testing and continuous delivery considerations
Because handlers are plain Rust functions registered against a TypeHandler, you can reuse the same handlers in unit tests, integration suites, and production processes. Tests that assert message payload shape and metadata are straightforward with an in-memory transport. For CI, prefer the memory or file endpoints to avoid flakiness tied to external service startup. When you do need to exercise broker behavior (ordering, durability, JetStream semantics) add a small set of integration tests that run against a disposable NATS or Kafka container.
Keeping config files under version control and sharing them between submitters and workers ensures consistency across environments. The example also suggests the possibility of separate publisher configuration in production (for topic names, group identifiers, or fan-out topics) while retaining common defaults for simpler deployments.
A forward-looking note on evolution
The example and accompanying library are intentionally narrow: they prioritize transport consistency and developer ergonomics over highly opinionated event-framework features. As the project matures, expect documentation and supported backends to expand. For teams that want a flexible messaging abstraction with typed handlers, mq-bridge offers a lightweight, config-driven path from local development to broker-backed production deployments; future iterations may continue to fill gaps around feature parity with larger frameworks while keeping the core goal — swap transports without changing handler code — intact.
