QueryScope: Merging Load Testing with LLM Observability to Diagnose Performance Regressions
QueryScope merges load testing with LLM observability, running benchmarks, indexing results, and producing grounded diagnostics so teams identify root causes of latency and errors.
A concrete problem meets a practical synthesis
QueryScope is an open-source tool that combines classic load testing with LLM-powered observability to help teams answer not just what happened during a benchmark but why it happened. Traditional tools like k6, JMeter, and Locust excel at generating traffic and reporting latency percentiles and error rates; observability platforms for generative AI—tools that track prompts, model responses, and token usage—typically sit on the sidelines and passively collect telemetry. QueryScope brings those worlds together: it executes controlled benchmarks against REST and LLM endpoints, indexes run summaries into a vector store, and uses retrieval-augmented generation to produce diagnostics grounded in actual benchmark data. That combination helps engineers move from detection to diagnosis without manually stitching disparate systems.
Why load testing and LLM observability belong together
Performance testing and model observability are complementary but historically separate disciplines. Load testing measures system behavior under stress—throughput, p50/p95/p99, and error trends—while LLM observability captures semantic context: prompt variations, response shapes, and content-level anomalies. When latency or error rates shift after a deploy, raw metrics alone rarely expose the causal chain. Did a change in prompt length inflate token usage and therefore latency? Did a third-party API begin throttling requests during peak concurrency? Combining load patterns with semantically rich, retrievable records of prior runs enables a more focused root-cause analysis. QueryScope addresses that gap by operationalizing a RAG-style pipeline that ties historical runs to recent benchmarks, letting language models reason about time-sensitive performance data rather than hallucinate from generic priors.
How QueryScope performs benchmarks and captures telemetry
At its core, QueryScope can target any RESTful or LLM endpoint with configurable request payloads and concurrency profiles. Users define the number of requests, concurrency level, and the request content for text-generation endpoints or APIs. A benchmark runner executes the workload and streams live metrics—throughput, error rate, and latency percentiles—into a dashboard for monitoring during the run.
What differentiates QueryScope is what it does after a run completes. The system synthesizes a human-readable plain-text summary that encodes key metrics (for example: request count, p50/p95/p99, throughput, and error percentage) and contextual metadata such as the target URL and HTTP method. Those summaries are embedded with a modern text-embedding model and indexed into a vector search service, enabling semantic retrieval of past runs that resemble a new query. By storing both the raw time-series and a concise natural-language synopsis, QueryScope lets LLMs reason over performance histories using language models’ affinity for natural text.
The RAG pipeline — indexing, retrieval, and grounded diagnosis
QueryScope uses a retrieval-augmented generation approach to produce diagnoses that are anchored to actual benchmark evidence. The pipeline has three tightly coupled stages:
-
Indexing: After each benchmark, the system generates a concise plain-text summary that reads naturally—phrases like “p99 spiked to 582ms” carry semantic weight for embedding models. That summary is converted into a vector embedding and upserted into a vector store so similar runs can be found by semantic similarity.
-
Retrieval: When an engineer asks a diagnostic question—either via a UI prompt or an automated agent—the query is embedded and used to fetch semantically similar historical runs from the vector index. Simultaneously, QueryScope adds the five most recent runs from the relational database directly into the prompt as explicit, time-ordered ground truth. This two-pronged retrieval solves a common vector-search shortcoming: similarity alone doesn’t respect recency demands such as “my last two runs.”
- Generation: The retrieved semantic neighbors and the explicit recent-run context are injected into a language model chain for analysis. By combining semantic relevance and recency, the model produces a diagnosis that cites concrete data points from the last runs and suggests plausible causes—configuration changes, tokenization differences, third-party latencies—while grounding its claims in the recorded benchmark data rather than speculative reasoning.
This hybrid of vector similarity plus direct database injection reduces hallucinations and increases the relevance and accuracy of the LLM-generated explanations.
MCP server and agentic benchmarking with Claude Desktop
QueryScope includes a Node.js component that implements the Model Context Protocol (MCP) to expose two callable tools: run_benchmark and query_runs. Registered with an MCP-capable client such as Claude Desktop, these tools permit an LLM to orchestrate benchmarks autonomously. For example, an engineer can prompt Claude Desktop to “benchmark this endpoint with 50 requests and concurrency 5.” The agent invokes the run_benchmark tool, which executes the workload and returns structured results; the agent then analyzes those results and can call query_runs to fetch related historical runs for a grounded assessment.
This design turns the LLM from a passive advisor into an execution driver: agents can trigger actual HTTP traffic, gather telemetry, and synthesize findings without a separate UI interaction. That model raises new possibilities for automated verification workflows (e.g., run smoke tests then benchmark during CI), continuous observability where agents periodically evaluate endpoints, and conversational SRE workflows where a human can iterate with an agent to narrow down root causes.
Full-stack architecture and deployment options
QueryScope is built as a modular stack so teams can adopt pieces that fit their existing infrastructure. Major components include:
-
An asynchronous REST API and benchmark runner built with FastAPI and async SQLAlchemy, backed by a relational database. The runner executes requests, aggregates metrics, and persists raw and summarized run data.
-
A vector indexing and retrieval layer using an embedding model to create vectors and a vector service (e.g., Azure AI Search) surfaced through LlamaIndex for upserts and queries.
-
A retrieval and explanation chain implemented with LangChain LCEL that injects both semantic neighbors and the most recent runs from Postgres into prompts, with the generative step executed by a capable LLM (GPT-4o-mini or similar).
-
A live dashboard front end built with React and visualization libraries for real-time polling of ongoing runs and historical comparisons.
- A Node.js MCP server that exposes callable tools for agent orchestration.
For local experimentation, QueryScope ships with a Docker Compose setup to start all components with a single command. Production deployments can scale workers via Kubernetes with Horizontal Pod Autoscalers for demand-driven concurrency. The design also includes adapters for different database backends—implementations use JSON columns for cross-database compatibility where native array types are not available—so teams can choose Postgres, MySQL, or other relational engines according to organizational standards.
Who should use QueryScope and common scenarios
QueryScope targets engineering teams responsible for APIs and LLM-based services that need both performance verification and explainable diagnostics. Typical use cases include:
-
Pre- and post-deploy verification: run benchmarks against new code or model versions and obtain a grounded explanation when tail latencies shift.
-
Third-party dependency validation: detect and explain regression patterns that correlate with downstream API changes or throttling.
-
Model prompt performance testing: quantify how prompt size or structure affects latency and token costs, and explain whether token growth explains observed slowdowns.
-
Continuous observability: schedule periodic benchmarks driven by an agent to identify regression windows and correlate them with releases or infra events.
- On-call triage augmentation: give on-call engineers a tool that not only shows that p99 increased but points to likely causes in recent runs, enabling faster mitigations.
Availability is immediate for teams willing to self-host: the project is published as open source and provides a demo walkthrough and a repository for cloning. That lowers barriers for teams to try the system in staging or internal test environments before integrating with production monitoring pipelines.
Developer considerations: integration, APIs, and customization
Integrating QueryScope into an existing toolchain involves a few practical decisions. Benchmarks should be parameterized to capture realistic traffic patterns, and payloads for LLM endpoints should reflect production prompt distributions to reveal authentic performance characteristics. Developers can customize the natural-language summary templates used for embeddings to include domain-specific context—such as tenant IDs, feature flags, or model names—which improves the semantic retrieval quality for later queries.
The RAG pipeline’s fidelity depends on embedding quality and the vector store’s configuration. Teams should choose an embedding model and vector index that balance accuracy and cost; for large historical datasets, index pruning or time-windowed retention strategies will control storage and query expense. Because the system injects the five most recent runs as ground truth, maintaining reliable and consistent run metadata in the relational database is critical: missing or inconsistent fields can reduce the grounding effectiveness.
For those connecting agent frameworks, the MCP tool surface is intentionally small and action-oriented: run_benchmark accepts parameters like target URL, concurrency, and request payload; query_runs returns semantically similar historical runs or the recent runs needed for explanation. That compact interface simplifies authorization and audit trails: agents’ actions are explicit and can be recorded for compliance.
Security, privacy, and cost trade-offs
Combining LLMs and telemetry opens policy questions. Benchmarks often include request payloads that may contain sensitive or proprietary data; teams must decide whether to redact or pseudonymize content before embedding. Embeddings themselves can leak information under certain threat models, so organizations with strict data governance should either avoid embedding sensitive strings or run embedding models in a controlled, on-premise environment.
Another consideration is cost: embedding models, vector index storage, and LLM generations add recurring expense. QueryScope’s design supports configurable retention and selective embedding strategies (for example, embed only summaries or only runs that meet certain anomaly criteria) to contain cost. Operational security also requires care when exposing an MCP endpoint to external agent clients; mutual TLS, fine-grained API keys, and role-based access control are advisable to avoid unauthorized benchmark execution.
How QueryScope compares to existing tools and ecosystems
QueryScope is not intended to replace k6, JMeter, or observability providers; instead, it complements them. Where traditional load tools measure and record, QueryScope layers semantic indexing and LLM-enabled diagnosis on top of that telemetry. In the AI observability space, platforms often ingest model logs and traces but do not actively exercise endpoints with controlled workloads. QueryScope fills that niche, bridging load generation and semantic observability in a single, extensible stack.
This integration also plays well with adjacent tooling ecosystems: data from QueryScope can feed incident management systems, notebook-driven postmortems, or performance dashboards. Teams using CI/CD can embed benchmark runs into pipelines as a regression gate. Security and AIOps tooling can consume diagnostic outputs to trigger automated rollbacks or throttles when grounded analysis points to deployment-induced regressions.
Broader implications for observability and software engineering practices
QueryScope’s synthesis of active testing and LLM-driven reasoning points toward a shift in how teams will approach performance diagnostics. Observability is evolving from passive telemetry collection to active interrogation: agents can run targeted experiments, retrieve semantically relevant history, and produce a human-readable diagnostic rationale tied to measurement evidence. For developers, that reduces the manual cognitive load of correlating timelines across monitoring systems, commit logs, and external factors.
For businesses, this approach promises faster mean-time-to-detect and mean-time-to-resolve for performance incidents, and a more repeatable process for validating model deployments and API changes. It also encourages closer collaboration between SREs, platform engineers, and ML teams: prompt engineering and model choices are now explicitly part of performance discussions because prompt content can meaningfully affect latency and cost.
From a tooling perspective, QueryScope exemplifies how language models can be responsibly applied as reasoning layers over deterministic data, provided pipelines include explicit mechanisms to ground outputs in verifiable records. This model—vector search for semantic similarity plus direct injection of recent, authoritative records—can serve as a template for other observability workflows where recency and semantic context both matter.
Best practices for teams adopting QueryScope
Teams should approach adoption iteratively. Start by running non-production benchmarks that mirror typical traffic and use the RAG diagnostics to identify obvious correlations—changes in prompt length, model version updates, or increased token usage. Instrument the benchmark runner to include necessary metadata (deploy tag, commit hash, model name) so diagnoses can point to concrete releases.
Next, tune the summary templates that get embedded. Small changes—adding a deploy timestamp or feature flag—can dramatically improve retrieval relevance for domain-specific queries. Finally, define data retention and privacy policies for embeddings and runs to balance investigatory power with compliance requirements.
Organizations should also test the MCP integration carefully in staging: granting an agent the ability to execute load must be paired with safeguards—rate limits, quotas, and scoped credentials—so that automated agents cannot accidentally generate disruptive traffic.
A growing number of observability needs—model performance, token-cost analysis, third-party dependency regressions—map well to a tool that can both execute reproducible tests and narrate the results in grounded, explainable terms. QueryScope demonstrates a pragmatic way to build that capability with existing open-source building blocks.
QueryScope’s approach places evidence and context at the center of LLM-driven explanations, and that makes it useful beyond benchmarking: any operational scenario where historical runs and recent measurements must be correlated—SLA investigations, cost optimization initiatives, or multi-tenant performance analysis—can benefit from a RAG-driven diagnostic layer.
Looking ahead, we can expect these agentic observability patterns to become more integrated into CI/CD and incident-management workflows: imagine pipelines that automatically run targeted benchmarks on canary deployments, synthesize a grounded report, and gate promotions based on diagnostic outputs. As vector indexing and retrieval capabilities mature and on-premise embedding options become more accessible, this class of tools will likely expand into richer data domains—traces, logs, and raw model outputs—enabling even more precise, explainable triage for complex systems.
