Evaluating multi-agent workflows with LLM-as-a-Judge

Author: Ilya Nartov, Machine Learning Engineer at Joom

The Trap of Manual Testing

Shipping LLMs to production without breaking existing systems is difficult. At Joom, we are building an internal AI assistant that interacts directly with our internal database. Its main job is to process natural language queries from business users and automatically pull data that would otherwise require manual extraction by our analytics team.

When building systems where the primary routing and reasoning engine is a non-deterministic language model, standard unit tests are insufficient. From the initial stages of development, it was clear that a comprehensive evaluation framework, or “evals”, was necessary to guarantee expected behavior across diverse user scenarios.

However, when we first started building the AI assistant, we did not yet have a heavy testing infrastructure. Adopting a massive off-the-shelf enterprise framework felt like overkill for a product still finding its footing. We needed a quick way to prove the value of automated testing before committing to complex architecture. This case study breaks down our move from slow manual output validation to a custom LLM-as-a-judge harness, and explains how we generated reliable evaluation datasets for a live changing database.

The Bottleneck of Manual Output Validation

During the initial iterations of our AI assistant, output validation was conducted entirely manually. The workflow was rudimentary and relied on an engineer inputting a test query, analyzing the generated response, subjectively assigning a pass or fail status, and logging the outcome. This approach was initially intended to save development time by delaying the construction of a dedicated testing infrastructure, but it quickly introduced severe operational inefficiencies.

Manual review of complex SQL queries and data outputs creates a significant bottleneck, preventing rapid testing cycles. Furthermore, without automated scoring, comparing the performance of different prompt versions or architectural adjustments was virtually impossible. The repetitive nature of manual validation also leads to developer friction, causing the team to delay testing and reducing the overall signal quality regarding model performance.

Any time we saved by skipping the initial infrastructure setup evaporated during manual reviews. Without automated evals, deploying an LLM-based system is basically guessing if your application works.

Architecting the LLM-as-a-Judge Harness

To resolve this testing bottleneck, we developed a dedicated evaluation harness. The implementation required approximately three hours of focused coding and fundamentally shifted our testing capabilities by replacing subjective human review with an automated, trace-based pipeline.

Before diving into the specific tooling, here is a brief solution overview: we built a lightweight local script designed to simulate user queries in batches, intercept the agent’s internal thought process and database interactions, and then pass that raw context to a separate evaluator model for a final grade.

Here is the actual execution flow. First, the system loads a batch of evaluation cases from a predefined JSON file. Next, the harness queries the AI agent and captures the complete execution trace rather than just the final answer. This full trace includes the generated SQL syntax, the executed BigQuery jobs, the model’s intermediate reasoning steps, and the final user-facing output.

The entire execution trace is then passed to a separate LLM configured specifically as a judge. This LLM-judge evaluates the trace against predefined criteria independently, outputting a definitive pass or fail for each check alongside a detailed explanation for its verdict. The system finally calculates the overall score as the average percentage of criteria successfully met.

The harness is executed locally, with all telemetry and scoring results written directly to BigQuery. A custom internal UI retrieves this data, providing the engineering team with clear visibility into version-over-version performance without requiring any manual intervention.

Dataset Engineering: The Core Challenge

While writing the evaluation harness was a rapid process, curating the actual evaluation dataset presented significant engineering challenges. An eval pack requires mapping specific user queries to expected model behaviors. Building this mapped dataset exposed three major architectural and operational hurdles that we had to solve to make the infrastructure viable:

• The (Non)Static Nature of Data: Standard testing protocols for SQL-engineering agents typically involve executing queries against a frozen staging database to ensure deterministic outputs. Investing in a static data environment was not feasible for our naive early-stage setup. Because our agent operates on a live, constantly updating database, identical queries return different numerical values depending on the execution date. The LLM-as-a-judge architecture resolved this by shifting the evaluation focus from the final data output to the execution logic itself. Instead of expecting a specific number, we implemented checklist-based criteria. The judge analyzes the trace to confirm operational requirements, verifying whether the agent queried the correct tables, applied the appropriate data filters, and included mandatory disclaimers.
• Multiple Solutions to a Single Query: Initially, the assumption was that these evaluation criteria could strictly dictate a single acceptable SQL path for a given query. However, real-world business queries are frequently ambiguous and can be accurately resolved using multiple valid SQL queries. To fix this, we expanded the eval dataset to include multiple valid logic paths. This allowed the LLM-judge to correctly pass the test even when it used a different SQL query that was still functionally correct.
• The Reluctance to Write Evals: Creating accurate evaluation criteria shifted the focus from prioritizing tests to actively securing business stakeholder participation. Our first attempt relied on synthetic data generation using Claude, but the resulting synthetic pack felt too artificial for rigorous production testing. The sustainable solution required establishing a feedback loop with end-users. We began collecting their actual prompts and their expected analytical outcomes. Technical experts then reviewed these requests for validity, allowing us to collaboratively build a robust dataset piece by piece.

Lessons Learned from a Zero-to-One Approach

Developing evaluation datasets is a highly repetitive component of AI agent engineering, but skipping it is not an option. Looking back at this initial iteration of our evaluation pipeline, several key lessons emerged. We quickly realized that building a quick custom harness is often more effective as a first step than spending weeks selecting and integrating heavy enterprise frameworks.

Furthermore, non-static target datasets can be easily worked around by evaluating the agent’s execution logic rather than hardcoded final numbers. We also learned that because there is rarely one correct path to a solution, evals must allow for multiple valid logic branches. Lastly, synthetic queries proved to be a dead end, making it mandatory to use actual business interactions to construct quality eval packs.

Ultimately, this naive local script — run directly from a single engineer’s laptop — proved the critical business value of automated evaluations. It allowed us to confidently track the impact of new architectural iterations and stop debugging broken queries in production.

But as the company grew, this rudimentary approach faced its own limits. That initial local script eventually evolved into the centralized, scalable testing infrastructure that serves the diverse needs of internal users across Joom today. We will share the details of that massive architectural shift in our next post.