How Joom Cut AI Moderation Cost 100x with a Two-Stage Gateway

Author: Sergey Smolonsky, Head of Moderation at Joom

Hitting the 20K Daily Limit with Legacy Models

Any time an e-commerce business decides to drastically expand its catalog assortment, some part of the backend infrastructure is going to snap. For us that exact bottleneck turned out to be pre-moderation. Our engineering team found themselves staring down a massive operational cliff: we had to figure out how to scale daily product processing from 20,000 all the way up to 2,000,000 items. Handling a 100x spike in volume is already a serious engineering challenge. Any team can scale moderation by throwing money at it. The real challenge is handling a 100x volume spike on the budget you already have. Dropping the per-item inference cost by a factor of 100 compared to the legacy manual process meant throwing out the old rules and designing a completely new architectural gateway.

Before anyone even mentioned an LLM, our pre-moderation workflow leaned heavily on a brittle mix of a legacy internal model and manual review. But over time, it had severely degraded. The human team simply couldn’t keep up with the sheer volume of borderline cases. At peak times, products waited in the manual moderation queue for weeks. In extreme scenarios, the backlog got so bad that some items were pushed to live without adequate checks just to keep the catalog moving.

The math broke down. Trying to push 2 million items a day through that manual interface would have required a massive, financially ruinous hiring surge. The goal was very clear. We needed a system that could parse millions of products, deliver consistent verdicts in under 24 hours, and actually explain rejection reasons to merchants — all while keeping baseline operational costs absolutely flat.

Bypassing High API Costs with a Two-Stage Setup

Trying to push 2 million daily items through a monolithic LLM pipeline is a fast track to burning your entire moderation budget. Our engineering team quickly realized that pinging a heavy model for every single product upload wasn’t going to work. To fix this, our ML Engineer and Backend Developer broke the moderation logic apart. We designed a highly optimized, two-tier architecture using a tuned Gemini-based gate and a larger Gemini main model.

Stage 1 acts as the frontline gatekeeper. Think of it as a fast “metal detector” — a high-volume, low-cost filter running in batch mode. In the world of asynchronous e-commerce pre-moderation, pushing maximum throughput matters far more than microsecond latency. By trading real-time inference speed for batch processing, we significantly cut our API execution costs. This stage aggressively weeds out the definitively clean items. Because of this, the vast majority of the catalog never even touches the expensive deep-inspection layer.

Stage 2 only wakes up if the first model cannot confidently mark the item as safe, like if the “metal detector” rings. This is the heavy, expensive inference layer. Instead of a rigid “Yes/No” final decision, Stage 2 acts as a diagnostic engine. It pinpoints exactly where the policy violation lives and feeds structured labels back to our internal business logic, which then decides the final action. For example, it can explicitly tell the backend that a trademark infringement is sitting on the fifth product image. That level of granular detail is exactly what drops merchant support tickets.

Translating LLM Outputs into Categorization Rules

The AI evaluates incoming items against a strict prompt that holds over 80 specific rules. But it doesn’t just stop at obvious red flags like weapons or hazardous chemicals. The real test of the system is how it handles dual-use or illicit-intent hardware.

Take things like GPS jammers, Wi-Fi blockers, or unauthorized signal boosters. To a basic keyword filter — or even a tired human moderator at the end of a shift — these just look like harmless consumer electronics. The LLM, on the other hand, actually analyzes the surrounding context. It flags them under the specific rule for goods intended for illegal activities. It’s not looking at superficial physical features; it’s evaluating the product’s actual intent.

Even better, the model doesn’t spit out raw paragraphs of text back to the engineering team. Instead, it generates highly structured confidence labels — for example, riskyauto_ip-brand_high, accompanied by specific parameters like brand=Nike and images=[…]. Our internal categorization engine immediately catches these automated labels and triggers specific business logic. So, rather than a rigid “block or approve” outcome by the AI itself, the system can dynamically hide an item from specific countries. Or, it can automatically freeze a product in a pending state until the merchant uploads the right CE certificate.

Staggered Deployment and the 100x Cost Drop

Handling over the keys to an AI gateway that dictates the flow of 2 million daily items comes with massive operational risk. Flipping the switch all at once could have triggered a catastrophic API burn rate or buried the platform under a mountain of falsely rejected products. To keep a tight leash on quality and monitor costs in real-time, our engineering team executed a strict, staggered rollout. We incrementally routed catalog traffic through the new Gemini pipeline, closely monitoring the system before finally fitting 100% coverage.

Budget was the absolute hard line during this entire process. We knew that if we couldn’t get the inference costs under control, we would be forced into a painful compromise — like only running a small fraction of the catalog through the AI and dumping the rest back onto the decaying internal model. But by walking away from a monolithic approach and building that two-tier, batch-processed gateway, we actually drove the localized cost of machine inference down by 100x.

Zoom out to the scale of the entire e-commerce catalog, and that single architectural decision is exactly how we managed to absorb a 100x volume surge without adding a single dollar to the baseline moderation budget.

Repurposing the Moderation Team for QA

Moving to AI automation at this scale completely shifted the role of our moderation team. Rather than replacing people with soulless machines, the AI actually rescued the team from an impossible backlog. Operators were pulled completely out of that repetitive, low-value manual queue that had been eating up their time and burning them out.

Instead of acting as a bottleneck, the operators stepped up into quality assurance and system management roles. Now they focus on the work that genuinely needs human judgment: auditing borderline AI decisions, refining the internal categorization rules, and building out “golden samples” — highly curated, perfect moderation examples that serve as the definitive ground truth. These baseline datasets are used to constantly evaluate, benchmark, and fine-tune the model against new edge cases. It’s less grindy, and far more meaningful work.

With the pre-moderation pipeline humming along at scale — and the internal quality control tools sitting at about 90% completion — our engineering team is already looking ahead. Our roadmap now focuses on pushing the AI’s mandate even further: building an advanced auto-categorizer to process the whole incoming product flow from scratch, and automating the verification of legal and merchant documents.

Conclusion: Architecture Over Brute Force

We believe that our pre-moderation case-study highlights a critical reality for scaling tech companies: deploying enterprise AI is rarely about simply plugging into the new LLM API. It requires deep architectural thinking to balance inference costs with massive throughput requirements.