May. Two weeks before peak season.
Search latency had crossed the five-second mark.
Not P95. Not the worst case.
Median.
Traffic was rising like it does every year. When the first warm weekends arrive, search volume doubles within days, editors maintain structured data continuously, and the system has to carry both at the same time. What had been good enough for years suddenly became the slowest part of the entire platform. I was running Sphinx. It had served well, until the requirements outgrew it.
When "it works" stops scaling
The original setup was conventional and had proven itself over years. MySQL as the source of truth, Sphinx for full-text indexing, PHP-FPM as the application layer, a nightly full reindex and a delta index every 15 minutes. For years the median search latency sat between 300 and 500 milliseconds, acceptable for an editorially run platform with seasonal traffic.
Then three things changed at once. Content volume doubled within 18 months. I introduced faceted filtering by manufacturer, technical specifications and model year. And editors started updating structured metadata continuously instead of in batches. Each of these changes would have been manageable on its own. Together they hit an architecture that was not built for exactly this usage pattern.
The symptoms crept in. The delta index fell behind the writes. Merge times went up. Memory pressure built. Occasional query spikes of three to five seconds appeared with no visible trigger. The real problem was not raw performance, it was operational fragility. Reindexing became an event I had to plan. Deployments had to be coordinated around indexing jobs. Search stopped being infrastructure and turned into ceremony. That was the architectural limit.
Why Manticore and not Elasticsearch
Elasticsearch was the obvious alternative. I rejected it for pragmatic reasons. Running the JVM means operational overhead that simply is not justified for a single platform of this size. Cluster complexity was far beyond my requirements. And I did not have a problem that demanded distributed multi-tenant analytics across petabytes; I had a problem with search queries taking too long on a grown PHP platform.
What I needed was low-latency full-text search, real-time updates without merge overhead, predictable memory behaviour and a minimal operational surface. Manticore, an actively developed fork of Sphinx, met those requirements and let me evolve incrementally instead of replacing the system wholesale. In this context that mattered more than feature breadth.
The architectural reset
The first deliberate decision was to stop thinking of the index as a replica of the relational database. In Sphinx my schema had gradually turned into a mirror of the MySQL tables. That felt safe because it was familiar. It was also expensive, because every search request triggered hydration logic and secondary lookups that had nothing to do with the actual search.
With Manticore I treated the index consistently as a projection optimized for retrieval. Human-readable manufacturer names were stored directly in the document instead of being resolved afterwards. Structured specifications were flattened. Publication status was held directly in the index. All filterable attributes were explicitly typed. The result was an index with no post-search joins and no enrichment layer.
Denormalization increased index size, and that was the deliberate trade-off. It cut request latency by 40 to 60 milliseconds under load and eliminated a whole class of N+1 problems that had crept in over months. The important shift was not technical but conceptual. Search is not storage. It is a read-optimized materialized view of the product intent, and it should be treated that way.
From batch indexing to write-through
Sphinx forced me into a compromise of full rebuild, delta index and periodic merge. Those merges were the system's hidden tax. As content volume grew, merge time grew proportionally. During peak traffic that became a real danger, because merge operations and heavy query load competed for the same resources.
With Manticore RT tables I moved to write-through indexing. Every successful database write triggered an immediate index update. I deliberately chose REPLACE over INSERT because it simplified the consistency model without requiring diff logic or conditional updates. The trade-off was higher write amplification and the possibility of drift when the search write failed, but that was a manageable problem, not a structural one.
So I made drift explicit instead of ignoring it. I introduced an asynchronous retry queue, a nightly reconciliation of updated_at timestamps between database and index, and a freshness metric that I exported to Prometheus. Search became deliberately eventually consistent, not by accident but as a conscious architectural decision. Operationally that removed all the index merge anxiety. No more indexing windows to coordinate deployments around. The system became continuous instead of episodic.
The measured outcome
Before the migration median latency was around 5.1 seconds, P95 above 6 seconds. Reindex windows blocked heavy writes for 12 to 18 minutes and had to be actively scheduled into the deployment process. After the migration median latency was 82 milliseconds, P95 130 milliseconds. No more planned reindex downtime.
The improvement was not magic and not a miracle of new technology. It was the consistent removal of architectural friction. The biggest gains came from eliminating delta merges and removing post-search hydration: two problems created by wrong base assumptions about the index, not by the technology itself.
The hard part: relevance is political
Performance problems are objectively measurable and can be backed with numbers. Relevance problems are political, and that is the uncomfortable truth no technology article likes to write down.
Within a week of go-live the first complaints arrived. New entries did not rank high enough. Exact model matches showed up below generic articles. Manticore uses BM25 by default, a solid baseline that works well for many use cases, but it does not know product priorities and cannot know them.
I needed title matches to outweigh body matches, exact matches to dominate partial ones, a slight recency bias for new content and strict filtering on publication status. Every adjustment improved one use case and degraded another. More recency bias improved the perception of freshness for current models but degraded historical research queries. Strong exact-match boosts made direct model names dominate, but sometimes hid valuable contextual content that users were actually looking for.
The breakthrough did not come from a better formula but from building a relevance regression suite. 50 canonical queries, manually defined expected top results, automated comparison before every deployment. Relevance stopped being opinion-driven and became testable behaviour. Without those regression tests, ranking adjustments are structured gambling: you improve something and have no idea what you break along the way.
The first production regression
Three months after go-live latency went from 80 to roughly 400 milliseconds. No traffic spike, no hardware change, no obvious trigger. The cause was a new filter on a high-cardinality string attribute for engine codes. Editors had started tagging inconsistently because there was no validation. What was meant as a structured filter behaved in practice like semi-free text, with the corresponding effect on query performance.
Fixing it required replacing the string attribute with a normalized integer mapping, rebuilding the index and enforcing validation at the write boundary so the problem could not come back. The lesson was uncomfortable but clear. Search performance degrades quietly and gradually when schema discipline erodes, and search infrastructure deserves the same care around schema changes as relational databases.
Running Manticore as infrastructure
The first lesson from operating it was that relevance rules need regression tests from day one. Not once stakeholders complain, but before the system goes live. Ranking changes without automated tests are blind. You do not know what you are improving and you do not know what you are breaking.
The second lesson was that attribute types have to be enforced at the write boundary. Not as a later optimization but as a structural requirement from the start. The production regression three months after go-live would not have happened with consistent validation at the entry point.
The third lesson was that index freshness has to be actively observable. Silent assumptions about the state of a system turn out wrong sooner or later. I versioned the index schema in Git and treated schema changes with the same care as database migrations: create a new index, run a backfill job, switch traffic, drop the old index.
Back to the starting point
The five-second median latency in May was not a temporary performance spike and not a random event. It was an architectural signal I had ignored for too long. Sphinx had reached its operational limits in my specific usage pattern, not because it is bad technology but because the requirements had changed without the architecture growing with them.
Manticore let me rethink the search subsystem from the ground up. The move from batch indexing to write-through, from a relational schema mirror to a retrieval-optimized projection, from implicit to explicit eventual consistency: those were the decisions that made the difference. Latency dropped from five seconds to under 100 milliseconds.
But the real change was a different one. Search stopped being a fragile subsystem I worked around and coordinated deployments about, and became infrastructure I understand, measure and deliberately evolve.
