Should you switch to Sphinx real time indexes?

Regular indexes problem

The main inconvenience of regular indexes is their update speed. In order to update one you should entirely rebuild it.

For large amounts of data we usually use main+delta indexes. The main contains the most of the data, and delta — only recent changes. So to keep whole index up-to-date we should rebuild delta index every 3-5 minutes. But the larger delta grows the longer it takes to rebuild it. That’s why we recommend to flush delta into main every day or week depending on your data growth rate.

But this approach has a couple of drawbacks:

• high average load of the system due to frequent index rebuilding;

• fresh data will be indexed only in several minutes in worst case.

Real-time indexes

Sphinx 1.10 introduces real-time index support. The main idea of it is the ability to insert and update index records on-the-fly. RT indexes are compatible with MySQL protocol which allows us to use existing MySQL client apps for work with them using SELECT, DELETE, INSERT and REPLACE operators.

Currently there are some performance issues with real-time indexes on large data sets. But for smaller ones (say, up to 500.000 Wikipedia documents) they show comparable to regular indexes speed.

So real-time indexes performance and simplicity makes them a preferable choice for storing relatively small but frequently changing data index.

Conclusion

Real-time indexes can be used as a replacement of main+delta regular index bundle. They can reduce server workload and simplify index updating routine.

Also we can make use of mixed indexes to plug in real-time indexes to the existing app.

Here’s the example of mixed index:

index distributed { type = distributed local = plain_main_index local = real_time_increment_index }

In this example we connect to both regular and real-time indexes using one distributed index. This way migration to real-time indexes can be performed seamlessly without significant modifications of production system.

Good luck and have fun with real-time indexes.

---

Need help with search infrastructure? Talk to our engineering team — we've built and maintained search systems for over 15 years.

Frequently Asked Questions

What are the advantages of Sphinx real-time indexes over main+delta?

RT indexes eliminate the rebuild cycle entirely — data appears in search results within milliseconds of insertion via SQL. With main+delta, fresh data has a 3-5 minute lag and rebuilds cause CPU spikes. RT indexes reduce server load by removing the indexer cron job and simplify operations. The trade-off is slightly higher memory usage and lower query performance on datasets above 500K documents.

How do I migrate from Sphinx main+delta to real-time indexes?

Use a mixed distributed index that references both your existing plain main index and a new RT index. Route new writes to the RT index via INSERT/REPLACE statements instead of rebuilding the delta. Gradually, the RT index accumulates all recent data. Once you've verified stability, rebuild the main index one final time including all data, then switch entirely to RT — or keep the hybrid setup permanently.

What are the performance limits of Sphinx real-time indexes?

RT indexes perform comparably to plain indexes up to approximately 500K documents. Beyond that, query latency increases because RT indexes maintain multiple disk chunks that must be scanned. For large datasets, use a hybrid setup: a plain index for the bulk of historical data plus an RT index for the most recent changes. Run OPTIMIZE on the RT index periodically to consolidate disk chunks and restore performance.

Related Reading

• How to speed up Sphinx Real Time indexes up to 5-10 times
• Sphinx in Action: Good and bad in Sphinx real time indexes
• Announcing DokuWiki Sphinx Search Plugin