Sphinx in Action: Good and bad in Sphinx real time indexes

This is a post from a series of posts called "Sphinx in Action". See all our Sphinx posts.

Sphinx has supported real time indexes since version 1.10 was released. Ever since, they have been getting more stable and robust, and now it is ready for use in production. Many people like this because it's really simple to understand (i.e. no indexing, crontasks, main + delta schemes, and so on), but anyone who wants to use them should also be aware of the drawbacks of this when comparing it to traditional monolithic indexes.

A real time index consists of 2 parts: one that is stored in memory, and the other is stored on a disk. Once a new insert/update query is sent to a real time index, it updates the memory as well. It works very fast, but the memory isn't unlimited. So as the number of queries begin to reach it's maximum and once rt_mem_limit is exceeded on the memory side, it gets converted into a disk index chunk, and so on. If you have a 10Gb index and rt_mem_limit is set to 1Gb, then you will end up with 10 disk chunks. If you set re_mem_limit to 10Gb you will have only 1 disk chunk. Now here's the dilemma: the more disk chunks you have the worse your search performance will be, but on the other hand less memory is needed to support the real time index. On the contrary, if you set rt_mem_limit to a high value you will have good search performance because you will have fewer disk chunks, but it will take up more memory on your server. Unfortunately, the amount of memory that a Sphinx real time index requires is much more than what a traditional index needs. This is because a traditional index only stores attributes and wordlists while a real time index stores everything else as well until it's converted into a disk chunk.

Here's what it looks like for the same data (1M docs) when rt_mem_limit is high enough:

Traditional index:

[root@SE01 snikolaev]# ls -sh idx.sp*
12M idx.spa
186M idx.spd
8.0K idx.sph
11M idx.spi
4.0K idx.spk
4.0K idx.spl
4.0K idx.spm
103M idx.spp
8.0K idx.sps

Real time index:

[root@SE01 snikolaev]# ls -sh idx_rt.*
8.0K idx_rt.kill
4.0K idx_rt.lock
8.0K idx_rt.meta
442M idx_rt.ram

The bolded lines are the things that are stored in memory. As you can see, the traditional index requires 23Mb while the real time index needs more than 400Mb of memory.

In practice, we usually recommend real time indexes in 2 cases:

• When the data volume is really small and is not going to grow quickly. Indeed, it doesn't actually matter whether you spend 5Mb or 100Mb if you have few gigs of memory and using the real time indexes will make sense for you because you will avoid having an indexing routine and will be able to synchronize your database and your search index on a data insert level.

• When the data volume is large and growing, but you want to reduce indexing latency (i.e. you want your data to appear in the index in a real time manner). And here comes the best part of using real time indexes. They can be combined with traditional indexes using a Sphinx distributed index. What you can do in this case is still use the main + delta scheme, but make the delta real time, once the main part is rebuilt you then should clean the real time delta index. Since the delta is real time it enables real time data accessibility in the index and because it's the delta, it doesn't require a lot of memory. The only routine is to flush it periodically to the main part. To empty the real time index the latest Sphinx builds provide "TRUNCATE RTINDEX" SphinxQL command.

---

Need help with search infrastructure? Talk to our engineering team — we've built and maintained search systems for over 15 years.

Frequently Asked Questions

When should you use Sphinx real-time indexes instead of traditional indexes?

We recommend real-time indexes in two situations: when your dataset is small, is not going to grow quickly and you want to skip the indexing routine entirely, or when you need sub-second data visibility for a large dataset using a main + RT delta hybrid.

How much more memory do Sphinx real-time indexes use compared to traditional indexes?

Significantly more. For 1M documents, a traditional Sphinx index uses roughly 23MB of RAM while the equivalent real-time index uses over 400MB when rt_mem_limit is set high enough to avoid disk chunks. This is because RT indexes store the full in-memory segment — attributes, wordlists, and all — until it flushes to disk.

What is rt_mem_limit and how does it affect search performance?

rt_mem_limit controls how much RAM Sphinx allocates for the in-memory portion of a real-time index before flushing to a disk chunk. A low limit (e.g., 512MB on a 10GB index) creates many disk chunks and degrades query performance.

How do you implement a main + delta hybrid with Sphinx real-time indexes?

Keep your historical data in a traditional monolithic index rebuilt nightly. Point a real-time index at only the delta — new and updated documents from the past 24 hours. Combine both in a Sphinx distributed index. When the main index rebuilds, run TRUNCATE RTINDEX to flush the delta. This gives you real-time insert visibility without the memory overhead of an all-RT setup.

How do you empty a Sphinx real-time index cleanly?

Use the TRUNCATE RTINDEX index_name SphinxQL command. This is the correct way to reset an RT index — it clears both the in-memory segment and any disk chunks. Run it as part of the main index rebuild cycle to reset the delta RT index after the traditional main index has been rebuilt.

Related Reading

• How to speed up Sphinx Real Time indexes up to 5-10 times
• Should you switch to Sphinx real time indexes?
• Converting Sphinx original indexes to real-time indexes