Indexer for real-time indexes

We’ve implemented a script which should help when you need to migrate from plain indexes to real-time (RT) indexes or simply make first-time RT indexing.

This script’s function is to fill and update RT Sphinx indexes. It does the similar work as sphinx indexer did for regular ones. Thus its config file (sources.ini) has almost the same syntax and content as the source section of sphinx.conf.

The script can:

• Populate index with data;

• Append new data to the index;

• Update existing index records;

• Make pre and post SQL queries.

To install:

• Download the recent version; - Extract it anywhere on your web server box.

Configuration:

• Edit sources.ini for that. It already has one demo section, use it as an example;

• You can make several sections, each for one data source for each RT index.

To append new data without rewriting existing records you should use id counter for data source. For this you should make such steps:

• Make a table for counters;

• Add $start and $end to sql_query config parameter;

• Modify sql_query_range to select min and max counter id from the counter table;

• Modify sql_query_post_index for counter table updating.

You can find more details on using counters at http://sphinxsearch.com/docs/current.html#delta-updates.

Having finished the script configuration you can run it as simple as:

php indexer.php Good luck, and let us know if you have any troubles with this script!

---

Need help with search infrastructure? See how Ivinco's development team can help →

Frequently Asked Questions

How do you migrate from plain Sphinx indexes to real-time indexes?

The PHP-based indexer script described in this post reads from your existing data sources and populates RT indexes. The script's config file (sources.ini) mirrors the source section of sphinx.conf, so migration is straightforward. It handles initial population, incremental appends, and record updates — the same operations the original sphinx indexer performed for plain indexes.

What is the difference between plain and real-time indexes in Sphinx?

Plain indexes require full rebuilds via the indexer command and are read-only between rebuilds. Real-time (RT) indexes accept INSERT, REPLACE, and DELETE operations on the fly without rebuilding. RT indexes are better for frequently changing data, while plain indexes can be faster for large, rarely-updated datasets.

How do you handle incremental updates with Sphinx RT indexes?

Set up a counter table in your database to track the last indexed ID. Configure sql_query with $start and $end placeholders, sql_query_range to read from the counter table, and sql_query_post_index to update it after each run. This way, each indexer run only processes new or changed records instead of re-indexing everything.

Related Reading

• Converting Sphinx original indexes to real-time indexes
• Five ways to configure Sphinx search engine
• Group by MVA in SphinxQL