SphinxSearch kill list feature
Mindaugas Zukas··Updated In using the SphinxSearch sql_query_killlist I am faced with a problem - the kill list doesn't always work as expected for several distributed indexes.
I found the cause of this problem in the order of the physical indexes within the distributed index.
Lets say we have a main index which contains the majority of the data and updates once a day. And we have delta index which contains the latest changes and updates every 3 minutes.
In order to ignore the old records in the main index we need to add a kill list into the delta index. The kill list should select the IDs which you want to remove from the delta index.
The problem is that the kill list will take effect only if the delta index is placed after the main index, so in a nutshell the kill list removes all IDs from all preceding indexes.
The correct index definition using a kill list should look like this:
index dist
{
type = distributed
local = main
local = delta
}
Need help with search infrastructure? Talk to our engineering team — we've built and maintained search systems for over 15 years.
Frequently Asked Questions
What is the Sphinx kill list and what problem does it solve?
The Sphinx kill list (sql_query_killlist) suppresses specific document IDs from preceding indexes in a distributed setup. In a main + delta scheme, when a document is updated, it exists in both the old main index and the delta. The kill list tells Sphinx to ignore the stale main index version and return only the updated delta copy, preventing duplicate results.
Why does the order of indexes in a distributed Sphinx index matter for kill lists?
Kill lists only suppress documents in indexes that appear before the index defining the kill list. If the delta index is listed before the main index in the distributed config, its kill list has no preceding indexes to suppress — and stale main index documents will still appear in results. Always list main first, delta second: local = main then local = delta.
How do you define a kill list query in Sphinx?
Add sql_query_killlist to the delta index source config. The query should return the IDs of all documents that have been updated or deleted since the last main index rebuild. For example: sql_query_killlist = SELECT id FROM documents WHERE updated_at > (SELECT last_rebuilt FROM index_meta). Sphinx reads this at index time and embeds the suppression list in the delta index files.
Does the kill list affect search performance?
Minimally for small kill lists. Sphinx applies kill list filtering at query time against the in-memory document ID set, which is an O(n) bitmap lookup. If the kill list grows large, rebuild the main index more often rather than letting the kill list accumulate.
Can kill lists be used with Sphinx real-time indexes?
No — kill lists are a feature of traditional disk-based indexes and are not supported by real-time indexes. For RT indexes, deletions and updates are handled directly via DELETE and REPLACE SphinxQL commands, which update the in-memory segment immediately. If you're using a main + RT delta hybrid, the RT delta handles its own update semantics without needing a kill list.
Related Reading
Related posts
Massive Indexing with Sphinx
Imagine the following situation: multiple Sphinx daemons are serving huge amount of data running on numerous hardware servers.
Sphinx: building 1M docs index having no one real doc
Just want to share an interesting trick on how to easily index something with Sphinx without need of populating database with a lot of data or doing smth...
Correct ordering of search result set using Sphinx
**Once you get results from Sphinx, it is important to maintain the way they are ordered. The typical search process used by Sphinx looks like the following:**