Sphinx replication

Sphinx doesn’t yet support any replication for plain or RT indexes out of the box, so you have to implement something by yourself if you need to have a copy of your Sphinx data somewhere. Why you may need it:

• you want to balance load on your servers (e.g. you can send half of all Sphinx queries to one server and the rest to another one)

• you want to have Sphinx index backups that are immediately available once the main Sphinx storage becomes unavailable for some reason

• you want to combine the both of the above with automatic problems detection and switching between servers (e.g. when one server crashes another one automatically starts handling all of the queries)

The easiest way to do plain indexes replication is just copy your Sphinx config to another server, index everything there and have searchd running, but what is bad in this case is that it’s difficult to provide good data synchronization level: Sphinx indexes will be rebuilt separately from the same source (e.g. your main database), they should be identical in the end, but it may happen with some latency, because some server is a bit more loaded or it has worse configuration, there may be a number of reasons.

Here’s how we can do it in another way which would provide minimal latency:

• make indexing only in one place (let’s call it MASTER)

• use rsync on SLAVEs for syncing from the MASTER

• tell Sphinx in all places it should start using the new rebuilt data

Here’s how it can be done in details:

1) Indexing

The challenge here is to NOT use ‘indexer –rotate’ which would make data in one place rebuilt while it’s still old in another place, but make some rotating in the end. For each Sphinx index you want to replicate, e.g.:

index idx
{
...
path      = /path/to/sphinx/indexes/idx
...
}

You need to add the inherited index with only one modification in ‘path’:

index idx_new : idx
{
path = /path/to/sphinx/indexes/idx.new
}

and when you want to reindex ‘idx’ you need to reindex ‘idx_new’ instead. Note you should use exactly ‘.new’ postfix in the new index path, because this way you can make Sphinx automatically rotate the indexes when you want it (see below).

How it works:

You make indexing:

[snikolaev@MASTER ~]$ indexer -c sphinx_simple.conf idx_new
Sphinx 0.9.10-dev (r1996)
Copyright (c) 2001-2009, Andrew Aksyonoff

using config file 'sphinx_simple.conf'...
indexing index 'idx_new'...
collected 1 docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done
total 1 docs, 28 bytes
total 0.007 sec, 3958 bytes/sec, 141.38 docs/sec
total 4 reads, 0.000 sec, 24.0 kb/call avg, 0.0 msec/call avg
total 6 writes, 0.000 sec, 4.7 kb/call avg, 0.0 msec/call avg

Here’s what you have in the dir after that:

[snikolaev@MASTER ~]$ ls -la sphinx_tmp/
total 184
drwxrwxr-x   2 snikolaev snikolaev  4096 Oct 14 07:23 .
drwx------  57 snikolaev snikolaev 16384 Oct 14 07:20 ..
-rw-r--r--   1 snikolaev snikolaev     0 Oct 14 07:23 idx.new.spa
-rw-r--r--   1 snikolaev snikolaev   211 Oct 14 07:23 idx.new.spd
-rw-r--r--   1 snikolaev snikolaev 28401 Oct 14 07:23 idx.new.sph
-rw-r--r--   1 snikolaev snikolaev   316 Oct 14 07:23 idx.new.spi
-rw-r--r--   1 snikolaev snikolaev     0 Oct 14 07:23 idx.new.spk
-rw-r--r--   1 snikolaev snikolaev     0 Oct 14 07:23 idx.new.spm
-rw-r--r--   1 snikolaev snikolaev     1 Oct 14 07:23 idx.new.spp
-rw-r--r--   1 snikolaev snikolaev     1 Oct 14 07:23 idx.new.sps
-rw-r--r--   1 snikolaev snikolaev     0 Oct 14 07:20 idx.spa
-rw-r--r--   1 snikolaev snikolaev   211 Oct 14 07:20 idx.spd
-rw-r--r--   1 snikolaev snikolaev 28401 Oct 14 07:20 idx.sph
-rw-r--r--   1 snikolaev snikolaev   316 Oct 14 07:20 idx.spi
-rw-r--r--   1 snikolaev snikolaev     0 Oct 14 07:20 idx.spk
-rw-------   1 snikolaev snikolaev     0 Oct 14 07:23 idx.spl
-rw-r--r--   1 snikolaev snikolaev     0 Oct 14 07:20 idx.spm
-rw-r--r--   1 snikolaev snikolaev     1 Oct 14 07:20 idx.spp
-rw-r--r--   1 snikolaev snikolaev     1 Oct 14 07:20 idx.sps

2) Data synchronization

There’re different ways how you can synchronize data between servers, I like rsync and here’s how it can be done with rsync:

[snikolaev@SLAVE ~]$ rsync -t --stats --exclude "*.spl" --rsh=ssh "MASTER:~/sphinx_tmp/*" ~/sphinx_tmp/ >> /tmp/sphinx_syncing 2>&1

You should exclude “spl” as .spl is a Sphinx lock file and it doesn’t make sense to copy it

You can add it to cron to run periodically. The period may be few minutes or even 1 minute, because rsync doesn’t make any hard work when there’s nothing to synchronize, i.e. when 2 dirs/files are euqal it won’t overload your servers and you don’t need to make some preliminary check whether the copies differ or not, rsync will do it ok. When you add it to cron remember that you need to prevent 2 instances of rsync from running, you can use some common locks algorithm for this.

3) Sphinx indexes rotating

Now when you have your data synchronized you need to rotate Sphinx instances in all places at the same time, here’s how it can be done if you have one SLAVE:

[snikolaev@SLAVE ~]$ test `tail -n 50 /tmp/sphinx_syncing|grep "Number of files transferred"|tail -1|awk -F: '{print $2}'|perl -p -e 's/ //g'` == '0' && (echo Rotating Sphinx instances on MASTER and SLAVE; echo MASTER; ssh MASTER "[ -f /home/snikolaev/sphinx.pid ] && cat /home/snikolaev/sphinx.pid|xargs kill -SIGHUP &"; echo SLAVE; [ -f /home/snikolaev/sphinx.pid ] && cat /home/snikolaev/sphinx.pid|xargs kill -SIGHUP)

You need to call this right after you make rsync, you can have some script for that.

It looks a bit complex, but it’s actually not: The following

test `tail -n 50 /tmp/sphinx_syncing|grep "Number of files transferred"|tail -1|awk -F: '{print $2}'|perl -p -e 's/ //g'` == '0'

checks whether we need to rotate, i.e. if we have “Number of files transferred: 0” in the end of the log rsync outputs to we don’t need to rotate otherwise we do

The following

ssh MASTER "[ -f /home/snikolaev/sphinx.pid ] && cat /home/snikolaev/sphinx.pid|xargs kill -SIGHUP"

opens ssh connection to the SLAVE, gets PID of the running Sphinx instance if it exists and sends SIGHUP signal to the process. There’s no need to use & or send the process to background by other means as Sphinx index rotating works very fast even on huge indexes.

And we do the rotating locally:

[ -f /home/snikolaev/sphinx.pid ] && cat /home/snikolaev/sphinx.pid|xargs kill -SIGHUP

And here’s what you have in the dir after the rotating:

[snikolaev@MASTER ~]$ ls -la sphinx_tmp/
total 108
drwxrwxr-x   2 snikolaev snikolaev  4096 Oct 14 07:24 .
drwx------  57 snikolaev snikolaev 16384 Oct 14 07:20 ..
-rw-r--r--   1 snikolaev snikolaev     0 Oct 14 07:23 idx.spa
-rw-r--r--   1 snikolaev snikolaev   211 Oct 14 07:23 idx.spd
-rw-r--r--   1 snikolaev snikolaev 28401 Oct 14 07:23 idx.sph
-rw-r--r--   1 snikolaev snikolaev   316 Oct 14 07:23 idx.spi
-rw-r--r--   1 snikolaev snikolaev     0 Oct 14 07:23 idx.spk
-rw-------   1 snikolaev snikolaev     0 Oct 14 07:24 idx.spl
-rw-r--r--   1 snikolaev snikolaev     0 Oct 14 07:23 idx.spm
-rw-r--r--   1 snikolaev snikolaev     1 Oct 14 07:23 idx.spp
-rw-r--r--   1 snikolaev snikolaev     1 Oct 14 07:23 idx.sps

as you can see idx.new.sp* were moved to idx.sp* by Sphinx and the old idx.sp* files were removed.

Notes:

• if your Sphinx instance restarts eariler than expected your replication will be broken, because Sphinx automatically rotates on start. To avoid this you might want to rename the ‘.new.’ indexes right after reindexing and back before rotating

• when you start Sphinx you will see the following warning:

• time spent for rsyncing may be big for huge indexes, but anyway data syncing on fs level is an easier task for your servers than using ‘indexer’.

• be careful with UpdateAttributes() usage if you use this scheme of Sphinx data replication, because once Sphinx flushes your updated attributes to disk in one place the data will become different compared to the second place and it will start your automation making synchronization, you can try to do UpdateAttributes() against the both places to avoid this