mysql> select tags, count(*) c from idx where match('word') group by tags order by c desc limit 10;
+--------------+------+
| tags         | c    |
+--------------+------+
| 210,348      |  366 |
| 204          |  116 |
| 206          |   73 |
| 132,348      |   71 |
| 210,348      |   40 |
| 29           |   36 |
| 25,29,270    |   30 |
| 208          |   28 |
| 180          |   24 |
| 25,348       |   23 |
+--------------+------+
10 rows in set (0.00 sec)

But as you can see the first column contains few values instead of one that you want. To see the needed value used by Sphinx for grouping you need to use special @groupby word:

mysql> select @groupby, tags, count(*) c from idx where match('word') group by tags order by c desc limit 10;
+----------+--------------+------+
| @groupby | tags         | c    |
+----------+--------------+------+
|      348 | 210,348      |  366 |
|      204 | 204          |  116 |
|      206 | 206          |   73 |
|      132 | 132,348      |   71 |
|      210 | 210,348      |   40 |
|       29 | 29           |   36 |
|      270 | 25,29,270    |   30 |
|      208 | 208          |   28 |
|      180 | 180          |   24 |
|       25 | 25,348       |   23 |
+----------+--------------+------+
10 rows in set (0.00 sec)

Now it’s clear that the ‘tags’ was not changed during grouping at all, this is just what you have in one record our of the group while the value which the dataset was grouped by can be seen in the @groupby column.
Hope this will help someone to save some time when he faces this situation.

This year we went to Turkey, to a beautiful 5 star all-inclusive hotel on the cost of Mediterranean Sea. When we arrived Turkey met us with a near-hurricane weather, which was a bit dissapointing (but allowed us to concentrate on our work plans better ), but after a few days it was all nice and sunny and we were able to relax on the beach and took one day-off our schedule for a boat trip near the old Alanya city with breathtaking nature.

Turkey was a great place to meet – we had a great relaxation and very good conditions for our group work sessions and presentations – hotel provided us a conference room with internet and projector.

In 2010 and 2009 we had company meetings in Moscow where we attended HighLoad and Sphinx User conferences where we gave a few talks about using Sphinx search engine.

Before Sphinx version 2.0.2-beta was available, the only way to update real-time (RT) indexes was to use the SphinxQL commands (INSERT/DELETE/REPLACE) which needed to be executed through the MySQL protocol to update RT indexes. At the same time, the original indexes used the indexer tool, which made updating indexes very simple. However, this kind of tool doesn’t exist for real-time indexes.

The only solution was to write a custom script.
Why isn’t a custom script good?

If we compare a custom script to the indexer: indexer was written in C++, it was very well optimized, and tested by many users on different amounts of data. If you want to write a custom script, do so using the language you know best, probably PHP/Ruby/Python, you will need to take care of:

• long term stability
• memory leaks
• data preparation and escaping
• performance optimization, nobody would wait a month to index a terabyte of data

Furthermore, this kind of script will load searchd which causes RT indexes to use a massive amount of memory. For example, 300 indexes with 100mb rt_mem_limit could require up to 30GB of free RAM.

Good news! Sphinx version 2.0.2-beta has added a new command which allows you to convert original indexes into real-time indexes. This command looks like: ATTACH INDEX diskindex TO RTINDEX rtindex.

After a successful ATTACH, the data originally stored in the source disk index becomes a part of the target RT index, and the source disk index becomes unavailable (until the next rebuild). ATTACH does not result in any index data changes. Basically, it just renames the files (making the source index a new disk chunk of the target RT index), and updates the metadata. So it is a generally quick operation which might (frequently) complete as fast as under a second.

So, now using the MySQL protocol you can convert the original index into a real-time one. In order to achieve this, you could write custom script which will execute the attach command for each index using MySQL client, but I found a better way.

I suggest solving this task in a more traditional way using the indexer tool.
To keep the example simple, I will show you how to convert just one index, you can use the code below to extend your sphinx.conf.

Some terminology I use in the sphinx.conf:

• orig – original type of index
• attach – special empty index which will executive attach command
• rtinex – real-time type of index

Before you start the real-time index should be empty and of course the original index should contain some data. The sphinx.conf structure should be as follows:

source orig
{
    type            = mysql

    sql_host        = localhost
    sql_user        = root
    sql_pass        =
    sql_db          = test
    sql_port        = 3306  # optional, default is 3306

    sql_query       = \
        SELECT id, group_id, UNIX_TIMESTAMP(date_added) AS date_added, title, content \
        FROM documents

    sql_attr_uint       = group_id
    sql_attr_timestamp  = date_added

    sql_query_info      = SELECT * FROM documents WHERE id=$id
}

index orig
{
    source          = orig
    path            = idx/orig
    docinfo         = extern
    charset_type        = sbcs
}

index rtindex
{
    type            = rt
    rt_mem_limit        = 32M

    path            = idx/rtindex
    charset_type        = utf-8

    rt_field        = title
    rt_field        = content
    rt_attr_uint        = group_id
    rt_attr_timestamp = date_added
}

source attach
{
    type            = mysql

    sql_host        = 127.0.0.1
    sql_user        =
    sql_pass        =
    sql_db          =
    sql_port        = 9306  # optional, default is 3306

    sql_query       = select 1 from testrt
    sql_query_post = ATTACH INDEX orig TO RTINDEX rtindex
}

index attach
{
    source          = attach
    path            = idx/attach
    docinfo         = extern
    charset_type        = sbcs
}

Notice that ‘attach’ index is connected to searchd not to mysql and
using sql_query_post executes the attach index command.

The configuration is ready, lets start converting:

Start searchd:

./bin/searchd  -c ./etc/sphinx.conf

Ignore the warning about the ‘attach’ index being empty.

I don’t have any data at ‘orig’ index, so I indexed it before running the transformation:
Skip this command if you have data in your ‘orig’ index:

./bin/indexer -c ./etc/sphinx.conf orig --rotate

Time to convert:

 ./bin/indexer -c ./etc/sphinx.conf attach

Pretty simple.

Let’s now check RT:

mysql -P9306 -h127.0.0.1
mysql> select * from rtindex;
+------+--------+----------+------------+
| id   | weight | group_id | date_added |
+------+--------+----------+------------+
|    1 |      1 |        1 | 1322419937 |
|    2 |      1 |        1 | 1322419937 |
|    3 |      1 |        2 | 1322419937 |
|    4 |      1 |        2 | 1322419937 |
+------+--------+----------+------------+

Yes, the data is in place, so the index was converted successfully!

Conclusion
The big improvement is that indexing and converting can now be done using the same indexer tool. One thing you need to be sure of is to extend the sphinx.conf generator script with the ‘attach’ and ‘rtindex’ indexes.

The trick is to use “pager” directive to redirect the output to a mail program:

mysql> pager mail -s "subject" yourname@yourdomain.com
PAGER set to 'mail -s "subject" yourname@yourdomain.com'

The above will redirect the output to yourname@yourdomain.com and the subject will be “subject”.

Then just start your long-lasting query to MySQL:

mysql> select count(*) from feed where ext_key like 'a%';
1 row in set (10 min 35.88 sec)

or SphinxQL:

mysql> select * from huge_index;
20 rows in set (1 min 14.00 sec)

and be informed via email.

For RT-index you can estimate memory consumption by calculating the size of all on-disk chunks (minus .spd & .spp sizes as noted above) plus RAM-chunk size (rt_mem_limit)

The formula looks good, but notice that they also added size of RAM-file into it.

What is that RAM-file?

The RAM-file is used to store all necessary data for Sphinx RT-indexes.
Sphinx keeps the RAM-file in memory to support real time updates for RT-indexes.
Before the RAM-file data is copied to a general chunk file, the RAM-file should reach rt_mem_limit in size.
Each RT-index has a personal RAM file.

So, what’s wrong with the RAM-file?

Imagine we have 30 indexes each 3Gb in size.
If we want to keep number of index-chunks low, i.e. less than 5, we need to set rt_mem_limit to 1Gb.
In this case we will have 3 chunks for each index.
Now let’s estimate how much memory we need to support this configuration.
For 30 indexes we will have 30 RAM-files multiplied by 1Gb (rt_mem_limit) which will make 30Gb.
30Gb of free memory is required to support our configuration and that’s without counting .spa and .spi files.
Of course in a real system with random data distribution Sphinx will consume much less memory, probably 1.5 times less.

Ok, lets decrease rt_mem_limit 10 times to 100Mb.
In this case we will get 3Gb memory required, but the number of chunks for each index will grow from
3 to 30 (overall it is 900 chunks for 30 indexes).
Now let’s imagine how fast Sphinx will query each of 30 or more chunks.
It will be very slow, because of many disk I/O operations, especially if we query more than one index at a time!

Conclusion
Low rt_mem_limit is good for memory, but hurts Sphinx performance.
High rt_mem_limit could gives good performance, but requires a lot of free memory.

The Sphinx team definitely needs to optimize this feature, i.e. add a new option to control RAM-file size (i.e. rt_RAM_limit).
rt_RAM_limit in conjunction with rt_mem_limit could give more flexibility to better setup the massive number of RT-indexes
on an average server with 16Gb of memory on board.
Another option is to exclude .spp and .spd data from RAM-file, so it keeps only those things in memory that are supposed to be there – .spa and .spi.

Update: according to the latest documentation SPH_RANK_SPH04 solves the problem. Anyway I suggest to read the article, it was interesting investigation.

Here’s the quote from Sphinx documentation:

Statistical rank is based on classic BM25 function which only takes word frequencies into account. If the word is rare in the whole database (ie. low frequency over document collection) or mentioned a lot in specific document (ie. high frequency over matching document), it receives more weight. Final BM25 weight is a floating point number between 0 and 1.

So it states that a word which is mentioned a lot receives more weight. But it’s not always true.
Let’s say we have the following table with some words repeated in every row:

mysql>select * from titles;

| id | title |

| 1 | Test Category - Test Article |
| 2 | Test CategoryA |
| 3 | Test CategoryB |

Please notice that “Test” occurs two times in document #1 and only one time in documents #2 and #3.
So we can expect that a search for the word “Test” will match #1 as the first result.

Let’s test it using SphinxQL (Sphinx query language):

mysql> select * from test1 where match('Test') ;

| id | weight |

| 2 | 1319 |
| 3 | 1319 |
| 1 | 1252 |

Hmm, something’s wrong here, document #1 got lower weight than documents #2 and #3.

As it was investigated the key problem is that the word “Test” occurs in each document. See the quote from Andrey Aksenoff (creator of Sphinx Search):

This is actually by BM25 design. It penalizes (!) the keywords that are overly frequent, ie. occur in more than 50% of the collection documents.

In my case the word “Test” occurred in 100% of documents and that’s why was penalized.

A possible workaround of this can be to use another ranking mode which doesn’t use BM25, i.e. SPH_RANK_WORDCOUNT or SPH_RANK_MATCHANY:

SPH_RANK_WORDCOUNT:

mysql> select * from test1 where match('Test') option ranker=wordcount;

| id   | weight |

|    1 |      2 |
|    2 |      1 |
|    3 |      1 |

SPH_RANK_MATCHANY:

mysql> select * from test1 where match('Test') option ranker=MATCHANY;

| id   | weight |

|    1 |      1 |
|    2 |      1 |
|    3 |      1 |

It looks like SPH_RANK_WORDCOUNT solves our problem.
In some cases it can be a solution, but in general it can’t, because for 99% of queries we might want to use as smart ranking algorithm as default SPH_RANK_PROXIMITY_BM25 ranking mode.

Percona Live is an intensive two-days MySQL summit. There are many tracks of expert speakers, including Percona consultants and hand-picked guests. The sessions are 100% technical—even the sponsored sessions.

It will be an honor to speak among many highest-level professionals working on the largest projects in the world – speaker list and schedule is really impressive, check it out at the official conference website.

Our session is titled “Building 50TB-scale search engine with MySQL and Sphinx” and shares the experience of using MySQL and Sphinx to build a search engine – the architecture of a stable, high-performance, easy-to-scale system that has low maintenance costs. We’ll discuss technical details starting from hardware and software configuration to maintenance, monitoring and high-availability solutions – how we ensure system is stable and what we do to to keep up with the data and usage growth.

If you are visiting the conference do not hesitate to say hello! If you are not attending, but in London during Oct 24-25 and would like to meet us, please let us know!

P.S. organizers provided a discount code “come-c-talk” – use it on registration to get £40 off for your tickets.

You can find a list of our projects at Ivinco Software page.

If you have any questions, feel free to contact us or leave a comment.

Since version 2.0.1 Sphinx has ability to build snippets in a parallel mode. It means that Sphinx will use several CPUs to do that. Below is the instruction on how to efficiently do that, but recommend this only if you need to build excerpts for large amount of text like 10-100 Mb.

Sphinx parallel processing is controlled by ‘dist_threads’ option, which tells searchd how many CPUs should be utilized for search processing. This parameter is also used by BuildExcerpts() API call in combination with ‘load_files’ option. By default the first parameter of function BuildExcerpts() is expected to be an array of text strings, but if the ‘load_files’ option is set to ’1′ then it should contain another thing – array of file names. Each file should contain the text for which you want to build an excerpt. These two options in combination allow Sphinx to build excerpts in a parallel mode which works much faster for huge amount of texts for processing.

But, this implementation has a bottleneck – it requires a file system to be used to read/write the files. If you use a disk it may be 1000 times slower than when it’s done in memory, so the right solution is to use file system in memory.

tmpfs does the job, it is a file system in memory, it is supported by the Linux kernel from version 2.4 and up. So I used this to workaround the files read/write performance issue.

File system

How to mount in-memory file system tmpfs:

mkdir /space
mount -t tmpfs -o size=1G,nr_inodes=10k,mode=0700 tmpfs /space

First I created the directory and then mounted tmpfs to that.
Among the parameters I specified file system size = 1 Gb and access permissions only for owner of the directory /space.

My BuildExcerpts() function based on files

function buildExcerptFile($documents, $options = array())
{
        foreach($documents as $doc){
            $file = "/space/".'snip_'.md5($doc).'_'.time();
            file_put_contents($file, $doc);
            $files[] = $file;
        }

        $client = new SphinxClient();
        $client->setServer('localhost', 9312);

        $res = $client->BuildExcerpts( $files, 'index', $keywords,
                array(
                    'around'=>10,
                    'limit' => 300,
                    'load_files' => 1
                    )
                );

        foreach($files as $file){
            unlink($file);
        }

        return $res;
}

Function works in three stages:

• 1. Convert text documents into temporary files. I choose dynamic file names to prevent file name collisions.
• 1. Call BuildExcerpts() function. The first parameter contains the list of file names instead of the list of documents and the third parameter contains ‘load_files’ option equal to ’1′, which tells BuildExcerpts() to process the documents as files.
• 1. Remove the temporary files to clear garbage from memory.

Setup dist_threads option

Add the following in the searchd section of your Sphinx config:

dist_threads = 2

I prefer to set dist_threads equal to number of CPUs in the system.

Conclusion

In my testing environment I gained two times better performance compared to the default BuildExcerpts() call performance.
Average size of documents was about 3-10 Mb. I passed 100 documents per one BuildExcerpts() call.

We’re glad to announce the third version of WordPress Sphinx Search plugin.

WordPress Sphinx Search plugin allows to use Sphinx Search Server power to enable ultra-fast and feature-rich search on WordPress-based websites. It is especially useful when your WordPress site becomes very large.

The new version enables a range of new tools that improve search quality and can help you increase your site’s rankings in search engines like Google and Bing.

With this version search results are more relevant, you can perform searches in posts, pages and comments using flexible search syntax sorting the results by freshness and relevance. This plugin comes with sidebar widgets to display the most recent searches, top searches and related searches.

New “freshness & relevance” search mode

New “freshness & relevance” search mode lets you sort results by relevance within time segments. It is perfect for blogs and news sites as new relevant posts will rank higher than old ones not allowing old information to take best positions in the search results. This mode is on by default.

Related search terms widget

This is a great tool for SEO and navigation on your site, it shows relevant search phrases for different sections of your blog:

• Search result pagess
• Posts
• Pages

For example if site visitor is on your product page the widget will analyze the page title and will show the most relevant search terms. When visitors search for “ipad”, other relevant search terms that include “ipad” will be displayed in this sidebar widget.
And the most tasty:

• This widget will help search engines like Google  and Bing to find more relevant pages on your site to improve your search engine rankings
• It can also help visitors find relevant information on your site easier

Search terms management

All searches, performed by your visitors are being tracked and we’ve developed a range of tools to manage the tracked search terms. You can control the search terms that are displayed in the widgets.
With the search terms management tools you can:

• Show only approved search terms
• Block search terms
• Make blacklist to hide all search terms containing the blacklisted phrases
• Import your own search terms which fit your blog site best of all
• Export search terms to Excel-like document

Search engine friendly URLs

Human readable or search engine friendly URLs are the URLs that make sense to both humans and search engines because they explain the path to the particular page they point to.

A few advantages of search engine friendly URLS:

• Letting users know what the URL is about
• Keyword-rich URLs impact SEO (can improve your page rankings on search engines like Google).
• URLs are cleaner, meaningful and descriptive, for example, http://yourblog.com/search/product+name/ looks better than http://yourblog.com/?s=product+name

How to enable search engine friendly URLs?

By default this option is turned off, so here’s how you can enable it.

Activate WordPress permalinks. Go to WP Admin → Settings → Permalinks and chose non default option.

Then download and install WordPress Sphinx Search plugin version 3.3 or higher. Follow this tutorial to install Sphinx Search plugin.

After installation is complete, go to Settings → Sphinx Search → Search settings (tab) and set “Enable friendly URLs” option to ON.

And many more…

• Sphinx in each widget component. All parts of the plugin now use Sphinx which means no more heavy MySQL full-text search queries. All works extremely fast.
• Search terms statistics. This is a good tool to analyze your search terms. It helps you better understand what your visitors look for on your site.
• Improved top searches widget. Lots of new settings have been added to the top searches widget. You can now take more control over what things will be shown on different types of pages on your site. Besides, it is now easier to manage it using the search terms management tool. You can also add your own terms to always display in top searches list — good ability to promote something on your site, for example your new product.

You can find more information on the plugin official page.

Support

This plugin is developed by Ivinco. If you need commercial support, or if you’d like WordPress Sphinx Search Plugin customized for your needs, we can help.