Just want to share an interesting trick on how to easily index something with Sphinx without need of populating database with a lot of data or doing smth like that. The below is a full Sphinx config which lets you build a 1M docs index consisting of random 3-char words and one numeric attribute with a random value. All you need is just any connection to any db (in this case ‘mysql -u root’ works).

source min
{
    type = mysql
    sql_host = localhost
    sql_user = root
    sql_pass =
    sql_db = test
    sql_attr_uint = attr
    sql_query_range = select 1, 1000000
    sql_range_step = 1
    sql_query = select $start, mid(md5(rand()), 1, 3) body, floor(rand() * 100000 * $end) attr
}
index idx
{
    path = idx
    source = min
}
searchd
{
    listen = 3306
    log = sphinx.log
    pid_file = sphinx.pid
}

As you can see the tricky part is to utilize Sphinx’ directives sql_query_range and sql_range_step to let Sphinx loop until it makes 1M docs collection. The drawback is slower indexing comparing to real fetching the same amount of data from db, but come on, you’re not going to use this in production, right?

I hope you’ll find it helpful when you decide to play with Sphinx.

Searching API

• Searches in Sphinx index
• gets id’s of matching records
• puts ids into MySQL query to get the rest of the information.

In order to demonstrate this, I have used the following table as my development environment:

`some_table` (
`id` int(10) unsigned NOT NULL auto_increment,
`some_text` text,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8

The table was filled by:

mysql> select * from some_table;
+----+----------------+
| id | some_text |
+----+----------------+
| 2 | test text |
| 4 | text test |
| 6 | something else |
| 8 | new row |
| 10 | new text |
| 12 | empty field |
| 14 | old text |
| 16 | result row |
| 18 | another row |
+----+----------------+

The Sphinx config file looks like:

source min
{
type = mysql
sql_host = localhost
sql_user =
sql_pass =
sql_db =
sql_query = select * from some_table;
}

index idx_min
{
path = idx/idx
source = min
}

searchd
{
listen = 39306:mysql41
log = logs/sphinx.log
pid_file = sphinx.pid
}

So, we get the results by using SphinxQL:

mysql> select * from idx_min where match ('text | new') ;
+------+--------+
| id | weight |
+------+--------+
| 10 | 1588 |
| 8 | 1568 |
| 2 | 1520 |
| 4 | 1520 |
| 14 | 1520 |
+------+--------+
5 rows in set (0.00 sec)

If those ids are put directly into the query, the results won’t keep the same order from the source (in this case relevance):

mysql> select * from some_table where id in (10,8,2,4,14);
+----+-----------+
| id | some_text |
+----+-----------+
| 2 | test text |
| 4 | text test |
| 8 | new row |
| 10 | new text |
| 14 | old text |
+----+-----------+

In this situation, the MySQL condition ‘ORDER BY FIELD’ can be helpful:

mysql> select * from some_table where id in (10,8,2,4,14) ORDER BY FIELD(id, 10,8,2,4,14);
+----+-----------+
| id | some_text |
+----+-----------+
| 10 | new text |
| 8 | new row |
| 2 | test text |
| 4 | text test |
| 14 | old text |
+----+-----------+

By using the method outlined in this brief tutorial, you can keep the order of the results.

Many of our customers want to improve their search engine by allowing users to refine their searches by showing them top or related queries from previous days or weeks. When they click on these links, they are taken to a corresponding search results page.

This works by storing all user queries to your database and then indexing them for the last N days with Sphinx. Then you simply make a query against the index to find the most popular related queries that match (or are similar) to the current query.
Here are a few tricks to add these features in the most efficient way:

1. Use the query text hash as the id, this will allow you to avoid grouping and therefore decrease the response time, Sphinx will ignore duplicated IDs during the indexing stage or when you insert (if you are using a real time index).
2. Some similar queries might differ slightly, but not significantly (eg. ‘MYSQL innodb’, ‘mysql innodb’ and ‘mysql, innodb’ mean the same). You probably want all of them to be merged into one. To do this, you need to normalize the texts. This is where you can use the ‘call keywords()’ command (which is named BuildKeywords() in Sphinx API), this can help you tokenize the query exactly the same way Sphinx will tokenize it while indexing, all you need to do is concatenate the returned keywords and you will have the query cleaned from everything insignificant.

So your index structure would be like this:

id: hash (normalized query)
field: normalized query
attr: count
attr: id in db

1. You can use the main+delta scheme to make indexing faster.
2. You can use UpdateAttributes() (or corresponding SphinxQL ‘UPDATE’ query) to increment the query hit count and index less frequent. This is especially useful if you don’t have a lot of new queries coming and mainly you need to update the existing ones. Remember one thing in this case: if you have a lot of hits per second you might encounter a collision issue since Sphinx cannot increment attributes instantly, it can only update it, so you will have to increment on the application side and if you do it simultaneously in a few processes you could end up with less increments than expected.
3. If you have a lot of memory and performance is more important for you than memory consumption you can put the query in its original state (before the normalization) into a string attribute. This way you won’t need to make an additional query to the db to find the query text and also you won’t need to store the query id in the index. The structure in this case is:

id: hash (normalized query)
field: normalized query
attr: count
attr: original query

Since the queries index is usually not very large you can use a real time index and then you won’t need to reindex it at all. If you do this, you might also want to periodically delete older queries. This is a good way to make the id based not just on hash(query), but also on the hash(query + date) and the hash(query) which will be an additional attribute as well as ‘date’.

So your index structure will be:

id: hash (normalized query concatenated with date)
field: normalized query
attr: count
attr: hash(normalized query)
attr: date
attr: id in db / original query

Then you need to do two more things in the Sphinx query:

1. Filter out older queries
2. Group by the hash (normalized query) and sort by sum(count)

One more thing to mention about related queries is that it makes sense to include only queries that actually produce results. People don’t like to click and be taken to “nothing is found” page. The problem here is there might be a query that previously returned results, but doesn’t find them any longer. You may have deleted them from your main search index. If you’re worried about this, it makes sense to do two things:

1. Add one more attribute ‘last_total_found’ to the queries index and update it along with updating the ‘count’ attribute. Thus the “nothing is found” issue can happen only for the first user who clicks on the link.
2. Periodically recheck of all your queries to be sure they’re still producing results even though this might take additional resources.

Many customers that we have helped with integrating search into their applications wanted their search to be more intelligent than just strictly matching a query with documents.
There are many ways to do this. Sphinx makes it very easy, as fuzzy matching is included out of the box. It consists of two main components:

1. Quorum operator:

   "computing and technology news"/2
   

   This means that at least two words of the phrase should match, i.e. this query would find texts containing both “computing news” and “technology news”.

2. Proximity search operator:

   "computing news"~3
   

   This means that there can be less than N non-matching words between the words from the query. Here are a few examples:
   If the text is “a b c d e f g h” :

   • “a h”~7 would find that since “b c d e f h” are 6 non matching words and this is less than 7
     however “a h”~6 wouldn’t find the text
   • “a d h”~6 would find that
   • “a d h”~5 wouldn’t

Many of our customers like the second pass logic when the first query to Sphinx is strict. If nothing is found or not enough results are returned, the second less strict query is issued. There also may be more advanced logic with 3rd and 4th passes. It just depends on your requirements and whether or not you want your user to at least find something in any case. Or, on the contrary, you can let them find something that exactly matches his query.

Sometimes it makes sense to do it the other way around and make the query stricter. For example, if your default matching strategy is ‘any word should match’ and your application doesn’t have any extended syntax to allow users to specify the best query themselves, it makes sense to first try the ‘all words should match’ strategy or even the ‘phrase should match’. This might significantly increase the quality of the search.

In some applications it makes sense to parallelize the 1st/2nd pass queries and so on. This can be easily done using Spinx multiquery. The logic here is to do the 2nd pass query beforehand and if nothing is found by the 1st pass query the results will be ready and since the queries were done simultaneously this improves performance. However, this depends on a lot of things. You should be careful, as it can reduce performance sometimes. These things are:

• What hardware are you using? If it’s not powerful enough to handle two queries at once or the response time is close to the response is double that of a single query, it makes little sense to use this technique.
• What is your load? If your Sphinx is already heavily loaded, you will get a worse response time.
• What are your statistics? If for 99% of queries the 1st pass the query returns results, there’s little point to make the 2nd pass query along with the 1st one, this will just waste resources.

When integrating full-text search into your application it’s important to think about tokenizing your text, or how it gets split up into words. This is the first process your data goes through once you’ve built the config and begun indexing.

Sphinx includes a variety of settings that give you full control of the way your text gets tokenized. They are:

• charset_table and ngram_chars; lets you define characters that will be treated as normal characters, everything else will be a separator.
  You can also use:
  • Ranges: a..z
  • Char mapping: A->a
  • Range mapping: A..Z->a..z
  • Char codes: U+410..U+42F
• ngram_chars; If you have to deal with chinese/japanese/korean text whose structure differs significantly from other languages, this can be useful as it allows you to insert a separator after each hieroglyph and instead of one long word you will have each hieroglyph indexed as a separate token, the same will happen with the query allowing your users to find what they’re looking for.
  Example:
  ngram_chars = U+3000..U+2FA1F
• ignore_chars; lets you totally ignore some characters.
• blend_chars; allows you make some characters both separators and normal characters as well. For example, if you have twitter-related things like @username in your data and want to allow users to search exactly for the twitter nicks you can easily do so by adding ‘@’ into the blend_chars, then the query ‘@username’ wouldn’t find text ‘username’ while the query ‘username’ would still find them both.
  Example:
  blend_chars = @

Exceptions, wordforms and stopwords

That concludes the character level commands. The next process the text goes through while getting indexed is the word handling level controlled by stopwords, word length directives, exceptions and wordforms.

• By using stopwords, you can make one or more lists of words that will not be indexed at all. They may be very frequently used words such as ‘a’, ‘the’, ‘and’, etc… There are a few goals for this: Firstly, it improves the search quality because once these words are excluded from the index and from the query, the latter will become more agile. For example if you have ‘on’, ‘a’, ‘the’ and ‘my’ in the stopwords, the query ‘search on my site’ will also find such phrases as ‘search on a site’, ‘search on the site’, ‘search my site’ etc. Secondly, it decreases the size of your index because indexing something which exists in almost all documents will take more resources and thus slow down performance.
• The related directive used to configure stopword behavior is stopword_step. If it is set to one (which is a default value) the query “search on my site” will match with any of the following queries: “search on a site”, “search on the site”, or in general “search + ANY_OTHER_STOPWORD + ANY_OTHER_STOPWORD + site”. However, if it is set to zero it will also match with “search my site” or “search site”. In other words, all stopwords will just be ignored.
• min_word_len; This specifies the minimal word length to index. You need to be careful to correctly use the related directive overshort_step. If it’s set to 1 and your min_word_len = 2 the query “search on site” will not match “search on a site”, however if it’s set to 0 it will match that phrase.
• prefix/infix directives help you to filter out things that shouldn’t be indexed and on the contrary index substrings that should be.
  Not only is it important to index whole words, but sometimes it makes sense to be able to search by close variant words, e.g. if you type in “dogs” you might also want to find “dog” or you want “hero” to find “superhero”. This is enabled by a variety of directives: min_prefix_len, min_infix_len, prefix_fields, prefix_fields, infix_fields, enable_star, expand_keywords. Using these directives, you can configure exactly how your words should be split into substrings. You can specify how many characters can be trimmed on the end or on the both ends of the word, but also what full-text field this should be applied to, whether to support wild-card syntax (e.g. dog*) or treat all query words as substrings. Be aware that using the prefixes and the infixes increases your index size and might affect the performance.
• Exceptions and wordforms
  Another thing Sphinx allows you to do is define list of words that should be mapped to each other which means you can tell Sphinx to treat USA, U.S.A, US, U.S, America, United Stated, United States of America as one and the same word, for example. To do this you should use the ‘exceptions’ directive. It works on very low level before even tokenizing the text.  Using this, you can map all the words related to a specific one, for example USA and the same will happen with the search query. This might dramatically increase your search quality especially if you have to deal with products that can have different names and abbreviations, but all mean the same (PlayStation, Play Station, PS, Sony PlayStation etc.). Another reason to use the exceptions is to index something which has a stopword or very short word which would otherwise be truncated. As an example, “The Matrix” would be converted to “Matrix” or “vitamin a” which would become “vitamin” when min_word_len = 2 or greater.
  *Note: Since exceptions work before tokenizing they have to be case sensitive (at least this is how it works now).

  Example:
  U.S.A. => USA
  U.S. => USA
  US => USA
  us => USA

  Sometimes it makes sense to map word to itself in the exceptions:
  AT&T => AT&T

  This allows to let the user search for ‘AT&T’ and find excactly ‘AT&T’, not separate words ‘AT’ and ‘T’ which would be return by the tokenizer if ‘&’ is a separator.

  The ‘wordforms’ directive is similar to the exceptions, with one difference. It’s applied after tokenizing so they’re case insensitive which is good, but you cannot use it to handle cases like ‘AT&T’ which is bad. However, the wordforms work much faster as they were designed to work with millions of different word forms and this can be especially useful when done along with stemming.

• Stemming
  Using stemming you can improve your search quality even more. For instance if you enable English stemmer ‘walking’, ‘walks’, ‘walked’ will be all converted to ‘walk’. The same will happen with the query and you will be able to find ‘he was walking on the street’ by searching for ‘walked’.
  Sphinx enables English, Russian and Czech stemming out of the box and it was built with ––with_libstemmer it supports anything else using the Snowball libstemmer library.

  These morphology processors are not perfect, but as I said the wordforms, they can be especially useful when combined with stemming. Because once the word is found in the wordforms it wont get processed later by the stemmer. Therefore, you can override something which doesn’t work perfectly. For example, ‘does’ gets converted to ‘doe’ by the English stemmer, but you can override this using the wordforms like this:

  does > do
  Likewise, it will also match ‘do’ when ‘does’ is typed in.

• html stripping
  Another nice feature is html strippping. Often Sphinx is used to search among web pages which are HTML documents and using the ‘html_strip’ directive you can do the html parsing job.
  This is important when you still need the markup in your datasource and don’t want to store both raw and stripped versions of the text.

To figure out exactly how your current index settings work, you can use the ‘call keywords()’ function in SphinxQL:

mysql> call keywords('abc a b c the AT&T A&BULL', 'idx');
+-----------+------------+
| tokenized | normalized |
+-----------+------------+
| abc       | abc        |
| b         | b          |
| c         | c          |
| AT&T      | AT&T       |
| bull      | bull       |
+-----------+------------+
5 rows in set (0.00 sec)

You can see that the text ‘abc a b c the AT&T A&BULL’ was split into words ‘abc’, ‘b’, ‘c’, ‘AT&T’ and ‘bull’. ‘a’ and ‘the’ were skipped, because they’re stopwords, AT&T is an exception and that’s why it was not split into ‘AT’ and ‘T’ as happened with ‘A&BULL’ which was split into ‘A’ and ‘BULL’ and then ‘A’ was not indexed, because it’s a stopword.
In the SphinxAPI this function is called ‘BuildKeywords()’.

Autocomplete provides suggestions as the user is typing in the search field. Everyone knows how it works, this is the first thing we see when using Google:

Many people like this and want it in their applications and Sphinx can be used to activate this functionality.

First, decide what things should be shown as suggestions. These can be:

• Blog post titles or anything else that makes sense in your application
• Previous/top user queries
• Separate words from your dataset prepared with ‘indexer –buildstops’, or by other means
• Anything else that works

We just need to gather the data we want to use in the autocomplete somewhere so it can be indexed by Sphinx or we can use Sphinx real time index to populate the autocomplete index (see below).

So once we have our suggestions data, we need to build the index. Many people find it difficult to do since Sphinx requires a unique id for each document and sometimes some documents don’t have ids or they can intersect if you want to merge them (eg. recent user queries or product names). We can handle this by using hashes instead of the ids. This can be crc32() if your autocomplete index is not very large or something else like half of md5() if your index is large enough to cause crc32() collisions.

The other thing we need to look at is how we will sort the suggestions. To do this, we can make a few attributes. We often use attributes such as suggestion length and suggestion word count (which can be easily calculated using the ‘sql_attr_str2wordcount’ directive). Here’s an example:

source keyword {
        sql_query = select crc32(name), name keyword, length(name) length from animals
        sql_attr_uint           = length
        sql_field_string        = keyword
        sql_attr_str2wordcount  = keyword_wc
...
}

index keyword {
        source                  = keyword
        path                    = keyword
        docinfo                 = extern
        min_prefix_len          = 1
...
}

The problem with using the hash is we will most likely have difficulty finding the text by hash in the db, but we won’t need this if we have all our strings set up right in the index using Sphinx string attributes (sql_field_string). There’s another reason to do so – users attack this index more frequently than the others because each typed character will generate a query against Sphinx. Thus we need to optimize it to provide the best performance. We can accomplish this nicely by storing the strings in memory. This removes the need to look in the db, and consequently, it is very fast.

It is important to make sure we allocate enough memory for the index so Sphinx doesn’t do unwanted reads from the disk:

index keyword {
...
        mlock                   = 1
}

Now, all we need to do is query the index like this (or use other attributes to sort the results):

mysql> select keyword from keyword where match('^b*') order by keyword_wc asc, length asc limit 10 option max_matches=10, ranker=none;
+-----------+
| keyword   |
+-----------+
| bee       |
| bat       |
| bear      |
| boar      |
| bison     |
| beaver    |
| buffalo   |
| bush baby |
+-----------+

If we don’t need to use text ranker (i.e. sorting by attributes is enough) we need to make sure to set ‘ranker=none’ option. This will improve performance.

It is good to apply the Sphinx real time index to autocomplete suggestions, because:

• We won’t need to rebuild this index if it is real time. We just need to insert into it in the same code we use to insert it into the db.
• When we use last user queries we can insert them into the real time index only (unless you need it in the db as well).

Although the Sphinx real time indexes have a few disadvantages they’re not very critical with respect to autocomplete. Since the index isn’t usually very big and we’re going to store the strings in memory anyway, the difference – when compared to a non-real time index, will be even less significant in terms of resources consumption.

Sphinx usage in any project usually starts with building the Sphinx configuration and indexing your data. In this article I’m going to point out some cool things that Sphinx provides regarding the configuration and how your data is indexed:

• First, I have to say that it supports inheritance. This is very useful because you can define things (like the connection to your database) only once and they will be reused for all the other sources, here’s an example:
  

  source text1
  {
          type            = mysql
          sql_host        = localhost
          sql_user        = b
          sql_pass        = u
          sql_db          = b
          sql_port        = 3306
          sql_query       = select id, body, published, lat, long, category from table
          sql_attr_timestamp      = published
          sql_attr_float  = lat
          sql_attr_float  = long
          sql_attr_uint   = category
  }
  
  source text2 : text1
  {
          sql_query       = select id, user_name, inserted from table2
          sql_attr_timestamp      = inserted
          sql_attr_float  =
          sql_attr_uint   =
  }

• If you’re a real hater of copy-paste technology, you’ll be happy to know that Sphinx config supports shebang which allows you to create your Sphinx config using your favorite scripting language, for example:

  #!/usr/bin/php
  <?php $m = new mysqli('maindb', 'user', 'password', 'main'); $res = $m--->query("select site_map.id, ip from site_map left join server on site_map.master_id = server.id");
  while ($row=$res->fetch_assoc()) {
          $n = $row['id'];
          $host = $row['ip'];
          echo "
  source chunk{$n} {
      type = mysql
      sql_host = {$host}
      sql_user = user
      sql_pass = pass
      sql_db = c{$n}
      sql_query = select id, {$n} chunk_id, body from a{$n} where id>=\$start AND id<=\$end and crawled=0
      sql_query_range     = SELECT MIN(id),MAX(id) FROM a$n
      sql_range_step = 100000
  }
  ";
  }
  ...

This creates many possibilities of making the Sphinx config really dynamic. It is especially important in large applications that have a lot of indexes and multiple servers. You can set up your config just once, and as you scale your project, it will rebuild itself automatically. One thing you will need to be careful is sending signal to the running searchd process so it starts using the new config. You can make life even easier by incorporating the signal sending into the config so that once you reindex, and see that the config is updated, you can inform your searchd process about this and it will switch to the new config.

Another cool thing you can do with data indexing is using a so-called main + delta scheme. Here’s what it does:

• It reduces the frequency that the main part of the index needs to be rebuilt.
• When you update a field in your data source and need to update this in your Sphinx index, you only need to rebuild the delta. When the delta is big enough and takes significant time to rebuild itself, you can dump all the data into the main index. This empties the delta and readies it to accept new data and start rebuilding fast again.
• It makes your updated data appear in the index much faster.
• By fetching less data from the database, it reduces the weight of the load on your server.

There are two main approaches when it comes to the main+delta:

• Split data by ‘id’:

  source main {
          sql_query_range = select min(id), max(id) from dogs
          sql_query       = select id, name from dogs where id >= $start and id <= $end
          sql_range_step = 1000
          sql_query_post_index = replace into sphinx_helper (type, const) values ('dogs', $maxid)'
  ...
  }
  
  source delta : main {
          sql_query_range = select const, (select max(id) from dogs)) from sphinx_helper where type = 'dogs'
          sql_query_post_index = replace into sphinx_helper (type, const) values ('dogs', $maxid)'
  ...
  }

  This is a more traditional method; however, the drawback is that if an older document gets updated and its ‘id’ is unchanged, it won’t be reindexed until main part of the index is rebuilt.

• Split data by ‘updated’:

  source main {
          sql_query_pre   = REPLACE INTO sphinx_helper set type = 'cats', tmp = (select max(updated) from cats)
          sql_query_range = select unix_timestamp(min(updated)), (select tmp from sphinx_helper where type = 'cats') from assets
          sql_query               = select id, name from cats where updated >= $start and updated <= $end
          sql_query_post_index = update sphinx_helper set const = tmp where type = 'cats'
          sql_range_step  = 3600 ...
  }
  source delta : main {
          sql_query_range = select const, unix_timestamp() from sphinx_helper where type = 'cats'
          sql_query_killlist = select id from cats u  where updated > (select const from sphinx_helper where type = 'cats')
          sql_query_post_index =
  ...
  }

  This aims to avoid the drawback of the split by id approach: once an old document gets updated its ‘updated’ field value will be updated as well, and it will be indexed as soon as the delta index is rebuilt. Technically the updated document will now be in both the delta and the main index parts. Using sql_query_killlist allows us to explicitly tell Sphinx to use the one in the delta.
  You should be careful to pick the best sql_range_step value to define the period of time used to fetch docs from the db at once. 3600 means one hour (3600 seconds). If the value is too low, there will be too many queries that don’t return results and it will waste resources and indexing time. On the other hand, if too much time is allowed, too much data could be fetched from the db, which could overload it.

Many of our customers want to incorporate “did you mean … ?” functionality into their applications. How it works is that when a typo is made in the query, a corrected version of the typo is suggested:

This can be done using a special technique with Sphinx that has been successfully used in many different projects.

There’s a demo of this technique in misc/suggest/ dir in the Sphinx source and an article (in Russian) where Andrew Aksyonoff describes the main idea behind this. The following is based on his original idea:

The technique is based on comparing the difference between the words in the current query and words from a dictionary. How this works is:

1. You should decide what dictionary of proven words would be most suitable. It can be some real dictionary, titles of your products, or you can even generate a new dictionary based on your current index using the ‘indexer –buildstops … –buildfreqs’ command.
2. You can modify each word in the dictionary by splitting it into characters and groups of characters: bigrams, trigrams and so on. For example, you would convert ‘mysql’ to ‘m y s q l _m my ys sq ql l_ __m _my mys ysq sql ql_ l__’. In this case the underscore is used to distinguish a letter in the beginning or end of the word from a letter in the middle.
3. Next, you will index your dictionary and word modifications. You can also index attributes like word length, or anything else that will help sort the results. Other examples might be: word frequency if using –buildfreqs indexer key, or product popularity rating.
4. Then you do the same with queried keywords, for example if it’s ‘mysslq’ it becomes ‘m y s s l q _m my ys ss sl lq q_ __m _my mys yss ssl sly lq_ q__’, you can put letters, bigrams and trigrams into separate fields in order to improve the quality.
5. Finally, you search for the calculated string in the index using the “…”/1 syntax (i.e. the quorum operator).

The idea is that the many parts of the mistyped query will match with their respective best suggestions.

To improve quality you can use ranker=wordcount (SPH_RANK_WORDCOUNT in the API) or ranker=proximity (SPH_RANK_PROXIMITY in the API) to calculate the weight based on proximity or the number of matching words, not the statistics based BM25 algorithm.

Not only can you sort by full-text weight, but also by the difference between the mistyped query and the suggestion (the less difference the better).

You can also filter by length to avoid suggestions that are too short or too long. It is unusual that the length of a mistyped query differs much from the correct one’s length (usually just 2-3 letters).

The example is:

	mysql> select keyword, len, freq, @weight + 2 - abs(7 - len) final from suggest where match('@trigrams "__m _ms mse sea eag age ge_ e__ "/1 @bigrams "_m ms se ea ag ge e_ "/1 @onegrams"m s e a g e "/1') and len >= 5 and len <= 9 order by final desc, freq desc limit 10 option ranker=wordcount;
	+-------+--------+------+------+----------+-------+
	| id | weight | freq | len | keyword | final |
	+-------+--------+------+------+----------+-------+
	| 3425  | 17 | 1560   | 7 | message | 19 |
	| 17492 | 17 | 288    | 7 | mileage | 19 |
	| 28521 | 16 | 163    | 7 | massage | 18 |
	| 10566 | 15 | 504    | 8 | messages| 16 |
	| 38476 | 14 | 114    | 7 | teenage | 16 |
	| 53885 | 14 | 74     | 7 | baggage | 16 |
	| 7198  | 13 | 755    | 7 | average | 15 |
	| 14641 | 14 | 350    | 8 | marriage| 15 |
	| 16844 | 14 | 301    | 6 | manage  | 15 |
	| 20092 | 13 | 246    | 7 | disease | 15 |
	+-------+--------+------+------+----------+-------+
	10 rows in set (0.05 sec)

To improve the quality even more, it makes sense to run additional calculations in the application AFTER you've fetched the docs from Sphinx. The simplest thing you can do is to calculate the levenshtein distance for the first ten words suggested by Sphinx.

You can also play with the weights of the fields (@letters, @bigrams, and @trigrams), the thresholds, and other attributes to find which settings will work best for your data.

Of course the suggested keywords can’t be correct 100% of the time, because there are some cases when two or more suggestions are correct. It produces good results in most cases. For example, each of the following typos of the word 'message' get successfully converted to 'message' or 'messages':

essage mssage mesage mesage messge messae messag mmessage meessage messsage messsage messaage messagge messagee emssage msesage mesasge messgae messaeg nessage jessage kessage mwssage m3ssage m4ssage mrssage mfssage mdssage msssage measage mewsage meesage medsage mexsage mezsage mesaage meswage meseage mesdage mesxage meszage messqge messwge messsge messxge messzge messafe messate messaye messahe messabe messave messagw messag3 messag4 messagr messagf messagd messags nmessage mnessage jmessage mjessage kmessage mkessage mwessage mewssage m3essage me3ssage m4essage me4ssage mressage merssage mfessage mefssage mdessage medssage msessage messsage meassage mesasage mewssage meswsage meessage mesesage medssage mesdsage mexssage mesxsage mezssage meszsage mesasage messaage meswsage messwage mesesage messeage mesdsage messdage mesxsage messxage meszsage messzage messqage messaqge messwage messawge messsage messasge messxage messaxge messzage messazge messafge messagfe messatge messagte messayge messagye messahge messaghe messabge messagbe messavge messagve messagwe messagew messag3e message3 messag4e message4 messagre messager messagfe messagef messagde messaged messagse messages

Sphinx has supported real time indexes since version 1.10 was released. Ever since, they have been getting more stable and robust, and now it is ready for use in production. Many people like this because it’s really simple to understand (i.e. no indexing, crontasks, main + delta schemes, and so on), but anyone who wants to use them should also be aware of the drawbacks of this when comparing it to traditional monolithic indexes.

A real time index consists of 2 parts: one that is stored in memory, and the other is stored on a disk. Once a new insert/update query is sent to a real time index, it updates the memory as well. It works very fast, but the memory isn’t unlimited. So as the number of queries begin to reach it’s maximum and once rt_mem_limit is exceeded on the memory side, it gets converted into a disk index chunk, and so on. If you have a 10Gb index and rt_mem_limit is set to 1Gb, then you will end up with 10 disk chunks. If you set re_mem_limit to 10Gb you will have only 1 disk chunk. Now here’s the dilemma: the more disk chunks you have the worse your search performance will be, but on the other hand less memory is needed to support the real time index. On the contrary, if you set rt_mem_limit to a high value you will have good search performance because you will have fewer disk chunks, but it will take up more memory on your server. Unfortunately, the amount of memory that a Sphinx real time index requires is much more than what a traditional index needs. This is because a traditional index only stores attributes and wordlists while a real time index stores everything else as well until it’s converted into a disk chunk.

Here’s what it looks like for the same data (1M docs) when rt_mem_limit is high enough:

Traditional index:
[root@SE01 snikolaev]# ls -sh idx.sp*
12M idx.spa
186M idx.spd
8.0K idx.sph
11M idx.spi
4.0K idx.spk
4.0K idx.spl
4.0K idx.spm
103M idx.spp
8.0K idx.sps


Real time index:
[root@SE01 snikolaev]# ls -sh idx_rt.*
8.0K idx_rt.kill
4.0K idx_rt.lock
8.0K idx_rt.meta
442M idx_rt.ram


The bolded lines are the things that are stored in memory. As you can see, the traditional index requires 23Mb while the real time index needs more than 400Mb of memory.

In practice, we usually recommend real time indexes in 2 cases:

1. When the data volume is really small and is not going to grow quickly. Indeed, it doesn’t actually matter whether you spend 5Mb or 100Mb if you have few gigs of memory and using the real time indexes will make sense for you because you will avoid having an indexing routine and will be able to synchronize your database and your search index on a data insert level.
2. When the data volume is large and growing, but you want to reduce indexing latency (i.e. you want your data to appear in the index in a real time manner). And here comes the best part of using real time indexes.  They can be combined with traditional indexes using a Sphinx distributed index. What you can do in this case is still use the main + delta scheme, but make the delta real time, once the main part is rebuilt you then should clean the real time delta index. Since the delta is real time it enables real time data accessibility in the index and because it’s the delta, it doesn’t require a lot of memory. The only routine is to flush it periodically to the main part. To empty the real time index the latest Sphinx builds provide “TRUNCATE RTINDEX” SphinxQL command.

This is first Sphinx event in USA and a great opportunity for Sphinx users in US to get together and meet each other and the Sphinx team. As the organizers say:

The aim of the Sphinx Search Day 2012 is to provide a technical forum to educate those who are new to Sphinx and drive innovation for long-time users. The majority of the talks will be technical in nature and more importantly delivered by real-world users and Sphinx community members.

I am also very glad that Ivinco was invited to this event. In our talk we will share our experience in building Sphinx systems from small projects to multi-terabyte search engines.

This is a free event – if you are around the Silicon Valley in April – go ahead and register. We look forward to meeting you at the Sphinx Search Day!