Solr AutoSuggest with TermsComponent and jQuery

I needed to implement an autosuggest/autocomplete search box for use with Solr. After a little research, I found the new TermsComponent feature in Solr 1.4. To use TermsComponent for suggestions, you need to provide set the prefix and lower bound to the input term and make the lower bound exclusive. Use the terms.fl parameter to set the source field. This means:

  • Set terms.lower to the input term
  • Set terms.prefix to the input term
  • Set terms.lower.incl to false
  • Set terms.fl to the name of the source field

Your resulting query should look something like this:

http://localhost:8983/solr/autoSuggest?terms=true&terms.fl=name&terms.lower=py&terms.prefix=py&terms.lower.incl=false&indent=true&wt=json

Note: This assumes you are using the default solrconfig.xml for Solr 1.4

In the example above I used “py” for my input term. You will then get output that looks similar to this:

{
 "terms":[
  "spell",[
	"pyblosxom",16,
	"pychm",16,
	"pyqt",16,
	"python",16]]}

Now that we have TermsComponent setup and working correctly its time to create the autosuggest/autocomplete search box. Since I am not one to reinvent the wheel, I did a quick search and found a jQuery UI plugin for autocomplete. The search frontend I was developing was already using jQuery, so this plugin was a perfect fit.

This autocomplete plugin is not in the current release of jQuery UI so I needed to grab it from their subversion repository. You can find instructions where to get it here.

The plugin supports AJAX calls for the data source. It expects the data source to return each suggestion on it’s own line, for example:

pyblosxom
pychm
pyqt
python

As you saw above, this is not what direct output from Solr looks like. On top of this, it is not a good idea to expose your backend server via your frontend code. Time to write a java servlet.

Unfortunately the java client for Solr, SolrJ, didn’t support TermsComponent yet. I decided to add this support, so please see this post for information on my patch.

Assuming you are using a version of SolrJ with my patch, here is a simple servlet that provides the functionality we need:

protected void doGet(HttpServletRequest req, HttpServletResponse res) throws ServletException, IOException {
        String q = req.getParameter("q");
        String limit = req.getParameter("limit");
	PrintWriter writer = res.getWriter();
	List<Term> terms = query(q, Integer.parseInt(limit));

	if (terms != null) {
		for (Term t : terms) {
			writer.println(t.getTerm());
		}
	}
}

And the query method:

private List<Term> query(String q, int limit) {
    List<Term> items = null;
    CommonsHttpSolrServer server = null;

     try {
         server = new CommonsHttpSolrServer("http://localhost:8983/solr");
     } catch(Exception e) { e.printStackTrace(); }

     // escape special characters
     SolrQuery query = new SolrQuery();
     query.addTermsField("spell");
     query.setTerms(true);
     query.setTermsLimit(limit);
     query.setTermsLower(q);
     query.setTermsPrefix(q);
     query.setQueryType("/terms");

     try {
         QueryResponse qr = server.query(query);
         TermsResponse resp = qr.getTermsResponse();
         items = resp.getTerms("spell");
     } catch (SolrServerException e) {
      	items = null;
     }

     return items;
}

Now you may be wondering why I used the “q” and “limit” parameters. I use these because this is what the jQuery autocomplete plugin sends to the servlet. “q” is the input term, and “limit” is the max number of suggestions to return.

Now to hook everything together. Insert the following javascript into the head of your search page and replace “#searchbox” with the id of the input box you want to use for autocompletion. Also insert the correct url to your servlet.

        	$(document).ready(function() {

        		$("#searchbox").autocomplete({ url: 'completion',
        			 max: 5,
        		});
          	});

Update your css file with required jQuery UI css:

/* Autocomplete
----------------------------------*/
.ui-autocomplete {}
.ui-autocomplete-results { overflow: hidden; z-index: 99999; padding: 1px; position: absolute; }
.ui-autocomplete-results ul { width: 100%; list-style-position: outside; list-style: none; padding: 0; margin: 0; } 

/* if  the width: 100%, a horizontal scrollbar will appear when scroll: true. */
/* !important! if line-height is not set, or is set to a relative unit, scroll will be broken in firefox */
.ui-autocomplete-results li { margin: 0px; padding: 2px 5px; cursor: default; display: block; font: menu; font-size: 12px; line-height: 16px; overflow: hidden; border-collapse: collapse; }
.ui-autocomplete-results li.ui-autocomplete-even { background-color: #fff; }
.ui-autocomplete-results li.ui-autocomplete-odd { background-color: #eee; }

.ui-autocomplete-results li.ui-autocomplete-state-default { background-color: #fff; border: 1px solid #fff; color: #212121; }
.ui-autocomplete-results li.ui-autocomplete-state-active { color: #000; background:#E6E6E6 url(images/ui-bg_glass_75_e6e6e6_1x400.png) repeat-x; border:1px solid #D3D3D3; }

.ui-autocomplete-loading { background: white url('images/ui-anim.basic.16x16.gif') right center no-repeat; }
.ui-autocomplete-over { background-color: #0A246A; color: white; }

Congratulations! You should now have a working Solr-based autocomple search box!
Solr AutoCompletion

Leave a comment ?

26 Comments.

  1. Otis Gospodnetic

    Nice.
    This one is not limited to terms and has a few other tricks up its sleeve:

    http://www.sematext.com/product-auto-complete.html

  2. Nice post. Your SOLR-1156 patch is useful as well.

    If your field is a string type, you can have multiple term suggestions.

  3. or you could also do autosuggest with faceting (and facet.prefix) – e.g. select/?q=*:*&facet=true&facet.field=spell&facet.prefix=i&rows=0

    • Yes you could use facets, however there was talk that TermsComponet is faster. The faster the better for autocompletion. I have no way to back this up, so take it with a grain of salt!

  4. Matt, you have a minor typo in your Servlet example. List should be List (3 occurrences, I believe).

  5. Angle brackets were stripped in above comment. List [term] should be List [Term]. Where [] should be angle brackets.

    • Thanks! I also updated it to reflect the latest changes to TermsComponent in trunk. More specifically I now use SetTermsLimit instead of SetTermsRows and updated to query type to be “/terms” as it is in the latest solrconfig.xml.

  6. Hi Matt, Thx for the post. Can you send me the runnable code via e-mail ? I don’t see a “download code” option.

    Also, do you know when your code will be added to SolrJ 1.4 ?

    Thx

    David

  7. Does your code support “inclusive” search (aka “like”) ? For example, if you type in “”niqu”, will it return “technique ?

  8. Mohsen Zainalpour

    thanks , very nice and useful post.
    I updated doGet method to support Unicode terms ( res.setCharacterEncoding(“UTF-8″); )

  9. Solr Digest, January 2010 « Sematext Blog - pingback on January 19, 2010 at 10:46 pm
  10. This would be nice to have in the WordPress plugin, too.. *hint* :)

    • It is in the next version along with much more cool stuff! Unfortunately my professional life has been pretty hectic lately and I have not had a chance to get it released!

  11. > On top of this, it is not a good idea to expose your backend server via your frontend code.

    In case someone still wants a direct communication between client and solr:

    http://code.drewwilson.com/entry/autosuggest-jquery-plugin

    Using this plugin one is able to define callback named retrieveComplete to transform the JSON data sent from solr into the finally required structure.

  12. hi Matt and thanks for this great and handy post,
    I want to use your modified version of solrJ that supports termsComponents but the link:
    http://www.mattweber.org/2009/04/30/solrj-termscomponent-support/
    you gave seems, for some reason, to be broken
    can you email me the code so I can use it?
    thanx in advance

    • That link works for me. Anyways, this code has been accepted into Solr. Just grab the latest version of Solr trunk and you will get a SolrJ with my patch already applied.

  13. Hi Matt, Thx for the post. Can you send me the runnable code via e-mail ? I don’t see a “download code” option.

    Thx in advance…..

  14. Hi,

    I am using solr4 and i configured terms component for auto suggest.Here, how can we configure for case insensitive auto suggest?

    Please help me..

    Regards,
    Siva

  15. Hello Mat,
    Nice post.
    I am building a autocomplete feature using Solr.
    I checked out the solr project from source control and built it using the dist command in ant to create multiple JAR files.
    I added these newly created JAR files in my project. So that, your patch for TermsResponse is available to me.
    It is giving me an exception at the following line:
    QueryResponse qr = server.query(query);
    This is the trace:
    org.apache.solr.client.solrj.SolrServerException: Error executing query
    at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:95)
    at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:119)
    at lucidlabs.searching.SolrJSearcherSpellCheckComp.search(SolrJSearcherSpellCheckComp.java:56)
    at lucidlabs.searching.RunSolrJSearcherSpellCheckComp.main(RunSolrJSearcherSpellCheckComp.java:22)
    Caused by: java.lang.RuntimeException: Invalid version or the data in not in ‘javabin’ format
    at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:99)
    at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:39)
    at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:477)
    at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244)
    at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89)
    … 3 more

    Can you please kindly help me find the problem here?
    I am sure it is some version mismatch but unable to determine where.
    Thanks a lot

  16. please show the steps for implementing AutoComplete in Solr-1.3. I have got front end written in JSP. It works well.

    I need UI for AutoComplete and the implementation .

    Please guide me through steps of implementing it.

    Thanks..

  17. Solr Auto Suggest | Maansu - pingback on April 10, 2011 at 1:16 pm
  18. How do I retrieve the suggestion with punctuation marks like {‘,’,'-’,';’,etc.,}. Can Any one help me to achieve this?

  19. Writing Terms Query for auto suggest in solr | My Blog - pingback on September 8, 2011 at 11:26 am
  20. Thank you so much for sharing on termscomponent for solrj. So important yet not found on its official documentation!

  21. Autosuggest using Solr | Kosmex : Empowering Cloud Applicaitons - pingback on June 30, 2012 at 11:57 pm

Leave a Comment


NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Trackbacks and Pingbacks: