Search-index

Search-index is a search index module for Node.js. Think "node version of Lucene, but much simpler".

Search-index is built with the soooperfast levelUP module, and the very useful Natural module.

The Point of Search-Index is to simplify set up and operation of an search engine. Search-index is essentially free from configuration- the index is dynamic and morphs into the structure that you require automatically, based on the documents that it is fed.

Search-index is in an alpha stage- meaning that it has been known to work quite well, but edge cases and portability may be challenging. Query-result is robust. See known issues and performance tips below.

#The Norch Search Engine

Search-index is currently the index powering the Norch search engine.

#Features

• Full text search
• Stopword removal
• Faceting
• Filtering
• Fielded search
• Field weighting
• Relevance weighting (tf-idf)
• Paging (offset and resultset length)

#Installation

Releases are listed here. Generally you will want the most recent one.

The easiest way to include search-index in your project is by using npm

npm install search-index

The module can then be initialised by including the line

si = require('search-index');

at the top of your app.

#API

##si.indexData([,callback])

Returns metadata about the state of the index. Metadata is accrued incrementally, so it is vulnerable to corruption. At any time metadata can be corrected by running the computationally demanding calibrate function.

si.indexData(function(msg) {
  console.log(msg);
});

##si.calibrate([,callback])

Iterate through entire index and count everything up. Tf-idf calculations are most precise when indexData is up to date. Only needs to be called if index has been altered externally, or if key collisions occur (mostly theoretical). A persistant running tally is kept by search-index which can be seen in the file norchindex.json.

si.calibrate(function(msg) {
  console.log(msg);
});

##si.indexPeek(start, stop [,callback])

Take a look at the raw index. Start is the start point and stop is the stop point. All keys in between will be returned. For debug purposes.

si.indexPeek(req.query['start'], req.query['stop'], function(msg) {
  console.log(msg);
});

##si.deleteDoc(req.body.docID [,callback])

Delete the document and all associated index entries.

si.deleteDoc(documentID, function(msg) {
  console.log(msg);
});

##si.search(query, [,callback])

Queries the search index

si.search(query, function(msg) {
  res.send(msg);
});

...where query is an object similar to (see Query Parameters for more info):

    {
    "query": [
      "usa"
    ],
    "offset": "0",
    "pagesize": "20",
    "facets": [
      "places",
      "organisations"
    ],
    "weight": {
      "title": [
        "10"
      ]
    },
    "filter": {
      "places": [
        "usa"
      ]
    }
  }

##si.index(batch, filters, [,callback])

Insets document into the index

si.index(batch, filters, function(msg) {
  res.send(msg);
});

Where batch is a JSON file containing one or more documents and formatted similar to:

{
  'doc1':{
    'title':'A really interesting document',
    'body':'This is a really interesting document',
    'metadata':['red', 'potato']
  },
  'doc2':{
    'title':'Another interesting document',
    'body':'This is another really interesting document that is a bit different',
    'metadata':['yellow', 'potato']
  }
}

...and filters is an array of field names that may be contained in the document that the index will use for building filters. A filter field must always be an array of single String tokens, for example ['metadata','places']

#Query Parameters

##query

A free text string containing one or many tokens. Search everything with ´´´´*´´´

    "query": [
      "usa"
    ]

##searchFields

Fields to search in. Query will only search in the fields specified here.

    "searchFields": [
      "title"
    ]

##offset

The starting point in the resultset of the results returned. Useful for paging

    "offset": 0

##pagesize

The maximum number of results in the resultset that can be returned. Counted from offset

    "pagesize": 20

##facets

Allows faceted navigation, the parameter is an array of fields. If no facets is sent, all possible facets are returned.

    "facets": [
      "topics",
      "places"
    ]

##weight

Sets a factor by which the score of a field should be weighted. Useful for building custom relevancy models

    "weight": {
      "body": [
        "10"
      ],
      "title": [
        "5"
      ]
    }

##filter

Used to return a resultset on a facet.

    "filter": {
      "places": [
        "ussr"
      ]
    }

#License

Search-index is released under the MIT license:

Copyright (c) 2013 Fergus McDowall

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.