Indexing book in hybrid cordova app using lunr.js

For a hybrid app I have a requirement to create an index of a book to implement a searching feature.

I decided to use lunr.js library to create index of books and afterwards to make search through this index.

First attempt was to make index at the front end and save it to the device's file system so later I could read needed index to perform a search. This straight forward idea works only for indexes < 10 mb. After reaching ~10 mb limit our hybrid application crashes. The problem occurs during the save process in cordova fileWriter function. Besides indexing on the front end could take much time depending on device's specs.

As our hybrid app has downloadable book content it is wise to provide index in the book package itself. So the next attempt was to move index creation process to the server side.

We want to use search on a dutch books so I tried to use lunr-languages library, which is language extension to the lunr.js. After playing around this extension we decided that we don't trust dutch stemmer because sometimes it creates huge non dutch words just by concatenating several words together. This point became very crucial because the index file grows significantly. Thus the only useful part of this extension is dutch stop words filter but it's simple enough to create custom word filter on top of the original lunr.js library so we decided to go only with lunr.js.

The filter function looks like this:

/* stop word filter function */
$lunr.stopWordFilter = function (token) {

   if (token.length <= 2) return undefined; //tokens of length less then 3

   if (!isNaN(+token) && isFinite(token)) return undefined; // skip numbers

   if (/\d+\.\d+\.\d+/.test(token)) return undefined; //numbers i.e 10.14.1

   if (/\d+\.\d+\.\d+\.\d+/.test(token)) return undefined; //numbers i.e 1.00.00.1
   if (/\d+\,\d+/.test(token)) return undefined; //numbers in dutch notation

   if ($lunr.stopWordFilter.stopWords.elements.indexOf(token) === -1) return token;
};

$lunr.stopWordFilter.stopWords = new $lunr.SortedSet();

$lunr.stopWordFilter.stopWords.length = 23;

$lunr.stopWordFilter.stopWords.elements = ' de en van ik te dat die in een hij het niet zijn is was op aan met als voor had er maar'.split(' ');

$lunr.Pipeline.registerFunction($lunr.stopWordFilter, 'stopWordFilter-dutch-custom');

Next step was to decrease the size of the index. At that moment I had 11mb index file size for 5mb html formatted book.

Playing with the stop words saved me 10% of the file size. The final stop word list consist of 200 words.

Exploring  the content of the index file I have notices that it contain number with a huge precision i.e "0.0012375940126393694". This numbers are the scores that index.search function returns along with ref string. Decreasing the precision of these numbers saved my another 20% of the file size. Decreasing were done in the server side as a post-processing of the stringified index:

var idx: string = JSON.stringify(this.index.toJSON());

idx = idx.replace(/(\d\.\d\d\d\d)\d\d\d\d\d\d\d\d\d\d\d\d\d\d\d/g, "$1");

idx = idx.replace(/(\d\.\d\d\d\d)\d\d\d\d\d\d\d\d\d\d\d\d\d\d/g, "$1");

idx = idx.replace(/(\d\.\d\d\d\d)\d\d\d\d\d\d\d\d\d\d\d\d\d/g, "$1");

idx = idx.replace(/(\d\.\d\d\d\d)\d\d\d\d\d\d\d\d\d\d\d\d/g, "$1");

idx = idx.replace(/(\d\.\d\d\d\d)\d\d\d\d\d\d\d\d\d\d\d/g, "$1");

idx = idx.replace(/(\d\.\d\d\d\d)\d\d\d\d\d\d\d\d\d\d/g, "$1");

idx = idx.replace(/(\d\.\d\d\d\d)\d\d\d\d\d\d\d\d\d/g, "$1");

idx = idx.replace(/(\d\.\d\d\d\d)\d\d\d\d\d\d\d\d/g, "$1");

idx = idx.replace(/(\d\.\d\d\d\d)\d\d\d\d\d\d\d/g, "$1");

idx = idx.replace(/(\d\.\d\d\d\d)\d\d\d\d\d\d/g, "$1");

idx = idx.replace(/(\d\.\d\d\d\d)\d\d\d\d\d/g, "$1");

Another helpful thing is to use a mapping for index items. Every time we add a new item to the index we should provide at least item id and some text. In my case I provide title, body, and a string id which is a combination of chapterId and paragraphId: {title:'this is title', id:"bookChapter36:bookparagrapg12", body:"this is body text"} Index will save this id as a reference. At this point we can create idMap array of string id's and save it to the disc along with the index file. Next we can add to the index an integer(index of idMap array) instead of the huge string. This procedure saved me another 10% of the file size.

As our app shows only titles as a result of a search we can include titles in the idMap file to avoid searching of all titles by reading book and searching for chapterId and paragraphId. This could be crucial for slow devices and big books.