November 24, 2014

VS can’t open project that contains npm package folder “node_modules”

Recently I have faced a problem with opening in Visual Studio a project that contains npm packages folder. My npm packages are placed in the folder “node_modules” inside my VS project and evidently VS is unable to read  this directory(after update VS13 upd4). I believe this is because the directory might have very deep path for one of the packages.

I did try to reinstall VS13 and even installed VS15 but it didn’t help.

The obvious solution is to exclude “node_modules” folder from the project BUT apparently it is impossible to do in VS for website-type project. Another solution is to move “node_modules” folder up in hierarchy so it would be next to your project folder but I need to keep npm packages locally and use specific and not the latest versions of some packages.

Finally I discovered that if you mark  “node_modules” folder as hidden VS will not index this folder which is the easiest solution to my problem.

November 14, 2014

Advanced file system manipulations using node.js scripting

Below you can find several functions that will help you to manage your files in more advanced way:

1. Synchronously create directory even if its parent directories don’t exist : createPathSync(‘a/b/c/d/e’);
2. Synchronously copy file: copyFileSync( source, target );
3. Synchronously copy directory with its content: copyFolderRecursiveSync( source, target );
4. Synchronously delete directory with its content: deleteFolderRecursiveSync(target);
5. Asynchronously zip folder: makeZip(source, target, zipName, callback);

first of all lets define dependencies:
archiver you only need to use zip functionality.

var fs = require('fs');
var path = require('path');
var archiver = require('archiver');    //npm install archiver
1. Synchronously create directory even if its parent directories don’t exist : createPathSync(‘a/b/c/d/e’);

function createPathSync(dirPath, mode, fullPath) { //usage: createPathSync('a/b/c/d', [mode]);
    if (typeof fullPath == 'undefined') fullPath = dirPath;    //initialization
    try {
        //try to make dir
        fs.mkdirSync(dirPath, mode);
        //if succeeded go recursive or finish
        if (!fs.existsSync(fullPath)) createPathSync(fullPath, mode, fullPath);
    } catch (err) {
        createPathSync(path.dirname(dirPath), mode, fullPath);
    }
}
2. Synchronously copy file: copyFileSync( source, target );

function copyFileSync(source, target) {
    var targetFile = target;
    if (fs.existsSync(target)) {
        if (fs.lstatSync(target).isDirectory()) {
            targetFile = path.join(target, path.basename(source));
        }
    }
    fs.createReadStream(source).pipe(fs.createWriteStream(targetFile));
}
3. Synchronously copy directory with its content: copyFolderRecursiveSync( source, target );

function copyFolderRecursiveSync(source, target) {
    var files = [];
    if (!fs.existsSync(target)) {    //assure that target path exists
        createPathSync(target);
    }
    if (fs.lstatSync(source).isDirectory()) {
        files = fs.readdirSync(source);
        files.forEach(function (file, index) {
            var curSource = path.join(source, file);
            var curTarget = path.join(target, file);
            if (fs.lstatSync(curSource).isDirectory()) {
                copyFolderRecursiveSync(curSource, curTarget);
            } else {
                //console.log(curSource+'       '+ curTarget);
                copyFileSync(curSource, curTarget);
            }
        });
    }
}
4. Synchronously delete directory with its content: deleteFolderRecursiveSync(target);

function deleteFolderRecursiveSync(path) {
    if (fs.existsSync(path)) {
        fs.readdirSync(path).forEach(function (file, index) {
            var curPath = path + "/" + file;
            if (fs.lstatSync(curPath).isDirectory()) { // recurse
                deleteFolderRecursiveSync(curPath);
            } else { // delete file
                fs.unlinkSync(curPath);
            }
        });
        fs.rmdirSync(path);
    }
};
5. Asynchronously zip folder: makeZip(source, target, zipName, callback);

function makeZip(from, to, zipName, callback) {
    var zipArchive = archiver('zip');
    var zipOutput = fs.createWriteStream(to + '/' + zipName);
    zipOutput.on('close', function () {
        console.log('zipping done: ', to + '/' + zipName);
        callback();
    });
    zipArchive.pipe(zipOutput);
    zipArchive.bulk([{ src: ['**/*'], cwd: from, expand: true }]);
    zipArchive.finalize(function (err, bytes) {
        if (err) throw err;
        console.log('done:', base, bytes);
    });
}

October 22, 2014

Optimizing performance of an accordion list for an ionic app.

One of the requirements for the app we develop was implementation of accordion list witch would represent table of content of a book. The first implementation was straight forward and was pretty much the copy of this CodePen example. The problem of that implementation is pure performance. Some of the books contain more than 80 chapters and each chapter contain a lot of paragraphs and some of the paragraphs could also contain sub-paragraphs and so on...

Lets analyze the code from CodePen:
1. it contains ng-repeat which obviously slows down our app
2. each item in this list contain ng-click, ng-class, ng-show and another ng-repeat.

This architecture lead us to very pure performance specially on relatively old devices (I have Samsung galaxy s2 for instance). And it is pretty clear that angular digest loop will suffer because of approximately 2000 watchers just for your table of content.

So the firs step was to get rid of all ng-repeat directives.
The idea is to pre-render table of content of a book as a plain HTML text and save it to device's file system.
After that plaine HTML text should be wired-up with a view using

$("#indexview").append($compile(plainHtml)($scope));
With a help of handlebars.js it's quite easy to pre-render plain HTML text. here are the recursive templates that I use for rendering table of content down to bottom paragraph level:

var chapterTocTemplate = '\
<ion-list>\
{{#each Chapters}}\
<ion-item id="i-{{Id}}" class="ii" chapter="{{Id}}" anchor="{{Id}}">\
<n>{{TitleNumber}}</n>\
<tc>{{TitleText}}</tc>\
<tb class="tb ion-chevron-down"></tb>\
</ion-item>\
<cp">\
{{#if Paragraphs}}{{> tocParagraphs }}{{/if}}\
</cp>\
{{/each}}\
</ion-list>';

var paragraphTocTemplate = '\
{{#each Paragraphs}}\
<pi chapter="{{ChapterId}}" anchor="{{Id}}">\
<n>{{TitleNumber}}</n>\
<tp>{{TitleText}}</tp>\
</pi>\
{{#if Paragraphs}}\
<sb>{{> tocParagraphs}}</sb>\
{{/if}}\
{{/each}}';
It is important here that you don’t use list of classes

<div class='classA classB classC classD'>{{TitleNumber}}</div>
because this will dramatically increase the size of the plain HTML text. Instead of that you should define styling for a new element:

<n>{{TitleNumber}}</n>
It’s good to know that scss style of a new element can extend existing styles:
n {
   @extend .classA;
   @extend .classB;
   ...
   @extend .classD;
}
As you have also noticed I don’t use any ng-click or ng-class directives in templates in order to reduce amount of watchers in the digest loop. The last step is compiling our plain text into the $scope of a view to wire-up ionic tags that we did use in our template with ionic scope. At this step we have solved our performance problem, but we still need to wire-up at least click actions: The easiest way is to register click listener to the whole view. The object that is passed through the callback event contains information about element that you have clicked on. I have created custom tags (see handlebars template) for all clickable elements and just compare in the callback

event.srcElement.tagName
with the tag I need. My clickable paragraphs also contain an additional attributes chapter and anchor. It is easy to have access to these attributes in my click event callback:

event.srcElement.attributes.chapter.value;
I got rid of ng-show and ng-class directives. Instead of those I have to use java script manipulations which is not the best practice in general but doing this I have got an additional profit in performance:

$('#someid').find('cp')[0].style.display = 'none';
event.srcElement.className = event.srcElement.className.replace(' ion-chevron-up', '') + ' ion-chevron-down';

October 13, 2014

Cordova-bootstrapper: Update of hybrid App or its content without pushing new releases to a web store.

For a hybrid App I had a requirement to be able to push update of the App or of its content not through the web store release. First of all it's important mainly because people which were not directly involved with developing could update a content. Second reason is that verifying a new release could take enormous amount of time for some platforms.

Thus, cordova-bootstrapper was created for this purpose. Basically cordova-bootstrapper is a "wrapper" system that allows you to push updates/new content for already installed cordova app.

First installation of the App includes installation of cordova-bootstrapper, cordova buid for specific platform, www zip folder of the actual App (embedded App) and content for the embedded app.

During each start of the App cordova-bootstrapper compares version file (version.json) in the App folder with version file on your remote server. If updates found cordova-bootstrapper detects it and installs new version of the embedded App or updates the content.

For more information visit :https://github.com/Qvatra/Bootstrapper

The flow diagram of the cordova-bootstrapper:


August 27, 2014

Indexing book in hybrid cordova app using lunr.js

For a hybrid app I have a requirement to create an index of a book to implement a searching feature.

I decided to use lunr.js library to create index of books and afterwards to make search through this index.

First attempt was to make index at the front end and save it to the device's file system so later I could read needed index to perform a search. This straight forward idea works only for indexes < 10 mb. After reaching ~10 mb limit our hybrid application crashes. The problem occurs during the save process in cordova fileWriter function. Besides indexing on the front end could take much time depending on device's specs.

As our hybrid app has downloadable book content it is wise to provide index in the book package itself. So the next attempt was to move index creation process to the server side.

We want to use search on a dutch books so I tried to use lunr-languages library, which is language extension to the lunr.js. After playing around this extension we decided that we don't trust dutch stemmer because sometimes it creates huge non dutch words just by concatenating several words together. This point became very crucial because the index file grows significantly. Thus the only useful part of this extension is dutch stop words filter but it's simple enough to create custom word filter on top of the original lunr.js library so we decided to go only with lunr.js.

The filter function looks like this:

/* stop word filter function */
$lunr.stopWordFilter = function (token) {

   if (token.length <= 2) return undefined; //tokens of length less then 3

   if (!isNaN(+token) && isFinite(token)) return undefined; // skip numbers

   if (/\d+\.\d+\.\d+/.test(token)) return undefined; //numbers i.e 10.14.1

   if (/\d+\.\d+\.\d+\.\d+/.test(token)) return undefined; //numbers i.e 1.00.00.1
   if (/\d+\,\d+/.test(token)) return undefined; //numbers in dutch notation

   if ($lunr.stopWordFilter.stopWords.elements.indexOf(token) === -1) return token;
};

$lunr.stopWordFilter.stopWords = new $lunr.SortedSet();

$lunr.stopWordFilter.stopWords.length = 23;

$lunr.stopWordFilter.stopWords.elements = ' de en van ik te dat die in een hij het niet zijn is was op aan met als voor had er maar'.split(' ');

$lunr.Pipeline.registerFunction($lunr.stopWordFilter, 'stopWordFilter-dutch-custom');
Next step was to decrease the size of the index. At that moment I had 11mb index file size for 5mb html formatted book.

Playing with the stop words saved me 10% of the file size. The final stop word list consist of 200 words.

Exploring  the content of the index file I have notices that it contain number with a huge precision i.e "0.0012375940126393694". This numbers are the scores that index.search function returns along with ref string. Decreasing the precision of these numbers saved my another 20% of the file size. Decreasing were done in the server side as a post-processing of the stringified index:

var idx: string = JSON.stringify(this.index.toJSON());

idx = idx.replace(/(\d\.\d\d\d\d)\d\d\d\d\d\d\d\d\d\d\d\d\d\d\d/g, "$1");

idx = idx.replace(/(\d\.\d\d\d\d)\d\d\d\d\d\d\d\d\d\d\d\d\d\d/g, "$1");

idx = idx.replace(/(\d\.\d\d\d\d)\d\d\d\d\d\d\d\d\d\d\d\d\d/g, "$1");

idx = idx.replace(/(\d\.\d\d\d\d)\d\d\d\d\d\d\d\d\d\d\d\d/g, "$1");

idx = idx.replace(/(\d\.\d\d\d\d)\d\d\d\d\d\d\d\d\d\d\d/g, "$1");

idx = idx.replace(/(\d\.\d\d\d\d)\d\d\d\d\d\d\d\d\d\d/g, "$1");

idx = idx.replace(/(\d\.\d\d\d\d)\d\d\d\d\d\d\d\d\d/g, "$1");

idx = idx.replace(/(\d\.\d\d\d\d)\d\d\d\d\d\d\d\d/g, "$1");

idx = idx.replace(/(\d\.\d\d\d\d)\d\d\d\d\d\d\d/g, "$1");

idx = idx.replace(/(\d\.\d\d\d\d)\d\d\d\d\d\d/g, "$1");

idx = idx.replace(/(\d\.\d\d\d\d)\d\d\d\d\d/g, "$1");

Another helpful thing is to use a mapping for index items. Every time we add a new item to the index we should provide at least item id and some text. In my case I provide title, body, and a string id which is a combination of chapterId and paragraphId: {title:'this is title', id:"bookChapter36:bookparagrapg12", body:"this is body text"} Index will save this id as a reference. At this point we can create idMap array of string id's and save it to the disc along with the index file. Next we can add to the index an integer(index of idMap array) instead of the huge string. This procedure saved me another 10% of the file size.

As our app shows only titles as a result of a search we can include titles in the idMap file to avoid searching of all titles by reading book and searching for chapterId and paragraphId. This could be crucial for slow devices and big books.