Side Project

Custom builds for jHERE

jHERE was born as a simple jQuery plugin with very few functionalities: add a map to a web page, add markers and events for markers, simple support for KML and heatmaps.

However, as I was using it for my hacks I realized there are other features that are not essentials but still nice to have. The original HERE Maps API has plenty of interesting functionalities, and not exposing them through jHERE would be a shame.

On the other hand I did not want the plugin to grow uncontrollably to a huge mess of code. Not only this would increase a lot the size of jHERE, but also due to size and complexity the codebase would have become hard to maintain.

Therefore I chose to make jHERE an extensible plugin. This means exactly what you are thinking. It is possible to make plugins for the plugin. Except I called them extensions. Plugins for plugins: how meta!

Allowing extensions was a great choice. I was able to easily create an autonit extension - that allows inexperienced web developers to add a map to their pages without having to write any JavaScript code - a routing extensions and a geocode/reverse geocode extension.

This leads unfortunately to another problem: all these extensions - plus the adapter for Zepto.JS - ship as separate files that have to be included with different script tags. This is great for development as it makes debugging easy, but once an application that uses jHERE plus one or more extensions reaches the production stage, having multiple script tags means multiple HTTP connections. This is a big performance hit. Therefore these files have to be combined.

That could be done ad build time, before or after the minification step. But again, building a big JS file with plugin and extensions inside is non sense. Why should my users download a bundle with jHERE and perhaps ten extensions when I only need the geocoding one? So that was not a possibility.

Bulding all the possible combinations of jHERE-core and extensions in not an option either, it would be crazy and completely nonsense.

Wouldn’t it be awesome if the users of the plugin could just choose the components that need to be part of the jHERE bundle? In other words, wouldn’t it be cool if developers could build a version of jHERE that is customized to their needs? This seemed definitely the way to go.

The first option I considered was a build script that can be called from the Makefile. This did not sound like a good option for non-advanced users, thus I almost immediately dropped this possibility.

The ideal choice seemed then a web page where one could simply click few checkboxes and download the custom build. And after this long introduction, this is exactly what this post is about: creating a web build tool for a JavaScript library that handles dependency management and returns customized and minified versions of the library ready for production. The result is live at custom.jhere.net.

Custom build tool

The tool I built has two components: one on the backend written in Node.js and the frontend that includes a simple static HTML page and some JavaScript. The backend is a pretty simple piece of software: given the url to two or more JavaScript files it downloads them, combines them together and sends the result to the browser. The frontend contains the list of available files, handles dependencies and builds the appropriate request to be passed to the backend. The custom build tool concatenates together plugin and extensions by fetching the minified version straight from Github.

Since this post is already too long I will now focus only on the backend. The frontend part is not very complicated, but I will probably analyze that in a separate post. The code can be found here: custom.jhere.net/main.js, in case you feel like having a look.

Backend

The backend takes care of serving the static files as well as exposing the endpoints for combining the files and flushing the cache. I used Express as the web framework that handles everything: it makes easy to set up routes, serve static files, parse query parameters, set headers and send responses back to the client.

The concatenation API is very simple. It is a GET or POST request to / with query parameter files set to file1+file2+…fileN. For example to download jHERE with the Zepto adapter you would request something like:

http://custom.jhere.net/
?files=https://raw.github.com/mmarcon/jhere/master/dist/zepto.adapter.min.js
+https://raw.github.com/mmarcon/jhere/master/dist/jhere.min.js

The first thing the code does when such a request is received is parsing the files parameter into an array of URLs that is passed to the concatenation function.

app.all('/', function(req, res, next){
    var files = req.query.files && req.query.files.split(' ');
    if (files && files.length) {
        return concatOnTheFly(files, res);
    }
    next();
});

Note that + in a URL are received as spaces: that is why I am splitting on spaces.

concatOnTheFly recursively fetches the remote files using the Request module.

function concatOnTheFly(files, res){
    var data = "", f;

    //Recursively get files until done.
    function fetch(file) {
        var f;
        request.get({url: file}, function(error, request, body){
            if(error){
                return res.send(500, {error: 'error while fetching' + file});
            }
            data += body;
            //Get next file
            f = files.shift();
            //Is there a next file?
            if(f) {
                //Cool then fetch it
                return fetch(f);
            }
            //Looks like I am done. Force the browser to
            //download what I've got.
            res.header('Pragma', 'no-cache');
            res.header('Content-Disposition', 'attachment; filename=jhere-custom.js');
            return res.send(data);
        });
    }

    //Let's start with the first file
    f = files.shift();
    fetch(f);
}

Requests are currently not being executed in parallel, but since files have to be combined one next to the other and in the given order it was just easier to implement it this way. This was a simple first implementation but after I tested it I noticed it was being pretty slow. I realized it was not necessary to download all the files every time and combine them together. To make it faster I added a simple caching mechanism that caches a specific combination of files the first time it is requested, so the next time it can be served from cache. The cache is a simple object where keys are the hashes of the requested files and values are the resulting combination.

function isCacheable(data){
    return data && data.length < 102400;
}

function concatOnTheFly(files, res){
    var data = "", f,
    key = crypto.createHash('md5').update(files.join('-')).digest("hex");

    //Is this already in cache?
    if(cache[key]) {
        //Yep, then serve it from there!
        res.header('Pragma', 'no-cache');
        res.header('Content-Disposition', 'attachment; filename=jhere-custom.js');
        return res.send(cache[key]);
    }
    //Nope, then get the files
    function fetch(file) {
        var f;
        request.get({url: file}, function(error, request, body){
            if(error){
                return res.send(500, {error: 'error while fetching' + file});
            }
            data += body;
            f = files.shift();
            if(f) {
                return fetch(f);
            }
            //Prevents maliciouos requests from
            //saturating memory with lots of different requests.
            if(cachesize >= MAX_CACHE_SIZE) {
                flushCache();
            }
            //Also defend against big files
            if(isCacheable(data)) {
                //OK, everything should be alright now.
                //Cache this stuff.
                cache[key] = data;
                cachesize++;
            }
            //And return it.
            res.header('Pragma', 'no-cache');
            res.header('Content-Disposition', 'attachment; filename=jhere-custom.js');
            return res.send(data);
        });
    }

    f = files.shift();
    fetch(f);
}

This simple caching layer improved performance a lot. However adding a cache created a potential issue. If files are already in cache and I push a new version to Github that is not going to be picked up unless I manually flush the cache. Normally I tend to forget to do things like this though, so I needed an automated way to flush the cache after every push. Therefore I added a new endpoint: /flush.

app.all('/flush', function(req, res){
    if(req.query.auth === AUTH_CODE) {
        flushCache();
        var server  = email.server.connect({
           user: MAIL_USER,
           password: MAIL_PASSWORD,
           host: "smtp.gmail.com",
           ssl: true
        });
        server.send({
           text:    "Cache flushed on " + Date(),
           from:    MAIL_SENDER,
           to:      MAIL_RECIPIENT,
           subject: "jHERE Custom - Cache Flushed"
        }, function(err, message) {
            res.send(200, {result: 'OK'});
        });
    } else {
        res.send(401, {result: 'unauthorized'});
    }
});

This enpoint accepts both GET and POST requests and requires an authentication code passed as a query parameter. If the authentication code is valid then the cache is flushed and an email is sent to my email address. This way I can make sure that the cache is flushed when it should be and also be notified in case of malicious activity. To handle emails I am using emailjs.

To complete the automation part I added a webhook from the Github admin panel. This way every time I push a change my custom build server is notified, the cache is flushed and I receive an email.

This works very well, and I am pretty satisfied with the result, especially considering that it was not more than three hours of work.

The complete code can be found here. You will notice that AUTH_CODE, MAIL_USER, MAIL_PASSWORD, MAIL_SENDER, and MAIL_RECIPIENT are not hardcoded but rather read as environment variables from process.env. This way the source code can be shared without disclosing passwords or other information that should not be exposed.

Custom.jhere.net is deployed on Appfog. Although I am not a big fan of their command line tool, their service is pretty awesome, and you can get a very good free forever plan.