Source Code

The Problem with non-deterministic dependencies

You set up a new Node JS project, installed your dependencies with npm install and your app runs smoothly.  A week later, you decided to share your awesome project with your colleagues or friends, they cloned it and install dependencies via npm install, then they run the app and all of sudden, errors everywhere! Puzzled, you looked at your code, and it’s working correctly.  They have the same dependencies and same code, why did the app not work on your friend’s computer?

The answer is bad dependency tree.  This is a common problem within the NPM ecosystem.

Before we go further, I’d like to go over the basics of semantic versioning in NPM:

in your package.json file, you may specify the type of updates you accept for a particular module when you run the install command, the following is from the NPM webpage about semver:

• Patch releases: 1.0 or 1.0.x or ~1.0.4
• Minor releases: 1 or 1.x or ^1.0.4
• Major releases: * or x

the version number x.x.x means Major, Minor, Patch (from left to right):

3	.	9	.	2
major		minor		patch

you can read more about Semver on this excellent guide.

So if you specify a module like this in package.json:

"jquery": "^3.0.1"

then the following are all eligible to be installed by NPM automatically: 3.1.1, 3.0.2, 3.0.4, 3.5.3, etc.

when installing a module with NPM install, by default you’re going to see these symbols for semver in your package.json.  This means your installations will be NON-DETERMINISTIC, because you are allowing the installer to install the version of the module that matches your rule AT THAT TIME.  This creates an enormous room for error when authors of modules release buggy or code-breaking updates relative to your app.

Nested Dependencies

Suppose you have a dependency with module A @version ~1.0.1, and module A depends on module B @version ^1.1.0 and module B depends on module C @version ^1.2.0

If module C decided to release a new version 1.3.0, then when you run NPM install, it will automatically upgrade it to 1.3.0 since you specify it as ^ in package.json.  The important thing is that this is SILENT because in your mind, you’re installing module A and you can’t really tell what type of dependencies Moduel A has on Module B and Module C.  This is the very reason why sometimes your project works on your computer and then a few weeks later, it failed on someone else’s computer.

NPM Shrinkwrap

Luckily, there’s a feature in NPM called NPM Shrinkwrap, shrinkwrap enables you to lock down your nested dependency tree.  This is critical if you want your code to work across all machines and at all times.  The way NPM Shrinkwrap works is that it takes a snapshot of all your modules and sub modules dependency tree, and generates a npm-shrinkwrap.json file in the root of your project directory. When you perform npm install, NPM will look at your shrinkwrap file to perform installation.

Let’s create a shrinkwrap file.

First, we need to install a couple of modules to our package.json, to be extra safe, we will be installing modules using the –save-exact flag, this will override the default semver range operator given by npm.  The downside of doing this is that you’ll have to remember to update/upgrade your modules regularly, but it’s a small price to pay to achieve deterministic installations.

npm i moment jquery bluebird express --save --save-exact

next we lock down our dependencies with npm shrinkwrap –dev command.  The reason we need the –dev command is because by default shrinkwrap DOES NOT include anything from devdependencies section, using –dev flag will include them.

In a few seconds, you should notice a new file created on your project’s root folder called npm-shrinkwrap.json

and that’s it! You may now check in your shrinkwrap file to source control so everyone will always install the same code.

How to install new modules

If you want to install new modules and would like shrinkwrap to auto update, you can need to run npm i –save, the –save flag will tell npm to update your shrinkwrap file as well.

protip:

If you want NPM to always save to your package.json file and always save an exact version when you run npm install, you can set up a config option to .npmrc like this:

echo save=true >> .npmrc
echo save-exact=true >> .npmrc

What’s this extraneous error?

Sometimes, when you run npm shrinkwrap –dev you might get the following error:

npm ERR! Darwin 14.4.0
npm ERR! argv "/Users/user/.nvm/versions/node/v5.6.0/bin/node" "/Users/user/.nvm/versions/node/v5.6.0/bin/npm" "shrinkwrap" "--dev"
npm ERR! node v5.6.0
npm ERR! npm  v3.7.5

npm ERR! Problems were encountered
npm ERR! Please correct and try again.
npm ERR! extraneous: ga@0.0.2 /Users/user/Desktop/Development/pentacode/shrinkwrap/node_modules/ga
npm ERR! 
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/user/Desktop/Development/pentacode/shrinkwrap/npm-debug.log

what this means is that you have a package named ga currently in your node_modules folder that is not listed in shrinkwrap or package.json file, it may be a test package or something you forgot about.  In that case, it’s very easy to remedy this, all you have to do is run:

npm uninstall package_name_here

and that module will be removed, or you can

rm -rf node_modules/package_name_here

Keep your module up to date

Using shrinkwrap will ensure you a deterministic build, but the downside is that your modules will get out of date over time, so to ensure you’re always up to date with critical releases you should run

npm outdated

to see what modules are out of date and update modules that are out of date and test it out with your app, after your tests, you can update your package.json file and generate a new shrinkwrap file and check into your repository.

NPM VET

NPM Vet is a simple CLI tool to help vet your npm package versions. NPM Vet can be used locally, or as a CI build-step to prevent builds passing with mismatched package versions.

This is a really useful tool to show you anything you have locally or on production that may be a mismatch when compared to your definition files.

To install NPM Vet globally:

npm install npmvet -g

There are several outputs to choose from, for a standard output:

npmvet -r inlinetable

and if you want a JSON output:

npmvet -r json

will show you a json output of its findings, so you can easily use this output as part of your internal vetting system or CI.

[
  {
    "name": "bluebird",
    "packageVersion": "3.5.0",
    "installedVersion": "3.5.0",
    "matches": true,
    "locked": true
  },
  {
    "name": "express",
    "packageVersion": "4.15.1",
    "installedVersion": "4.15.1",
    "matches": true,
    "locked": true
  },
  {
    "name": "jquery",
    "packageVersion": "3.1.0",
    "installedVersion": "3.1.0",
    "matches": true,
    "locked": true
  },
  {
    "name": "lodash",
    "packageVersion": "4.17.3",
    "installedVersion": "4.17.3",
    "matches": true,
    "locked": true
  },
  {
    "name": "moment",
    "packageVersion": "2.17.0",
    "installedVersion": "2.17.0",
    "matches": true,
    "locked": true
  },
  {
    "name": "babel",
    "packageVersion": "6.23.0",
    "installedVersion": "6.23.0",
    "matches": true,
    "locked": true
  }
]

There are other tools out there such as Yarn package manager which automatically creates a lock file for you when you install modules, I have an intro and comparison tutorial here. I hope this tutorial provided you enough info about how to lock down your dependencies within the NPM ecosystem.