How to Watch for Files Changes in Node.js

In a previous article on counting unique items in a JavaScript array, I introduced you to the system I am creating that enables our family to log when the fish 🐟 in our aquarium have been fed. The feeding times are logged to a file by pressing a push-button on a circuit board connected to a Raspberry Pi, pressing an Amazon Dash button, or clicking a button through a web interface. The resulting log file looks like this:

2018-5-21 19:06:48|circuit board
2018-5-21 10:11:22|dash button
2018-5-20 11:46:54|web

Our next challenge is to watch this log file for changes as button pushes are streamed in from one of our three sources (Amazon dash button, circuit board push-button, Web button) and take action.

In this article, we’ll learn how to watch for file changes in Node.js (and take action when those file changes occur) using a real IoT project as a learning context. We’ll explore a few techniques for watching files and ultimately arrive at the best solution.

Article contents

I need a quick solution

Do you haz teh codez? Yes, I do.😉 We’ll be exploring how to use Node’s built-in file watching capabilities. Truth to be told, file watching can be accomplished without taking a dependency on an external package. If you’d rather install an npm package and move on, I’d recommend either chokidar or node-watch. These are both excellent packages that leverage Node’s built-in file watching functionality.

For those that want to learn how to use Node’s built-in file watching capabilities and get a little closer to the metal, keep reading!

First steps

To explore Node’s different file watching options, let’s first set up a project. For starters, create a folder and navigate into that folder from the terminal.

Run the following command to create a package.json file for Node.js:

$ npm init -y

Next, install the log-timestamp package from NPM and save it as a dependency in the package.json file:

$ npm install --save log-timestamp

The log-timestamp package prepends the timestamp to any messages that we log to the console using console.log and enables us to see the timing of the file watching events that our code generates. We’re using log-timestamp purely for educational purposes and is not necessary for any production-ready solutions.

Using fs.watchfile

The built-in fs-watchFile method seems like a logical choice to watch for changes in our log file. The callback listener will be invoked each time the file is changed. Let’s give that a try:

const fs = require('fs');
require('log-timestamp');

const buttonPressesLogFile = './button-presses.log';

console.log(`Watching for file changes on ${buttonPressesLogFile}`);

fs.watchFile(buttonPressesLogFile, (curr, prev) => {
  console.log(`${buttonPressesLogFile} file Changed`);
});

In the code, we watch for changes to the button-presses.log file. The listener is invoked any time the file changes.

The fs.Stats object of the current state of the file (curr) and the previous state of the file (prev) are passed as arguments to the listener so you could, for example, get the previous modified time of the file using prev.mtime.

Open the button-presses.log file in an editor and make a change. Sure enough, the listener is invoked as shown in the following output:

$ node file-watcher.js
[2018-05-21T00:54:55.885Z] Watching for file changes on ./button-presses.log
[2018-05-21T00:55:04.731Z] ./button-presses.log file Changed

You may observe a delay between the time you make the change to the log file and the time you see the listener call the code and print to the console. Why? The fs.watchFile method polls for file changes every 5.007 seconds by default. We can change the default polling time by supplying an options object containing an interval property:

fs.watchFile(buttonPressesLogFile, { interval: 1000 }, (curr, prev) => {
  console.log(`${buttonPressesLogFile} file Changed`);
});

We supply an interval of 1000 milliseconds to specify that the log file should be polled every 1 second for changes.

Node: the fs-watchFile documentation indicates that the callback listener will be invoked each time the file is accessed. I’m currently running Node v9.8.0, and this has not been my experience. I’m only seeing the callback listener get invoked when the file I am watching actual changes.

Using fs.watch

A much better option for watching files is to use fs.watch. Whereas fs.watchFile ties up system resources conducting file polling, fs.watch relies on the underlying operating system to provide a way to be notified of filesystem changes. As cited in the documentation, Node uses inotify on Linux systems, FSEvents on macOS, and ReadDirectoryChangesW on Windows to receive asynchronous notifications whenever files change (in lieu of synchronous file polling). The performance gains of fs.watch become even more significant when watching for file changes in entire directories since the first argument supplied can be either a specific file or a directory.

Let’s give fs.watch a try:

const fs = require('fs');
require('log-timestamp');

const buttonPressesLogFile = './button-presses.log';

console.log(`Watching for file changes on ${buttonPressesLogFile}`);

fs.watch(buttonPressesLogFile, (event, filename) => {
  if (filename) {
    console.log(`${filename} file Changed`);
  }
});

In the code, we watch for changes to the log file and write the result to the console when we see changes.

Let’s change the log file and see what happens. I am running these examples on a Raspberry Pi (Raspbian), so your results might vary if you are running on other OS platforms.

$ node file-watcher.js
[2018-05-21T00:55:52.588Z] Watching for file changes on ./button-presses.log
[2018-05-21T00:56:00.773Z] button-presses.log file Changed
[2018-05-21T00:56:00.793Z] button-presses.log file Changed
[2018-05-21T00:56:00.802Z] button-presses.log file Changed
[2018-05-21T00:56:00.813Z] button-presses.log file Changed

Whoa! I made one change and the listener was fired four times! Depending on the underlying OS platform, multiple events may be generated, perhaps because the file write operation takes X amount of time and multiple changes are detected within the X time frame. We need to modify our solution to be less sensitive.

Technical nuance: fs.watch listens for file modifications that occur as a result of either renaming a file or changing the contents of the file. If we wanted to be super rigorous and watch only for changes to the file contents and not ‘rename’ events, we would modify the above code as follows:

const fs = require('fs');
require('log-timestamp');

const buttonPressesLogFile = './button-presses.log';

console.log(`Watching for file changes on ${buttonPressesLogFile}`);

fs.watch(buttonPressesLogFile, (event, filename) => {
  if (filename && event ==='change') {
    console.log(`${filename} file Changed`);
  }
});

Alas, I am not concerned with being this rigorous—but maybe you are 😄. Furthermore, your mileage may vary in terms of support for the rename event. In my testing, the file rename event was detected when running Node on Windows, but not on Raspbian.

Improvement attempt #1: compare file modification times

We only want our listener to get fired for each real change to the log file. Let’s attempt to improve on our fs.watch code by watching the file modification time to capture bona fide changes to the log file:

const fs = require('fs');
require('log-timestamp');

const buttonPressesLogFile = './button-presses.log';

console.log(`Watching for file changes on ${buttonPressesLogFile}`);

let previousMTime = new Date(0);

fs.watch(buttonPressesLogFile, (event, filename) => {
  if (filename) {
    const stats = fs.statSync(filename);
    if (stats.mtime.valueOf() === previousMTime.valueOf()) {
      return;
    }
    previousMTime = stats.mtime;
    console.log(`${filename} file Changed`);
  }
});

We set a value for the previous modified time (previousMTime) and invoke a console.log whenever the file modified time actually changes. Seems like it should work, shouldn’t it?

Let’s give it a try:

$ node file-watcher.js
[2018-05-21T00:56:50.167Z] Watching for file changes on ./button-presses.log
[2018-05-21T00:56:55.611Z] button-presses.log file Changed
[2018-05-21T00:56:55.629Z] button-presses.log file Changed
[2018-05-21T00:56:55.645Z] button-presses.log file Changed

Whaaat? 😕 The results don’t look much better. There appears to be a lot of chatter and the underlying OS (Raspbian) in this case is generating multiple events as the file is in the process of being saved. Let’s try something else:

Improvement attempt #2: compare file MD5 checksums

Let’s create an MD5 hash (checksum) of the file contents initially and then create another MD5 checksum whenever fs.watch detects an alleged file change. If the file has not really changed, perhaps we won’t receive a false positive.

As a first step, we’ll install the md5 package from npm:

$ npm install --save md5

Next, let’s write some code to detect whether alleged file changes are actual changes with the help of md5 hashing:

const fs = require('fs');
const md5 = require('md5');
require('log-timestamp');

const buttonPressesLogFile = './button-presses.log';

console.log(`Watching for file changes on ${buttonPressesLogFile}`);

let md5Previous = null;

fs.watch(buttonPressesLogFile, (event, filename) => {
  if (filename) {
    const md5Current = md5(fs.readFileSync(buttonPressesLogFile));
    if (md5Current === md5Previous) {
      return;
    }
    md5Previous = md5Current;
    console.log(`${filename} file Changed`);
  }
});

In the code above, we use a similar approach when comparing the file modification times, but we create an md5 hash of the file to look for real changes. Let’s give it a go:

$ node file-watcher.js
[2018-05-21T00:56:50.167Z] Watching for file changes on ./button-presses.log
[2018-05-21T00:59:00.924Z] button-presses.log file Changed
[2018-05-21T00:59:00.936Z] button-presses.log file Changed

Oh noes! This is not what I had hoped for. The OS appears to be emitting events multiple times as the file save operation is in process. I was hoping for a victory here!

The recommended way to use fs.watch

We’ve attempted some different options along the way and failed.☹️ We have not really failed, however, because we have learned a lot along the way.🤔 Let’s see if we can get it right this time around by introducing a small delay (debounce function) so we don’t capture superfluous file change events within a given window of time:

const fs = require('fs');
require('log-timestamp');

const buttonPressesLogFile = './button-presses.log';

console.log(`Watching for file changes on ${buttonPressesLogFile}`);

let fsWait = false;
fs.watch(buttonPressesLogFile, (event, filename) => {
  if (filename) {
    if (fsWait) return;
    fsWait = setTimeout(() => {
      fsWait = false;
    }, 100);
    console.log(`${filename} file Changed`);
  }
});

We implement a debounce function here thanks to some help from our friends on Stack Overflow. A delay of 100 milliseconds appears to work well to emit only one file change event for a given file change while allowing time for multiple file change events to be captured if a file is saved on a fairly frequent basis. Let’s see how it works:

$ node file-watcher.js
[2018-05-21T00:56:50.167Z] Watching for file changes on ./button-presses.log
[2018-05-21T01:00:22.904Z] button-presses.log file Changed

Very awesome – it works! We have found the magic formula for file watching. If you review file watching npm packages, you will find that many implement debounce type functions behind the scenes in concert with fs.watch to accomplish the goal. We’re able to accomplish the same goal ourselves and learn something in the process.

As a final note, you could combine the debounce function with an md5 to only emit events when the file has really changed, and someone did not simply hit the save button without changing the file contents:

const fs = require('fs');
const md5 = require('md5');
require('log-timestamp');

const buttonPressesLogFile = './button-presses.log';

console.log(`Watching for file changes on ${buttonPressesLogFile}`);

let md5Previous = null;
let fsWait = false;
fs.watch(buttonPressesLogFile, (event, filename) => {
  if (filename) {
    if (fsWait) return;
    fsWait = setTimeout(() => {
      fsWait = false;
    }, 100);
    const md5Current = md5(fs.readFileSync(buttonPressesLogFile));
    if (md5Current === md5Previous) {
      return;
    }
    md5Previous = md5Current;
    console.log(`${filename} file Changed`);
  }
});

This borders on the esoteric and would not be necessary in over 99% of contexts, but it presents an interesting thought exercise nonetheless.

Conclusion

We can listen for file changes in Node.js and run code in response to those file changes!🎉 I’m able to detect button presses in my fish feeding application by watching for changes to the log file. There are many contexts in which file watching can be useful once you know that the capability exists.

Do not use fs.watchFile since it polls the system at regular intervals to detect file change events. Use fs.watch instead with a debounce function as I describe.

Go build some awesome Node.js projects that watch for file changes!

Follow @thisDaveJ (Dave Johnson) on Twitter to stay up to date with the latest tutorials and tech articles.

Additional articles

How to Count Unique Items in JavaScript Arrays,
Guide to Installing Node.js on a Raspberry Pi
Using TOML Config Files in Your Node.js Applications
Guide to Using Redis with Node.js

Last updated May 22 2018