How to Test a Node.js Command-Line Tool

Published in

JavaScript in Plain English

11 min readMar 3, 2022

You can use Jest to test command-line tools.

This article focuses on some advanced techniques for front-end testing, as well as a lot of demo code to show how to do integration testing for more complex command-line tools.

Why test cases are needed

Advantages

1. Code quality assurance and increased trust

Looking at Github as a whole, a mature tool library must have:

• Well-developed test cases (Jest/Mocha)

• Friendly documentation (official website/demo)

• Type declaration file d.ts

• Continuous integration environment (git action/circleci)

Without these elements, users may encounter a variety of bugs when using the product, and it is clear that they certainly do not want to see this happen, which ultimately makes it difficult for users to accept your product.

The most important point of test cases is to improve the quality of the code so that others have the confidence to use the tools you develop (the relationship of trust generated by confidence is crucial for software engineers).

In addition, the test cases can be directly regarded as ready-made debugging environments, which will gradually make up for the unthought-of cases in the requirement analysis when writing test cases.

2. Reorganization of the guarantee

Having well-developed test cases can play a critical role in refactoring when the code needs to be updated for major releases.

Choose the test case design method of black-box testing, only care about the input and output, do not need to care about what the test case does internally.

For refactoring, if the final api exposed to the user does not change, then almost no changes are needed, and the previous test cases are reused directly.

Therefore, if the code has good test cases, it can greatly enhance the confidence of refactoring, without the concern that the original function will not work due to code changes.

3. Increase code readability

For developers who want to understand the project source code, reading the test cases is an efficient way.

Test cases provide a very visual representation of the tool’s functionality and behavior in various case scenarios.

In other words, a test case is a “document” for the software developer to read.

// Vue.js test cases
// https://github.com/vuejs/core/blob/main/packages/reactivity/__tests__/computed.spec.ts#L24
it('should compute lazily', () => {
  const value = reactive<{ foo?: number }>({})
  const getter = jest.fn(() => value.foo)
  const cValue = computed(getter)  // lazy
  expect(getter).not.toHaveBeenCalled()expect(cValue.value).toBe(undefined)
  expect(getter).toHaveBeenCalledTimes(1)  // should not compute again
  cValue.value
  expect(getter).toHaveBeenCalledTimes(1)  // should not compute until needed
  value.foo = 1
  expect(getter).toHaveBeenCalledTimes(1)  // now it should compute
  expect(cValue.value).toBe(1)
  expect(getter).toHaveBeenCalledTimes(2)  // should not compute again
  cValue.value
  expect(getter).toHaveBeenCalledTimes(2)
})

Disadvantages

There are two sides to everything, so after talking about the advantages, let’s talk about the disadvantages to help you better determine whether you should write test cases for your project.

No time

When developers generally do not write test cases, the most common point is that they are too cumbersome.

I write test cases of time, the code has long been written, you say testing? Leave it to QA, that is their responsibility.

This situation is completely understandable, usually, the development time is too late, how can we spare time to write test cases?

So you need to judge the value of writing test cases according to the type of the project.

For projects with frequent UI modifications and short life cycles, such as official websites and event pages, I do not recommend writing test cases.

Because they are generally time-sensitive, frequent changes in the structure of the page will directly lead to frequent changes in the test cases, in addition, such projects are generally equipped with QA resources, to a certain extent to ensure the quality of the project (self-testing is still necessary).

On the contrary, for tool libraries and component libraries, since there are few functional changes and generally no QA resources, it is recommended to supplement test cases if a certain user scale already exists.

Can’t write

Writing test cases requires learning the syntax of the test framework, so it requires some learning costs (no time to learn is also the cause of not writing)

The good thing is that the mainstream testing framework on the market is more or less the same, the overall idea tends to be consistent, while the breaking change itself is not much. Ordinary developers can get started in a week, two weeks to advance. After learning to be able to use in any front-end project (learn once, write everywhere).

Compared to the learning cost of Vue.js and React, and combined with the advantages of the previous, is it a very good deal?

After the advantages and disadvantages, let’s share some experiences in writing test cases for a complex command-line tool.

Integration testing of command-line tools

The difference between integration tests and unit tests is that the former is broader and the latter is more fine-grained, and integration tests can also be combined from multiple unit tests.

Since it is a command-line tool, the first thing to think about is how to simulate the behavior of the user using the command-line.

Run Command-line directly

At first, my understanding of the test cases was to simulate the original user input as much as possible. So my thinking was to run the command-line tool directly in the test case.

Run the command-line tool in the child process, then print the output of the child process to the parent process, and finally determine whether the print result is as expected.

Advantage: More in line with the way users use the command-line.

Disadvantage: When you need to debug test cases, the debugger starts with very poor performance due to the dependence on sub-processes, and the test cases often time out or even swallow errors or output some system errors that are not related to the test cases themselves.

Function Run

The last solution lagged too badly and was forced to think of other solutions.

Since I use commander to implement the command-line tool for Node.js, the test cases essentially just need to get the action behind the command to execute.

The commander documentation mentions that calling the parse method with command-line arguments will trigger the action callback.

So we expose a bootstrap startup function that takes command-line arguments and passes them into parse.

Advantages: No dependence on child processes, run test cases directly in the current process, debugger also no problem, successfully solve the performance bottleneck.

Disadvantages: There are side effects in the code, all test cases share the same program instance, test cases can be used individually without problems, but multiple test cases may interfere with each other.

Factory function operation

Having learned from the last time, we expose a factory function that generates command-line tools directly.

This way, each time you run a test case, you create a separate program, making the test cases isolated from each other.

After addressing the initialization of the command-line tool, let’s look at a few special test cases for the command-line case.

Test help command

When testing help commands ( — help, -h), or testing the validation of command-line arguments, commander outputs the prompt text to the process as an error log and calls process.exit to exit the current process.

This will cause the test case to exit early, so this behavior needs to be rewritten.

Commander internally provides an overridden function exitOverride, which throws a JavaScript error instead of the original process exit.

After overriding the exit behavior, to validate the help command text, you also need to use the configureOutput provided by commander.

Then modify the test case:

Testing asynchronous use cases

Command-line tools may have asynchronous callbacks, and test cases need to support asynchronous cases.

The good thing is that Jest works out of the box for asynchronous test cases, so let’s take the help command as an example.

For asynchronous test cases, it is recommended to set a timeout to prevent waiting for test results due to code writing errors.

Jest has a default timeout of 5000ms, which can also be overridden via the configuration file/test case.

In addition to the timeout, the number of assertions added is also a point to ensure the success of asynchronous test cases.

expect.assertions can specify the number of times a single test case will trigger an assertion, which is useful for testing exception catching scenarios.

Both timeouts and the expected number of times do not match will make the test case fail.

Variables in test runs

In a normal test scenario, the value of a variable can be verified by running the return value of the exported function.

However, command-line tools may rely on contextual information (arguments, options) and are not well suited to disassembling and exporting individual functions internally, so how do you test the values of variables during runtime?

I used debug + jest.doMock + toHaveBeenCalled

1. Use the debug module to print parameters that need to be validated (somewhat invasive in code, but the debug module can also be used for logging).

2. Test case runtime hijacks the debug module with jest.doMock so that debug execution returns jest.fn.

3. Validate the entry of jest.fn with toHaveBeenCalled.

Why use jest.doMock instead of jest.mock?
jest.mock declares lifting during runtime, making it impossible to use the external variable f
https://github.com/facebook/jest/issues/2567

Simulate command-line interaction

Command-line tools with command-line interaction are a very common scenario.

Inspired by vue-cli, simulating user input in test cases becomes very simple and not intrusive in any code.

1. Create __mock__/inquirer.js, hijack and proxy the prompt module, and add Jest’s assertion statement to the reimplemented prompt function.

2. Simulate the user’s questions and answers via expectPrompts before running the test case (conditions for creating assertions).

3. When the code runs inquirer.prompt, the proxy jumps to the __mock__/inquirer.js self-defined prompt, and the prompt will match (consume data) in order based on the questions and answers created by the previous expectPrompts.

4. The final proxy prompt will return the same answers object as the real prompt, making the final behavior consistent.

Summary

Ensure that test cases are written independently of each other, do not affect each other, have no side effects, and have idempotency.

This can be done from the following perspectives:

• Each time you run a test case, create a new commander instance.

• Allow a single test case to use singleton mode, do not allow multiple test cases to use the same singleton.

• File System Isolation.

Other Testing Tips

Simulation Job Catalog

jest.spyOn(process, ‘cwd’).mockImplementation(() => mockPath))

jest.spyOn tracks the calls to process.cwd, and jest.mockImplementation rewrites the behavior of process.cwd for the purpose of mocking the working directory.

If you don’t rely on the Jest API, you can also pass the mock working directory as a parameter to the createProgram factory function.

Simulation of file systems

File reading and writing are also operations that contain side effects, and since command-line tools may involve file modifications, there is no guarantee of a clean environment every time a test case is run. To ensure that the test cases are independent of each other, a real file system needs to be simulated.

Choose memory-fs here, which converts real filesystem operations to virtual files in memory.

Create a new __mocks__ folder in the project root.

Add fs.js to the __mocks__ folder to export the memfs module.

Jest treats the files in the __mocks__ folder as modules that can be emulated by default.

Run jest.mock(fs) in the test case to hijack the fs module and proxy it under __mocks__/fs.js.

The file system side effect problem is solved by proxying fs to memfs.

Silent Error Log

jest.mockImplementation does not pass any arguments and will silently process the function after the mock.

Implement the function of not outputting error logs during running tests, making the test cases run cleaner.

Not outputting the error log is not swallowing the error, you can still use try/catch to verify the error scenario.

The error log can also be rewritten using program.configureOutput as mentioned earlier.

Before use.

After use.

Test case life cycle hooks

Jest provides the following hooks

• beforeAll

• beforeEach

• afterEach

• afterAll

Lifecycle hooks are triggered before/after each/all test cases, and by adding public code to the hooks, you can reduce the amount of code to some extent.

For example, with the beforeEach hook, the code is mocked before each test case is run, and after it is finished, it is restored with jest.retoreAllMock.

TypeScript Support

Adding TypeScript support to test cases allows for stronger type hints and allows pre-checking of code types before running test cases.

Add ts-jest , typescript, @types/jest type declaration file.

npm i ts-jest typescript @types/jest -D

2. Add the tsconfig.json file, and add the previously installed @types/jest to the list of declaration files.

{
 “compilerOptions”: {
   “types”: [ “jest” ],
 }
}

3. Modify the test case filename suffix index.spec.js → index.spec.ts and change the CommonJS introduction to ESM.

Test Coverage

Visual representation of test cases run, code not run, number of lines.

Add the coverage parameter to the end of the test command.

jest --coverage

The folder that generates the coverage after running contains the test coverage report.

In addition, test coverage can be integrated with CI/CD platform to generate test coverage report and upload to CDN after each tool release.

Achieve an increasing trend of test coverage per tool release.

Summary

Writing test cases is a way to invest more time upfront (learning test case syntax) and gain a lot later (continuous code quality assurance and improved refactoring confidence).

Suitable for products with fewer changes and fewer QA resources, such as command-line tools, tool libraries.

A tip for writing test cases is to refer to the test cases on the Github of the corresponding tool, often the official test cases are more complete.

Integration testing of command-line tools requires test cases to be isolated from each other to ensure idempotency.

Expose a factory function that creates a commander instance, creating a brand new instance each time the test case is run.

Use Jest’s built-in api, such as jest.spyOn , mockImplementation, jest.doMock, to proxy npm modules or built-in functions, which is less invasive to the code.