“As of today, npm audit is a stain on the entire npm ecosystem. The best time to fix it was before rolling it out as a default. The next best time to fix it is now.”
According to Abramov, 99% of the vulnerabilities flagged by the command are false alarms in common usage scenarios. And this appears to be a fairly widespread sentiment among npm users.
To some extent, the situation is unavoidable given the attack surface in the Node.js ecosystem, where the installation of an average npm package means trusting around 80 other packages due to transitive dependencies. But for Abramov, npm audit produces security warnings in contexts where the risks are not a real concern and the alert overload doesn’t help anyone involved. He wrote:
“The root of the issue is that npm added a default behaviour that, in many situations, leads to a 99+% false positive rate, creates an incredibly confusing first programming experience, makes people fight with security departments, makes maintainers never want to deal with Node.js ecosystem ever again, and at some point will lead to actually bad vulnerabilities slipping in unnoticed.”
Kat Marchán, who helped create npm audit fix and is now a senior software engineer at Microsoft, responded via Twitter, “This isn’t wrong,” while going on to explore some of the tradeoffs involved in security alerts and the decisions that led to the current state of affairs, some of which had to do with NPM’s management and labour challenges in 2018 and 2019. Marchán explained:
“The feature overall, for the company, was kind of a marketing (scare-ish) tactic to promote its private registry service. This isn’t to say it was malicious: there was, and still is, a loud clamour for this kind of security visibility/improvement of the ecosystem.”
Rebecca Turner, also involved in the creation of npm’s auditing feature and now a principal engineer at Microsoft, also responded to Abramov’s broadside, acknowledging that NPM’s need for revenue shaped some design decisions. Turner wrote in a Twitter thread:
“The start of the theatre was the report of the number of vulnerabilities found. It didn’t report the number of vulnerable modules, but the number of things ever depending on the vulnerable module, producing numbers often larger than the number of modules in the tree.”
Turner said she’d have pushed back on that at the time if she’d realized the consequences, but testing didn’t show an excessive number of advisories. Turned also said:
“No further development of this feature happened in the main-line life of npm. Priorities and resources were shifted elsewhere in the push for profitability. Discussion around how to make results more manageable was starting to happen the next year … but between the business pushing to develop those as premium features and the firing of half the CLI team for union organizing they never went anywhere.”
It’s not quite so simple as union-busting broke npm audit. Crafting security alerts that provide just the right amount of information at just the right time in the appropriate context is a challenge. As Marchan put it,
“I personally spent a long time workshopping the CLI messages around audit to make them as unobtrusive as possible. But sometimes noise is noise, no matter how much you try to reduce it.”
Further code adjustments being considered could improve the situation by providing a manual way to resolve audit warnings, as could Abramov’s call for a way to exclude certain transitive dependencies from generating security warnings.
But calibrating the level of security concern to be appropriate for every individual and situation is a thankless task – dialling back too much on the frantic hand waving and suppressed vulnerability mitigation advice might just lead to the next SolarWinds or Kaseya compromise.