Last month, JEP 411 proposed deprecating Java’s Security Manager for eventual removal through a process of gradual functional degradation. The Security Manager should be removed because the high cost of maintaining it is no longer justified by its benefits, which have dropped drastically over time as the deployment and threat environment changed.
Security measures are deployed to defend against a certain threat landscape. As a security measure, the Security Manager was designed to defend against threats posed by untrusted code — code you believe might be malicious.
This is a primary threat that browsers running JavaScript on web sites face, as did Java Applets, the technology for which the Security Manager was originally designed as a sandbox. A sandbox is a mechanism that restricts the operations available for a piece of code, and we’ll discuss it at length later, but the threat landscape facing server-side applications is quite different. Servers tend to mostly run trusted code — code assumed to be benign — and the threats facing it are exploits of vulnerabilities that cause trusted code to perform unintended operations through carefully manipulated inputs.
Java’s security mechanisms for trusted code include a suite of cryptographic protocols, secure XML processing, JAR signing, and serialization filters, but also inherent VM properties, like memory safety — which precludes array overflows and use-after-free — and will increasingly rely on the module system’s encapsulation. Not every feature that assists in security looks like a dedicated security feature; even Loom’s virtual threads help prevent vulnerabilities caused by secrets leaked through ThreadLocals accidentally shared by multiple unrelated tasks. The Security Manager is not a central component for securing trusted server-side code, a good thing, too, because few systems use it and it doesn’t defend against some of the most common and dangerous exploits. In fact, the Security Manager is already crippled on the common pool (and on Loom’s virtual threads) because setting up appropriate security contexts would defeat the performance requirements of those constructs, and using it with CompletableFutures or any asynchronous (or “reactive”) context requires the developer to carefully capture and reestablish security contexts as operations travel from one thread to another. Nevertheless, a sandbox can serve as an additional effective protection layer for trusted code by blocking unintended operations triggered by exploits, but the Security Manager, despite its powerful theoretical capabilities, has been found over the years to be an ineffective sandbox for trusted code.
A sandbox could restrict which API elements are directly available to sandboxed code. Such a sandbox can be called shallow, because it performs access checks close in the call stack to the sandboxed code. In contrast, a deep sandbox blocks operations further away in the call-stack, close to where the operation is actually performed, perhaps when interacting with the OS or with the hardware. A simple deep sandbox blocks certain operations, like writing to a particular file, regardless of how they’re performed; in contrast, a path-dependent (or stack-dependent) deep sandbox might block or allow a particular operation depending on the code path taken to perform it, by combining the different permissions granted to different layers of the call stack.
For applications and libraries that make use of a large set of complex APIs, a deep sandbox can provide better security because only specific operations — such as interaction with the file systems — need to be analysed for their security implications and then restricted, rather than possible hundreds of thousands of API elements. But herein lies the problem with employing the Security Manager for that purpose: it is a path-dependent deep sandbox, which means it is very complex, and complexity is an enemy to security. For one, the set of permissions a complex application requires can be very large, and it is hard to evaluate whether it truly provides the requisite measure of security; Amazon uses formal methods to analyse policy files even for their simple sandboxes. For another, that set depends on internal implementation details of both the application and its dependencies, so it needs to be recomputed and re-analysed with every update to the application or any of its dependencies, greatly increasing the maintenance burden. Finally, the Security Manager’s path-dependence complicates things further, requiring judicious use of AccessController.doPrivileged; if a library doesn’t make use of doPrivileged, the permissions need to be granted to all of its callers on the call stack as well. The result is that the Security Manager is so complex that few applications use it, and those that do, more often than not do it incorrectly.
Shallow Java Sandboxes
Some have pointed out another use case, that of sandboxing server-side plugins. Plugins are normally trusted (even a popular IDE like VSCode treats plugins as trusted code) but sandboxing them is not intended to defend against malicious code so much as to protect the functional integrity of the application by restricting the APIs available to a plugin as a means of controlling its operation. This use case is too narrow and too rare to justify the high cost of the continued maintenance of the Security Manager, and mixing trusted and untrusted code in the same process is a hard problem, but I would argue that it is better served by a shallow sandbox, anyway, allowing the plugin to interact with its environment only through a very limited set of APIs.
It relies on the module system, and layers in particular, which can be thought of as a shallow sandbox, too, but is insufficient to create a restrictive sandbox of the kind we want because the base module alone (java.base) already grants far too much power than we’d like the plugin to have (but modules will allow us to control reflection without blocking altogether). The module system’s coarse granularity when it comes to the java.base module is the main problem that requires most of the code below. We will refine it by employing a custom class loader that restricts the API at class granularity, and then refine it further yet by employing bytecode instrumentation, using the ASM library, to restrict the API at method granularity, because some useful public classes also expose dangerous public methods.
