AWS announced that Fortuna, an open source toolkit for quantifying the uncertainty of ML models, is generally available.
Any trained neural network can be used with the calibration methods offered by Fortuna, such as conformal prediction, to produce calibrated uncertainty estimates, Infoq wrote on the subject.

There are numerous documented methods for estimating or calibrating prediction uncertainty, but current uncertainty quantification tools and libraries are limited in scope and do not provide a comprehensive collection of methods. This results in large overhead and makes it difficult to incorporate uncertainty into production systems. Fortuna fills this gap by bringing together well-known techniques, making them accessible to users through a standardized and user-friendly interface.

Fortuna is an open source library for quantifying uncertainty in ML models. Fortuna provides calibration methods, such as conformal prediction, that can be applied to any trained neural network to obtain calibrated uncertainty estimates. In addition, the library supports a number of Bayesian inference methods that can be applied to deep neural networks written in Flax.

Fortuna offers three modes of use: 1/ Run from Flax models, 2/ Run from model outputs, and 3/ Run from uncertainty estimates. Their pipelines are depicted in the following figure, each starting from one of the green panels. The code sample above is an example of using Fortuna starting from Flax models, which allows model training using Bayesian inference procedures. Alternatively, you can start either from the model outputs or directly from your own uncertainty estimates. Both of these latter modes are framework independent and help you obtain calibrated uncertainty estimates starting from a trained model.

Applications that require critical decision making depend on an accurate estimate of the expected uncertainty. When there is uncertainty, it is possible to judge the accuracy of model predictions, defer to human judgment, or determine whether the model can be used safely.