An overview of the learning-to-bound framework.
This overview introduces the Learning-To-Bound framework, a method for estimating performance bounds (like CRB or Barankin) directly from data. Leveraging recent advances in machine learning, particularly generative models, this framework begins with a general problem definition and notation, followed by a discussion of research papers addressing various performance bounds.
Let,
Then, given a dataset $\mathcal{D}=\set{\theta_i,x_i}_{i=1}^{N}$ of $N$ i.i.d parameter-measurement pairs, $\mathcal{D}$ of size $N$, lower bound the mean square error of $\hat{\theta}(\cdot)$ by a learned bound ${LB}(\mathcal{D})$, satisfying: \(\textbf{non-Bayesian: }\mathbb{E}[\epsilon\epsilon^T|\theta]\succeq {B}(\theta)\xleftarrow{\substack{\infty\leftarrow{}N}}{LB}(\theta;\mathcal{D})\quad \text{a.s},\) \(\textbf{Bayesian:}\mathbb{E}[\epsilon\epsilon^T]\succeq {B} \xleftarrow{\substack{\infty\leftarrow{}N}}{LB}(\mathcal{D})\quad \text{a.s},\) where ${B}$ is the theoretical bound.
From here in the rest of this blog, we provide the abstract, paper and code of several Learned Bounds:
Abstract: The Cramér-Rao bound (CRB), a well-known lower bound on the performance of any unbiased parameter estimator, has been used to study a wide variety of problems. However, to obtain the CRB, requires an analytical expression for the likelihood of the measurements given the parameters, or equivalently a precise and explicit statistical model for the data. In many applications, such a model is not available. Instead, this work introduces a novel approach to approximate the CRB using data-driven methods, which removes the requirement for an analytical statistical model. This approach is based on the recent success of deep generative models in modeling complex, high-dimensional distributions. Using a learned normalizing flow model, we model the distribution of the measurements and obtain an approximation of the CRB, which we call Generative Cramér-Rao Bound (GCRB). Numerical experiments on simple problems validate this approach, and experiments on two image processing tasks of image denoising and edge detection with a learned camera noise model demonstrate its power and benefits.
For more details, please checkout:
Abstract: The estimation of the frequency of a single tone signal is a classical problem. The Cramér-Rao lower bound (CRB) on the frequency estimates has been well studied for the case of additive Gaussian noise. In practical applications, however, the probability density function of the noise is rarely Gaussian, or known. Moreover, non-linear effects, as quantization, are often present, making the Gaussian CRB unreachable. In this paper we propose a data-driven approach for evaluating the CRB on frequency estimation with unknown noise and other degradation. Using a learned normalizing flow model, we model the distribution of the measurements by a neural network and obtain an approximate CRB, referred to as a Generative CRB (GCRB). We demonstrate the GCRB on frequency estimation both in cases where the CRB has been previously evaluated, showing the accuracy of the GCRB empirically, and on complex cases where the CRB cannot be evaluated analytically or numerically.
For more details, please checkout:
Abstract:Misspecified Cram ́er-Rao lower bound (MCRB) provides a lower bound on the performance of any unbiased estimator of parameter vector θ under model misspecification. An approximation of the MCRB can be numerically evaluated using a set of i.i.d samples of the true distribution at θ. However, obtaining a good approximation for multiple values of θ requires collocating an unrealistically large number of samples. In this paper, we present a method for approximating the MCRB using a Generative Model, referred to as a Generative Misspecified Lower Bound (GMLB), in which we train a generative model on data from the true measurement distribution. Then, the generative model can generate as many samples as required for any θ, and therefore the GMLB can use a limited set of training data to achieve an excellent approximation of the MCRB for any parameter. We demonstrate the GMLB on two examples: a misspecified Linear Gaussian model; and a Non-Linear Truncated Gaussian model. In both cases, we empirically show the benefits of the GMLB in accuracy and sample complexity. In addition, we show the ability of the GMLB to approximate the MCRB on unseen parameters.
For more details, please checkout:
Abstract: We introduce the Generative Barankin Bound (GBB), a learned Barankin Bound, for evaluating the achievable performance in estimating the direction of arrival (DOA) of a source in non-asymptotic conditions, when the statistics of the measurement are unknown. We first learn the measurement distribution using a conditional normalizing flow (CNF) and then use it to derive the GBB. We show that the resulting learned bound approximates the analytical Barankin bound well for the case of a Gaussian signal in Gaussian noise, Then, we evaluate the GBB for cases where analytical expressions for the Barankin Bound cannot be derived. In particular, we study the effect of non-Gaussian scenarios on the threshold SNR.
For more details, please checkout:
Abstract: The Bayesian Cramér-Rao bound (BCRB) is a crucial tool in signal processing for assessing the fundamental limitations of any estimation problem as well as benchmarking within a Bayesian frameworks. However, the BCRB cannot be computed without full knowledge of the prior and the measurement distributions. In this work, we propose a fully learned Bayesian Cramér-Rao bound (LBCRB) that learns both the prior and the measurement distributions. Specifically, we suggest two approaches to obtain the LBCRB: the Posterior Approach and the Measurement-Prior Approach. The Posterior Approach provides a simple method to obtain the LBCRB, whereas the Measurement-Prior Approach enables us to incorporate domain knowledge to improve the sample complexity and interpretability. To achieve this, we introduce a Physics-encoded score neural network which enables us to easily incorporate such domain knowledge into a neural network. We study the learning errors of the two suggested approaches theoretically, and validate them numerically. We demonstrate the two approaches on several signal processing examples, including a linear measurement problem with unknown mixing and Gaussian noise covariance matrices, frequency estimation, and quantized measurement. In addition, we test our approach on a nonlinear signal processing problem of frequency estimation with real-world underwater ambient noise.
For more details, please checkout: