A minimal sufficient statistic is a function of every other sufficient statistic.

The notion of minimal sufficient statistics was introduced by Lehmann and Scheff´e (Lehmann and Scheff´e, 1950) as the simplest sufficient statistics, or the coarsest sufficient partition of the sample space which captures the relevant components of the sample with respect to the parameter.

Pitman-Koopman-Darmois theorem showed that exact sufficient statistics with bounded dimensionality exist only for distributions of exponential form (Koompan, 1936).

Kullback and Leibler (Kullback and Leibler) related suf- ficiency to Shannon’s information theory, showing that suf- ficiency is equivalent to preserving mutual information on the parameter, while minimal sufficient statistics minimize the mutual information with the sample due to the dataprocessing inequality (Cover and Thomas, 1991). The Information bottleneck (IB) method, introduced in (Tishby, Pereira and Bialek, 1999), is an information theoretic generalization of the minimal-sufficient-statistic concept to general distributions of two variables, X and Y .