Bounded differences inequality

cosmos 16th May 2019 at 9:43pm
Concentration inequality

a.k.a. McDiarmid's inequality

Let Z1,...,ZmZZ_1,...,Z_m \in \mathcal{Z} (remember for random variables \in is interpreted as they take values in, or the codomain of the r.v. is ) be mm i.i.d. Random variables, and let h:ZmRh: \mathcal{Z}^m \to \mathbb{R} be a function satisfying, for every k{1,...,m}k \in \{1,...,m\} and for all z1,...,zm,zkZz_1,...,z_m,z'_k \in \mathcal{Z}:

h(z1,...,zk1,zk,zk+1,...,zm)h(z1,...,zk1,zk,zk+1,...,zm)c|h(z_1,...,z_{k-1},z_k,z_{k+1},...,z_m) - h(z_1,...,z_{k-1},z_k,z'_{k+1},...,z_m)| \leq c

so changing any of the arguments of hh doesn't change the output much.. Then,

P(h(Z1,...,Zm)Eh(Z1,...,Zm)+t)exp(2t2mc2)\mathbf{P}(h(Z_1,...,Z_m) \geq \mathbf{E}h(Z_1,...,Z_m)+t) \leq \exp{\left(-\frac{2t^2}{mc^2}\right)}

or, equivalently, with probability at least 1δ1-\delta we have

h(Z1,...,Zm)Eh(Z1,...,Zm)+cm2log1δh(Z_1,...,Z_m) \leq \mathbf{E}h(Z_1,...,Z_m) + c\sqrt{\frac{m}{2}\log{\frac{1}{\delta}}}

See version using variance here