Cross entropy

cosmos 4th November 2016 at 2:43pm
Information measures

In information theory, the cross entropy between two probability distributions p and q over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set, if a coding scheme is used that is optimized for an "unnatural" probability distribution q, rather than the "true" distribution p.

H(p,q)=Ep[logq]=H(p)+DKL(pq),H(p, q) = \text{E}_p[-\log q] = H(p) + D_{\mathrm{KL}}(p \| q),\!

https://www.wikiwand.com/en/Cross_entropy

Cross entropy is used in Machine learning, as a Loss function