The entropy, $H(X)$, of a Random variable, $X$, with Probability distribution $p$ is defined as

$H(X) = - \sum_x p(x) \log{p(x)}$

Intuitively, entropy is the number of yes/no questions you expect you need to ask to identify the state of the world, under a Model of the world (Probability distribution over states of the world). I.e. how ignorant I think I am about the world.

video

Entropy and Kolmogorov complexity: thesis

Concavity of entropy