The notion of probability is defined mathematically as a Probability measure on a Sigma-algebra, and corresponds to the intuitive notion of uncertainty in the outcome on any instantiation of the random experiment whose possible outcomes the sigma-algebra models.
Simple but counterintuitive example (similar to problems in statMech) – similar Example showing an important subtlety – related thing on statmech