A System, often a mathematical or computational one, that simulates/represents another system, i.e. it represents features of the original system, both structural, and behavioural/dynamic.

See these notes on computer graphics:

In Computational mathematics a very useful abstraction paradigm consists of devising four abstraction levels, called universes: the physical universe, the mathematical universe, the representation universe and the implementation universe (Figure 1.4).

The physical universe contains the objects from the real (physical) world which we intend to study; the mathematical universe contain the abstract mathematical description of the objects from the physical world; the representation universe contains discrete descriptions of the objects from the mathematical universe; and the implementation universe contains data structures and machine models of computation so that objects from the representation universe can be associated with algorithms in order to obtain an implementation in the computer.

Note that using this paradigm we associate to each object from the real (physical) world three mathematical models: a continuous model in the mathematical universe, a discrete model in the representation universe and a finite model in the implementation universe. This abstraction paradigm is called the four-universe paradigm (Gomes & Velho, 1995).

Basically, we go from the physical world back to the physical world, going through a peak of abstraction reached at the mind..

The modeling process consists in choosing an object from the physical world, associate to it a mathematical model, discretize it and implement it. This implementation should provide solutions to properly posed problems involving the initial physical object. Note that the loss of information we mentioned above is a critical factor in this chain.

In Reinforcement learning, a model is a method for calculating next states and rewards, given a current state and action. It can be:

• Distribution model: consists of the probabilities of next states and rewards for possible actions, given current state.
• Sample model: produces single transitions and rewards generated according to these probabilities.

Dynamic programming methods require a distribution model because it uses Full backups. A sample model is what is needed to simulate interacting with the environment during which sample backeups, like those used by many Model-free reinforcement learning algorithms can be used. Sample models are generally much easier to obtain than distribution models.