A parametric Probabilistic model of functions, given by a neural network. That is, a Bayesian neural network is a probability distribution over a set of functions (all functions implementable by a given neural network architecture $N$), induced by a probability distribution on the weights/Parameters of the neural network $N$.

Given the architecture, we can just focus on the distribution over parameters $P(\theta)$. This is the Prior distribution. As per Bayes' theorem, the posterior distribution over parameters is given by the normalized product of the prior and the Likelihood (the probability of the data (input output pairs) given a parameter $\theta$).

http://mlg.eng.cam.ac.uk/yarin/blog_2248.html