2 Foundations of probability

Probability theory is the discipline concerned with the study of uncertain (or random) phenomena and probability is the mathematical language adopted for quantifying uncertainty. Such phenomena, although not predictable in a deterministic fashion, may present some regularities and consequently be described mathematically by idealized probabilistic models. These models consist of a list of all possible outcomes together with the respective probabilities. The theory of probability makes possible to infer from these models the patterns of future behaviour.

This chapter presents the basic notions of probability which serves as a necessary background to understand the statistical aspects of machine learning. We ask the reader to become acquainted with two aspects: the notion of random variable as a compact representation of uncertain knowledge and the use of probability as an effective formal tool to manipulate and process uncertain information. In particular, we suggest the reader give special attention to the notions of conditional and joint probability. As we will see in the following, these two related notions are extensively used by statistical modelling and machine learning to define the dependence and the relationships between random variables.