Probability and random variables form the backbone of advanced signal processing. These concepts provide a mathematical framework for analyzing signals with unpredictable components, enabling us to model and process real-world data effectively.

This section covers key probability theory foundations, types of random variables, and their distributions. We'll explore expectation, moments, and multivariate random variables, essential tools for understanding and manipulating complex signals in various applications.

Probability theory foundations

• Probability theory provides a mathematical framework for analyzing random phenomena, which is essential in advanced signal processing for modeling and processing signals with random components
• Understanding the foundational concepts of probability theory allows for the development of statistical signal processing techniques and the analysis of stochastic processes

Experiments, sample spaces and events

• An experiment is a procedure that yields one of a set of possible outcomes, with the outcome determined by chance
• The sample space is the set of all possible outcomes of an experiment, often denoted as $\Omega$
• An event is a subset of the sample space, representing a collection of outcomes (rolling a die and getting an even number)

Axioms of probability

• Non-negativity: The probability of an event $A$, denoted $P(A)$, is always non-negative, i.e., $P(A) \geq 0$
• Normalization: The probability of the entire sample space is equal to 1, i.e., $P(\Omega) = 1$
• Countable additivity: For a countable sequence of mutually exclusive events $A_1, A_2, \ldots$, the probability of their union is equal to the sum of their individual probabilities, i.e., $P(\bigcup_{i=1}^{\infty} A_i) = \sum_{i=1}^{\infty} P(A_i)$

Conditional probability

• Conditional probability measures the probability of an event $A$ given that another event $B$ has occurred, denoted as $P(A|B)$
• It is defined as $P(A|B) = \frac{P(A \cap B)}{P(B)}$, where $P(A \cap B)$ is the probability of the intersection of events $A$ and $B$
• Conditional probability is used to update probabilities based on new information or observations (probability of a signal being present given a certain measurement)

Statistical independence

• Two events $A$ and $B$ are statistically independent if the occurrence of one does not affect the probability of the other
• Mathematically, $A$ and $B$ are independent if and only if $P(A \cap B) = P(A)P(B)$
• Independent events simplify probability calculations and are often assumed in signal processing models (noise samples being independent of the signal)

Bayes' theorem applications

• Bayes' theorem describes the probability of an event based on prior knowledge and new evidence
• It states that $P(A|B) = \frac{P(B|A)P(A)}{P(B)}$, where $P(A)$ is the prior probability, $P(B|A)$ is the likelihood, and $P(B)$ is the marginal probability of $B$
• Bayes' theorem is used in signal processing for parameter estimation, hypothesis testing, and signal detection (estimating the probability of a signal being present given noisy measurements)

Random variables

• A random variable is a function that assigns a numerical value to each outcome in a sample space, providing a mathematical description of a random phenomenon
• Random variables are essential in advanced signal processing for modeling and analyzing signals with random components

Discrete vs continuous types

• Discrete random variables take on a countable set of values, often integers (number of bit errors in a digital communication system)
• Continuous random variables can take on any value within a specified range or interval (amplitude of a noisy analog signal)
• The type of random variable determines the appropriate probability functions and mathematical tools used for analysis

Probability mass functions (PMFs)

• For a discrete random variable $X$, the probability mass function $p_X(x)$ gives the probability that $X$ takes on a specific value $x$
• The PMF satisfies two conditions: $p_X(x) \geq 0$ for all $x$, and $\sum_x p_X(x) = 1$
• PMFs are used to calculate probabilities, expected values, and other statistical properties of discrete random variables (PMF of the number of photons detected in a quantum optics experiment)

Cumulative distribution functions (CDFs)

• The cumulative distribution function $F_X(x)$ of a random variable $X$ gives the probability that $X$ takes on a value less than or equal to $x$
• For a discrete random variable, $F_X(x) = \sum_{t \leq x} p_X(t)$; for a continuous random variable, $F_X(x) = \int_{-\infty}^x f_X(t) dt$, where $f_X(t)$ is the probability density function
• CDFs are used to calculate probabilities and quantiles of random variables (CDF of the signal-to-noise ratio in a communication system)

Probability density functions (PDFs)

• For a continuous random variable $X$, the probability density function $f_X(x)$ is a non-negative function whose integral over any interval gives the probability of $X$ falling within that interval
• The PDF satisfies two conditions: $f_X(x) \geq 0$ for all $x$, and $\int_{-\infty}^{\infty} f_X(x) dx = 1$
• PDFs are used to calculate probabilities, expected values, and other statistical properties of continuous random variables (PDF of the amplitude of a noisy sinusoidal signal)

Joint probability distributions

• Joint probability distributions describe the probability of two or more random variables taking on specific values simultaneously
• For discrete random variables $X$ and $Y$, the joint PMF is denoted as $p_{X,Y}(x,y)$; for continuous random variables, the joint PDF is denoted as $f_{X,Y}(x,y)$
• Joint distributions are used to study the relationships between multiple random variables and to calculate joint probabilities and conditional distributions (joint distribution of the in-phase and quadrature components of a complex signal)

Expectation and moments

• Expectation and moments are statistical measures that summarize the properties of random variables, providing insights into their average behavior and dispersion
• These concepts are fundamental in advanced signal processing for characterizing signals and estimating their parameters

Expected value of a random variable

• The expected value (or mean) of a random variable $X$, denoted as $E[X]$, is a measure of its central tendency
• For a discrete random variable, $E[X] = \sum_x x p_X(x)$; for a continuous random variable, $E[X] = \int_{-\infty}^{\infty} x f_X(x) dx$
• The expected value is used to estimate the average behavior of a random variable (mean of a noisy signal)

Variance and standard deviation

• The variance of a random variable $X$, denoted as $\text{Var}(X)$ or $\sigma_X^2$, measures its dispersion around the mean
• $\text{Var}(X) = E[(X - E[X])^2] = E[X^2] - (E[X])^2$
• The standard deviation, denoted as $\sigma_X$, is the square root of the variance
• Variance and standard deviation quantify the spread of a random variable (variance of the noise in a signal)

Moments and moment-generating functions

• The $n$-th moment of a random variable $X$ is defined as $E[X^n]$, with the first moment being the mean
• The moment-generating function (MGF) of $X$ is defined as $M_X(t) = E[e^{tX}]$, which generates the moments of $X$ through differentiation
• Moments and MGFs are used to characterize the distribution of a random variable and to derive its properties (MGF of a Gaussian noise process)

Characteristic functions

• The characteristic function (CF) of a random variable $X$ is defined as $\phi_X(t) = E[e^{itX}]$, where $i$ is the imaginary unit
• CFs are an alternative to MGFs for uniquely characterizing the distribution of a random variable
• CFs are particularly useful for studying the properties of sum of independent random variables (CF of the Fourier transform of a random signal)

Common discrete distributions

• Discrete probability distributions are used to model random variables that take on a countable set of values
• These distributions are frequently encountered in advanced signal processing applications, such as digital communications and image processing

Bernoulli and binomial distributions

• A Bernoulli random variable models a single trial with two possible outcomes (success with probability $p$ and failure with probability $1-p$)
• The binomial distribution models the number of successes in a fixed number of independent Bernoulli trials (number of bit errors in a fixed-length digital transmission)
• The PMF of a binomial random variable $X$ with parameters $n$ and $p$ is given by $p_X(k) = \binom{n}{k} p^k (1-p)^{n-k}$ for $k = 0, 1, \ldots, n$

Poisson distribution

• The Poisson distribution models the number of occurrences of a rare event in a fixed interval of time or space
• It is characterized by a single parameter $\lambda$, which represents the average number of occurrences in the interval
• The PMF of a Poisson random variable $X$ with parameter $\lambda$ is given by $p_X(k) = \frac{\lambda^k e^{-\lambda}}{k!}$ for $k = 0, 1, 2, \ldots$ (number of photons detected in a fixed time interval)

Geometric and negative binomial distributions

• The geometric distribution models the number of Bernoulli trials needed to achieve the first success, with success probability $p$
• The PMF of a geometric random variable $X$ is given by $p_X(k) = (1-p)^{k-1}p$ for $k = 1, 2, \ldots$
• The negative binomial distribution generalizes the geometric distribution to model the number of trials needed to achieve a fixed number of successes (number of transmissions needed to successfully deliver a message)

Hypergeometric distribution

• The hypergeometric distribution models the number of successes in a fixed number of draws from a finite population without replacement
• It is characterized by three parameters: the population size, the number of successes in the population, and the number of draws
• The PMF of a hypergeometric random variable $X$ is given by $p_X(k) = \frac{\binom{K}{k}\binom{N-K}{n-k}}{\binom{N}{n}}$, where $N$ is the population size, $K$ is the number of successes in the population, and $n$ is the number of draws (number of defective items in a sample drawn from a lot)

Common continuous distributions

• Continuous probability distributions are used to model random variables that can take on any value within a specified range or interval
• These distributions are essential in advanced signal processing for modeling and analyzing analog signals and noise processes

Uniform distribution

• The uniform distribution models a random variable with equal probability of taking on any value within a specified interval $[a, b]$
• The PDF of a uniform random variable $X$ is given by $f_X(x) = \frac{1}{b-a}$ for $x \in [a, b]$ and zero elsewhere
• Uniform distributions are used to model signals with constant amplitude over a certain range (phase of a sinusoidal signal)

Exponential and gamma distributions

• The exponential distribution models the time between occurrences of rare events in a Poisson process, with rate parameter $\lambda$
• The PDF of an exponential random variable $X$ is given by $f_X(x) = \lambda e^{-\lambda x}$ for $x \geq 0$ and zero elsewhere
• The gamma distribution generalizes the exponential distribution and models the waiting time until the $k$-th occurrence of an event in a Poisson process (time between packet arrivals in a network)

Normal (Gaussian) distribution

• The normal (or Gaussian) distribution is the most widely used continuous distribution, characterized by its bell-shaped PDF
• It is specified by two parameters: the mean $\mu$ and the variance $\sigma^2$
• The PDF of a normal random variable $X$ is given by $f_X(x) = \frac{1}{\sqrt{2\pi\sigma^2}} e^{-\frac{(x-\mu)^2}{2\sigma^2}}$ for $x \in \mathbb{R}$
• Normal distributions are used to model noise processes and the sum of many independent random variables (thermal noise in electronic circuits)

Beta distribution

• The beta distribution models random variables with values between 0 and 1, making it useful for modeling proportions and probabilities
• It is characterized by two shape parameters, $\alpha$ and $\beta$, which control the shape of the PDF
• The PDF of a beta random variable $X$ is given by $f_X(x) = \frac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha, \beta)}$ for $x \in [0, 1]$, where $B(\alpha, \beta)$ is the beta function (proportion of time a signal spends above a certain threshold)

Chi-square, t, and F distributions

• The chi-square distribution models the sum of squares of independent standard normal random variables, with degrees of freedom parameter $\nu$
• The t-distribution arises when estimating the mean of a normally distributed population with unknown variance, with degrees of freedom parameter $\nu$
• The F-distribution models the ratio of two independent chi-square random variables, with numerator and denominator degrees of freedom parameters $\nu_1$ and $\nu_2$
• These distributions are used in hypothesis testing and confidence interval estimation for signal parameters (testing for the presence of a signal in noise)

Multivariate random variables

• Multivariate random variables are used to model and analyze the joint behavior of multiple random variables, which is essential in advanced signal processing for studying the relationships between signals and their components

Joint probability density functions

• For continuous random variables $X$ and $Y$, the joint PDF $f_{X,Y}(x,y)$ gives the probability density of $X$ and $Y$ taking on specific values simultaneously
• The joint PDF satisfies two conditions: $f_{X,Y}(x,y) \geq 0$ for all $x$ and $y$, and $\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} f_{X,Y}(x,y) dx dy = 1$
• Joint PDFs are used to calculate joint probabilities, marginal and conditional distributions, and other statistical properties of multivariate random variables (joint PDF of the real and imaginary parts of a complex signal)

Marginal and conditional distributions

• The marginal distribution of a random variable $X$ is obtained by integrating the joint PDF over the other variable, e.g., $f_X(x) = \int_{-\infty}^{\infty} f_{X,Y}(x,y) dy$
• The conditional distribution of $X$ given $Y=y$ is defined as $f_{X|Y}(x|y) = \frac{f_{X,Y}(x,y)}{f_Y(y)}$, where $f_Y(y)$ is the marginal PDF of $Y$
• Marginal and conditional distributions are used to study the individual behavior of random variables and their relationships (conditional distribution of a signal given the presence of a certain type of noise)

Covariance and correlation coefficient

• The covariance between two random variables $X$ and $Y$ is defined as $\text{Cov}(X,Y) = E[(X-E[X])(Y-E[Y])]$, measuring their linear dependence
• The correlation coefficient $\rho_{X,Y}$ is a normalized version of the covariance, given by $\rho_{X,Y} = \frac{\text{Cov}(X,Y)}{\sigma_X \sigma_Y}$, where $\sigma_X$ and $\sigma_Y$ are the standard deviations of $X$ and $Y$
• Covariance and correlation coefficient quantify the strength and direction of the linear relationship between random variables (correlation between the in-phase and quadrature components of a complex signal)

Linear combinations of random variables

• A linear combination of random variables is a weighted sum of the form $Z = aX + bY$, where $a$ and $b$ are constants
• The mean and variance of a linear combination can be expressed in terms of the means, variances, and covariance of the individual random variables: $E[Z] = aE[X] + bE[Y]$ and $\text{Var}(Z) = a^2\text{Var}(X) + b^2\text{Var}(Y) + 2ab\text{Cov}(X,Y)$
• Linear combinations are used to analyze the properties of sums and differences of random signals (sum of a signal and noise)

Central limit theorem

• The central limit theorem (CLT) states that the sum of a large number of independent and identically distributed (i.i.d.)