Please enable JavaScript.

Coggle requires JavaScript to display documents.

MAST30020 Probability for Inference (Characteristic functions (Theorems (…

- - - - Has mass character eg. could be repeated many times, in theory
      - Outcomes are uncertain (to the best of our prior knowledge)
      - Has some statistical regularly - the relative frequencies of outcomes stablize around some values as # independent repeations grows
    - - Events are subsets of the outcome space, which may or may not be in the \(\sigma\)-algebra.
        Events must be measurable in order to calculate their probability
      - Indicator functions
        
        Definition
        
        \(I_A(w) := \begin{cases} 1, & w \in A \\ 0, & w\not\in A \end{cases} \)
        
        Operations with indicator functions
        
        \(\begin{align} A \lor B \equiv&& A \cup B \equiv && \max\{I_A, I_B\} && \approx \exists \\ A \land B \equiv&& A \cap B \equiv && I_A I_B && \approx \forall \\ \lnot A \equiv&& A^c \equiv && 1- I_A && \\ A \oplus B \equiv&& (A \cup B) \setminus (A \cap B) \equiv && |I_A - I_B| \end{align}\)
      - Complements, unions, intersections are events
      - Special cases
        
        Event \(A\): \( A_1, A_2, \dots \) occurred infinitely often (i.o.)
        \( \qquad \equiv A = \bigcap_{n=1}^\infty \bigcup_{k=n}^\infty A_k \)
        \( \qquad \equiv A \text{ occurred iff } \forall n, \exists k\geq n, s.t. [A_k \text{ occured}]\)
        
        Event \( A: A_1, A_2, \dots \) occurred finitely often (f.o.):
        \( \qquad \equiv A = \bigcup_{n=1}^\infty \bigcap_{k=n}^\infty A_i^c \)
        \( \qquad \equiv A \text{ occurred iff } \exists n, \forall k\geq n, [A_k \text{ did not occur} ]\)
      - Independence
        
        Definition: Events \( A_1, A_2, ... \) are independent iff
        \( \qquad \textbf{P}(\bigcap_i A_i) = \prod_i \textbf{P}(A_i) \)
    - - Probability function
        
        Basic properties
        
        :star: Theorem 1.25:
        \( \qquad \textbf{P} \text{ satisfies } P.3 \quad \equiv \quad A_n \uparrow A \Rightarrow P(A_n) \uparrow P(A)\quad \equiv \quad A_n \downarrow A \Rightarrow P(A_n) \downarrow P(A)\quad \equiv \quad A_n \downarrow \emptyset \Rightarrow P(A_n) \downarrow 0 \)
        
        Disjointification:
        \(\qquad A_1 \subset A_2 \subset A_2 ....\\ \qquad B_n = A_n \setminus \bigcup_{i=1}^{n-1} A_i \\ \Rightarrow\\ \qquad \bigcup_{i=1}^n A_i = \bigcup_{i=1}^n B_i \\ \qquad B_n \subset A_n \)
        
        Monotonicity:
        \( \qquad A \subset B \Rightarrow P(A) \leq P(B) \)
        
        :star: Theorem 1.24 Boole's Inequality:
        \(\qquad \forall A_1, A_2, ... \in \mathcal{F}, P(\bigcup_{j=1}^\infty A_i) \leq \sum_{j=1}^\infty P(A_i) \)
        
        :star: Theorem 1.27, (1st) Borel-Cantelli Lemma
        If \( \sum_{n=1}^\infty P(A_n) \leq \infty, \text{ then } P(A_n, i.o.) = 0 \)
        Proof:
        \( \qquad P(A_n, i.o.)\overset{def i.o.}{=} P(\bigcap_{n\geq 1} \bigcup_{k\geq n} A_k) \overset{1.25c}{=} \lim_{n\rightarrow \infty} P(\bigcup_{k \geq n} A_k) \overset{Boole}{ \leq} \lim_{n\rightarrow \infty} \sum_{k \geq n}P(A_k) \overset{*}{=} 0 \)
        (*) since by assumption \( \sum_{i\geq 1} P(A_k) < \infty \)
        
        Axioms to satisfy
        
        :red_flag: P.3) Countable additivity (for disjoint \( A_1, A_2, \dots \))
        \( \qquad \textbf{P} (\bigcup_{i=1}^{\infty} A_i) = \sum_{i=1}^{\infty} \textbf{P} (A_1) \)
        
        :red_flag: P.1) Non-negativity:
        \( \qquad \textbf{P} (A) \geq 0, A \in \mathcal{F} \)
        
        :red_flag: P.2) Law of total probability:
        \( \qquad \textbf{P} (\Omega) = 1 \)
        
        PF.3) - finite version of P.3
        
        Defined on measurable subsets of \( \Omega\)
      - Sigma-algebra
        
        Smallest generated: \(g \subset g'\) for all others
        
        Examples
        
        Borel set: \(\mathcal{F} = \mathcal{B}(\mathbb{R}) = \sigma\{ (a,b] : a,b \in \mathbb{R}, a < b \} \)
        
        Generated by event: \( \mathcal{F} = \{ \emptyset, A, A^c, \Omega \} \)
        
        Powerset: \( \mathcal{F} = \mathscr{P}(\Omega) \)
        
        Trivial: \(\mathcal{F} = \{ \emptyset, \Omega \} \)
        
        Must satisfy
        
        :red_flag: A.3) Closed under countable union:
        \(\qquad A ,B \in \mathcal{F} \Rightarrow A \cup B \in \mathcal{F} \) etc
        
        :red_flag: A.2) Closed under complementation:
        \( \qquad A \in \mathcal{F} \Rightarrow A^c \in \mathcal{F} \)
        
        :red_flag: A.1) Contains the outcome space itself:
        \( \qquad \Omega \in \mathcal{F} \)
        
        A set of subsets of the outcome space / a subset of the powerset of the outcome space
      - Outcome space
        
        The set of all possible outcomes (may be finite, countable, uncountable). Elements of the outcome space are individual outcomes.
      - \( (\Omega, \mathcal{F}, \textbf{P} ) \)
        
        \(\Omega\): outcome space
        \(\mathcal{F} \subset \mathscr{P}(\Omega)\) : a \(\sigma\)-algebra of \( \Omega\)
        \( \textbf{P} : \mathcal{F} \rightarrow [0,1]\): a probability function from measurable sets to [0-1]
  - - - Probability space is \( ( \mathbb{R}, \mathbb{B}(\mathbb{R}), P) \)
      - Probability function is defined in a "simpler" way - by the cdf, which defines every \( (a,b] \) interval hence defines the borel sets
    - - The "distribution function" of \( P \) is an economical way to define it where \(F(t) = P(\{-\infty, t]) \), which
      - :star: Theorem 1.33 Definition of DF,
        For any DF of P, \(F(t)\) satisfies:
        
        :red_flag: 1) \(F\) is non-decreasing, so it has one sided limits:
        \( \qquad F(t-) = \lim_{s \uparrow t} F(t) \\ \qquad F(t+) = \lim_{s \downarrow t} F(t) \)
        Proof: monotonicity of \(P, (\infty, s] \subset (\infty, t], s < t, P( (\infty, s]) \leq P((\infty, t]), F(s) \leq F(t) \)
        
        :red_flag: 2) \( F \) is right continuous
        \( \qquad F(t) = F(t+) \)
        Proof: Right-continuity of \( P: A_n \downarrow A = (\infty, t], P(A_n) \downarrow P(A), F(t_n) \downarrow F(t) \)
        
        3) \( \lim_{t\rightarrow -\infty} F(t) = 0, \lim_{t \rightarrow \infty} F(t) = 1 \) :red_flag:
        Proof: \( A_n = (-\infty, t] \downarrow \emptyset, F(t) \downarrow P(\emptyset) = 0 \\ A_n = (-\infty, t] \uparrow \mathbb{R}, F(t) \uparrow P(\mathbb{R}) = 1 \)
      - :star: Theorem 1.36
        Any function \(F \) which satisfies 1.33 defines a unique \(\textbf{P}\) on \( \mathbb{B}(\mathbb{R}), F = F_P\)
        This means we can completely define a probability on \( \mathbb{R} \) by the DF.
    - - Absolutely continuous (AC) probabilities on \( \mathbb{R} \)
        
        These ones have densities / pdf's!
        
        \( F_P(t) \) is AC iff exists \(f(t)\) s.t. \( F_P(t) = \int_{-\infty}^t f(s) ds \)
        
        Any integral \(f(t) \geq 0 \) which integrates to 1 is a density, hence defines a distribution
      - Discrete probabilities on \( \mathbb{R} \)
        
        \(P(C) = 1\) for some countable set \( C \subset \mathbb{R} \)
        
        Equivalent characterizations:
        
        \(P\) is discrete
        
        \(P = \sum_i p_i \epsilon_{t_i}, \sum_i p_i = 1 \)
        
        \(F_P(t) = \sum_i p_i I_{t_i \leq t} \)
        
        for some \( \{ t_i \}_{i \geq 1} \subset \mathbb{R} \)
      - Mixed distributions
        
        Any convex combination of \(P_1, P_2\) defines a new probability function:
        \( P = pP_1 + (1-p)P_2, p \in [0,1] \)
      - Singular distributions
        
        Has continuous DF, but not AC! Ie. lacks a density
      - :star: Theorem 1.52 Lebesgue's decomposition
        Every probability function \(P\) can be represented as the weighted combination of some discrete, AC, and singular distirbutions
  - - - Definition
        
        This means the probability \( \textbf{P}(X \in B) \) is defined for any \( B \in \mathbb{B}(\mathbb{R}) \)
        
        Since \( X^{-1}(B) \) preserves all set operations and disjointness, it is sufficient to show that:
        \( \qquad \{X \in (-\infty, t] \} \equiv X^{-1}((-\infty, t]) \equiv \{ \omega \in \Omega : X(\omega) \in (-\infty, t] \} \in \mathcal{F} , \forall t \in \mathbb{R}\)
        
        :red_flag: \( X(w) : \Omega \rightarrow \mathbb{R} \), s.t. the inverse image of any \( B \in \mathbb{B}(\mathbb{R}) \) must be in \(\mathcal{F}\):
        \( \qquad \{X \in B \} \overset{def}{\equiv} X^{-1}(B) \equiv \{ \omega \in \Omega : X(\omega) \in B \} \in \mathcal{F}, \forall B \in \mathbb{B}(\mathbb{R}) \)
        Also called "measurable" function.
      - RVs preserve set operations :red_flag: Prop. 2.2
        
        Subsets: \( B_\alpha \subset B_\beta \Rightarrow X^{-1}(B_\alpha) \subset X^{-1}(X_\beta) \subset \Omega \)
        
        Intersection: \( \bigcap_{\alpha\in I} X^{-1}(B_i) = X^{-1} (\bigcap_{\alpha \in I} B_i) \)
        
        Unions: \( \bigcup_{\alpha\in I} X^{-1}(B_i) = X^{-1} (\bigcup_{\alpha \in I} B_i) \)
        
        Disjointness: \( B_\alpha \cap B_\beta = \emptyset \Rightarrow X^{-1}(B_\alpha) \cap X^{-1}(B_\beta) = \emptyset \)
        
        Complements: \( X^{-1}(B_\alpha^c) =[X^{-1}(B_\alpha)]^c \)
        
        :red_flag: Prop 2.9, given an RV \( X \), \( \sigma(X) = \{ X^{-1}(B) : B\in \mathbb{B}(\mathbb{R}) \} \) defines a \( \sigma\)-algebra
      - Types of RVs
        
        Simple RVs: \(X = \sum_{i=1}^n a_i I_A\)
        
        Random vectors:
        
        Complex valued RVs: \( Z : \Omega \rightarrow \mathbb{C}, Z = X + iY \)
      - Probability functions of RVs
        
        \(P_X(B) := \textbf{P}(X \in B), B \in \mathbb{B}(\mathbb{R}) \) is called the distribution of \(X\)
        
        This defines a probability function
        
        So the DF of \(X\) is \( F_X(t) = P_X((-\infty, t]) = \textbf{P}(X \in (-\infty, t])) \)
        
        "Survival tail" of \(X\) is \(S_X(t) = 1 - F_X(t)\)
    - - Functions of RVs
        
        General fact: if \(g(x)\) is a continuous function then \(g(X)\) is a RV
        
        Constant multipliers, linear combinations, products of RVs are RVs
      - :star: Prop 2.40 if \( g \) is increasing and continuous function then:
        \( g(X) \sim F_X(g^{-1}(t)) \)
      - :star: Theorem 2.41 if \(X\) is RV, \(g\) is continuously differentiable on and open set, then density is:
        \( g(X) \sim f_X(g^{-1}(t))|\tfrac{d}{dt}g^{-1}(t)| \)
      - General transformation is:
        \( Y = g(X), F_Y(t) = P(Y \leq t) = P(g(X) \leq t) \) etc
      - :star: Theorem 2.43 For RVecs, replace |...| with the Jacobian of the transform (determinant)
    - - \( X = (X_1, X_2, ... X_d) \in \mathbb{R}^d\)
        Equivalently, \(X^{-1}(B) \in \mathcal{F}, B \in \mathbb{B}(\mathbb{R}^d) \)
      - Distributions of RVecs are:
        \( \qquad F_X(t_1,...t_d) = P(X_1 \leq t_1, ... X_d \leq t_d) \)
      - In terms of joint density:
        \( \qquad F_X(t_1, ... t_d) = \int_{-\infty}^{t_1} ... \int_{-\infty}^{t_d} f_X(t_1,...t_d) dt_d...dt_1 \)
      - :star: Prop 2.28 RVecs comprised of discrete RVs are discrete
      - :star: Prop 2.29 Marginal density of \(X_j\) in RVec can be found by integrating over all the other variables
    - - Definition
        
        RVs \( X_1,...,X_n \) are independent if the joint probabilities can always be factored into individual probabilities:
        \( \qquad \forall B_1, ... B_n \in \mathbb{B}(\mathbb{R^n}), P(X_1 \in B_1, ..., X_n \in B_n) = P(X_1 \in B_1) ...P(X_n \in B_n) \)
      - Theorems
        
        Continuous RVs
        
        :star: Theorem 3.3, RVs are independent if DFs factorize:
        \( \qquad \forall t_1, ..., t_n \in \mathbb{R}^n, F(t_1, ... , t_n) = F_{X_1}(t_1) ... F_{X_n}(t_n) \)
        
        :star: Theorem 3.5, Rvs are independent iff joint density factorizes:
        \( \qquad \forall t_1, ..., t_n \in \mathbb{R}^n, f(t_1, ... , t_n) = f_{X_1}(t_1) ... f_{X_n}(t_n) \)
        
        Discrete RVs
        
        :star: Theorem 3.4, discrete RVs are independent if joint probability factorizes:
        \( \qquad \forall t_1, ..., t_n \in \mathbb{R}^n, \textbf{P}(X_1 = t_1, ... ,X_n = t_n) = \text{P}(X_1 = t_1) ... \textbf{P}(X_n = t_n) \)
        
        General
        
        :star: Functions of independent RVs are independent
        
        Of events
        
        :star: Definition 3.19: Events \( A_1, A_2, ... \) are independent iff
        \( \qquad \textbf{P}(\bigcap_i A_i) = \prod_i \textbf{P}(A_i) \)
        Equivalently, if their indicator RVs are independent RVs as per definitions above. Proof:
        \( \qquad \textbf{P}(\bigcap_i A_i ) = \textbf{P} (I_{A_1} = 1, I_{A_2}, = 1, ...) \overset{ind.}{=} \textbf{P}(I_{A_1} = 1)\textbf{P}(I_{A_2} = 1).... = \prod_i \textbf{P} (A_1) \)
        
        :star: Corollary 3.21: Events \( A_1, ... A_n \) are independent iff \( A_1^c, ... A_n^c \)
- - - - Discrete case
        
        Definition in terms of summation: (discrete)
        \( E(X) = \sum_i t_i P(X = t_i) \)
        
        Interpretation in terms of relative frequency (r.f.):
        \( \qquad \overline{X_n} := \frac1n \sum_j X_j = \frac1n \sum_j \underbrace{ \sum_i t_i I(X_j = t_i) }_{=X_j} = \frac1n \sum_i t_i \underbrace{\sum_j I(X_j = t_i) }_{=n_i} = \sum_i t_i \underbrace{\frac{n_i}{n}}_{=r.f.} \approx \sum_i t_i P(X = t_i) \)
        
        Interpretation in terms of indicators:
        \( \qquad E(X) := E(\sum_i t_i I(X_i = t_i)) = \sum_i t_i E(I(X_i = t_i)) = \sum_i t_i P(X_i = t_i) \)
        
        Definition in terms of integrals:
        \( E(X) = \int_\Omega X(\omega) P(d\omega) = \int_\Omega X(\omega) dP(\omega) = \int_\Omega X(\omega) dP \)
      - Continuous case
        
        \( E(X) = \int x f_X(x) dx \)
      - Properties of expectation
        
        Constants: Expectations of constants are the constants themselves eg. \(X=c \Rightarrow E(X) = c \)
        
        Linearity: \(E(aX+b) = aE(X) + b\)
        
        Monotonicity: \( X \leq Y \Rightarrow E(X) \leq E(Y) \)
      - Examples
        
        Simple RV:
        \( \qquad E(X) = E(\sum_i a_i I_{A_i}) = \sum_i a_i E(I_{A_i}) = \sum_i a_i P(A_i) \)
        
        Indicators:
        \( \qquad E(I_A) = P(I_A = 0) \cdot 0 + P(I_A = 1)\cdot 1 = P(1_A=1) = P(A) \)
      - Expectation over an event
        
        Defined as \( E(X;A) = E(X I_A) \)
        
        \(E(X;A) \leq E(X) \)
        
        \( \sum_i E(X;A_i) = E(X) \) for some \(\Omega\) partition \(A_i \)
        .... reminiscent of the LTP \( \sum_i P(A \cap B_i) = P(A) \)
      - Random Vectors & Complex numbers
        
        Defined element-wise
    - - :warning: Def 4.12 An RV is "integrable" iff \( E|X| < \infty \)
        :warning: Notation: \(X\) is integrable \(\equiv X \in L^1 \)
      - Integrable RVs can have expectation defined as:
        \( \qquad E(X) := EX^+ - EX^- \)
        where \( X = X^+ - X^-\) (so \( |X| = X^+ + X^- \)). This definition still works if at most one of the terms is infinite.
      - :star: Cor 4.14: if \( X \in L^1 \Rightarrow |EX| \leq E|X| \)
        Proof: \( |EX| = |E(X^+ - X^-)| \overset{tri.}{\leq} |E(X^+)| + |EX^-| \overset{pos.}{=} E(X^+) + E(X^-) \overset{lin.}{=} E(X^+ + EX^-) = E|X| \)
      - While Reimann integration partitions the domain, Lebesgue integration partitions the range:
        \( \int g(x) dF(x) \)
    - - :star: Thm 4.23 For \( X \geq 0\):
        \( \qquad E(X) = \int_0^\infty (1 - F_X(x))dx \)
        \( \qquad E(X) = \sum_{n\geq1} nP(X=n) = \sum_{n\geq 1}P(X \geq n) \)
      - :star: 4.39 Jensen's inequality:
        Let \( X\in L^1, g\) be convex, then, \( g(EX) \leq Eg(X) \)
        (special case: \( |EX| \leq E|X| \)
        If concave, then \( g(EX) \geq Eg(X)\)
      - :star: Cor 4.37 Lyapunov's inequality. For \( 0 < r \leq s\):
        \( \qquad (E|X|^r)^{1/r} \leq (E|X|^s)^{1/s} \)
        NOTE: implies that if k'th moment is finite, all smaller ones are too!
      - :star: Thm 4.40 Chebyshev's / Markov's inequality
        if \(g\) non-decreasing function, then for any \(X, a \in \mathbb{R} \):
        \( \qquad P(X \geq a) \leq \frac{Eg(X)}{g(a)} \)
        Proof:
        \( P(X \geq a) = E(I(X\geq a)), I(X\geq a) \leq X/a \\ \Rightarrow I(X \geq a) \leq g(X)/g(a) \Rightarrow E(I(X \geq a) \leq E(g(X)/g(a)) \Rightarrow P(X \geq a) \leq E(g(X)) / g(a) \)
      - :star: Cor 4.39 Cauchy-Bunyakovsky inequality
        \( \qquad E|XY| \leq \sqrt{EX^2 EY^2} \)
    - - Functions of RVs
        
        :star: Cor 4.30: if \( X_1, X_2 \) independent then \( \qquad E(g(X_1),h(X_2)) = E(g(X_1)) E(h(X_2)) \)
        
        \(Eg(X) = \sum_i g(t_i) P(X = t_i) \)
        \(Eg(X) = \int g(x) f_X(x) dx \)
      - Moments
        
        :red_flag: Def. k'th moment: \(E(X^k)\)
        
        :red_flag: Def. k'th central moment: \( E((X - E(X))^k) \)
        
        :red_flag: Def. k'th absolute moment: \( E|X - E(X)|^k \)
        
        if \( E(|X|^p) < \infty \) then \(X \in L^p, p > 0 \)
      - Covariance
        
        :red_flag: Def: For \( X,Y \in L^2\), \( Cov(X,Y) \ E((X-EX)(Y-EY)) = E(XY) - E(X)E(Y)\)
        
        :red_flag: Def: \( Corr = \frac{Cov(X,Y)}{\sqrt{Var(X)Var(Y)}}\). NOTE: \(|Corr(X,Y)| \leq 1 \)
        
        Notes
        
        Correlation is a measure of linear association
        
        Uncorrelated RVs are like orthogonal vectors; cos of angle between u,v ~~ correlation between X,Y - where 1 means zero angle and perfect correlation etc
        \( |Corr(X,Y)| = 1 \Leftrightarrow P(Y = aX+b) = 1 \)
        
        Covariance matrices
        
        \( M= Cov_X^2 = E((X-EX)^T (X-EX)) \)
        \( M_ij = Cov(X_i, X_j) \)
        
        :warning: CovM.1 \( Cov_X^2 \) is symmetry
        
        :warning: CovM.2 \( Cov_X^2 \) is positive/non-negative definite: \( x Cov_X^2 x^T \geq 0 \)
        
        Multivariate normal (MVN)
        
        Let \( X \sim N(0,1)^d \), with density \( \frac{1}{(2\pi)^{d/2}} \exp \{ -\tfrac12 xx^T \} \)
        
        Let \( Y = \mu + XA \), then:
        \( C_Y^2 = E((Y - EY)^T(Y - EY)) = E((XA)^T(XA)) = E(A^TX^T X A) = A^T I A = A^T A \)
        
        Then \( f_Y(y) = \frac{1}{(2\pi)^{m/2} \sqrt{det(C_Y^2)}} \exp \{ -\tfrac12 (y-\mu) (C_Y^2)^{-1} (y - \mu)^T \} \)
  - - - The idea
        
        However when we do have information about \( X \) then we can improve the best case via conditional expectation
        
        Conditional expectation is not a numerical value; rather it is a function of the observed RVs
        
        Conditional expectation \( E(X|A) \) minimizes the quadratic error of the conditional event:
        \( g(a) := E((X-a)^2; A) = E((X-a)^2 I_A) = E(X^2 I_A) - 2aE(X I_A) + a^2 E(I_A) \)
        \( \tfrac{d}{da} g(a) = -2E(X I_A) + 2a P(A) \overset{set}{=} 0 \)
        \( \min_a g(a) = E(X I_A)/P(A) = E(X; A)/P(A) = E(X | A) \)
        
        The values of \(Y\) don't matter for \(\hat{X} = E(X|Y) \); the partition generated by the values is what matters.
        
        If \(A_i := \{Y = y_i\}\) is observed then the best guess at \(X\) is:
        \( \hat{X} = E(X|Y=y_i) = E(X|A_i) = \frac{E(X; A_i)}{P(A_i)} := x_i \)
        
        Expectation is the "best guess". Also it minimizes the mean squared error, ie. \( EX = \min_a (X - a)^2 \), in the absence of extra information about \( X\)
      - Axioms
        
        :red_flag: CE.2 On the atoms of \( Y \), \(X, \hat{X} \) are indistinguishable ie. \( E(X|A) = E(E(X|Y)|A) = E(\hat{X}|A) \)
        
        :red_flag: CE.1 The RV \( \hat{X}\) is flat on the atoms of \( Y \) ie. is a RV of \( \sigma(Y) \)
        i.e. \( \hat{X} =E(X | Y_1, ... , Y_n) = h(Y_1, ..., Y_n) \) is a function of \( Y_1, ... Y_n \) and nothing else
    - - If \( \psi(x) \) is a one to one function then \(E(X|Y) = E(X|\psi(Y)) \). So the RV \(Y\) is less important than the information it contained
      - :red_flag: CEP.1 Linearity: \( E(aX + bZ | Y) = aE(X|Y) + bE(Z|Y) \)
      - :red_flag: CEP.2 Monotonicity: \( X \leq Z \Rightarrow E(X|Y) \leq E(Z|Y) \)
      - :red_flag: CEP.3 functions of Y behave like constants when conditioning on Y:
        \(Z = h(Y) \Rightarrow E(ZX|Y) = ZE(X|Y) \)
      - :red_flag: CEP.4 Independence: if \( X,Y\) independent then \( E(X|Y) = E(X) \)
      - :red_flag: CEP.5 Double expectation law:
        \( E(E(X|Y_1, Y_2) | Y_1) = E(X | Y_1) \)
        \( E(E(X|Y)) = E(X) \)
    - - Conditional distribution of \( X|Y \), \( P(X \in B | Y) = P(I(X \in B) | Y) = E(I(X \in B) | Y) \)
      - For AC distributions:
        \( f_{X|Y}(x|y) = \frac{f_{X,Y} (x,y)}{f_Y(y)}, f_Y = \int f_{X,Y}(x,y) dx\)
        \(E(X|Y) = \int x f_{X|Y}(x|y) dx = g(Y)\)
- - - - Pointwise convergence of the sequence of function on a set of probability 1
      - \( X_n(\omega) \rightarrow X(\omega), \forall \omega \in A, P(A) = 1 \)
    - - The probability of the gap between \(X_n, X\) being greater than any \( \epsilon > 0 \) goes to 0:
      - \(P(|X_n - X| > \epsilon) \rightarrow 0,\forall \epsilon > 0, P(A) = 1 \)
    - - The distribution function approaches the limit at all continuity points:
      - \( F_{X_n}(t) \rightarrow F_X(t), n\rightarrow \infty, \forall t s.t. F_X(t-) = F_X(t) \)
      - Discrete case: \(P(X_n = k) \rightarrow P(X = k)\)
    - - Expectation of absolute difference goes to 0
      - \(E|X_n - X| \rightarrow 0, n \rightarrow \infty\)
    - - Expectation of squared difference goes to 0
      - \(E(X_n - X)^2 \rightarrow 0, n \rightarrow \infty\)
- - - - :warning: Cor (of NF): the MLE is a function of a sufficient statistic:
        \( \max_\theta f_\theta(X) = \max_\theta h(X)\psi(S(x),\theta) = h(X) \max_\theta \psi(S(x)) \)
      - :warning: Def. \(\hat{\theta}= argmax_\theta f_\theta(x) \)
    - - :warning: Theorem: An efficient estimator in \(K_b\) is unique
      - :warning: Rao-Blackwell Theorem: If you take an estimator, and take its expectation conditioned on S (sufficient statistic), then the bias remains the same and the variance will be less or equal:
        Let \(\theta^* \in \mathcal{K}_b, S\) is \(SS\), and let \(\theta_S^* = E-\theta(\theta^* | S )\).
        Then \( \theta_S^* \in \mathcal{K}_b, E(\theta_S^* - \theta)^2 \leq E(\theta^* - \theta)^2, \forall \theta \)
      - To apply RB theorem:
        1) Identify the conditional distribution of the estimator on the statistic
        2) Compute the mean and variance of it
        3) Should be better than the variance of the estimator!