Mathematical Statistics
John Travis
CHAPTER 3
Defn: The kth moment of the random variable X about the value a is given by E( (X-a)k ). If a=0, we call this the kth moment about the origin.
Remark: The first moment is the mean. The second moment about the mean is the variance. The other moments don't have as much of a practical interpretation but can be used to describe the distribution under consideration. We would like to have a simple way to generate these higher moments.
Defn: The probability generating function of the random variable X is given by h(t) = E( tX ).
Result: h(t) can be expressed as a series whose coefficients are the values of the p.d.f.
Pf: Assuming the space of X to start at zero,
h(t) = E( tX ) = S tx f(x) = t0 f(0) + t1 f(1) + t2 f(2) + t3 f(3) + ...
h(0) = f(0), h'(0) = f(1), h''(0) = 2! f(2), h'''(0) = 3! f(0), and in general
f(r) = h(r)(0)/r!
Results: Given a probability generating function h(t),
1. h(1) = 1
2. h'(1) = E[X] = m
3. h''(1) = E[X(X-1)] = s2 - m + m2
Results: Probability generating functions for several discrete distributions are listed below:
· Geometric: h(t) = pt / [1 - (1-p)t ]
· Binomial: h(t) = [ (1-p) + pt ]n
· Negative Binomial: h(t) = (pt)r / [1 - (1-p)t ]r
· Poisson: h(t) = el(t-1).
Defn: The moment generating function M(t) for the random variable X is defined to be E[etX]. We say that a moment generating function for X exists if there is a positive constant b such that M(t) is finite for |t| < b.
Result: M( ln(t) ) = E[ e ln(t)X ] = E[ tX ] = h(t). Hence, one can find probability generating functions by using moment generating functions and vice versa.
Result: M(t) can be expressed as an infinite series which contains all moments for a given distribution.
Pf: Notice, by McLaurin expansion, etx = 1 + (tx) + (tx)2/2! + (tx)3/3! + ... + (tx)n/n! + ...
Hence, assuming each moment is finite, we have
E(etx) = Setxf(x) = S{1 + (tx) + (tx)2/2! + (tx)3/3! + ... + (tx)n/n! + ...} f(x)
= S f(x)+ t S x f(x)+ (t2/2!) S x2 f(x)+ (t3/3!) S x3 f(x)+ ... + (tn/n!) S xn f(x) + ...
Results:
1. M(0) = 1.
2. M'(0)=m.
3. M''(0)=s2 + m2.
4. In general, the kth moment about the origin for a given distribution equals M(k)(0).
Results: Moment generating functions for several discrete distributions:
· Geometric: M(t) = p et / [1 - (1-p)et ]
· Binomial: M(t) = [ (1-p) + p et ]n
· Negative Binomial: M(t) = (pt)r / [1 - (1-p)t ]r
· Poisson: M(t) = exp(l (et - 1) ), which implies h(t) = M( ln(t) ) = exp(l(t- 1) )
Result: If two random variables have the same moment generating function, then they have the same distribution.
Results: Moment generating functions for several continuous distributions:
· Exponential: M(t) = 1 / (1 – t/l)
· Normal: M(t) = emt + sstt/2
Gamma-Type Distribution: Consider a Poisson process with mean l such that X measures the waiting time till the a-th change. If a=1, then we have the exponential distribution. If a>1, then as with the exponential distribution, the distribution function for X is given by (for x>0)
F(x) = P(X < x) = 1 - P(X > x) = 1 - P( fewer than a changes in the interval [0,x] )
= 1 - Sk=0...a-1 (lx)ke-lx / k!
To get f(a) for a continuous distribution, use the fact f(x) = F'(x) to get
f(x) = l(lx)a-1 e-lx / (a-1)!
Gamma Function: G(t) = ,/`[0,oo] yt-1 e-y dy
Result: G(a) = (a-1)G(a-1), for any a >1. So, G(a) behaves like a continuous analogue of the factorial function. Indeed, G(n+1) = n!.
Result: The p.d.f. for the gamma distribution can be written
f(x) = xa-1 e-lx {la/ G(a)}, with
M(t) = 1 / (1 - t/l)a, and with
m = a/l and s2 = a / l2.
Remark: The integral of the gamma-type f(y) can be performed exactly for a=1, a=2, ... easily using integration by parts and will yield a sum of certain Poisson random variables. However, when a is not an integer, a closed-form expression is not possible.
Chi-Square Distribution: Let r be a positive integer. A random variable Y is said to have a chi-square distribution c2(r) with r degrees of freedom if and only if Y is a gamma-type variable with parameters a = r/2 and l = 1/2.
Result: c2(r) has the following properties:
·
M(t) = (1-2t)-r/2
·
m = r
· s2 = 2r
Remark: Probabilities of the form P(a < x < b) can be found for a variety of values using Table IV in the back of the book. Notice, for a given probability, several possible intervals [a,b] are possible. Also, several probabilities of the form P(x > c) = a are listed. The value of c, for a given a, is denoted c2(a).
Normal Distribution: Discuss p.d.f. and how to compute probabilities by converting to normal.
Results:
Theorem 4.4.1: If X is N(m,s2), then Z=(X-m)/s is N(0,1).
Pf:
P(Z < z) = P(X < zs +m) =
Theorem 4.4.2: If X is N(m,s2), then V = Z2 = (X-m)2/s2 is c2(1).
Pf: Since v>0, G(v) = P(V < v) = P(Z2 < Öv ) = P(-Öv < Z < Öv )
= 2 ,/`[0, Ö(v)] exp(-z2/2) / Ö(2p) dz, and by letting z = Öy or z2 = y, we get
2 ,/`[0,v] exp(-y/2) / Ö(2 y p) dy.
So, by the FTOC, we get for 0 < v < oo
g(v) = G'(v) = exp(-v/2) / Ö(2 v p) = v1/2 - 1 exp(-v/2) / { Ö2 Öp) }.
Since G(v) is a distribution function, then g(v) must be a p.d.f. and so
,/`[0,oo] v1/2 - 1 exp(-v/2) / { Ö2 Ö p } = 1
Letting x = v/2 yields
1 = ,/`[0,oo] x1/2 - 1 exp(-x) / Öp = G(1/2) / Öp
Hence, G(1/2) = Öp. This gives
g(v) = v1/2 - 1 exp(-v/2) / { Ö2 G(1/2)}
which is the p.d.f. for c2(1).
Tchebysheff's Theorem: Let Y be a random variable with finite mean m and variance s2. Then, for any positive constant k,
P( |Y-m| < ks ) > 1-1/k2 , and
P( |Y-m| > ks ) < 1/k2
Proof:
For Y a continuous random variable, write s2 as three integrals with the same integrand but over
the intervals (-¥
, m-ks), (m-ks, m+ks) and (m+ks, ¥
). Replace the middle integral by a lower bound of zero and on the others, note
(y-m) > k2s2.
Remark: This result generally gives a rough set of bounds on the given probabilities. The most important consequence of these is that they hold for any kind of distribution, regardless of shape, for which the mean and variance are known.