Zrjydtj

How do I hide Chekhov's Gun?

Define, (actually define) the "stability" and "energy" of a compound

Be in awe of my brilliance!

How do I interpret this "sky cover" chart?

Will a pinhole camera work with instant film?

Rules about breaking the rules. How do I do it well?

Identifying the interval from A♭ to D♯

Science-fiction short story where space navy wanted hospital ships and settlers had guns mounted everywhere

How to write cleanly even if my character uses expletive language?

Where is the 1/8 CR apprentice in Volo's Guide to Monsters?

2D counterpart of std::array in C++17

Happy pi day, everyone!

Possible Leak In Concrete

Theorems like the Lovász Local Lemma?

It's a yearly task, alright

Why does Deadpool say "You're welcome, Canada," after shooting Ryan Reynolds in the end credits?

Good allowance savings plan?

When do we add an hyphen (-) to a complex adjective word?

What is the greatest age difference between a married couple in Tanach?

Co-worker team leader wants to inject his friend's awful software into our development. What should I say to our common boss?

Latest web browser compatible with Windows 98

Should we release the security issues we found in our product as CVE or we can just update those on weekly release notes?

Is having access to past exams cheating and, if yes, could it be proven just by a good grade?

How to simplify this time periods definition interface?

Distribution of Maximum Likelihood Estimator

Maximum likelihood of function of the mean on a restricted parameter spaceMaximum Likelihood Estimator (MLE)Maximum Likelihood Estimator of the exponential function parameter based on Order StatisticsFind maximum likelihood estimateMaximum Likelihood Estimation in case of some specific uniform distributionsMaximum likelihood estimate for a univariate gaussianFind the Maximum Likelihood Estimator given two pdfsMaximum Likelihood Estimate for a likelihood defined by partsVariance of distribution for maximum likelihood estimatorHow is $P(D;theta) = P(D|theta)$?

1

$begingroup$

Why is the Maximum Likelihood Estimator Normally distributed? I can't figure out why it is true for large n in general. My attempt (for single parameter)

Let $L(theta)$ be the maximum likelihood function for the distribution $f(x;theta)$

Then after taking sample of size n

$$L(theta)=f(x_1;theta)cdot f(x_2theta)...f(x_n;theta)$$

And we want to find $theta_max$ such that $L(theta)$ is maximized and $theta_max$ is our estimate (once a sample has actually been selected)

Since $theta_max$ maximizes $L(theta)$ it also maximizes $ln(L(theta))$

where

$$ln(L(theta))=ln(f(x_1;theta))+ln(f(x_2;theta))...+ln(f(x_n;theta))$$

Taking the derivative with respect to $theta$

$$fracf'(x_1;theta)f(x_1;theta)+fracf'(x_2;theta)f(x_2;theta)...+fracf'(x_n;theta)f(x_n;theta)$$

$theta_max$ would be the solution of the above when set to 0 (after selecting values for all $x_1,x_2...x_n$) but why is it normally distributed and how do I show that it's true for large n?

probability distributions normal-distribution estimation sampling

share|cite|improve this question

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

add a comment | 

1

$begingroup$

Why is the Maximum Likelihood Estimator Normally distributed? I can't figure out why it is true for large n in general. My attempt (for single parameter)

Let $L(theta)$ be the maximum likelihood function for the distribution $f(x;theta)$

Then after taking sample of size n

$$L(theta)=f(x_1;theta)cdot f(x_2theta)...f(x_n;theta)$$

And we want to find $theta_max$ such that $L(theta)$ is maximized and $theta_max$ is our estimate (once a sample has actually been selected)

Since $theta_max$ maximizes $L(theta)$ it also maximizes $ln(L(theta))$

where

$$ln(L(theta))=ln(f(x_1;theta))+ln(f(x_2;theta))...+ln(f(x_n;theta))$$

Taking the derivative with respect to $theta$

$$fracf'(x_1;theta)f(x_1;theta)+fracf'(x_2;theta)f(x_2;theta)...+fracf'(x_n;theta)f(x_n;theta)$$

$theta_max$ would be the solution of the above when set to 0 (after selecting values for all $x_1,x_2...x_n$) but why is it normally distributed and how do I show that it's true for large n?

probability distributions normal-distribution estimation sampling

share|cite|improve this question

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

add a comment | 

$begingroup$

Why is the Maximum Likelihood Estimator Normally distributed? I can't figure out why it is true for large n in general. My attempt (for single parameter)

Let $L(theta)$ be the maximum likelihood function for the distribution $f(x;theta)$

Then after taking sample of size n

$$L(theta)=f(x_1;theta)cdot f(x_2theta)...f(x_n;theta)$$

And we want to find $theta_max$ such that $L(theta)$ is maximized and $theta_max$ is our estimate (once a sample has actually been selected)

Since $theta_max$ maximizes $L(theta)$ it also maximizes $ln(L(theta))$

where

$$ln(L(theta))=ln(f(x_1;theta))+ln(f(x_2;theta))...+ln(f(x_n;theta))$$

Taking the derivative with respect to $theta$

$$fracf'(x_1;theta)f(x_1;theta)+fracf'(x_2;theta)f(x_2;theta)...+fracf'(x_n;theta)f(x_n;theta)$$

$theta_max$ would be the solution of the above when set to 0 (after selecting values for all $x_1,x_2...x_n$) but why is it normally distributed and how do I show that it's true for large n?

probability distributions normal-distribution estimation sampling

share|cite|improve this question

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

share|cite|improve this question

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

share|cite|improve this question

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

asked 3 hours ago

Colin HicksColin Hicks

1353

New contributor

Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

asked 3 hours ago

Colin HicksColin Hicks

1353

1 Answer
1

active

oldest

votes

3

$begingroup$

MLE requires $$fracpartial ln L(theta)partial theta = sum_i=1^n frac f'(x_i;theta)f(x_i;theta),$$
where $f'(x_i;theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;theta)=frac f'(x;theta)f(x;theta).$ Then $g(x_i;theta)_i=1^n$ is a new iid sequence of random variables, with $Eg(x_1;theta)=0$. If $Eg(x_1;theta)g(x_1;theta)'<infty$, CLT implies,
$$sqrtn(barg_n(theta)-Eg(x_1;theta))=sqrtnbarg_n(theta) rightarrow_D N(0,E(g(x;theta)g(x;theta)'),$$
where $barg_n(theta)=frac1n sum_i=1^n g(x_i;theta).$ The ML estimator solves the equation
$$barg_n(theta)=0.$$
It follows that the ML estimator is given by
$$hattheta=barg_n^-1(0).$$
So long as the set of discontinuity points of $barg_n^-1(z)$, i.e. the set of all values of $z$ such that $barg_n^-1(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $theta$.

share|cite|improve this answer

edited 2 hours ago

answered 3 hours ago

dlnBdlnB

3916

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  $fracf'(x)f(x)$ is just another function of x so central limit theorem applies thank you for that
  $endgroup$
  – Colin Hicks
  3 hours ago
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  You're welcome :)
  $endgroup$
  – dlnB
  2 hours ago
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "65"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

Colin Hicks is a new contributor. Be nice, and check out our Code of Conduct.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f397619%2fdistribution-of-maximum-likelihood-estimator%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

3

$begingroup$

MLE requires $$fracpartial ln L(theta)partial theta = sum_i=1^n frac f'(x_i;theta)f(x_i;theta),$$
where $f'(x_i;theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;theta)=frac f'(x;theta)f(x;theta).$ Then $g(x_i;theta)_i=1^n$ is a new iid sequence of random variables, with $Eg(x_1;theta)=0$. If $Eg(x_1;theta)g(x_1;theta)'<infty$, CLT implies,
$$sqrtn(barg_n(theta)-Eg(x_1;theta))=sqrtnbarg_n(theta) rightarrow_D N(0,E(g(x;theta)g(x;theta)'),$$
where $barg_n(theta)=frac1n sum_i=1^n g(x_i;theta).$ The ML estimator solves the equation
$$barg_n(theta)=0.$$
It follows that the ML estimator is given by
$$hattheta=barg_n^-1(0).$$
So long as the set of discontinuity points of $barg_n^-1(z)$, i.e. the set of all values of $z$ such that $barg_n^-1(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $theta$.

share|cite|improve this answer

edited 2 hours ago

answered 3 hours ago

dlnBdlnB

3916

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  $fracf'(x)f(x)$ is just another function of x so central limit theorem applies thank you for that
  $endgroup$
  – Colin Hicks
  3 hours ago
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  You're welcome :)
  $endgroup$
  – dlnB
  2 hours ago
  

  
  
  
  

add a comment | 

3

$begingroup$

MLE requires $$fracpartial ln L(theta)partial theta = sum_i=1^n frac f'(x_i;theta)f(x_i;theta),$$
where $f'(x_i;theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;theta)=frac f'(x;theta)f(x;theta).$ Then $g(x_i;theta)_i=1^n$ is a new iid sequence of random variables, with $Eg(x_1;theta)=0$. If $Eg(x_1;theta)g(x_1;theta)'<infty$, CLT implies,
$$sqrtn(barg_n(theta)-Eg(x_1;theta))=sqrtnbarg_n(theta) rightarrow_D N(0,E(g(x;theta)g(x;theta)'),$$
where $barg_n(theta)=frac1n sum_i=1^n g(x_i;theta).$ The ML estimator solves the equation
$$barg_n(theta)=0.$$
It follows that the ML estimator is given by
$$hattheta=barg_n^-1(0).$$
So long as the set of discontinuity points of $barg_n^-1(z)$, i.e. the set of all values of $z$ such that $barg_n^-1(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $theta$.

share|cite|improve this answer

edited 2 hours ago

answered 3 hours ago

dlnBdlnB

3916

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  $fracf'(x)f(x)$ is just another function of x so central limit theorem applies thank you for that
  $endgroup$
  – Colin Hicks
  3 hours ago
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  You're welcome :)
  $endgroup$
  – dlnB
  2 hours ago
  

  
  
  
  

add a comment | 

$begingroup$

MLE requires $$fracpartial ln L(theta)partial theta = sum_i=1^n frac f'(x_i;theta)f(x_i;theta),$$
where $f'(x_i;theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;theta)=frac f'(x;theta)f(x;theta).$ Then $g(x_i;theta)_i=1^n$ is a new iid sequence of random variables, with $Eg(x_1;theta)=0$. If $Eg(x_1;theta)g(x_1;theta)'<infty$, CLT implies,
$$sqrtn(barg_n(theta)-Eg(x_1;theta))=sqrtnbarg_n(theta) rightarrow_D N(0,E(g(x;theta)g(x;theta)'),$$
where $barg_n(theta)=frac1n sum_i=1^n g(x_i;theta).$ The ML estimator solves the equation
$$barg_n(theta)=0.$$
It follows that the ML estimator is given by
$$hattheta=barg_n^-1(0).$$
So long as the set of discontinuity points of $barg_n^-1(z)$, i.e. the set of all values of $z$ such that $barg_n^-1(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $theta$.

share|cite|improve this answer

edited 2 hours ago

answered 3 hours ago

dlnBdlnB

3916

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

share|cite|improve this answer

edited 2 hours ago

answered 3 hours ago

dlnBdlnB

3916

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

answered 3 hours ago

dlnBdlnB

3916

New contributor

dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

answered 3 hours ago

dlnBdlnB

3916