Decomposing estimation error into bias and varianceVariance of sum of square differences corrupted by noiseVariance of a Population of Two Indpendent Random VariablesProve that $int k(w)o(h^2w^2)dw=o(h^2)$ for $int k(w)dw=1$statistics of aggregated random variableError in estimating additive factor based on samples?How to compute distribution conditional on two random variablesBias-variance decompositionEstimating the Bias of a Coin using Chebyshev's inequalityError of Linear vs. Non-Linear Least Mean SquareSimultaneous estimation of counting and delay distribution with observations reported during limited time period.
Go Pregnant or Go Home
Can criminal fraud exist without damages?
How do I keep an essay about "feeling flat" from feeling flat?
Coordinate position not precise
Is exact Kanji stroke length important?
Is there any reason not to eat food that's been dropped on the surface of the moon?
Modify casing of marked letters
is this a spam?
What's the purpose of "true" in bash "if sudo true; then"
Hostile work environment after whistle-blowing on coworker and our boss. What do I do?
How can I use the arrow sign in my bash prompt?
How do I define a right arrow with bar in LaTeX?
Do I need a multiple entry visa for a trip UK -> Sweden -> UK?
What would happen if the UK refused to take part in EU Parliamentary elections?
Why are on-board computers allowed to change controls without notifying the pilots?
Bash method for viewing beginning and end of file
Can a monster with multiattack use this ability if they are missing a limb?
What't the meaning of this extra silence?
The plural of 'stomach"
What would be the benefits of having both a state and local currencies?
I'm in charge of equipment buying but no one's ever happy with what I choose. How to fix this?
Why did Kant, Hegel, and Adorno leave some words and phrases in the Greek alphabet?
How does it work when somebody invests in my business?
Generic lambda vs generic function give different behaviour
Decomposing estimation error into bias and variance
Variance of sum of square differences corrupted by noiseVariance of a Population of Two Indpendent Random VariablesProve that $int k(w)o(h^2w^2)dw=o(h^2)$ for $int k(w)dw=1$statistics of aggregated random variableError in estimating additive factor based on samples?How to compute distribution conditional on two random variablesBias-variance decompositionEstimating the Bias of a Coin using Chebyshev's inequalityError of Linear vs. Non-Linear Least Mean SquareSimultaneous estimation of counting and delay distribution with observations reported during limited time period.
$begingroup$
Let $Bbb X$ be an outcome space and let $P_theta : theta in Theta$ be a class of probability distributions on $Bbb X$. Let $X$ be an observed outcome generated by $P_theta$ for some $theta$ that is unknown to us. We want to estimate $g(theta)$ for a function $g$ on $Theta$ using some decision rule $delta$. Show that for any decision rule $delta$, the estimation error $Bbb E_theta[(g(theta) - delta(X))^2] = (textBias_theta(delta))^2 + textVar_theta(delta)$ where $$textBias_theta(delta) equiv g(theta) - Bbb E_theta[delta(X)] text and textVar_theta(delta) equiv Bbb E_theta[(Bbb E_theta[delta(X)] - delta(X))^2]$$
I first set them equal and simplify to get:
$$Bbb E_theta[g(theta)^2] - 2 Bbb E_theta[g(theta)delta(X)] + Bbb E_theta[delta(X)^2] $$ $$=$$ $$g(theta)^2 - 2g(theta) Bbb E_theta[delta(X)] + Bbb E_theta[delta(X)]^2 + Bbb E_theta[delta(X)]^2 - 2 Bbb E_theta[delta(X)]^2 + Bbb E_theta[delta(X)^2]$$
Then I reduce again to get:
$$Bbb E_theta[g(theta)^2] - 2 Bbb E_theta[g(theta)delta(X)] = g(theta)^2 - 2g(theta) Bbb E_theta[delta(X)]$$
But from here I can't see how to proceed.
Does anyone have any ideas?
probability machine-learning
asked Mar 17 at 13:41
Oliver GOliver G
1,3651632
$endgroup$
add a comment |
$begingroup$
Let $Bbb X$ be an outcome space and let $P_theta : theta in Theta$ be a class of probability distributions on $Bbb X$. Let $X$ be an observed outcome generated by $P_theta$ for some $theta$ that is unknown to us. We want to estimate $g(theta)$ for a function $g$ on $Theta$ using some decision rule $delta$. Show that for any decision rule $delta$, the estimation error $Bbb E_theta[(g(theta) - delta(X))^2] = (textBias_theta(delta))^2 + textVar_theta(delta)$ where $$textBias_theta(delta) equiv g(theta) - Bbb E_theta[delta(X)] text and textVar_theta(delta) equiv Bbb E_theta[(Bbb E_theta[delta(X)] - delta(X))^2]$$
I first set them equal and simplify to get:
$$Bbb E_theta[g(theta)^2] - 2 Bbb E_theta[g(theta)delta(X)] + Bbb E_theta[delta(X)^2] $$ $$=$$ $$g(theta)^2 - 2g(theta) Bbb E_theta[delta(X)] + Bbb E_theta[delta(X)]^2 + Bbb E_theta[delta(X)]^2 - 2 Bbb E_theta[delta(X)]^2 + Bbb E_theta[delta(X)^2]$$
Then I reduce again to get:
$$Bbb E_theta[g(theta)^2] - 2 Bbb E_theta[g(theta)delta(X)] = g(theta)^2 - 2g(theta) Bbb E_theta[delta(X)]$$
But from here I can't see how to proceed.
Does anyone have any ideas?
probability machine-learning
asked Mar 17 at 13:41
Oliver GOliver G
1,3651632
$endgroup$
$begingroup$
Are you trying to show that last equation is true? If so, note that $g(theta)$ is a constant.
$endgroup$
– Minus One-Twelfth
Mar 17 at 13:44
$begingroup$
$newcommandEmathbbE_theta$Basically the way to show this result is to 1) note that $Eleft[(Y+Z)^2right] = Eleft[Y^2right] + Eleft[Z^2right]$ if $E[YZ] = 0$, and 2) use this result with $Y = g(theta) - Eleft[delta(X)right]$ and $Z = E[delta(X)] - delta(X)$, showing that indeed $E[YZ] = 0$ here (use that $Y$ is a constant here, so you have $E[YZ] = YE[Z]$ and just have to show $E[Z] = 0$).
$endgroup$
– Minus One-Twelfth
Mar 17 at 13:51
$begingroup$
Why is $g(theta)$ a constant? It looks like it's a function of $theta$ and I'm taking an expectation $g(theta)$ with respect to $theta$ so wouldn't that not be constant?
$endgroup$
– Oliver G
Mar 17 at 14:24
$begingroup$
I think $theta$ is an (unknown) constant and $mathbbE_theta$ means taking expectation with respect to $P_theta$.
$endgroup$
– Minus One-Twelfth
Mar 17 at 20:02
$begingroup$
So if we assume $g(theta)$ is constant, then is my way of proof correct or do I need to show the way you presented?
$endgroup$
– Oliver G
Mar 17 at 23:11
add a comment |
$begingroup$
Let $Bbb X$ be an outcome space and let $P_theta : theta in Theta$ be a class of probability distributions on $Bbb X$. Let $X$ be an observed outcome generated by $P_theta$ for some $theta$ that is unknown to us. We want to estimate $g(theta)$ for a function $g$ on $Theta$ using some decision rule $delta$. Show that for any decision rule $delta$, the estimation error $Bbb E_theta[(g(theta) - delta(X))^2] = (textBias_theta(delta))^2 + textVar_theta(delta)$ where $$textBias_theta(delta) equiv g(theta) - Bbb E_theta[delta(X)] text and textVar_theta(delta) equiv Bbb E_theta[(Bbb E_theta[delta(X)] - delta(X))^2]$$
I first set them equal and simplify to get:
$$Bbb E_theta[g(theta)^2] - 2 Bbb E_theta[g(theta)delta(X)] + Bbb E_theta[delta(X)^2] $$ $$=$$ $$g(theta)^2 - 2g(theta) Bbb E_theta[delta(X)] + Bbb E_theta[delta(X)]^2 + Bbb E_theta[delta(X)]^2 - 2 Bbb E_theta[delta(X)]^2 + Bbb E_theta[delta(X)^2]$$
Then I reduce again to get:
$$Bbb E_theta[g(theta)^2] - 2 Bbb E_theta[g(theta)delta(X)] = g(theta)^2 - 2g(theta) Bbb E_theta[delta(X)]$$
But from here I can't see how to proceed.
Does anyone have any ideas?
probability machine-learning
asked Mar 17 at 13:41
Oliver GOliver G
1,3651632
$endgroup$
Let $Bbb X$ be an outcome space and let $P_theta : theta in Theta$ be a class of probability distributions on $Bbb X$. Let $X$ be an observed outcome generated by $P_theta$ for some $theta$ that is unknown to us. We want to estimate $g(theta)$ for a function $g$ on $Theta$ using some decision rule $delta$. Show that for any decision rule $delta$, the estimation error $Bbb E_theta[(g(theta) - delta(X))^2] = (textBias_theta(delta))^2 + textVar_theta(delta)$ where $$textBias_theta(delta) equiv g(theta) - Bbb E_theta[delta(X)] text and textVar_theta(delta) equiv Bbb E_theta[(Bbb E_theta[delta(X)] - delta(X))^2]$$
I first set them equal and simplify to get:
$$Bbb E_theta[g(theta)^2] - 2 Bbb E_theta[g(theta)delta(X)] + Bbb E_theta[delta(X)^2] $$ $$=$$ $$g(theta)^2 - 2g(theta) Bbb E_theta[delta(X)] + Bbb E_theta[delta(X)]^2 + Bbb E_theta[delta(X)]^2 - 2 Bbb E_theta[delta(X)]^2 + Bbb E_theta[delta(X)^2]$$
Then I reduce again to get:
$$Bbb E_theta[g(theta)^2] - 2 Bbb E_theta[g(theta)delta(X)] = g(theta)^2 - 2g(theta) Bbb E_theta[delta(X)]$$
But from here I can't see how to proceed.
Does anyone have any ideas?
probability machine-learning
probability machine-learning
asked Mar 17 at 13:41
Oliver GOliver G
1,3651632
asked Mar 17 at 13:41
Oliver GOliver G
1,3651632
asked Mar 17 at 13:41
Oliver GOliver G
1,3651632
asked Mar 17 at 13:41
Oliver GOliver G
1,3651632
asked Mar 17 at 13:41
Oliver GOliver G
1,3651632
1,3651632
$begingroup$
Are you trying to show that last equation is true? If so, note that $g(theta)$ is a constant.
$endgroup$
– Minus One-Twelfth
Mar 17 at 13:44
$begingroup$
$newcommandEmathbbE_theta$Basically the way to show this result is to 1) note that $Eleft[(Y+Z)^2right] = Eleft[Y^2right] + Eleft[Z^2right]$ if $E[YZ] = 0$, and 2) use this result with $Y = g(theta) - Eleft[delta(X)right]$ and $Z = E[delta(X)] - delta(X)$, showing that indeed $E[YZ] = 0$ here (use that $Y$ is a constant here, so you have $E[YZ] = YE[Z]$ and just have to show $E[Z] = 0$).
$endgroup$
– Minus One-Twelfth
Mar 17 at 13:51
$begingroup$
Why is $g(theta)$ a constant? It looks like it's a function of $theta$ and I'm taking an expectation $g(theta)$ with respect to $theta$ so wouldn't that not be constant?
$endgroup$
– Oliver G
Mar 17 at 14:24
$begingroup$
I think $theta$ is an (unknown) constant and $mathbbE_theta$ means taking expectation with respect to $P_theta$.
$endgroup$
– Minus One-Twelfth
Mar 17 at 20:02
$begingroup$
So if we assume $g(theta)$ is constant, then is my way of proof correct or do I need to show the way you presented?
$endgroup$
– Oliver G
Mar 17 at 23:11
add a comment |
$begingroup$
Are you trying to show that last equation is true? If so, note that $g(theta)$ is a constant.
$endgroup$
– Minus One-Twelfth
Mar 17 at 13:44
$begingroup$
$newcommandEmathbbE_theta$Basically the way to show this result is to 1) note that $Eleft[(Y+Z)^2right] = Eleft[Y^2right] + Eleft[Z^2right]$ if $E[YZ] = 0$, and 2) use this result with $Y = g(theta) - Eleft[delta(X)right]$ and $Z = E[delta(X)] - delta(X)$, showing that indeed $E[YZ] = 0$ here (use that $Y$ is a constant here, so you have $E[YZ] = YE[Z]$ and just have to show $E[Z] = 0$).
$endgroup$
– Minus One-Twelfth
Mar 17 at 13:51
$begingroup$
Why is $g(theta)$ a constant? It looks like it's a function of $theta$ and I'm taking an expectation $g(theta)$ with respect to $theta$ so wouldn't that not be constant?
$endgroup$
– Oliver G
Mar 17 at 14:24
$begingroup$
I think $theta$ is an (unknown) constant and $mathbbE_theta$ means taking expectation with respect to $P_theta$.
$endgroup$
– Minus One-Twelfth
Mar 17 at 20:02
$begingroup$
So if we assume $g(theta)$ is constant, then is my way of proof correct or do I need to show the way you presented?
$endgroup$
– Oliver G
Mar 17 at 23:11
$begingroup$
Are you trying to show that last equation is true? If so, note that $g(theta)$ is a constant.
$endgroup$
– Minus One-Twelfth
Mar 17 at 13:44
$begingroup$
Are you trying to show that last equation is true? If so, note that $g(theta)$ is a constant.
$endgroup$
– Minus One-Twelfth
Mar 17 at 13:44
$begingroup$
$newcommandEmathbbE_theta$Basically the way to show this result is to 1) note that $Eleft[(Y+Z)^2right] = Eleft[Y^2right] + Eleft[Z^2right]$ if $E[YZ] = 0$, and 2) use this result with $Y = g(theta) - Eleft[delta(X)right]$ and $Z = E[delta(X)] - delta(X)$, showing that indeed $E[YZ] = 0$ here (use that $Y$ is a constant here, so you have $E[YZ] = YE[Z]$ and just have to show $E[Z] = 0$).
$endgroup$
– Minus One-Twelfth
Mar 17 at 13:51
$begingroup$
$newcommandEmathbbE_theta$Basically the way to show this result is to 1) note that $Eleft[(Y+Z)^2right] = Eleft[Y^2right] + Eleft[Z^2right]$ if $E[YZ] = 0$, and 2) use this result with $Y = g(theta) - Eleft[delta(X)right]$ and $Z = E[delta(X)] - delta(X)$, showing that indeed $E[YZ] = 0$ here (use that $Y$ is a constant here, so you have $E[YZ] = YE[Z]$ and just have to show $E[Z] = 0$).
$endgroup$
– Minus One-Twelfth
Mar 17 at 13:51
$begingroup$
Why is $g(theta)$ a constant? It looks like it's a function of $theta$ and I'm taking an expectation $g(theta)$ with respect to $theta$ so wouldn't that not be constant?
$endgroup$
– Oliver G
Mar 17 at 14:24
$begingroup$
Why is $g(theta)$ a constant? It looks like it's a function of $theta$ and I'm taking an expectation $g(theta)$ with respect to $theta$ so wouldn't that not be constant?
$endgroup$
– Oliver G
Mar 17 at 14:24
$begingroup$
I think $theta$ is an (unknown) constant and $mathbbE_theta$ means taking expectation with respect to $P_theta$.
$endgroup$
– Minus One-Twelfth
Mar 17 at 20:02
$begingroup$
I think $theta$ is an (unknown) constant and $mathbbE_theta$ means taking expectation with respect to $P_theta$.
$endgroup$
– Minus One-Twelfth
Mar 17 at 20:02
$begingroup$
So if we assume $g(theta)$ is constant, then is my way of proof correct or do I need to show the way you presented?
$endgroup$
– Oliver G
Mar 17 at 23:11
$begingroup$
So if we assume $g(theta)$ is constant, then is my way of proof correct or do I need to show the way you presented?
$endgroup$
– Oliver G
Mar 17 at 23:11
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "69"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3151555%2fdecomposing-estimation-error-into-bias-and-variance%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3151555%2fdecomposing-estimation-error-into-bias-and-variance%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
Are you trying to show that last equation is true? If so, note that $g(theta)$ is a constant.
$endgroup$
– Minus One-Twelfth
Mar 17 at 13:44
$begingroup$
$newcommandEmathbbE_theta$Basically the way to show this result is to 1) note that $Eleft[(Y+Z)^2right] = Eleft[Y^2right] + Eleft[Z^2right]$ if $E[YZ] = 0$, and 2) use this result with $Y = g(theta) - Eleft[delta(X)right]$ and $Z = E[delta(X)] - delta(X)$, showing that indeed $E[YZ] = 0$ here (use that $Y$ is a constant here, so you have $E[YZ] = YE[Z]$ and just have to show $E[Z] = 0$).
$endgroup$
– Minus One-Twelfth
Mar 17 at 13:51
$begingroup$
Why is $g(theta)$ a constant? It looks like it's a function of $theta$ and I'm taking an expectation $g(theta)$ with respect to $theta$ so wouldn't that not be constant?
$endgroup$
– Oliver G
Mar 17 at 14:24
$begingroup$
I think $theta$ is an (unknown) constant and $mathbbE_theta$ means taking expectation with respect to $P_theta$.
$endgroup$
– Minus One-Twelfth
Mar 17 at 20:02
$begingroup$
So if we assume $g(theta)$ is constant, then is my way of proof correct or do I need to show the way you presented?
$endgroup$
– Oliver G
Mar 17 at 23:11