Fdtxjr

Student evaluations of teaching assistants

Displaying the order of the columns of a table

Increase performance creating Mandelbrot set in python

Is HostGator storing my password in plaintext?

How do I rename a LINUX host without needing to reboot for the rename to take effect?

Will it be accepted, if there is no ''Main Character" stereotype?

There is only s̶i̶x̶t̶y one place he can be

What's the purpose of "true" in bash "if sudo true; then"

Is expanding the research of a group into machine learning as a PhD student risky?

What are the ramifications of creating a homebrew world without an Astral Plane?

What is difference between behavior and behaviour

Everything Bob says is false. How does he get people to trust him?

Why are on-board computers allowed to change controls without notifying the pilots?

Is a roofing delivery truck likely to crack my driveway slab?

How will losing mobility of one hand affect my career as a programmer?

Is there any reason not to eat food that's been dropped on the surface of the moon?

Trouble understanding overseas colleagues

How was Earth single-handedly capable of creating 3 of the 4 gods of chaos?

Hide Select Output from T-SQL

Why does John Bercow say “unlock” after reading out the results of a vote?

Why Were Madagascar and New Zealand Discovered So Late?

How can a jailer prevent the Forge Cleric's Artisan's Blessing from being used?

Is it correct to write "is not focus on"?

Have I saved too much for retirement so far?

Confused about Nesterov momentum gradient descent algorithm

Gradient descent with constraintsA constrained gradient descent algorithmIntuition for gradient descent with Nesterov momentumProjected gradient descent with momentumgradient descent algorithm definitionGradient Descent DivergenceWhy is gradient descent used?Momentum in gradient descentReason for differences in EWA equations and Momentum equationsGetting to the gradient descent algorithm

0

$begingroup$

I've found a variety of variations of writing Nesterov but I cannot understand why they cannot simply be expanded into a one liner.

Here is one I found that can just be re-arranged, can someone explain why I am wrong?

$theta_t = y_t - gamma nabla f(y_t) \
y_t+1 = theta_t + rho (theta_t - theta_t-1)$

Plug first equation into second,

$y_t+1 = y_t - gamma nabla f(y_t) + rho (theta_t - theta_t-1)$

Let $Delta y_t = y_t+1 - y_t$ then it simply becomes

$$Delta y_t = - gamma nabla f(y_t) + rho (y_t - gamma nabla f(y_t) - y_t-1 + gamma nabla f(y_t-1) \
= - gamma nabla f(y_t) + rho (Delta y_t-1 + gamma (nabla f(y_t-1) - nabla f(y_t)) $$

so what am I doing wrong?

In fact I've found a similar form in a paper: Ning Qian. On the momentum term in gradient descentlearning algorithms.Neural Networks, 12(1):145 – 151,1999.

where gradient descent with momentum is defined as

$$Delta theta_t = - gamma nabla f(theta) + rho Delta theta_t-1 $$ (I'm also not sure why it's $f(theta)$ and not $f(theta_t)$)

optimization numerical-optimization gradient-descent

share|cite|improve this question

edited Mar 17 at 12:49

Alexis Drakopoulos

asked Mar 17 at 12:44

Alexis DrakopoulosAlexis Drakopoulos

1013

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Why do you think it’s wrong to write it in one line?
  $endgroup$
  – David M.
  Mar 17 at 16:24
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  What about the update for the momentum term in the one-liner?
  $endgroup$
  – user3658307
  2 days ago
  

  
  
  
  

add a comment | 

0

$begingroup$

I've found a variety of variations of writing Nesterov but I cannot understand why they cannot simply be expanded into a one liner.

Here is one I found that can just be re-arranged, can someone explain why I am wrong?

$theta_t = y_t - gamma nabla f(y_t) \
y_t+1 = theta_t + rho (theta_t - theta_t-1)$

Plug first equation into second,

$y_t+1 = y_t - gamma nabla f(y_t) + rho (theta_t - theta_t-1)$

Let $Delta y_t = y_t+1 - y_t$ then it simply becomes

$$Delta y_t = - gamma nabla f(y_t) + rho (y_t - gamma nabla f(y_t) - y_t-1 + gamma nabla f(y_t-1) \
= - gamma nabla f(y_t) + rho (Delta y_t-1 + gamma (nabla f(y_t-1) - nabla f(y_t)) $$

so what am I doing wrong?

In fact I've found a similar form in a paper: Ning Qian. On the momentum term in gradient descentlearning algorithms.Neural Networks, 12(1):145 – 151,1999.

where gradient descent with momentum is defined as

$$Delta theta_t = - gamma nabla f(theta) + rho Delta theta_t-1 $$ (I'm also not sure why it's $f(theta)$ and not $f(theta_t)$)

optimization numerical-optimization gradient-descent

share|cite|improve this question

edited Mar 17 at 12:49

Alexis Drakopoulos

asked Mar 17 at 12:44

Alexis DrakopoulosAlexis Drakopoulos

1013

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Why do you think it’s wrong to write it in one line?
  $endgroup$
  – David M.
  Mar 17 at 16:24
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  What about the update for the momentum term in the one-liner?
  $endgroup$
  – user3658307
  2 days ago
  

  
  
  
  

add a comment | 

$begingroup$

I've found a variety of variations of writing Nesterov but I cannot understand why they cannot simply be expanded into a one liner.

Here is one I found that can just be re-arranged, can someone explain why I am wrong?

$theta_t = y_t - gamma nabla f(y_t) \
y_t+1 = theta_t + rho (theta_t - theta_t-1)$

Plug first equation into second,

$y_t+1 = y_t - gamma nabla f(y_t) + rho (theta_t - theta_t-1)$

Let $Delta y_t = y_t+1 - y_t$ then it simply becomes

$$Delta y_t = - gamma nabla f(y_t) + rho (y_t - gamma nabla f(y_t) - y_t-1 + gamma nabla f(y_t-1) \
= - gamma nabla f(y_t) + rho (Delta y_t-1 + gamma (nabla f(y_t-1) - nabla f(y_t)) $$

so what am I doing wrong?

In fact I've found a similar form in a paper: Ning Qian. On the momentum term in gradient descentlearning algorithms.Neural Networks, 12(1):145 – 151,1999.

where gradient descent with momentum is defined as

$$Delta theta_t = - gamma nabla f(theta) + rho Delta theta_t-1 $$ (I'm also not sure why it's $f(theta)$ and not $f(theta_t)$)

optimization numerical-optimization gradient-descent

share|cite|improve this question

edited Mar 17 at 12:49

Alexis Drakopoulos

asked Mar 17 at 12:44

Alexis DrakopoulosAlexis Drakopoulos

1013

$endgroup$

share|cite|improve this question

edited Mar 17 at 12:49

Alexis Drakopoulos

asked Mar 17 at 12:44

Alexis DrakopoulosAlexis Drakopoulos

1013

share|cite|improve this question

edited Mar 17 at 12:49

Alexis Drakopoulos

asked Mar 17 at 12:44

Alexis DrakopoulosAlexis Drakopoulos

1013

asked Mar 17 at 12:44

Alexis DrakopoulosAlexis Drakopoulos

1013

asked Mar 17 at 12:44

Alexis DrakopoulosAlexis Drakopoulos

1013