Fdtxjr

The following signatures were invalid: EXPKEYSIG 1397BC53640DB551

Why not take a picture of a closer black hole?

Keeping a retro style to sci-fi spaceships?

Do working physicists consider Newtonian mechanics to be "falsified"?

Didn't get enough time to take a Coding Test - what to do now?

Why are PDP-7-style microprogrammed instructions out of vogue?

Is 'stolen' appropriate word?

Do I have Disadvantage attacking with an off-hand weapon?

Loose spokes after only a few rides

Homework question about an engine pulling a train

"is" operation returns false even though two objects have same id

What do I do when my TA workload is more than expected?

Button changing its text & action. Good or terrible?

Why doesn't a hydraulic lever violate conservation of energy?

How did the audience guess the pentatonic scale in Bobby McFerrin's presentation?

Why can't wing-mounted spoilers be used to steepen approaches?

"... to apply for a visa" or "... and applied for a visa"?

Was credit for the black hole image misappropriated?

Huge performance difference of the command find with and without using %M option to show permissions

University's motivation for having tenure-track positions

Student Loan from years ago pops up and is taking my salary

Sub-subscripts in strings cause different spacings than subscripts

Python - Fishing Simulator

Why did Peik Lin say, "I'm not an animal"?

Directional Derivatives and Jacobian of a Linear Neural Network

1

$begingroup$

I have to compute the following double derivative:

$$ partial _x_i nabla_W sigma(f(W,x))$$

where $W = (W_1, W_2, dots, W_L)$ is the set of weight matrices, $f(W,x)$ is a $linear$ neural network, hence $f(W,x) = xW_1W_2 dots W_L$, the value $x_i$ is the $i$-th entry of the vector $x$ and $sigma$ is the softmax activation function,

$$sigma(x)_i = frac e^x_isum_j e^x_j.$$

I know that $fracpartial sigmapartial x$ is a matrix $J_i,j(x) = sigma(x)_i(delta_i,j - sigma_j(x))$, hence

$$nabla_W sigma(f(W,x)) = J(f(W,x))cdot x $$

First of all, is it right?

So, now, how can I compute

$$partial_x_i J(f(W,x))cdot x ?$$

Let's assume $z = f(W,x)$ for an easy notation, my guess would be to compute the matrix $A = partial_x_i J(z)$ s.t.

$$A_i,j = sigma(z)_i(1-sigma(z)_i (1-2sigma(z)_i) text if i=j $$

$$A_i,j = sigma(z)_i sigma(z)_j(sigma(z)_i + sigma(z)_j textif ineq j $$

where $A$ is the second derivative of the softmax function.

Hence, $$partial_x_iJ(z)cdot x = J(z)cdot x^(i) + Acdot partial_x_iz cdot x$$

I don't know wether it's right or not, and second I don't know how to keep going. Does someone have suggestions?

Thank you very much!!

derivatives neural-networks

share|cite|improve this question

edited Mar 26 at 21:44

Alfred

asked Mar 23 at 15:17

AlfredAlfred

418

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  A linear neural network doesn't make much sense as all the layers will become equivalent to a single layer ( with $left( prod_l=1^L W_l right)$ as the equivalent weight). Mostly ReLU is employed to introduce non linearity to every layer.
  $endgroup$
  – Balakrishnan Rajan
  Mar 24 at 19:59
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Yes, I know this. My question is part of a bigger problem I'm facing, but I'm having troubles with some computations like this one.
  $endgroup$
  – Alfred
  Mar 24 at 21:19
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Umm. What is $deltai,j$? Why are you performing $fracpartial sigmapartial x$? $Delta_W sigma(f(W,x))$ is akin to differentiating (partial ?) $sigma(f(W,x))$ with respect to $W$. I lost you there. Also if you are trying to understand back propogation, then bear in mind that there is a separate loss function $L(y, haty)$ which is actually minimized w.r.t weights not the activation.
  $endgroup$
  – Balakrishnan Rajan
  Mar 27 at 8:07
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Perhaps you can add more info about the dimensionality of your tensors.
  $endgroup$
  – Balakrishnan Rajan
  Mar 27 at 8:08
  

  
  
  
  

add a comment | 

1

$begingroup$

I have to compute the following double derivative:

$$ partial _x_i nabla_W sigma(f(W,x))$$

where $W = (W_1, W_2, dots, W_L)$ is the set of weight matrices, $f(W,x)$ is a $linear$ neural network, hence $f(W,x) = xW_1W_2 dots W_L$, the value $x_i$ is the $i$-th entry of the vector $x$ and $sigma$ is the softmax activation function,

$$sigma(x)_i = frac e^x_isum_j e^x_j.$$

I know that $fracpartial sigmapartial x$ is a matrix $J_i,j(x) = sigma(x)_i(delta_i,j - sigma_j(x))$, hence

$$nabla_W sigma(f(W,x)) = J(f(W,x))cdot x $$

First of all, is it right?

So, now, how can I compute

$$partial_x_i J(f(W,x))cdot x ?$$

Let's assume $z = f(W,x)$ for an easy notation, my guess would be to compute the matrix $A = partial_x_i J(z)$ s.t.

$$A_i,j = sigma(z)_i(1-sigma(z)_i (1-2sigma(z)_i) text if i=j $$

$$A_i,j = sigma(z)_i sigma(z)_j(sigma(z)_i + sigma(z)_j textif ineq j $$

where $A$ is the second derivative of the softmax function.

Hence, $$partial_x_iJ(z)cdot x = J(z)cdot x^(i) + Acdot partial_x_iz cdot x$$

I don't know wether it's right or not, and second I don't know how to keep going. Does someone have suggestions?

Thank you very much!!

derivatives neural-networks

share|cite|improve this question

edited Mar 26 at 21:44

Alfred

asked Mar 23 at 15:17

AlfredAlfred

418

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  A linear neural network doesn't make much sense as all the layers will become equivalent to a single layer ( with $left( prod_l=1^L W_l right)$ as the equivalent weight). Mostly ReLU is employed to introduce non linearity to every layer.
  $endgroup$
  – Balakrishnan Rajan
  Mar 24 at 19:59
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Yes, I know this. My question is part of a bigger problem I'm facing, but I'm having troubles with some computations like this one.
  $endgroup$
  – Alfred
  Mar 24 at 21:19
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Umm. What is $deltai,j$? Why are you performing $fracpartial sigmapartial x$? $Delta_W sigma(f(W,x))$ is akin to differentiating (partial ?) $sigma(f(W,x))$ with respect to $W$. I lost you there. Also if you are trying to understand back propogation, then bear in mind that there is a separate loss function $L(y, haty)$ which is actually minimized w.r.t weights not the activation.
  $endgroup$
  – Balakrishnan Rajan
  Mar 27 at 8:07
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Perhaps you can add more info about the dimensionality of your tensors.
  $endgroup$
  – Balakrishnan Rajan
  Mar 27 at 8:08
  

  
  
  
  

add a comment | 

$begingroup$

I have to compute the following double derivative:

$$ partial _x_i nabla_W sigma(f(W,x))$$

where $W = (W_1, W_2, dots, W_L)$ is the set of weight matrices, $f(W,x)$ is a $linear$ neural network, hence $f(W,x) = xW_1W_2 dots W_L$, the value $x_i$ is the $i$-th entry of the vector $x$ and $sigma$ is the softmax activation function,

$$sigma(x)_i = frac e^x_isum_j e^x_j.$$

I know that $fracpartial sigmapartial x$ is a matrix $J_i,j(x) = sigma(x)_i(delta_i,j - sigma_j(x))$, hence

$$nabla_W sigma(f(W,x)) = J(f(W,x))cdot x $$

First of all, is it right?

So, now, how can I compute

$$partial_x_i J(f(W,x))cdot x ?$$

Let's assume $z = f(W,x)$ for an easy notation, my guess would be to compute the matrix $A = partial_x_i J(z)$ s.t.

$$A_i,j = sigma(z)_i(1-sigma(z)_i (1-2sigma(z)_i) text if i=j $$

$$A_i,j = sigma(z)_i sigma(z)_j(sigma(z)_i + sigma(z)_j textif ineq j $$

where $A$ is the second derivative of the softmax function.

Hence, $$partial_x_iJ(z)cdot x = J(z)cdot x^(i) + Acdot partial_x_iz cdot x$$

I don't know wether it's right or not, and second I don't know how to keep going. Does someone have suggestions?

Thank you very much!!

derivatives neural-networks

share|cite|improve this question

edited Mar 26 at 21:44

Alfred

asked Mar 23 at 15:17

AlfredAlfred

418

$endgroup$

share|cite|improve this question

edited Mar 26 at 21:44

Alfred

asked Mar 23 at 15:17

AlfredAlfred

418

share|cite|improve this question

edited Mar 26 at 21:44

Alfred

asked Mar 23 at 15:17

AlfredAlfred

418

asked Mar 23 at 15:17

AlfredAlfred

418

asked Mar 23 at 15:17

AlfredAlfred

418