Skip to content

softmax.md

Latest commit

 

History

History
130 lines (82 loc) · 3.8 KB

softmax.md

File metadata and controls

130 lines (82 loc) · 3.8 KB
Raw

Softmax

is called the softmax function.

Practical issues: Numeric stability

When you’re writing code for computing the Softmax function in practice, the intermediate terms ef may be very large due to the exponentials.

def softmax( f ):
    # Bad: Numeric problem, potential blowup
    return np.exp(f) / np.sum(np.exp(f))
    
>>>
>>> softmax( np.array([123, 456, 789])  )
__main__:3: RuntimeWarning: overflow encountered in exp
__main__:3: RuntimeWarning: invalid value encountered in divide

Notice that if we multiply the top and bottom of the fraction by a constant C and push it into the sum, we get the following (mathematically equivalent) expression:

We are free to choose the value of C. This will not change any of the results. That is , only the relative difference of one action over another is important.

>>> softmax( np.array([12,25,44]) )
array([  1.26641655e-14,   5.60279641e-09,   9.99999994e-01])
>>> # add 100 to all entries , it keeps the same result
>>> softmax(  np.array([12+100,25+100,44+100]) )
array([  1.26641655e-14,   5.60279641e-09,   9.99999994e-01])

A common choice for C is to set logC=−max(f). This simply states that we should shift the values inside the vector f so that the highest value is zero.

def softmax( f ):
    # instead: first shift the values of f so that the highest number is 0:
    f -= np.max(f) # f becomes [-666, -333, 0]
    # you may need set `axis=x`  for non 1-D case
    return np.exp(f) / np.sum(np.exp(f) )  # safe to do, gives the correct answer

>>> softmax( np.array([12,25,44]) )
array([  1.26641655e-14,   5.60279641e-09,   9.99999994e-01])
# Vectorized version
# f: normally a 2D array, X*W ( NxC )
#   N: number of training example
#   C: number of classes
# return NxC , element is softmax value
def softmax( f, aggregate_axis = 0 ):
    # instead: first shift the values of f so that the highest number is 0:
    f_max = np.max(f, axis= aggregate_axis )
    f -= f_max.reshape( f_max.shape + (1,) )
    f_exp = np.exp(f)
    f_sum = np.sum( f_exp, axis=aggregate_axis )
    return f_exp / f_sum.reshape( f_sum.shape + (1,) )

cs231n softmax notes

Softmax & Sigmoid

Sigmoid function:

def sigmoid(X):
   return 1/(1+np.exp(-X))

What is the softmax distribution in case of 2 weights [a,b] ?

Suppose a≥b, Then:

Yes, the soft-max distribution is the same as that given by the sigmoid.

>>> softmax( np.array([14,12]) )
array([ 0.88079708,  0.11920292])
>>> sigmoid( 12-14 )
0.11920292202211755

Derivative

Softmax is fundamentally a vector function. It takes a vector as input and produces a vector as output; in other words, it has multiple inputs and multiple outputs.

Therefore, we cannot just ask for "the derivative of softmax"; We should instead specify:

  1. Which component (output element) of softmax we're seeking to find the derivative of.
  2. Since softmax has multiple inputs, with respect to which input element the partial derivative is computed.

Take the derivative in multiple direction in space is very complicated if you think about it in the mathematical way.

求导&证明