Skip to content

Express a softmax over a dynamic axis

Frank Seide edited this page Aug 27, 2016 · 8 revisions

A softmax over a dynamic axis can be done via a recurrence:

SoftMaxOverSequence (z) = [
    # define a recursive expression for \log(\sum_{i=1}^t \exp(z_i))
    runningLogSumExp = LogPlus (z,
                           PastValue (0, runningLogSumExp, defaultHiddenActivation=-1e30)) 
    logSumExp = Boolean.If (BS.Loop.IsLast (runningLogSumExp), # if last entry
                /*then*/ runningLogSumExp,                     # then copy that
                /*else*/ FutureValue (0, logSumExp))     # else just propagate to the front
    result = Exp (z - logSumExp)
].result
Clone this wiki locally