BS Functions

Functions

Functions are BrainScript's way of creating reusable modules. Functions are parameterized expressions, and can be defined in a straight-forward manner, e.g.:

Sqr (x) = x * x

Functions are first-class data types in the sense that functions can accept and return other functions.

Defining Functions

There are two ways to define functions, standard syntax and anonymous lambda syntax.

Standard Syntax

Named functions are record members that are defined as record-member assignments of this form:

   f (arg1, arg2, ..., optionalArg1=optionalArg1Default, ...) = expression of args and optional args

The function arguments create a name scope like a record; i.e. function arguments are as if they were defined as members in an enclosing record.

Optional parameters

Functions can have optional parameters. These are passed by preceding their value by name=. If an optional parameter is not passed, its default value is used. In the function definition, optional parameters are declared using the form name=defaultValue.

Here is an example for a Softplus function that takes an optional steepness parameter:

Softplus (x, steepness=1) =
    if steepness == 1                # use the simpler formula if no steepness specified
    then Log (Constant(1) + Exp (x)) # softplus(x)
    else [                           # version with steepness: 1/s softplus (s x)
        steepnessAsConstant = Constant (steepness)
        y = Reciprocal (steepnessAsConstant) .* Softplus (steepnessAsConstant .* x)
        # Note: .* denotes an elementwise product
    ].y

Note: The current BrainScript version has a problem with passing an optional parameter's value to a function as an optional parameter of the same name. E.g. to pass the value of a variable tag to a function's optional parameter which is also named tag one would say f(tag=tag), causing a circular reference. Currently, one should first assign the value to a different name, and use that; for example: tag1 = tag; f(tag = tag1). This is planned to be addressed in an upcoming update.

Lambda Syntax

In addition to the function-definition syntax, BrainScript allows anonymous lambdas using the syntax (x => f(x)), which is meant to mimic C#. The following two definitions are equivalent:

sqr(x) = x * x
sqr = (x => x * x) 

Actually, the BrainScript parser internally transforms the latter into the former form.

The lambda syntax is currently constrained to one parameter, and also does not support optional parameters. Lambdas with more than one parameter or with optional parameters must be defined as named functions.

Invoking Functions

Functions are invoked as you expect: By passing the arguments in parentheses; and passing optional arguments as named parameters. For example, a self-stabilizer (a scaling technique somewhat similar to batch normalization) using Softplus with steepness 4 would be written with the following two function invocations:

beta = ParameterTensor (1, init='fixedValue', value=1)
xStabilized = x .* Softplus (beta, steepness=4)

Functions are executed upon invocation. Specifically, if a function defines model parameters inside (that is, the variants of the Parameter() function and any function that has calls to it inside), each invocation of the function yields an independent instance of these model parameters. Often, the result of a function invocation is then assigned to a variable or passed as a function argument. Multiple uses of such value do not result in repeated execution.

Purity/Referential Transparency

Although BrainScript has commonalities with functional languages, be aware that BrainScript functions are not entirely pure, in that can have one very specific side effect: that of instantiating new model parameters. For example, while the expression 2 * sqr(x) is equivalent to sqr(x) + sqr(x) (with sqr(x) = x*x), that is not the case for 2 * Parameter(N) vs. Parameter(N) + Parameter(N).

Functions as Values

In BrainScript, functions are values. A named function can be assigned to a variable and passed as an argument. In fact, named functions are just record members that have the type "function." Function-name lookup is the same as record-member lookup. This allows, for example, to group functions into "namespaces" by defining them inside a record expression.

Named Functions as Values

If a function name is used without following parentheses, it will refer to the function itself as an object.

Partial Application/Currying

BrainScript currently does not have syntactic support partial application or currying. To achieve a similar effect, one can define a new function that calls another function with bound/captured parameters:

Layer (x, m, n, f) = f (Parameter (m, n) * x + Parameter (n))
Sigmoid512Layer (x) = Layer (x, 512, 512, Sigmoid)

Examples

The following example defines a simple hierarchy of common network layer types:

Layers = [
    # base feed-forward without and with parameterized energy function
    # AffineLayer() defines model parameters inside.
    AffineLayer (x, m, n) = Parameter (m, n, init='uniform') * x
                            + Parameter (n, init='fixedValue', value=0)
    # NLLayer applies a non-linearity on top of an affine layer
    NLLayer (x, m, n, f) = f (AffineLayer (x, m, n))
    # a few non-linearities
    SigmoidLayer (x, m, n)  = NLLayer (x, m, n, Sigmoid) # pass Sigmoid() function as the non-linaerity
    SoftplusLayer (x, m, n) = NLLayer (x, m, n, (x => Log (Constant(1) + Exp(x)))/*Softplus as lambda*/)
    ReLULayer (x, m, n)     = NLLayer (x, m, n, RectifiedLinear)
    SoftmaxLayer (x, m, n)  = NLLayer (x, m, n, Softmax)
]

which could be used like this:

# constants defined
# Sample, Hidden, and Label dimensions
SDim = 28*28 # feature dimension
HDim = 256   # hidden dimension
LDim = 10    # number of classes

features = Input (SDim, tag='feature')
labels   = Input (LDim, tag='label')

# layers
h = Layers.ReLULayer (features, HDim, SDim)
z = Layers.AffineLayer (h, LDim, HDim) # input to softmax; same as log softmax without normalization  

# output and criteria
P    = Softmax (z, tag='output')
ce   = CrossEntropyWithSoftmax (labels, z, tag='criterion')
errs = ErrorPrediction         (labels, z, tag='evaluation')

This is the same as the initial example in Basic Concepts, but using functions.

NDLNetworkBuilder (Deprecated)

Earlier versions of CNTK used the now deprecated NDLNetworkBuilder instead of BrainScriptNetworkBuilder. NDLNetworkBuilder's function definition differed in a few ways. The following definition is valid in NDLNetworkBuilder:

FF (X1, W1, B1) # no equal sign
{ # both curly braces and brackets are allowed
    T = Times (W1, X1)
    FF = Plus (T, B1)
} # return value is FF, and not the entire record

Especially, the return value is returned via the local variable that has the same name as the function; or if no such variable is found, the variable defined last will be returned. In BrainScript this must now be written as:

FF (X1, W1, B1) =
[
    T = Times (W1, X1)
    Y = Plus (T, B1)
].Y

Especially, the return value Y must be explicitly selected at the end using . syntax, otherwise the function's value would be the entire record.