Best practices for training latency-coded SNNs on TSC task with supervised STDP

[ElisaNguyen]

Dear All,

I am currently working on training a latency-coded SNN (i.e. earliest spike for classification) on a time-series classification task with supervised STDP (voltage clamp). As there are several hyperparameters, I was wondering if you have any best practices to share for this case?

Currently, I am working with the UCI Binary ADL dataset, where I have transformed the data to binary spike trains with a timestep of 1 second.
This is the configuration of my network right now:

• Input Layer of size 13 (=number of input dimensions)
• Output LIF Layer of size 2 (first, I wanted to train it on a binary classification task, i.e. sleeping or not sleeping)
• Both layers are fully connected, with the PrePost update rule and no weight decay
• Learning rates nu=(1,1)

I am feeding the data in time windows of 1 second to the network, without resetting the state variables in between (i.e. continuous prediction). Due to the size of the dataset, training time is quite long to try different hyperparameters. As I am quite inexperienced, I wanted to ask if you have any recommendations for learning rates, weight decay, weight boundaries, etc. for me.

Thank you very much!
Elisa

[Hananel-Hazan]

Thanks Elisa for describing your task.

Searching the hyper parameters is never fun and never a short task.
I would start with smaller nu value, one seems to me too big, moreover, I would stick to asymmetric nu where one side is bigger then the other side.
Then you can use a particle swarm optimization on couple of parameters:

• nu
• Threshold level in case of LIF, in case of adaptive LIF threshold decay parameters.
• Weights bound [-1,1], [-10,10]....
• Time window of input, you need to make sure that the encoding is beneficial to the network.

Let me know how it goes.

[ElisaNguyen]

Thank you for the quick reply! What reason does it have to stick to an asymmetric nu?

Do you perhaps also have an idea on how to record true training accuracy? Currently, I am running the input through the network with the clamp, which then gives me a training accuracy of 100% all the time. However, if I run the same input again without the clamp to get the network's prediction at that point, I am basically presenting the same data twice to the network and the voltage of the neurons would be skewed. So, step 2 in the code below would not be the right way to get the training accuracy in my opinion... ?

      ### Step 1: Run the network with the provided inputs and clamp in train mode###
      inputs = {'Input': x}
      network.run(inputs=inputs, time=time, clamp=clamp)

      ### Step 2: Disable train mode and rerun to get prediction###
       #network.train(False)
       #network.run(inputs=inputs, time=time)

[Hananel-Hazan]

What reason does it have to stick to an asymmetric nu?

The natural STDP has an asymmetric shape link, so I would start from here and change only if needed. Note that STDP Pre and Post timing is also asymmetric.

Do you perhaps also have an idea on how to record true training accuracy?

There is no relevance to test the presented data during the training process since the program clamps the right answer. However, by releasing the clamp and presenting the same data, the network output can give a crude evaluation of the synaptic change. In any case, the important performance is not how good the training data has been trained, rather what is the accuracy of the testing data.

[ElisaNguyen]

Thank you for your answer! It does indeed make sense not to track training accuracy when using voltage clamping as a training method.

I have another, general design question. If the output neurons at test time do not spike at all, until now I have used sampling from a uniform distribution across all classes to predict the output class. However, this is limited as the network might lose learning progress on the way. Have you come across a similar problem?

[Hananel-Hazan]

There are a times when the input is not strong enough, in the SOM example you can see that some MNIST images dont elicit any spike, at this case the input is amplified and fed back again to the network.