Unit 4.5 - Is the activation function necessary for using MLP as a regression model?

[melrefaey]

Hi,

In a Unit 4.5 lecture, it was mentioned that to convert an MLP Classifier into a Regression model, you should remove the activation function. However, while experimenting with the PyTorchMLP class, the activation function RLU was there in the notebook, and I kept the torch.nn.ReLU() activation function, and the model performed quite well. The curve fit the data points smoothly, and the loss was around 0.01.

Out of curiosity, I removed the activation function to observe any changes, and I noticed the loss increased slightly to 0.02, and the curve didn’t fit as smoothly. The difference wasn’t major, but it raised the question: Is removing the activation function always necessary for regression tasks, or could it still be beneficial, especially with larger datasets? Any thoughts on how this might impact performance on bigger datasets?

Thanks in advance!

[rasbt]

Hi there,

here, I meant removing the activation function after the last layer. Otherwise, you will restrict the model outputs. So, I wouldn't remove any ReLU function in the middle of a network.

[melrefaey]

Awesome! That makes total sense. Thank you for your prompt response.