Time-Delayed Variables

[TadeuNP]

Hi,

I'm trying to use PySR to identify some equations with a certain $\tau$ dependence on the past, similar to the following delayed differential equation,

$$\begin{align} \frac{\mathrm{d}x}{\mathrm{d}t} = \frac{\beta}{1+(x(t-\tau))^n} - x. \end{align}$$

One way of doing it is by introducing a new column with shifted data, but I was hoping for a method that allowed PySR itself to look for $\tau$. I thought about defining a binary operator B(x, tau) that returns the value of x at tau time steps ago, but looking at the file that defines the operators, it seems they only work with a single scalar value as input, so it looks incompatible.

Is there a way? Any suggestion?

Thanks.

[MilesCranmer]

Hi @TadeuNP,

Very interesting idea. I think implementing this really depends on specific requirements. If you just want to query x in the past, you could certainly create a unary operator that takes a scalar argument, converts it to integer inside the function (type conversion is allowed - see the prime number example in the docs page on code examples), and indexes the correct row. However, all operators assume they are applied element-wise without changing functional form, so if you need the row index of the x, you will need to do some hacking.

If what you mean is a functional, rather than a numerical operator (ie the operator should change all of the variables in the subtree), that is not supported and would take a large amount of work. (edit - see comment)

Cheers,
Miles

[MilesCranmer]

Thanks. This is a bit difficult because operators don’t have access to that information. You can see in the call here: https://github.com/SymbolicML/DynamicExpressions.jl/blob/34271cb68c7286d90fe5600d6df173e5c05381bc/src/EvaluateEquation.jl#L153 - since the index “j” is not being passed to the operator, I’m not sure you could set this up easily.

Probably the easiest solution is to simply pass shifted variables as new columns. It might just work!

An alternative is to do some kind of symbolic distillation: https://speakerdeck.com/milescranmer/symbolic-distillation-of-neural-networks, where the NN is able to select the time delay. This requires more work on your end though.

[MilesCranmer]

Actually one way to get it to work is set up GenericOperatorEnum (https://github.com/SymbolicML/DynamicExpressions.jl/tree/34271cb68c7286d90fe5600d6df173e5c05381bc#generic-types) to work with SymbolicRegression.jl. Then the operator would actually have access to the entire array at once.

[TadeuNP]

I will take a look at all your new suggestions, but let me remark that it indeed works if I simply introduce shifted variables, and the most important thing is that PySR feature selection seems to be quite good and fast (I had not tried it before), so the problem of having too many shifted variables may not be that problematic after all!

edit: okay, the feature selection part is not working as well as it seemed to, but even with ~20 features it can still recover the expression if simple enough :)

[MilesCranmer]

Awesome!! Do you mean you are using select_k_features? Interestingly I actually don't usually use this feature - I find that just letting the GA run for a bit is enough to select the important features. I think the crossovers are good at doing the feature selection.

(I should really turn off the warning for >10 features)

[TadeuNP]

Yes, I was just testing it for the first time, but it seems to struggle with discriminating between very similar features , as is the case when $x_2$ is just a small shift of $x_1$.

As you mentioned, it turns out that just running for a bit longer is enough to find the correct variables, even when I had 15+ delayed features introduced.