[Feature]: Custom loss function implementation in python #673

xnerhu · 2024-07-16T22:14:23Z

xnerhu
Jul 16, 2024

Feature Request

Hello. I would like to have a feature, where I can pass custom function loss implemented in python instead of in julia. I'm not able to implement it in julia as it part of bigger optimization pipeline. I believe this feature can be implemented by passing python function pointer to julia and executing it from there.

MilesCranmer · 2024-07-16T22:52:34Z

MilesCranmer
Jul 16, 2024
Maintainer

Maybe you could try PythonCall.jl? https://juliapy.github.io/PythonCall.jl/stable/pythoncall/ It lets you call Python functions from Julia. See https://astroautomata.com/PySR/examples/#7-julia-packages-and-types for an example of using an external Julia package in the loss function.

0 replies

xnerhu · 2024-07-17T09:12:07Z

xnerhu
Jul 17, 2024
Author

Maybe you could try PythonCall.jl? https://juliapy.github.io/PythonCall.jl/stable/pythoncall/ It lets you call Python functions from Julia. See https://astroautomata.com/PySR/examples/#7-julia-packages-and-types for an example of using an external Julia package in the loss function.

It somewhat worked thanks.

import os


os.environ["PYTHON_JULIACALL_THREADS"] = "1"
from datetime import datetime
import numpy as np
from pysr import PySRRegressor
from pysr import jl

from julia.api import Julia


x = None


def custom_loss(y_true, y_pred):
    # global x
    # print(y_true, y_pred, x)
    # if x is None:
    #     x = np.random.rand()
    return float((y_true - y_pred) ** 2)


jl.seval(
    """
import Pkg
Pkg.add("PythonCall")
"""
)
jl.custom_loss_function = custom_loss

jl.seval(
    """
function custom_loss_wrapper(y_true, y_pred)
    py_obj = PythonCall.pycall(custom_loss_function, y_true, y_pred)
    return PythonCall.pyconvert(Float32, py_obj)
end
"""
)

X = np.random.rand(100, 2)
y = X[:, 0] * X[:, 1] + np.random.rand(100) * 0.1

model = PySRRegressor(
    niterations=40,
    binary_operators=["+", "-", "*", "/"],
    unary_operators=["cos", "exp"],
    elementwise_loss="custom_loss_wrapper",
)

model.fit(X, y)
print(model)
predictions = model.predict(X)
print(predictions)

Though I had to do os.environ["PYTHON_JULIACALL_THREADS"] = "1" from #661

because of

  Resolving package versions...
  No Changes to `C:\Users\xnerhu\.julia\environments\pyjuliapkg\Project.toml`
  No Changes to `C:\Users\xnerhu\.julia\environments\pyjuliapkg\Manifest.toml`
Compiling Julia backend...

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7ff85e1587a5 --  at 0x7ff85e1587a5 -- ION with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7ff85e163a4d -- PyObject_GC_Malloc at C:\Python310\python310.dll (unknown line)  
nd any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7ff85e1587a5 --  at 0x7ff85e1587a5 -- IONC:\Python310\python310.dll (unknown lin expression starting at none:0
\python310.dll (unknown line)
in expression starting at none:0
in expression starting at none:0
\python310.dll (unknown line)
ne)
in expression starting at none:0
in expression starting at none:0
\python310.dll (unknown line)
PyEval_EvalFrameDefault at C:\PytPyEval_EvalFrameDefault at C:\Python310\python310.dll (unknown line)
PyFunction_Vectorcall at C:\Python310\python310.dll (unknown line)

PyFunction_Vectorcall at C:\Python310\python310.dll (unknown linyFunction_Vectorcall at C:\Python310\python310.dll (unknown line)
PyVectorcall_Call at C:\Python310\python310.dll (unknown line)
e)
PyVectorcall_Call at C:\Python310\python310.dll (unknown line)
PyVectorcall_Call at C:\Python310\python310.dll (unknown line)
PyObject_Call at efault at C:\Python310\python310.dll (unknown PyObject_Call at C:\Python310\python310.dll (unknown line)   
wn line)
PyObject_Call at C:\Python310\python310.dll (unknown line)
PyFunction_Vectorcall at C:\Python310\python310.dll (unknowPyFunction_Vectorcall at C:\Python310\python310.dll (unknown line)
PyObject_CallObject at C:\Users\xnerhu\.julia\packages\PythonCall\S5MOg\src\C\pointers.jl:297 [inlined]
macro expansion at C:\Users\xnerhu\.julia\packages\PythonCall\S5MOg\src\Core\Py.jl:132 [inlined]
pycallargs at C:\Users\xnerhu\.julia\packages\PythonCall\S5MOg\src\Core\builtins.jl:212
unknown function (ip: 00000138b4d57ddf)
#pycall#21 at C:\Users\xnerhu\.julia\packages\PythonCall\S5MOg\src\Core\builtins.jl:230
#pycall#21 at C:\Users\xnerhu\.julia\packages\PythonCall\S5MOg\#pycall#21 at C:\Users\xnerhu\.julia\packages\PythonCall\S5MOg\src\Core\builtins.jl:230
nlined]
ined]
macro expansion at C:\Users\xnerhu\.julia\packages\PythonCamacro expansion at C:\Users\xnerhu\.julia\packages\PythonCall\S5MOg\src\Core\Py.jl:132 [inlined]
pycallargs at C:\Users\xnerhu\.julia\packages\PythonCall\S5MOg\src\Core\builtins.jl:212
jl_apply at C:/workdir/src\julia.h:1982 [inlined]
do_apply at C:/workdir/src\builtins.c:768
do_apply at C:/workdir/src\builtins.c:76pycall at C:\Users\xnerhu\.julia\packages\PythonCall\S5MOg\src\Core\builtins.jl:220 
pycall#21 at C:\Users\xnerhu\.julia\packages\PythonCall\S5MOg\src\Core\builtins.jl:230
:297 [inlined]
macro expansion at C:\Users\xnerhu\.julia\packages\PythonCall\S5MOg\src\Core\Py.jl:132 [inlined]
pycallargs at C:\Users\xnerhu\.julia\packages\PythonCall\S5MOg\src\Core\builtins.jl:212
unknown function (ip: 00000138b4d57ddf)
ages\PythonCall\S5MOg\src\Core\builtins.jl:212
unknown function (ip: 00000138b4unknown function (ip: 00000138b4d57ddf)
l at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\LossFunctions.jl:19 [inlined]
#4 at C:\Users\xnerhu\.julia\environments\pyjuliapkg\pyjuliapkg\install\share\julia\stdlib\v1.10\Statistics\src\Statistics.jl:187 [inlined]
mapreduce_impl at .\reduce.jl:262
mapreduce_impl at .\reduce.jl:262
nments\pyjuliapkg\pyjuliapkg\install\share\julia\stdljl_apply at C:/workdir/src\julia.h:1982 [inlined]
do_apply at C:/workdir/src\builtins.c:768
pycall at C:\Users\xnerhu\.julia\packages\PythonCall\S5MOg\src\Core\builtins.jl:220
custom_loss_wrapper at .\none:2
l at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\LossFunctions.jl:19 [inlined]
#4 at C:\Users\xnerhu\.julia\environments\pyjuliapkg\pyjuliapkg\install\share\julia\stdlib\v1.10\Statistics\src\Statistics.jl:187 [inlined]
mapreduce_impl at .\reduce.jl:262
mapreduce_impl at .\reduce.jl:277 [inlined]
_mapreduce at .\reduce.jl:447
jl_apply at C:/workdir/src\julia.h:1982 [inlined]
do_apply at C:/workdir/src\builtins.c:768
pycall at C:\Users\xnerhu\.julia\packages\PythonCall\S5MOg\src\Core\builtins.jl:220
pycall at C:\Users\xnerhu\.julia\packages\Pytho#mapreduce#821 at .\reducedim.jl:357 [inlined]
mapreduce at .\reducedim.jl:357 [inlined]
#_sum#831 at .\reducedim.jl:1015 [inlined]
_sum at .\reducedim.jl:1015 [inlined]
#sum#829 at .\reducedim.jl:1011 [inlined]
sum at .\reducedim.jl:1011 [inlined]
_mean at C:\Users\xnerhu\.julia\environments\pyjuliapkg\pyjuliapkg\install\share\julia\stdlib\v1.10\Statistics\src\Statistics.jl:187
_mean at C:\Users\xnerhu\.julia\_mean at C:\Users\xnerhu\.julia\environments_mapreduce at .\reduce.jl:447
ia\environments\pyjuliapkg\pyjuliapkg\install\share\julia\stdlib\v1.10\Statistics\src\Statistics.jl:104 [inlined]
mean at C:\Users\xnerhu\.julia\environments\pyjuliapkg\pyjuliapkg\install\share\julia\stdlib\v1.10\Statistics\src\Statistics.jl:104 [inlined]
_loss at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\LossFunctions.jl:20
unknown function (ip: 00000138b4df92cf)
unknown function (ip: 00000138_mapreduce_dim at .\reducedim.jl:365 [inlined]
#mapreduce#821 at .\reducedim.jl:357 [inlined]
mapreduce at .\reducedim.jl:357 [inlined]
#_sum#831 at .\reducedim.jl:1015 [inlined]
_sum at .\reducedim.jl:1015 [inlined]
#sum#829 at .\reducedim.jl:1011 [inlined]
sum at .\reducedim.jl:1011 [inlined]
_mean at C:\Users\xnerhu\.julia\environments\pyjuliapkg\pyjuliapkg\install\share\julia\stdlib\v1.10\Statistics\src\Statistics.jl:187
_mean at C:\Users\xnerhu\.julia\environments\pyjuliapkg\pyjuliapkg\install\share\julia\stdlib\v1_mean at C:\Users\xnerhu\.julia\environments\pyjuliapkg\pyjuliapkg\install\share\julia\stdlib\v1.10\Statistics\src\Statistics.jl:186
_eval_loss at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\LossFunctions.jl:67
#mean#1 at C:\Users\xnerhu\.julia\environments\pyjuliapkg\pyjuliapkg\install\share\julia\stdlib\v1.10\Statistics\src\Statistics.jl:104 [inlined]
mean at C:\Users\xnerhu\.julia\environments\pyjuliapkg\pyjuliapkg\install\share\julia\stdlib\v1.10\Statistics\src\Statistics.jl:104 [inlined]
_loss at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\LossFunctions.jl:20
jl:105
_loss at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\LossFunctions.jl:20
unknown function (ip: 00000138b4df92cf)
unknown function (ip: 00000138b4df92cf)
SymbolicRegression\FtJSD\src\LossFunctions.jl:20
jl:105
Statistics\src\Statistics.jl:104 [inlined]
]
mean at C:\Users\xnerhu\.julia\environments\pyjuliapkg\pyjuliapkg\install\share\julia\stdlib\v1.10\Statistics\src\Statistics.jl:104 [inlined]
_loss at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\LossFunctions.jl:20
unknown function (ip: 00000138b4df92cf)
SymbolicRegression\FtJSD\src\LossFunctions.jl:20
b\v1.10\Stati#score_func#5 at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\LossFunctions.jl:164 [inlined]
score_func at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\LossFunctions.jl:161 [inlined]
#PopMember#2 at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\PopMember.jl:99
#PopMember#2 at C:\Users\xnerhu\.julia\p#PopMember#2 at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\PopMember.jl:99
1#eval_loss#3 at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\LossFunctions.jl:105
_eval_loss at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\LossFunctions.jl:67
eval_loss at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\LossFunctions.jl:97 [inlined]
#score_func#5 at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\LossFunctions.jl:164 [inlined]
score_func at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\LossFunctions.jl:161 [inlined]
#PopMember#2 at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\PopMember.jl:99
#PopMember#2 at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\PopMember.jl:99
1 [ieval_loss at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\LossFunctions.jl:97 [inlinPopMember at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\PopMember.jl:88 [inlined]
#2 at .\none:0 [inlined]
iterate at .\generator.jl:47 [inlined]
ages\SymbolicRegression\FtJSD\src\PopMember.jl:88 [inlined]
PopMember at C:\Users\xnerhu\.julia\paccollect at .\array.jl:834
collect at .\array.jl:834
u\.julia\packages\SymbolicRegression\FtJSD\src\PopMember.jl:88 [inlined]
#2 at .\none:0 [inlined]
iterate at .\generator.jl:47 [inlined]
collect at .\array.jl:834
#Population#1 at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\Population.jl:49 [inlined]#score_func#5 at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\LossFunctions.jl:164 [inlined]
score_func at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\LossFunctions.jl:161 [inlined]
#PopMember#2 at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\PopMember.jl:99
#PopMember#2 at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\PopMember.jl:99
1 [inlined]Population at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\Population.jl:35 [inlined]
macro expansion at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\SymbolicRegression.jl:765 [inlined]
#54 at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\SearchUtils.jl:116
#54 at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\SearchUtils.jl:116
8 [inlined]
PopMember at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\PopMember.jl:88 [inlined]
#2 at .\none:0 [inlined]
iterate at .\generator.jl:47 [inlined]
collect at .\array.jl:834
collect at .\array.jl:834
7 [inlined]
ages\SymbolicRegression\FtJSD\src\PopMember.jl:88 [inlined]
:765Population at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\Population.jl:35 [inlined]
macro expansion at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\SymbolicRegression.jl:765 [inlined]
#54 at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\SearchUtils.jl:116
jfptr_YY.54_21945 at C:\Users\xnerhu\.julia\compiled\v1.10\SymbolicRegression\X2eIS_JE3Mg.dll (unknown line)
#Population#1 at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\Population.jl:49 [inlined]
Population at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\Population.jl:35 [inlined]
macro expansion at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\SymbolicRegression.jl:765 [inlined]
#54 at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\SearchUtils.jl:116
#54 at C:\Users\xnerhu\.julia\packages\SymbolicRegression\FtJSD\src\SearchUtils.jl:116
Regression.jl:765 [injl_apply at C:/workdir/src\julia.h:1982 [inlined]
start_task at C:/workdir/src\task.c:1238
Allocations: 17550813 (Pool: 17526745; Big: 24068); GC: 21
jfptr_YY.54_21945 at C:\Users\xnerhu\.julia\compiled\v1.10\SymbolicRegression\X2eIS_JE3Mg.dll (unknown line)
jl_apply at C:/workdir/src\julia.h:1982 [inlined]
start_task at C:/workdir/src\task.c:1238
Allocations: 17550813 (Pool: 17526745; Big: 24068); GC: 21

also, how can I do loss_function intead of elemenetwise_loss? I think seval needs to have explicit types in args

juliacall.JuliaError: MethodError: no method matching custom_loss_wrapper(::Node{Float32}, ::Dataset{Float32, Float32, Matrix{Float32}, Vector{Float32}, Nothing, @NamedTuple{}, Nothing, Nothing, Nothing, Nothing}, ::Options{Int64, DynamicExpressions.OperatorEnumModule.OperatorEnum, Node, false, false, nothing, StatsBase.Weights{Float64, Float64, Vector{Float64}}})

1 reply

MilesCranmer Jul 17, 2024
Maintainer

from julia.api import Julia

Note that julia is from the older PyJulia, whereas nowadays, PySR uses JuliaCall. I would avoid mixing them because it will cause segfaults.

xnerhu · 2024-07-17T09:24:36Z

xnerhu
Jul 17, 2024
Author

btw is it possible to do multivaritive prediction (multiple outpouts)?

0 replies

MilesCranmer · 2024-07-17T11:25:21Z

MilesCranmer
Jul 17, 2024
Maintainer

Though I had to do os.environ["PYTHON_JULIACALL_THREADS"] = "1" from #661

So if you are calling back into Python, you can't use multithreading or multiprocessing because of Python's global interpreter lock. So you will need to run PySR with multithreading=False, procs=0 as options to PySRRegressor. For this reason, and because of the overhead of Python in general, it will be much slower than if you were to write your code as a Julia function.

0 replies

MilesCranmer · 2024-07-17T11:27:28Z

MilesCranmer
Jul 17, 2024
Maintainer

also, how can I do loss_function intead of elemenetwise_loss? I think seval needs to have explicit types in args

seval does not need explicit types. The error is because loss_function takes 3 arguments whereas you have implemented a 2-argument function. If you read through the other discussion threads there are several examples of complex loss_function implementations: https://github.com/MilesCranmer/PySR/discussions. Also the docs have an example: https://astroautomata.com/PySR/examples/#9-custom-objectives

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature]: Custom loss function implementation in python #673

{{title}}

Replies: 5 comments 1 reply

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

[Feature]: Custom loss function implementation in python #673

xnerhu Jul 16, 2024

Feature Request

Replies: 5 comments · 1 reply

MilesCranmer Jul 16, 2024 Maintainer

xnerhu Jul 17, 2024 Author

MilesCranmer Jul 17, 2024 Maintainer

xnerhu Jul 17, 2024 Author

MilesCranmer Jul 17, 2024 Maintainer

MilesCranmer Jul 17, 2024 Maintainer

xnerhu
Jul 16, 2024

Replies: 5 comments 1 reply

MilesCranmer
Jul 16, 2024
Maintainer

xnerhu
Jul 17, 2024
Author

MilesCranmer Jul 17, 2024
Maintainer

xnerhu
Jul 17, 2024
Author

MilesCranmer
Jul 17, 2024
Maintainer

MilesCranmer
Jul 17, 2024
Maintainer