More dask fixes #16

lithomas1 · 2024-11-23T18:49:56Z

Reference issue

What does this implement/fix?

Additional information

Add a new optional parameter `callback` to least_squares. A user-defined function that is called on each iteration, can stop optimization by returning True or raising StopIteration. Same signature and API as in eg. scipy.optimize.differential_evolution. Only implemented for trf and dogleg methods so far.

Fix linter errors

lucascolley · 2024-11-23T19:29:46Z

scipy/fft/tests/test_basic.py

+        # Trick to get the array-api-compat namespace for dask
+        # (otherwise the "naked" dask.array asarray does not respect
+        # the input dtype)


I think I fixed this in dask/dask#11288?

I skimmed that patch really quick - but at a quick glance it looks like that only fixes the issue when the input is a dask array.

When the input is a numpy array it looks like we go into the from_array codepath which is not passed the dtype option?

If my patch didn't fix it then this should be a new bug report for Dask - we shouldn't have to work around that.

I'll poke around more inside dask to double check if this is still an issue.

Seems to be fixed in 2024.12.1 at least - gonna take this out.
Maybe I was on too old of a dask version.

Fixed in dask/dask#11586

lucascolley · 2024-11-23T19:30:36Z

scipy/fft/tests/test_real_transforms.py

+    # use array-api-compat namespace for dask
+    # since dask asarray never makes a copy
+    # which makes xp_copy silently a no-op
+    xp = array_namespace(x)


could we adjust xp_copy to special-case and copy for Dask instead?

We are currently passing in xp into xp_copy in these tests which is the problem.

i.e. xp_copy(arr, xp=<bad dask.array xp>)

If we don't pass in xp_copy, I've changed array_namespace in our vendored array-api-compat to use the wrapped dask.array namespace by default (will upstream this later).

It would be easier to just do these copies NumPy side so we don't need to involve xp_copy at all:

x = np.random.rand(100) x2 = x.copy() x = xp.asarray(x) x2 = xp.asarray(x2)

We can also drop the xp argument to xp_copy (which would be a smaller diff).

happy with that!

actually, no - x2 should be an array from the unwrapped namespace before it is passed into func.

I think I've changed all of these back to using numpy, let me know if there's any more left.

lithomas1 · 2024-12-05T18:44:46Z

Going to hold off on this until
data-apis/array-api-compat#209 is resolved.

Looks like dasks is switching to always copying in asarray?

…d warnings

…is slow due to a bug with std::priority_queue

…rce)

…s.test_shortest_path

Remove trailing spaces, unused variables and imports, change reference of Fibonacci heap to priority queue

Add missing spaces between tests.

model the structure on scipy.ndimage: - add `_delegators.py` with *_signature functions - add _signal_api.py to collect "bare" imports from _private modules - add _support_alternative_backends.py to decorate "bare" functions - in __init__.py, import decorated names from _support_alternative_backends.py

[skip ci]

lithomas1 · 2024-12-22T23:00:31Z

stats is now fully passing as well.

lucascolley · 2024-12-22T23:04:36Z

scipy/_lib/_util.py

+    if is_dask_namespace(xp) or is_jax_namespace(xp):
+        # TODO: verify for jax
+        return xp.where(cond, f(arrays[0], arrays[1]), f2(arrays[0], arrays[1]) if not fillvalue else fillvalue)


cf. scipy#22070, cc @crusaderky.

I think @mdhaber was also interested in changes to _lazywhere.

(Sorry for the slow ping!)

This looks fine to me if you need something that works with Dask and JAX.

f2 can be None. IMHO _lazywhere should be moved to array-api-extras before such backend-specific hacks take place.

I think I was try to hack around _lazywhere since IIRC @mdhaber mentioned _lazywhere didn't seem to be much of an optimization compared to just doing the regular where, and so might be removed.

Happy to wait for your jax/array-api-extra changes to land, though.

Thanks for the pointer about f2, though.
I'll patch in a later commit, but strange that it wasn't hit in tests.

lucascolley · 2024-12-22T23:06:05Z

scipy/fft/tests/test_real_transforms.py

-    x2 = xp_copy(x, xp=xp)
+    x2 = xp.asarray(x_np.copy())


could you remind me why xp_copy doesn't work for dask?

I think xp_copy does an asarray(..., copy=True) to make a copy, but dask silently ignores the copy flag in its asarray implementation.

hmm, this may be for older Dask only? data-apis/array-api-compat#211 seemed to suggest that copies are now always made even when copy=None

This is a test-specific issue where xp is the naked dask module. The wrapper returned by array_namespace has a functioning copy=True.

scipy/sparse/linalg/_propack/PROPACK

MAINT: improve overflow handling in factorial functions

TST: array types: enforce namespace in tests

…hinx` 0.17 (scipy#22161) * DOC, MAINT: Drop `convert_notebooks.py` * DOC: Use Markdown notebooks with JupyterLite * MAINT: Pin jupyterlite-sphinx to >=0.17.1 * DOC: Use the `NotebookLite` directive instead * DOC: Add a "Download" button for the notebooks [docs only]

lithomas1 · 2025-01-03T22:39:05Z

@lucascolley
This should be close to green now, and ready for re-integration back into your main PR.
I've added skips for all the other modules (e.g. cluster, ndimage)

Are you able to do one last rebase to pick up the lazywhere changes for jax on scipy main?

[skip cirrus] [skip circle]

Co-authored-by: Guido Imperiale <[email protected]>

lucascolley · 2025-01-03T23:54:15Z

oops, the rebase went wrong for some reason. Let me try to fix it.

lucascolley · 2025-01-04T00:00:16Z

I'll open a new PR to SciPy

vpecanins added 5 commits October 16, 2024 02:59

Merge remote-tracking branch 'upstream/main'

2409045

ENH: Add callback for scipy.optimize.least_squares

37f4316

Fix linter errors

Merge remote-tracking branch 'upstream/main'

6a3a7b9

Fix linter errors (again)

a3a440f

lithomas1 marked this pull request as ready for review November 23, 2024 18:52

lithomas1 mentioned this pull request Nov 23, 2024

ENH: array types: add dask.array support scipy/scipy#20956

Closed

lucascolley reviewed Nov 23, 2024

View reviewed changes

windows-server-2003 and others added 15 commits December 12, 2024 08:12

MAINT: Change matrix type specification in dijkstra benchmark to avoi…

3c98c18

…d warnings

ENH: Speed up sparse.csgraph.dijkstra by replace FibonacciHeap which …

c27bcbc

…is slow due to a bug with std::priority_queue

MAINT: Unify dijkstra functions(directed/undirected, single/multi-sou…

aab90d2

…rce)

MAINT: Reorder the testcases and add comments for sparse.csgraph.test…

f1dea56

…s.test_shortest_path

MAINT: Fix signed-compare warnings

98e7576

TST: add 'directed=False' parameter to undirected yen test

d486810

MAINT: PR review fixes

14c41db

Remove trailing spaces, unused variables and imports, change reference of Fibonacci heap to priority queue

TST: Add star graph test.

5647be2

Add missing spaces between tests.

CI: Run 'Checkout scipy' and 'Check for skips' only on Github actions CI

4f1a46e

CI: Add comment in 'Check for skips'

13202c7

BUG: signal: only wrap functions, not classes

c471c8d

MAINT: signal: silence mypy noise

d862029

BUG: signal: fix delegation for LTI-consuming polymorphism

50c1dfe

ENH: linalg.clarkson_woodruff_transform: add batch support

0ff9da3

[skip ci]

lucascolley reviewed Dec 22, 2024

View reviewed changes

lucascolley force-pushed the dask branch from baa7b21 to e02e9ce Compare December 23, 2024 22:56

h-vetinari and others added 2 commits December 24, 2024 10:36

MAINT: improve overflow handling in factorial functions

2550697

Merge pull request scipy#22172 from h-vetinari/factorial_inf

93d11b2

MAINT: improve overflow handling in factorial functions

lucascolley and others added 3 commits January 3, 2025 20:30

Merge pull request scipy#22070 from crusaderky/xp_at

4ce4ef4

Merge pull request scipy#22132 from crusaderky/array_api_tests

9851880

TST: array types: enforce namespace in tests

lucascolley and others added 19 commits January 3, 2025 23:40

ENH: array types: add dask.array support

3bc5fe4

BUG: fixes for Dask

0c6adf1

[skip cirrus] [skip circle]

add some skips [skip ci]

5cc98c0

Fixes for fft/special modules (scipy#14)

78f3e58

fix some CI failures

d14105d

[skip cirrus] [skip circle]

fix array-api-compat install

e44d04d

[skip cirrus] [skip circle]

ENH: array types: add dask.array support

f397f4d

Fixes for fft/special modules (scipy#14)

d89c8ce

fix array-api-compat install

a9c2500

[skip cirrus] [skip circle]

fft/linalg/special passing tests

af2bb98

ENH: array types: add dask.array support

2ce086b

address comments

fa915e8

stats fully passing

3b7cda3

Update scipy/_lib/_util.py

6bff4d4

Co-authored-by: Guido Imperiale <[email protected]>

remove accidental subproject changes

c12f1a3

small fix and rollback array-api-compat hack

920a880

skip all remaining failing tests

5a6d17a

fix ndimage and misc tests

41dbc4b

revert lazywhere hack

da9c068

lucascolley force-pushed the dask branch from 3defd90 to da9c068 Compare January 3, 2025 23:51

lucascolley requested a review from tupui as a code owner January 3, 2025 23:51

lucascolley removed the request for review from tupui January 3, 2025 23:53

lucascolley closed this Jan 4, 2025

lithomas1 deleted the dask branch January 21, 2025 16:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

More dask fixes #16

More dask fixes #16

lithomas1 commented Nov 23, 2024

lucascolley Nov 23, 2024

lithomas1 Nov 23, 2024

lucascolley Nov 23, 2024

lithomas1 Nov 24, 2024

lithomas1 Dec 21, 2024

crusaderky Dec 24, 2024

lucascolley Nov 23, 2024

lithomas1 Nov 23, 2024

lucascolley Nov 23, 2024

lithomas1 Nov 24, 2024

lucascolley Nov 24, 2024

lucascolley Nov 24, 2024

lithomas1 Dec 22, 2024

lithomas1 commented Dec 5, 2024

lithomas1 commented Dec 22, 2024

lucascolley Dec 22, 2024

lithomas1 Dec 22, 2024

mdhaber Dec 22, 2024

crusaderky Dec 24, 2024

lithomas1 Dec 25, 2024 •

edited

Loading

lucascolley Dec 22, 2024

lithomas1 Dec 22, 2024

lucascolley Dec 22, 2024

crusaderky Dec 24, 2024

lithomas1 commented Jan 3, 2025

lucascolley commented Jan 3, 2025

lucascolley commented Jan 4, 2025

More dask fixes #16

More dask fixes #16

Conversation

lithomas1 commented Nov 23, 2024

Reference issue

What does this implement/fix?

Additional information

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lithomas1 commented Dec 5, 2024

lithomas1 commented Dec 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lithomas1 Dec 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lithomas1 commented Jan 3, 2025

lucascolley commented Jan 3, 2025

lucascolley commented Jan 4, 2025

lithomas1 Dec 25, 2024 •

edited

Loading