Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support fury as an optional serializer #78

Open
wants to merge 9 commits into
base: main
Choose a base branch
from

Conversation

codingl2k1
Copy link
Contributor

@codingl2k1 codingl2k1 commented Sep 13, 2023

What do these changes do?

Related issue number

import xorbits
import time
from xorbits.remote import spawn
from xoscar.serialization.core import FurySerializer
from dataclasses import dataclass
from typing import List

@dataclass
class Foo:
    f1: int
    f2: int
    f3: float
    f4: int
    f5: int
    f6: List[int]


FurySerializer.register(Foo)

import logging

logging.basicConfig(level=logging.DEBUG)

if __name__ == "__main__":
    xorbits.init(config={"third_party_modules":["t1"]})


    def f1(x):
        return x


    foos = [Foo(f1=5000, f2=2 ** 32, f3=1 / 3, f4=100, f5=200, f6=[11, 1234]) for i in range(500)]

    start = time.time()
    results = []
    for _ in range(100):
        r1 = spawn(f1, foos)
        results.append(r1)

    xorbits.run(results)

    print(f"cost: {time.time() - start}")

This test code may be crash when fury deserializing, but if it does not crash, the performance is better than Mars default serializers.

Use fury(if not crash): 9.764425039291382
Mars default: 13.590802669525146

Fixes #xxxx

Check code requirements

  • tests added / passed (if needed)
  • Ensure all linting tests pass

@XprobeBot XprobeBot added this to the v0.2.0 milestone Sep 13, 2023
@codecov
Copy link

codecov bot commented Sep 13, 2023

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (2a041ae) 89.12% compared to head (d43d8f4) 88.98%.

Files Patch % Lines
python/xoscar/serialization/pyfury.py 83.33% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #78      +/-   ##
==========================================
- Coverage   89.12%   88.98%   -0.15%     
==========================================
  Files          48       48              
  Lines        4010     4012       +2     
  Branches      764      764              
==========================================
- Hits         3574     3570       -4     
- Misses        351      356       +5     
- Partials       85       86       +1     
Flag Coverage Δ
unittests 88.80% <83.33%> (-0.15%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@codingl2k1 codingl2k1 marked this pull request as ready for review September 15, 2023 04:12


def register_classes(*args):
def register_class_to_fury(obj_type):

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only register class for current thread, fury in other threads doens't get registerd. The deserialization may raise error if the register are not consisitent.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the root cause of random crash?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, another issue tis that the class register order must be consistent between serializatio and deserialization.

Is class registration constsitent always? Could you print all class registration too.

@chaokunyang
Copy link

Is there any things I can help for this PR?

@codingl2k1
Copy link
Contributor Author

Is there any things I can help for this PR?

fury Python is hard to use, could you help fix the crash issue?

@XprobeBot XprobeBot modified the milestones: v0.3.0, v0.3.1, v0.3.2, v0.3.3 Jul 10, 2024
@XprobeBot XprobeBot modified the milestones: v0.3.3, v0.3.4 Aug 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants