Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Common functions for interfacing with python-irodsclient #202

Merged
merged 31 commits into from
Dec 8, 2023

Conversation

sellth
Copy link
Contributor

@sellth sellth commented Nov 7, 2023

fixes #200

  • add new class iRODSTransfer
  • put files to iRODS
  • trigger remote checksum computation
  • fix tests
  • also get files from iRODS
  • create common class with utility functions
  • use iRODS session manager implemented in irods.check
  • create sub-collections (recursive put)
  • rsync-like functionality (--sync?) which skips files present on remote
  • file progress as (n / n_total)
  • check if session is properly handled by contextmanager
  • handle CAT_PASSWORD_EXPIRED for long transfers

Note: Uploads cannot be cancelled by KeyboardInterrupt (caused by multiprocessing in irods client)

@sellth sellth self-assigned this Nov 7, 2023
Copy link

codecov bot commented Nov 7, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (9026388) 77.84% compared to head (bab45a0) 79.84%.
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #202      +/-   ##
==========================================
+ Coverage   77.84%   79.84%   +2.00%     
==========================================
  Files          98      100       +2     
  Lines        7330     7701     +371     
==========================================
+ Hits         5706     6149     +443     
+ Misses       1624     1552      -72     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@Nicolai-vKuegelgen Nicolai-vKuegelgen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks very good overall only a few minor comments.

Would be good if someone who understands mock testing in python better than me could review that part.

cubi_tk/irods_common.py Outdated Show resolved Hide resolved
cubi_tk/irods_common.py Outdated Show resolved Hide resolved
cubi_tk/irods_common.py Outdated Show resolved Hide resolved
Copy link
Contributor

@Nicolai-vKuegelgen Nicolai-vKuegelgen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

though as of now this will be unused right?
Should pick a 1-2 functions and implement it there and then start rolling out where needed/possible

@sellth
Copy link
Contributor Author

sellth commented Nov 13, 2023

I could dress up my "SODAR ingest anything" function as a pilot use case, but maybe we want to separate those PRs? Might be worth implementing the get functionality though.

@bihealth bihealth deleted a comment from github-actions bot Nov 17, 2023
@sellth sellth merged commit 67e6e49 into main Dec 8, 2023
8 checks passed
@sellth sellth deleted the irods_utils branch December 8, 2023 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Switch from icommand usage to the python irods client
3 participants