Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

COMP: Update the CI package dependancies. #108

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

hjmjohnson
Copy link
Member

No description provided.

@hjmjohnson hjmjohnson requested review from dzenanz and thewtex February 1, 2025 15:45
Copy link
Member

@dzenanz dzenanz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but Matt should review, too.

README.md Show resolved Hide resolved
@hjmjohnson hjmjohnson force-pushed the update-to-allow-ci-passing branch from 6366a05 to bdc6af9 Compare February 2, 2025 14:31
@hjmjohnson
Copy link
Member Author

@thewtex -- @dzenanz requested your review.

# "_2_34-x86_64" -- most commonly used platform for developers (largest use case)
# "_2_34-aarch64" -- developers with aarch64 likely use quite modern os
# "2014-x86_64" -- fallback packages for deployments to older OS's (i.e. computational cluster environments)
default: '["_2_34-x86_64","_2_34-aarch64","2014-x86_64"]'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Default here is _2_34

- macOS 11.0+ x86_64 platforms (macos-14)
- macOS 13.0+ ARM64 platforms (macos-15)
- Linux glibc 2.17+ (E.g. Ubuntu 18.04+) x86_64 platforms (See: https://github.com/pypa/manylinux for details)
- Linux glibc 2.28+ (E.g. Ubuntu 20.04+) aarch64 (ARMv8) platforms (See: https://github.com/pypa/manylinux for details)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In readme we mention versions different from the default (_2_34). Is this intentional?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand how the readme is related to _2_34. Can the readme be simplified to avoid the bookkeeping overhead of keeping the code insync with the readme?

<< Warning rant begin >>

To be honest, I don't understand all the magic that's going on. I have been trying to get 1 PR for a remote module to pass CI, but after nearly a week of tracking down required changes across a maze of separate repos, and resources pulled from various branches (main/master, release, dashboard, or specific hashes), I'm getting to the end of my ability to provide more time trying to make this work.

I recognize the rational for not repeating code in separate packages, but I'm begining to believe that replication would be much easier to maintain than all the complex interactions between different repos that each require PR's to update and test. Additionally, trying to debug the CI failures is very confusing when the resources needed for a single CI run span many different repos. I have downloaded and learned several repos in the process so far.

I think debugging could be a lot easier if we inline the CI content into the packages that depend on them:
ITKClangFormatLinterAction
ITKApplyClangFormatAction

<< rant end >> :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have been following your efforts, and trying to support you. I gave SphinxExamples a shot on Friday, but gave up soon - so many moving pieces.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand your frustration. And I agree that "inlining" some of the things is less elegant, but would require less knowledge to maintain.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dzenanz: Thank you for tolerating my rant above.

@thewtex @dzenanz @blowekamp I am going to "unravel and inline" the CI for InsightSoftwareConsortium/ITKLabelErodeDilate#40 (a simple case) and try to minimize the learning curve for maintaining the remote repositories.

AN OBSERVATION NOT A CRITIQUE: It is perfectly reasonable to request @thewtex to review, and it is perfectly reasonable to need to allow a week for that review to happen. However, that means that I can not proceed with the effort of pushing the project forward during my 1 free weekend this month.

Things I am going to try:

  • Move config files needed by remote modules CI config into the ITK proper to minimize the number of hashes/branches/repos that need to be synchronously modified.
  • Prefer diffing different remote .github/workflow/*.yml to syncronize rather than depending on dynamicly downloaded external resources.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hjmjohnson I understand your frustration. There is a lot of complexity. This naturally arises from the need to support ARM64, AMD64, MacOS, Linux, Windows, a stable version that most people are using in production to address a number of important real-world uses cases, and a development version where developers that make experimental improvements the long haul, and a GitHub infrastructure that we can all collaborate on. And we are continuously making strides to reduce that complexity as much as possible.

And consolidation of some configuration files in the ITK repository could be a good step forward, where that makes sense.

However, please do not go about breaking existing infrastructure that does work.

This will slow you down much more and it slows everyone else down, too. Burning everything down due to frustration with the status quo is not a recipe to make progress better or faster.

@dzenanz
Copy link
Member

dzenanz commented Feb 4, 2025

Trying to use this here, I run into:

Run sudo xcode-select -s "/Applications/Xcode_15.0.1.app"
  sudo xcode-select -s "/Applications/Xcode_15.0.1.app"
  shell: /bin/bash -e {0}
xcode-select: error: invalid developer directory '/Applications/Xcode_15.0.1.app'
Error: Process completed with exit code 1.

and into:

HEAD is now at ca1a7e5 ENH: Only extract required directories from the ITKPythonBuilds tar
HEAD detached at ca1a7e5
nothing to commit, working tree clean
~/work/ITKMontage/ITKMontage
ERROR: can not find required binary './ITKPythonPackage/scripts/dockcross-manylinux-build-module-wheels.sh'
Building module wheels
./dockcross-manylinux-download-cache-and-build-module-wheels.sh: line 76: ./ITKPythonPackage/scripts/dockcross-manylinux-build-module-wheels.sh: No such file or directory
Error: Process completed with exit code 127.

@hjmjohnson hjmjohnson force-pushed the update-to-allow-ci-passing branch from bdc6af9 to aba88ff Compare February 4, 2025 21:11
@hjmjohnson hjmjohnson force-pushed the update-to-allow-ci-passing branch from 3e5b39f to f80ec68 Compare February 4, 2025 22:06
itk-python-package-tag:
# See https://github.com/InsightSoftwareConsortium/ITKPythonPackage
description: 'Git tag or commit hash for ITKPythonPackage build scripts to use'
required: false
type: string
# v5.4.0 + fixes
default: 'ca1a7e5809dd9dbc1ac3486ff601b5eb1b58184b'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use a tagged, tested version on master.

- macOS 11.0+ x86_64 platforms (macos-14)
- macOS 13.0+ ARM64 platforms (macos-15)
- Linux glibc 2.17+ (E.g. Ubuntu 18.04+) x86_64 platforms (See: https://github.com/pypa/manylinux for details)
- Linux glibc 2.28+ (E.g. Ubuntu 20.04+) aarch64 (ARMv8) platforms (See: https://github.com/pypa/manylinux for details)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hjmjohnson I understand your frustration. There is a lot of complexity. This naturally arises from the need to support ARM64, AMD64, MacOS, Linux, Windows, a stable version that most people are using in production to address a number of important real-world uses cases, and a development version where developers that make experimental improvements the long haul, and a GitHub infrastructure that we can all collaborate on. And we are continuously making strides to reduce that complexity as much as possible.

And consolidation of some configuration files in the ITK repository could be a good step forward, where that makes sense.

However, please do not go about breaking existing infrastructure that does work.

This will slow you down much more and it slows everyone else down, too. Burning everything down due to frustration with the status quo is not a recipe to make progress better or faster.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants