ENH: Allow environments from docker build contexts

[samb-t]

Currently environments can only be constructed from conda files. This PR allows adds the optional argument build_context to submit_to_azure_if_needed which can either be a DockerBuildContext for SDK v1 or a BuildContext for SDK v2.

Build contexts are dockerfiles or folders containing a dockerfile that set up the docker environment.

For example to set up uv, the following dockerfile can be used

FROM mcr.microsoft.com/azureml/openmpi4.1.0-cuda11.8-cudnn8-ubuntu22.04
WORKDIR /workspace
RUN curl -LsSf https://astral.sh/uv/install.sh | sh
ENV PATH="/root/.local/bin:$PATH"
CMD ["/bin/bash"]

The build context folder could contain a pyproject.toml file and uv sync added to the dockerfile. Alternatively, uv sync can be run at the start of a job, or just uv run can be used to execute the script and install all depencies.

The argument python_launch_command has been added to submit_to_azure_if_needed. It defaults to "python" and is used to form the command to be run in AzureML. Expected uses would be "torchrun" or "uv run".

Tests for new/modified functions:

• load_and_hash_directory
• create_run_configuration
• create_script_run
• submit_run_v2
• submit_to_azure_if_needed

[fepegar]

Added a couple of comments and re-triggered the failing CI job.

[ant0nsc]

Thanks a lot for making those changed! Support for Docker is great to have.
Left a few questions, mostly around: Do we really need to add another argument python_exec?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.92%. Comparing base (f4e21b7) to head (b541757).
Report is 12 commits behind head on main.

Additional details and impacted files

Flag	Coverage Δ
hi-ml	?
hi-ml-azure	?
hi-ml-cpath	70.92% <ø> (-5.30%)	⬇️
hi-ml-multimodal	?

Flags with carried forward coverage won't be shown. Click here to find out more.

see 87 files with indirect coverage changes

[samb-t]

For awareness, I have changed how the section of code that creates an Environment from an SDK v1 DockerBuildContext. It turns out for SDK v1 the class stores the location of the context in blob rather than the the local path. As such, I changed it to create the environment name by hashing the URI. This is potentially problematic if the underlying files change - the environment won't update automatically. Better solutions welcome

hi-ml/hi-ml-azure/src/health_azure/himl.py

Lines 266 to 276 in 9c12e58

	elif docker_build_context is not None:
	# Set the name to be the name of the directory in blob storage
	location = docker_build_context.location
	environment_name = generate_unique_environment_name(location)
	new_environment = Environment.from_docker_build_context(
	name=environment_name,
	docker_build_context=docker_build_context,
	)
	registered_env = register_environment(workspace, new_environment)
	environment = registered_env
	elif conda_environment_file is not None:

[fepegar]

This is potentially problematic if the underlying files change

The chances of this are probably tiny, so I wouldn't worry.

[fepegar]

Thanks, @samb-t!