[ci] Example Dockerfiles install from source / PyPI

[ryan-williams]

Changes

docker/: example Dockerfiles that pip install tiledbsoma

• Contains Ubuntu and Debian examples, includes system deps.
• README.md includes build-debugging tips:
  • pip install --no-clean tiledbsoma
  • VCPKG_FORCE_SYSTEM_BINARIES=1 (required by vcpkg on ARM)

Add python-dockers.yml GHA

Tests building and running docker/*.dockerfile:

• Runs "nightly" on Ubuntu 24.04, AMD and ARM.
• Verifies python -c 'import tiledbsoma; tiledbsoma.show_package_versions()' or python scripts/show-versions.py on each image.

Improve GHA path-filters

I also improved GHAs' path-filtering, so they don't run on changes to other workflow .ymls or docker/**.

However, path-filters apparently don't take effect on the PR that adds them, so I canceled all unrelated GHAs here, to save runner quota.

Notes for Reviewer

• Does "nightly" seem like the right schedule? "Every PR" felt like overkill.
• Are there other base images we should add?

Punted: macos Docker builds

• I tried to test building Docker images on macOS here, but that may not be possible in GHAs, currently.
• "Nested virtualization" is only supported on macOS-15 and ≥M3, but the current macos-15 runners only have M1 (or possibly M2 Pro?) chips.
• Instead, python-dockers.yml builds+runs on Ubuntu ARM and AMD platforms.

Example failure

https://github.com/single-cell-data/TileDB-SOMA/actions/runs/13714468862/job/38356765298:

colima start fails with:

colima version 0.8.1
git commit: 96598cc5b64e5e9e1e64891642b91edc8ac49d16
time="2025-03-07T05:34:05Z" level=info msg="starting colima"
time="2025-03-07T05:34:05Z" level=info msg="runtime: containerd"
time="2025-03-07T05:34:08Z" level=info msg="creating and starting ..." context=vm
time="2025-03-07T05:34:09Z" level=info msg="downloading disk image ..." context=vm

                                                                           0.0%
#                                                                          1.5%
###                                                                        5.1%
######                                                                     9.5%
#########                                                                 12.7%
##########                                                                15.2%
#############                                                             18.8%
################                                                          22.8%
###################                                                       27.2%
#####################                                                     30.5%
########################                                                  34.0%
###########################                                               38.0%
##############################                                            42.9%
#################################                                         46.9%
###################################                                       49.4%
#####################################                                     52.7%
#######################################                                   55.0%
#########################################                                 58.3%
#############################################                             63.2%
################################################                          66.7%
##################################################                        70.5%
#####################################################                     74.1%
########################################################                  78.6%
###########################################################               83.0%
##############################################################            86.6%
#################################################################         91.3%
####################################################################      95.7%
#######################################################################   99.4%
######################################################################## 100.0%
time="2025-03-07T05:34:14Z" level=info msg="Terminal is not available, proceeding without opening an editor"
time="2025-03-07T05:34:15Z" level=info msg="Starting the instance \"colima\" with VM driver \"vz\""
time="2025-03-07T05:34:15Z" level=info msg="Attempting to download the image" arch=aarch64 digest= location=/Users/runner/Library/Caches/colima/caches/fc55147f0d53562b3fca013da024db1e250eaeb[60](https://github.com/single-cell-data/TileDB-SOMA/actions/runs/13714468862/job/38356765298#step:3:61)029e2632c0867d86f40bb25
time="2025-03-07T05:34:15Z" level=info msg="Downloaded the image from \"/Users/runner/Library/Caches/colima/caches/fc55147f0d535[62](https://github.com/single-cell-data/TileDB-SOMA/actions/runs/13714468862/job/38356765298#step:3:63)b3fca013da024db1e250eaeb60029e2632c0867d86f40bb25\""
time="2025-03-07T05:34:15Z" level=info msg="Converting \"/Users/runner/.colima/_lima/colima/basedisk\" (qcow2) to a raw disk \"/Users/runner/.colima/_lima/colima/diffdisk\""
time="2025-03-07T05:34:19Z" level=info msg="Expanding to 100GiB"
time="2025-03-07T05:34:20Z" level=info msg="[hostagent] hostagent socket created at /Users/runner/.colima/_lima/colima/ha.sock"
time="2025-03-07T05:34:20Z" level=info msg="[hostagent] Starting VZ (hint: to watch the boot progress, see \"/Users/runner/.colima/_lima/colima/serial*.log\")"
time="2025-03-07T05:34:20Z" level=fatal msg="exiting, status={Running:false Degraded:false Exiting:true Errors:[] SSHLocalPort:0} (hint: see \"/Users/runner/.colima/_lima/colima/ha.stderr.log\")"
time="2025-03-07T05:34:20Z" level=fatal msg="error starting vm: error at 'creating and starting': exit status 1"
Error: Process completed with exit code 1.

If we cat the ha.stderr.log before exiting:

{"level":"debug","msg":"ResolveVMType: resolved VMType \"vz\" (explicitly specified in []*LimaYAML{o,y,d}[1])","time":"2025-03-07T05:35:59Z"}
{"level":"debug","msg":"Creating iso file /Users/runner/.colima/_lima/colima/cidata.iso","time":"2025-03-07T05:36:00Z"}
{"level":"debug","msg":"Using /var/folders/r4/ph658m2d3vs0vj00g75q0zyr0000gn/T/diskfs_iso2776112454 as workspace","time":"2025-03-07T05:36:00Z"}
{"level":"debug","msg":"Failed to detect CPU features. Assuming that AES acceleration is available on this Apple silicon.","time":"2025-03-07T05:36:00Z"}
{"level":"debug","msg":"OpenSSH version 9.8.1 detected","time":"2025-03-07T05:36:00Z"}
{"level":"debug","msg":"AES accelerator seems available, prioritizing [email protected] and [email protected]","time":"2025-03-07T05:36:00Z"}
{"level":"info","msg":"hostagent socket created at /Users/runner/.colima/_lima/colima/ha.sock","time":"2025-03-07T05:36:00Z"}
{"level":"info","msg":"Starting VZ (hint: to watch the boot progress, see \"/Users/runner/.colima/_lima/colima/serial*.log\")","time":"2025-03-07T05:36:00Z"}
{"level":"debug","msg":"Kernel file \"/Users/runner/.colima/_lima/colima/kernel\" not found","time":"2025-03-07T05:36:00Z"}
{"level":"debug","msg":"Using EFI Boot Loader","time":"2025-03-07T05:36:00Z"}
{"level":"fatal","msg":"Error Domain=VZErrorDomain Code=2 Description=\"Invalid virtual machine configuration. Virtualization is not available on this hardware.\" UserInfo={\n    NSLocalizedFailure = \"Invalid virtual machine configuration.\";\n    NSLocalizedFailureReason = \"Virtualization is not available on this hardware.\";\n}","time":"2025-03-07T05:36:00Z"}

(source)

xrefs

actions/runner-images/#2150 (Add Docker to macOS)

I am closing the issue since we can't bake this tool into the image due to license issue. Hope something will be changed in future.

discussions#69211 (GitHub Actions: Apple silicon (M1) macOS runners are now available in public beta!)

That is correct. As indicated in our docs, this is a limitation of Apple's Virtualization Framework, which our hypervisor uses. Nested-virtualization is not supported by arm64 runners. You can run nested virtualization using larger Linux runners:
https://github.blog/changelog/2023-02-23-hardware-accelerated-android-virtualization-on-actions-windows-and-linux-larger-hosted-runners/

douglascamata/setup-docker-macos-action

Apple is adding support for nested virtualization on M3 processors and beyond on macOS 15 (see Apple Developer docs). This action will be updated to support it as soon as Github starts to update the runners accordingly.

setup-docker-macos-action#35

It's impossible to support macOS 15 because it runs on arm64 and we don't have nested virtualization enabled on M2 Pro (the latest chip GH action workers use).

actions/runner-images#6216 (comment)

does any of you know why this is removed from macos 13?
it's also missing on the new macos-14 image as well

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.17%. Comparing base (aefd8ed) to head (f858e1d).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3763      +/-   ##
==========================================
+ Coverage   89.15%   89.17%   +0.01%     
==========================================
  Files          54       54              
  Lines        6419     6419              
==========================================
+ Hits         5723     5724       +1     
+ Misses        696      695       -1     

Flag	Coverage Δ
python	89.17% <ø> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
python_api	89.17% <ø> (+0.01%)	⬆️
libtiledbsoma	∅ <ø> (∅)

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[johnkerl]

Does "nightly" seem like the right schedule? "Every PR" felt like overkill.

@ryan-williams nightly sounds delightful! :)

[ryan-williams]

Note this addition to OP:

Improve GHA path-filters

I also improved GHAs' path-filtering, so they don't run on changes to other workflow .ymls or docker/**.

However, path-filters apparently don't take effect on the PR that adds them, so I canceled all unrelated GHAs here, to save runner quota.

The new GHA passed:

I canceled the other GHAs, hence the ❌.