GitHub Actions for Windows-latest don't run tests

[dirkf]

Checklist

• I've verified that I'm running GitHub master 7009bb9
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar bug reports including closed ones
• I've read bugs section in FAQ

Verbose log

Download test extract:

2022-09-22T23:06:49.5018407Z ##[group]Run ./devscripts/run_tests.bat
2022-09-22T23:06:49.5018813Z �[36;1m./devscripts/run_tests.bat�[0m
2022-09-22T23:06:49.5061425Z shell: C:\Program Files\PowerShell\7\pwsh.EXE -command ". '{0}'"
2022-09-22T23:06:49.5061792Z env:
2022-09-22T23:06:49.5062098Z   pythonLocation: C:\hostedtoolcache\windows\Python\3.2.5\x64
2022-09-22T23:06:49.5062456Z   YTDL_TEST_SET: download
2022-09-22T23:06:49.5062752Z ##[endgroup]
2022-09-22T23:06:49.8525808Z 
2022-09-22T23:06:49.8526899Z D:\a\youtube-dl\youtube-dl>rem Keep this list in sync with the `offlinetest` target in Makefile 
2022-09-22T23:06:49.8527471Z 
2022-09-22T23:06:49.8528040Z D:\a\youtube-dl\youtube-dl>set DOWNLOAD_TESTS="age_restriction^|download^|iqiyi_sdk_interpreter^|socks^|subtitles^|write_annotations^|youtube_lists^|youtube_signature" 
2022-09-22T23:06:49.8531304Z 
2022-09-22T23:06:49.8531770Z D:\a\youtube-dl\youtube-dl>if "download" == "core" (
2022-09-22T23:06:49.8532497Z set test_set="-I test_(""age_restriction^|download^|iqiyi_sdk_interpreter^|socks^|subtitles^|write_annotations^|youtube_lists^|youtube_signature"")\.py"  
2022-09-22T23:06:49.8533099Z  set multiprocess_args="" 
2022-09-22T23:06:49.8533423Z )  else if "download" == "download" (
2022-09-22T23:06:49.8534010Z set test_set="-I test_(?!""age_restriction^|download^|iqiyi_sdk_interpreter^|socks^|subtitles^|write_annotations^|youtube_lists^|youtube_signature"").+\.py"  
2022-09-22T23:06:49.8534742Z  set multiprocess_args="--processes=4 --process-timeout=540" 
2022-09-22T23:06:49.8535048Z )  else (
2022-09-22T23:06:49.8535278Z echo YTDL_TEST_SET is not set or invalid  
2022-09-22T23:06:49.8535510Z  exit /b 1 
2022-09-22T23:06:49.8535678Z ) 
2022-09-22T23:06:49.8538596Z 
2022-09-22T23:06:49.8539309Z D:\a\youtube-dl\youtube-dl>nosetests test --verbose -I test_(?!age_restriction|download|iqiyi_sdk_interpreter|socks|subtitles|write_annotations|youtube_lists|youtube_signature).+\.py --processes=4 --process-timeout=540 
2022-09-22T23:06:51.2059548Z 
2022-09-22T23:06:51.2559790Z ----------------------------------------------------------------------
2022-09-22T23:06:51.2560988Z Ran 0 tests in 0.376s
2022-09-22T23:06:51.2582071Z 
2022-09-22T23:06:51.2582970Z OK

Core test similar apart from this line, which shows that the batch file logic is correct:

2022-09-22T23:06:15.5233910Z D:\a\youtube-dl\youtube-dl>nosetests test --verbose -I test_(age_restriction|download|iqiyi_sdk_interpreter|socks|subtitles|write_annotations|youtube_lists|youtube_signature)\.py  

Description

Apparently the GitHub CI tests, neither core nor download, haven't been running on the Windows-latest platform for some longish time (months?). The relevant action log extract above is what GH shows when the ./devscripts/run_tests.bat is modified with @echo on.

Is it just this:

2022-09-22T23:06:49.5061425Z shell: C:\Program Files\PowerShell\7\pwsh.EXE -command ". '{0}'"

But supposedly GH made PowerShell the default at least 2 years ago, which seems too long for that to be the problem.

The batch file is running. The nose output shows that it just doesn't find the tests. The code that @dstftw committed is unchanged since switching to GH Actions so presumably the command-line escaping of eg ( and ! must have worked then (logs expired).

Any ideas?

[dirkf]

This is how GitHub Actions runs the commands that you specify for the steps of the action, ie {0} is replaced by the command. The Unix tests have

shell: /bin/bash -e {0}

See https://github.blog/changelog/2019-10-17-github-actions-default-shell-on-windows-runners-is-changing-to-powershell/.

It's possible to set the default shell to bash for all test platforms since GH includes a bash in its Windows builds. If you do that (and run the .sh version of the tests), again nothing happens:

Tests (windows-latest, 3.2, cpython, download, sh)
...
2022-09-25T08:45:25.3864717Z ##[group]Run ./devscripts/run_tests.sh
2022-09-25T08:45:25.3865180Z �[36;1m./devscripts/run_tests.sh�[0m
2022-09-25T08:45:25.3876386Z shell: C:\Program Files\Git\bin\bash.EXE --noprofile --norc -e -o pipefail {0}
2022-09-25T08:45:25.3876706Z env:
2022-09-25T08:45:25.3877035Z   pythonLocation: C:\hostedtoolcache\windows\Python\3.2.5\x64
2022-09-25T08:45:25.3877382Z   YTDL_TEST_SET: download
2022-09-25T08:45:25.3877606Z ##[endgroup]
2022-09-25T08:45:26.6952787Z 
2022-09-25T08:45:26.6955131Z ----------------------------------------------------------------------
2022-09-25T08:45:26.6960038Z Ran 0 tests in 0.391s
2022-09-25T08:45:26.6982769Z 
2022-09-25T08:45:26.6983705Z OK

So maybe the problem is that the Windows commands aren't finding the test scripts.

[gamer191]

The code that @dstftw committed is unchanged since switching to GH Actions so presumably the command-line escaping of eg ( and ! must have worked then (logs expired).

You can tell whether it worked based on how long the tests took. Based on that, it broke sometime between https://github.com/ytdl-org/youtube-dl/actions/runs/1885214664 and https://github.com/ytdl-org/youtube-dl/actions/runs/1892945259

You may also notice that the warning:

windows-latest workflows will use windows-2022 soon. For more details, see https://github.com/actions/virtual-environments/issues/4856

is present for the first link, but not the second

So I'm almost certain (but don't have time to test right now) that pinning the workflow to windows-2019 would fix this issue

[dirkf]

YES!

I observed a bug in nose where it doesn't find any tests because it normalises the path and case of the test filenames in two incompatible ways. Quite possibly this might have been a latent bug revealed by the newer version of the Windows runner.

After unsuccessfully trying to deploy modified versions of nose to test this, I came to the conclusion that yt-dlp's adoption of pytest was the answer. But in the short term this is a lot simpler:

--- old/youtube-dl/.github/workflows/ci.yml
+++ new/youtube-dl/.github/workflows/ci.yml
@@ -15,12 +15,12 @@
         run-tests-ext: [sh]
         include:
         # python 3.2 is only available on windows via setup-python
-        - os: windows-latest
+        - os: windows-2019
           python-version: 3.2
           python-impl: cpython
           ytdl-test-set: core
           run-tests-ext: bat
-        - os: windows-latest
+        - os: windows-2019
           python-version: 3.2
           python-impl: cpython
           ytdl-test-set: download