Staging site no longer finds components without registry_url in idf_component.yml (PACMAN-1062)

[gojimmypi]

The Component Manager version

2.1.1

ESP-IDF Version

v5.2

python Version

Python 3.10.12

Operating System

WSL

Browser (for https://components.espressif.com Issues)

No response

Description

This issue is related to #78, in particular #78 (comment).

TL;DR: without explicit registry_url: https://components-staging.espressif.com in main/idf_component.yml, staging site components are not found.

I believe this was introduced in a recent version of the idf_component_manager. I'm using v2.1.1

I can fix this moving forward, but all prior staging site components are now inoperable.

To Reproduce

Details

Without the IDF_COMPONENT_REGISTRY_URL staging site setting, the component is of course not found at https://components.espressif.com/

$ idf.py create-project-from-example "gojimmypi/mywolfssh=1.4.19-preview1c:wolfssh_echoserver"
Executing action: create-project-from-example
ERROR: Component "gojimmypi/mywolfssh" not found
gojimmypi:/mnt/c/testc
$  echo $IDF_COMPONENT_REGISTRY_URL

And of course with a IDF_COMPONENT_REGISTRY_URL staging site value, the library is found:

gojimmypi:/mnt/c/testc
$ idf.py create-project-from-example "gojimmypi/mywolfssh=1.4.19-preview1c:wolfssh_echoserver"
Executing action: create-project-from-example
NOTICE: Example "wolfssh_echoserver" successfully downloaded to /mnt/c/testc/wolfssh_echoserver
Done
gojimmypi:/mnt/c/testc
$ echo $IDF_COMPONENT_REGISTRY_URL
https://components-staging.espressif.com

BUT - try the same thing with a prior version, mywolfssh=1.4.19-preview1a:

gojimmypi:/mnt/c/testc
$ cd test1a
gojimmypi:/mnt/c/testc/test1a
$ idf.py create-project-from-example "gojimmypi/mywolfssh=1.4.19-preview1a:wolfssh_echoserver"
Executing action: create-project-from-example
NOTICE: Example "wolfssh_echoserver" successfully downloaded to /mnt/c/testc/test1a/wolfssh_echoserver
Done

The project is fetched, but no component source code yet. (fetched at first-build time)

$ cd test1a
gojimmypi:/mnt/c/testc/test1a


$ idf.py create-project-from-example "gojimmypi/mywolfssh=1.4.19-preview1a:wolfssh_echoserver"
Executing action: create-project-from-example
NOTICE: Example "wolfssh_echoserver" successfully downloaded to /mnt/c/testc/test1a/wolfssh_echoserver
Done


gojimmypi:/mnt/c/testc/test1a
$ ls
wolfssh_echoserver
gojimmypi:/mnt/c/testc/test1a
$ ls wolfssh_echoserver
CMakeLists.txt  README.md  VisualGDB  main  partitions_singleapp_large.csv  sdkconfig.defaults
gojimmypi:/mnt/c/testc/test1a
$ ls wolfssh_echoserver/main
CMakeLists.txt  echoserver.c  idf_component.yml  include  main.c  time_helper.c  wifi_connect.c
gojimmypi:/mnt/c/testc/test1a

These are known to be bad version, we'll edit, later:

$ cat wolfssh_echoserver/main/idf_component.yml
dependencies:
  gojimmypi/mywolfssh: ^1.4.17-Preview6
  gojimmypi/mywolfssl: ^5.7.1-Preview2d
  idf:
    version: '>=4.1.0'
gojimmypi:/mnt/c/testc/test1a

Edit the wolfssh_echoserver/main/idf_component.yml with recent libraries known to exist:

$ nano wolfssh_echoserver/main/idf_component.yml
gojimmypi:/mnt/c/testc/test1a
$ cat wolfssh_echoserver/main/idf_component.yml
dependencies:
  gojimmypi/mywolfssh: ^1.4.19-preview1c
  gojimmypi/mywolfssl: ^5.7.4-preview1i
  idf:
    version: '>=4.1.0'
gojimmypi:/mnt/c/testc/test1a

See

• gojimmypi/mywolfssh/versions/1.4.19-preview1c
• gojimmypi/mywolfssl/versions/5.7.4-preview1i

gojimmypi:/mnt/c/testc/test1a
$ cd wolfssh_echoserver
gojimmypi:/mnt/c/testc/test1a/wolfssh_echoserver
$ idf.py build
Executing action: all (aliases: build)
Running cmake in directory /mnt/c/testc/test1a/wolfssh_echoserver/build

[ ... snip ... ]

NOTICE: Dependencies lock doesn't exist, solving dependencies.
.HINT: Please check manifest file of the following component(s): main
CMake Error at /mnt/c/SysGCC/esp32/esp-idf/v5.2/tools/cmake/build.cmake:544 (message):
  WARNING: Component "gojimmypi/mywolfssh" not found

  WARNING: Component "gojimmypi/mywolfssl" not found

  ERROR: Because project depends on gojimmypi/mywolfssl (^5.7.4-preview1i)
  which doesn't match any versions, version solving failed.

Call Stack (most recent call first):
  /mnt/c/SysGCC/esp32/esp-idf/v5.2/tools/cmake/project.cmake:605 (idf_build_process)
  CMakeLists.txt:122 (project)


-- Configuring incomplete, errors occurred!
See also "/mnt/c/testc/test1a/wolfssh_echoserver/build/CMakeFiles/CMakeOutput.log".
cmake failed with exit code 1, output of the command is in the /mnt/c/testc/test1a/wolfssh_echoserver/build/log/idf_py_stderr_output_4255 and /mnt/c/testc/test1a/wolfssh_echoserver/build/log/idf_py_stdout_output_4255


Edit the `main/idf_component.yml` to explicitly name the `registry_url` (even though we already have `IDF_COMPONENT_REGISTRY_URL` set)


gojimmypi:/mnt/c/testc/test1a/wolfssh_echoserver
$ cat main/idf_component.yml
dependencies:
  gojimmypi/mywolfssh:
    version: ^1.4.19-preview1c
    registry_url: https://components-staging.espressif.com
  gojimmypi/mywolfssl:
    version: ^5.7.4-preview1i
    registry_url: https://components-staging.espressif.com
  idf:
    version: '>=4.1.0'

$ echo $IDF_COMPONENT_REGISTRY_URL
https://components-staging.espressif.com


Observe components now all found, and project builds successfully:


gojimmypi:/mnt/c/testc/test1a/wolfssh_echoserver
$ idf.py build
Executing action: all (aliases: build)
Running cmake in directory /mnt/c/testc/test1a/wolfssh_echoserver/build
Executing "cmake -G Ninja -DPYTHON_DEPS_CHECKED=1 -DPYTHON=/home/gojimmypi/.espressif/python_env/idf5.2_py3.10_env/bin/python -DESP_PLATFORM=1 -DCCACHE_ENABLE=0 /mnt/c/testc/test1a/wolfssh_echoserver"...

[ ... snip ... ]

-- Building ESP-IDF components for target esp32
NOTICE: Dependencies lock doesn't exist, solving dependencies.
...NOTICE: Updating lock file at /mnt/c/testc/test1a/wolfssh_echoserver/dependencies.lock
NOTICE: Processing 3 dependencies:
NOTICE: [1/3] gojimmypi/mywolfssh (1.4.19-preview1c)
NOTICE: [2/3] gojimmypi/mywolfssl (5.7.4-preview7aj)
NOTICE: [3/3] idf (5.2.0)

[ ... snip ... ]

[1053/1054] Generating binary image from built executable
esptool.py v4.8.1
Creating esp32 image...
Merged 2 ELF sections
Successfully created esp32 image.
Generated /mnt/c/testc/test1a/wolfssh_echoserver/build/wolfssh_echoserver.bin
[1054/1054] cd /mnt/c/testc/test1a/wolfssh_echoserver/build/es...c/testc/test1a/wolfssh_echoserver/build/wolfssh_echoserver.bi
wolfssh_echoserver.bin binary size 0xf7e40 bytes. Smallest app partition is 0x177000 bytes. 0x7f1c0 bytes (34%) free.

Project build complete. To flash, run:
 idf.py flash
or
 idf.py -p PORT flash
or
 python -m esptool --chip esp32 -b 460800 --before default_reset --after hard_reset write_flash --flash_mode dio --flash_size 2MB --flash_freq 40m 0x1000 build/bootloader/bootloader.bin 0x8000 build/partition_table/partition-table.bin 0x10000 build/wolfssh_echoserver.bin
or from the "/mnt/c/testc/test1a/wolfssh_echoserver/build" directory
 python -m esptool --chip esp32 -b 460800 --before default_reset --after hard_reset write_flash "@flash_args"

This is all with the same ISP-IDF and component manager versions:

$  idf.py --version
ESP-IDF v5.2-dev-3903-g66992aca7a-dirty
gojimmypi:/mnt/c/testc/test1a/wolfssh_echoserver
$ python -m idf_component_manager --help
Usage: python -m idf_component_manager [OPTIONS] COMMAND [ARGS]...

Options:
  -W, --warnings-as-errors  Treat warnings as errors.
  -h, --help                Show this message and exit.

Commands:
  autocomplete  Generate tab-completion scripts for the specified shell.
  cache         Group of commands to manage cache of the IDF Component...
  component     Group of commands to interact with components.
  manifest      Group of commands to manage manifest of the project.
  project       Group of project related commands
  registry      Group of commands to work with component registry
  version       Print version of the IDF Component Manager.
gojimmypi:/mnt/c/testc/test1a/wolfssh_echoserver
$ python -m idf_component_manager version
2.1.1

Expected behaviour

Previously, staging components would be found without registry_url setting in idf_component.yml.

If there's a IDF_COMPONENT_REGISTRY_URL environment setting, that should be the default for missing registry_url values.

It is a potentially cool feature (mix staging and release components? alternate, provider-specific sources?).

Additional info

See #78 (comment)

Test on Staging Registry Documentation at

https://docs.espressif.com/projects/idf-component-manager/en/latest/guides/packaging_components.html#test-on-staging-registry

I have checked existing issues and online Documentation

• I confirm I have checked existing issues and online Documentation.