Redesign dependency-list creation for github prepare action #6255

hgy59 · 2024-10-05T20:26:39Z

Description

This redesign speedsup the evaluation of dependent packages in github prepare action.
Instead of 3 to 5 minutes it now takes only a few seconds to create the dependency list that is required.

add dependency-list.sh script to evaluate the dependencies
use new script in global Makefile for the dependency-list target
use global Makefile to create dependency-list in prepare.sh
avoid variables in DEPENDS and BUILD_DEPENDS definitions

This script is designed to be used for the github build action only, to evaluate the packages to build and download sources for.
We still need the dependency-list target to find real dependencies for packages to build. The script uses all dependencies (it doesn't care about conditional dependencies) but when building packages we need the real dependencies and must not built dependencies not supported for specific archs or DSM versions.

Mandatory framework changes

makefile variables like $(SPK_NAME) are not allowed in dependency definitions anymore

Checklist

Build rule all-supported completed successfully
New installation of package completed successfully
Package upgrade completed successfully (Manually install the package again)
Package functionality was tested
Any needed documentation is updated/created

Type of change

Includes framework changes

- add script to evaluate the dependencies - use new script in global Makefile for the dependency-list target - use global Makefile to create dependency-list in prepare.sh - avoid variables in DEPENDS definitions - make dependency-list.sh executable

hgy59 · 2024-10-05T21:36:44Z

@th0ma7 I want to remove some obsolete dependencies.

Can't we remove native/nasm and native/yasm since those are prerequisites for the development environment (and included in the spksrc docker image)?

The same applies to native/cmake but here you have recent updates to intel-vc-intrinsics that needs USE_NATIVE_CMAKE, so I guess we must keep native/cmake.

Another one is native/cmake-legacy. This one is used with USE_NATIVE_CMAKE_LEGACY and is used for mysql-connector-c only.
Probably we can drop the mysql-connector and use mariadb-connector with mysql compatibility (CMAKE_ARGS += -DWITH_MYSQLCOMPAT=ON) instead?

th0ma7 · 2024-10-06T13:05:28Z

Can't we remove native/nasm and native/yasm since those are prerequisites for the development environment (and included in the spksrc docker image)?

yes certainly. although a fair bit of testing may be needed along with adapting the framework. Just to manage expectations, I can certainly review code but my next focus really needs to be python...

The same applies to native/cmake but here you have recent updates to intel-vc-intrinsics that needs USE_NATIVE_CMAKE, so I guess we must keep native/cmake.

That's probably an oversight as I updated our debian image during the time I was workding on intel opencl and this may have be there while using the older image. I'll check if that can be removed.

Another one is native/cmake-legacy. This one is used with USE_NATIVE_CMAKE_LEGACY and is used for mysql-connector-c only. Probably we can drop the mysql-connector and use mariadb-connector with mysql compatibility (CMAKE_ARGS += -DWITH_MYSQLCOMPAT=ON) instead?

Feel free to remove that altogether. I've been trying to keep mysql-connector-c alive as prerequisite for other packages that i ended-up migrating, mainly related to python if i recall. There used to be chromaprint but I believe it's now solved.

Although my 2 cents are, cmake native and legacy have proved to be quite useful. I'd be tempted to keep the code for newer cmake as i'm sure we'll hit this again later in a year or two.

Also, I believe it would be time to start stripping off broken packages that never received any love since pre-DSM6, along with deleting them from our repository online. As well, I would remove all the older toolchains as no longer needed and clean-up the code related to tc and tk accordingly. This would reduce the framework weight for a bit and accelerated archs evaluation.

hgy59 · 2024-10-06T13:39:10Z

@th0ma7 thank you for the feedback.

I think I will hold back this PR for prior creating some smaller PRs for:

removal of variables in DEPENDS definitions (multiple PRs to touch less packages in each)
removal of native/nasm native/yasm
removal of python2
...

Then, later on, in this PR:

use scripts for all dependency lists (dependency-list and dependency-flat)
remove dependency-tree. It is not very usefull (not readable) with such a lot of dependencies. It was the only source for dependency evaluation before we introduced dependency-list.

Finally, to remove OPTIONAL_DEPENDS, I would prefer separated PRs again.

…n_dependency_list

- incr. package revision - avoid variable in DEPENDS (for SynoCommunity#6255) - restore ownership of package directories for SRM/DSM < 7

…n_dependency_list

* tvheadend: fix path to ffmpeg(7) * tvheadend: minor adjustments - incr. package revision - avoid variable in DEPENDS (for #6255) - restore ownership of package directories for SRM/DSM < 7 * update changelog

Co-authored-by: hgy59 <[email protected]>

…n_dependency_list

- libicu 74 is the latest version that builds for DSM 6 - remove make variable in DEPENDS for SynoCommunity#6255

- avoid make variables in DEPENDS for SynoCommunity#6255

- libicu 74 is the latest version that builds for DSM 6 - remove make variable in DEPENDS for #6255

…n_dependency_list

- some changes taken from PR SynoCommunity#6255 to reduce number of dependent packages - changes that trigger ffmpeg* - changes that trigger python311 (except vim, separated in PR SynoCommunity#6398)

- some changes taken from PR #6255 to reduce number of dependent packages - changes that trigger ffmpeg* - changes that trigger python311 (except vim, separated in PR #6398)

…n_dependency_list

- follow up to SynoCommunity#6004 - PLIST must contain own files only - update revision.h patching - remove variable in DEPENDS for SynoCommunity#6255

* gpac: fix build - follow up to #6004 - PLIST must contain own files only - update revision.h patching - remove variable in DEPENDS for #6255

…n_dependency_list

- avoid conflicting changes with SynoCommunity#6366

- avoid conflicting changes with SynoCommunity#6398

- avoid conflicting changes with SynoCommunity#6394

hgy59 · 2025-01-18T18:48:06Z

@th0ma7, @mreid-tt This is now ready to merge.

Are there any objections?

mreid-tt · 2025-01-18T19:00:04Z

@hgy59, no objections from me (but I haven't reviewed it in great detail).

With regard to the failures for lirc, I assume this is because the Makefile is too old? Should the package be marked as broken?

hgy59 · 2025-01-18T19:11:33Z

With regard to the failures for lirc, I assume this is because the Makefile is too old? Should the package be marked as broken?

lirc is a package that depends on kernel sources (REQUIRE_KERNEL = 1), so it is expected to fail when building for generic archs.
And it was never mirgated to DSM 7 (still using the deprecated installer script).
And I guess it will never run on DSM 7 due to restrictions for USB devices.

It successfully builds for hi3535 and evansport for DSM 6.2.4.
qoriq is not supported and ARMv5 (88f6281) is lacking the kernel sources (this might be an issue).

licr has no other dependencies and is only triggerd to build when cross/lirc* or spk/lirc was changed, so no need to declare the package as broken.

th0ma7

Lgtm, script code is nice and clean, well done. Maybe missing in your sort is a -u in case there are duplicates such as eventually depending on both ffmpeg and python.

Also, i would suggest modifying our wiki (no clue where?) for the cross/appname to make that mandatory that they must match and no makefile variables are to be used.

Lastly, having any timing difference between run with your new script vs old method? (On my cell phone so maybe you already provided in your pr description).

hgy59 · 2025-01-18T20:13:49Z

Lastly, having any timing difference between run with your new script vs old method? (On my cell phone so maybe you already provided in your pr description).

Yes it is in the description
Dependency evaluation is now typ. 8-10 sec (vs 5-6 min)

hgy59 · 2025-01-18T20:22:58Z

Lgtm, script code is nice and clean, well done. Maybe missing in your sort is a -u in case there are duplicates such as eventually depending on both ffmpeg and python.

The functions get_python_dependency and get_ffmpeg_dependency add only cross/python* and cross/ffmpeg* to the depencencies.
Those are added for futher evaluation and duplicates will be removed (get_dependencies always removes duplicates and is called until no additional deps are found)

hgy59 · 2025-01-18T21:09:39Z

Also, i would suggest modifying our wiki (no clue where?) for the cross/appname to make that mandatory that they must match and no makefile variables are to be used.

now added to makefile variables wiki page at https://github.com/SynoCommunity/spksrc/wiki/Makefile-variables

- was not updated in SynoCommunity#6255

* fix dependency-list.sh - script was lacking some dependencies (not recursing into all dependent Makefiles) * cross/ntopng: avoid variable in DEPENDS - was not updated in #6255 * speedup dependency-list.sh - remove duplicates in cumulated dependencies early * optimize dependency-list script - remove obsolete unique filtering - apply python and ffmpeg dependencies on top level only

hgy59 added the framework label Oct 5, 2024

hgy59 mentioned this pull request Oct 7, 2024

github action: omit broken packages in dependency-list #6261

Merged

6 tasks

Merge branch 'master' of https://github.com/hgy59/spksrc into redesig…

93fd1d5

…n_dependency_list

th0ma7 mentioned this pull request Oct 8, 2024

vc-intrinsics: Update to v0.20.1 and remove unused USE_NATIVE_CMAKE #6264

Merged

10 tasks

This was referenced Oct 14, 2024

dynamic crossenv + python3*-wheels + python310-313 updates #6282

Merged

Creation of a new spk fails to trigger build #6284

Closed

ps3netsrv: Update to ps3netsrv 20240709 #6213

Merged

This was referenced Oct 28, 2024

Jellyfin: Update to v10.10.0 #6293

Merged

bazarr: update 1.4.2 => 1.4.5 and migrate to ffmpeg 7 #6164

Merged

hgy59 added a commit to hgy59/spksrc that referenced this pull request Nov 3, 2024

tvheadend: minor adjustments

a1d82f7

- incr. package revision - avoid variable in DEPENDS (for SynoCommunity#6255) - restore ownership of package directories for SRM/DSM < 7

Merge branch 'master' of https://github.com/hgy59/spksrc into redesig…

4313817

…n_dependency_list

hgy59 mentioned this pull request Jan 6, 2025

Feng Office: Update to v3.11.4.14 #6377

Merged

6 tasks

mreid-tt added a commit to mreid-tt/spksrc that referenced this pull request Jan 6, 2025

Avoid makefile variables in DEPENDS for SynoCommunity#6255

61f966f

Co-authored-by: hgy59 <[email protected]>

mreid-tt added a commit to mreid-tt/spksrc that referenced this pull request Jan 6, 2025

Avoid makefile variables in DEPENDS for SynoCommunity#6255

bf98833

mreid-tt added a commit to mreid-tt/spksrc that referenced this pull request Jan 7, 2025

Avoid makefile variables in DEPENDS for SynoCommunity#6255

7404ea6

Merge branch 'master' of https://github.com/hgy59/spksrc into redesig…

94361bf

…n_dependency_list

hgy59 added a commit to hgy59/spksrc that referenced this pull request Jan 17, 2025

libICU: update from v71.1 to v74.2

9f91b9b

- libicu 74 is the latest version that builds for DSM 6 - remove make variable in DEPENDS for SynoCommunity#6255

hgy59 mentioned this pull request Jan 17, 2025

libICU: update to v74.2 #6396

Merged

6 tasks

hgy59 added a commit to hgy59/spksrc that referenced this pull request Jan 17, 2025

java-11-openjdk: udpate Makefile

6f9132b

- avoid make variables in DEPENDS for SynoCommunity#6255

hgy59 added a commit that referenced this pull request Jan 17, 2025

libICU: update from v71.1 to v74.2 (#6396)

584d8af

- libicu 74 is the latest version that builds for DSM 6 - remove make variable in DEPENDS for #6255

hgy59 added 2 commits January 17, 2025 15:41

Merge branch 'master' of https://github.com/hgy59/spksrc into redesig…

3b78cfe

…n_dependency_list

Merge branch 'master' of https://github.com/hgy59/spksrc into redesig…

efcd4a5

…n_dependency_list

hgy59 mentioned this pull request Jan 18, 2025

prepare redesign of dependency list #6402

Merged

6 tasks

hgy59 added a commit that referenced this pull request Jan 18, 2025

prepare redesign of dependency list (#6402)

d92586f

- some changes taken from PR #6255 to reduce number of dependent packages - changes that trigger ffmpeg* - changes that trigger python311 (except vim, separated in PR #6398)

Merge branch 'master' of https://github.com/hgy59/spksrc into redesig…

20665d6

…n_dependency_list

hgy59 added a commit to hgy59/spksrc that referenced this pull request Jan 18, 2025

gpac: fix build

9ffeec2

- follow up to SynoCommunity#6004 - PLIST must contain own files only - update revision.h patching - remove variable in DEPENDS for SynoCommunity#6255

hgy59 mentioned this pull request Jan 18, 2025

gpac: fix build #6405

Merged

6 tasks

hgy59 added a commit that referenced this pull request Jan 18, 2025

gpac: fix build (#6405)

c6a078a

* gpac: fix build - follow up to #6004 - PLIST must contain own files only - update revision.h patching - remove variable in DEPENDS for #6255

hgy59 added 6 commits January 18, 2025 16:30

stockfish-webgui: fix digests

d53b54c

Merge branch 'master' of https://github.com/hgy59/spksrc into redesig…

d195ef8

…n_dependency_list

dependency-list: add python and ffmpeg dependency based on include file

ed7f0c6

revert changes in sabnzbd

1a6fd0e

- avoid conflicting changes with SynoCommunity#6366

revert changes in vim

7997a82

- avoid conflicting changes with SynoCommunity#6398

revert changes in java-11-openjdk

372f972

- avoid conflicting changes with SynoCommunity#6394

hgy59 changed the title ~~redesign dependency-list creation for github prepare action~~ Redesign dependency-list creation for github prepare action Jan 18, 2025

th0ma7 approved these changes Jan 18, 2025

View reviewed changes

hgy59 merged commit bdc87a5 into SynoCommunity:master Jan 18, 2025
15 of 19 checks passed

hgy59 deleted the redesign_dependency_list branch January 18, 2025 23:54

hgy59 mentioned this pull request Jan 19, 2025

rutorrent: upgrade 4.3.7 => 5.1.5 #6404

Draft

7 tasks

hgy59 added a commit to hgy59/spksrc that referenced this pull request Jan 21, 2025

cross/ntopng: avoid variable in DEPENDS

b272e39

- was not updated in SynoCommunity#6255

hgy59 mentioned this pull request Jan 21, 2025

Fix dependency list #6411

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Redesign dependency-list creation for github prepare action #6255

Redesign dependency-list creation for github prepare action #6255

hgy59 commented Oct 5, 2024 •

edited

Loading

hgy59 commented Oct 5, 2024 •

edited

Loading

th0ma7 commented Oct 6, 2024

hgy59 commented Oct 6, 2024

hgy59 commented Jan 18, 2025

mreid-tt commented Jan 18, 2025

hgy59 commented Jan 18, 2025

th0ma7 left a comment

hgy59 commented Jan 18, 2025

hgy59 commented Jan 18, 2025

hgy59 commented Jan 18, 2025

Redesign dependency-list creation for github prepare action #6255

Redesign dependency-list creation for github prepare action #6255

Conversation

hgy59 commented Oct 5, 2024 • edited Loading

Description

Mandatory framework changes

Checklist

Type of change

hgy59 commented Oct 5, 2024 • edited Loading

th0ma7 commented Oct 6, 2024

hgy59 commented Oct 6, 2024

hgy59 commented Jan 18, 2025

mreid-tt commented Jan 18, 2025

hgy59 commented Jan 18, 2025

th0ma7 left a comment

Choose a reason for hiding this comment

hgy59 commented Jan 18, 2025

hgy59 commented Jan 18, 2025

hgy59 commented Jan 18, 2025

hgy59 commented Oct 5, 2024 •

edited

Loading

hgy59 commented Oct 5, 2024 •

edited

Loading