Add package validation command #64

landerweit-phytec · 2023-08-02T08:07:59Z

Add a new command for package validation validate.
The goal of package validation is to make sure a package can be written to a device without writing to the device. Additionally, remove most validation steps from the install command to reduce the time i takes to write to the device.
Open tasks:

Check if URI points into archive
Size calculation for uncompressed archives
Option to use validate without passing a device (maybe use a loop device and pass eMMC size as argument)
Ensure the partitions fit the device
Test validate command
Add commit messages

src/pu-utils.c

mschwan-phytec

A first glimpse on your PR...

src/pu-utils.c

tests/emmc.c

src/pu-input.c

src/pu-input.h

src/pu-emmc.c

mschwan-phytec · 2023-09-22T12:38:24Z

You may want to consider squashing some commit fixing earlier stuff of your PR, e.g.:

Check file size and raw overlap only in validate

There is code in src/pu-emmc.c:pu_emmc_new that gets added earlier and then removed. Maybe you can squash/rearrange your commits, so changes are correct in the first place.

doc/usage.rst

mschwan-phytec · 2023-09-22T12:44:28Z

doc/usage.rst

+   -s, --skip-checksums         Skip checksum verification for all input files
+   -w, --write-file             Install the package to a file as an additional validation step
+   -k, --keep-file-as=FILENAME  Keep the file. Implies --write-file


From a user perspective I think it is not clear, what you mean exactly with "file" in the options --write-file and --keep-file-as as the term collides with the input files.

mschwan-phytec · 2023-09-22T12:46:57Z

src/pu-main.c

+    { "skip-checksums", 's', G_OPTION_FLAG_NONE, G_OPTION_ARG_NONE,
+        &arg_validate_skip_checksums, "Skip checksum verification for all input files", NULL },
+    { "size", 0, G_OPTION_FLAG_NONE, G_OPTION_ARG_STRING,
+        &arg_validate_file_size, "Size of loop device. Default is 1G", "SIZE" },


Again, a user may not be familiar with loop devices, so this description is not easy to understand without looking at the inner workings of partup.

src/pu-main.c

mschwan-phytec

Please rebase your changes, just in case.

src/pu-utils.c

doc/usage.rst

mschwan-phytec · 2024-01-30T09:23:11Z

I tried running partup with the validate command and somehow my RAM got clogged up. I guess there is a memory leak or similar in your code? I checked for leftover mounted loop device but that does not seem to be the issue on first sight. When running partup I get the following generic error message:

10:07:13.989678 CRITICAL partup: Error writing to file: No space left on device

However, running anything that requires RAM then throws a similar error message, e.g. GCC:

../src/pu-loopdev.c:88:1: fatal error: error writing to /tmp/ccScBcPv.s: No space left on device

The commands I ran before are something like this:

sudo partup-git validate -d rauc-image.partup --size 2G -s

Maybe partup must return with an error code before this issue appears. Should happen if there are no checksums in the layout config and we do not provide the -s option.

Add a new function pu_spwan_command_line_sync_result() which returns the output of stdout. Also, add a corresponding unit test. Signed-off-by: Leonard Anderweit <[email protected]>

Only append the source filename in case the destination is a directory. Signed-off-by: Leonard Anderweit <[email protected]>

Add functions to create and remove loop devices. Signed-off-by: Leonard Anderweit <[email protected]>

Signed-off-by: Leonard Anderweit <[email protected]>

Separate input files from flash specific information as they only contain general information like filename, checksums and size on target. Signed-off-by: Leonard Anderweit <[email protected]>

Signed-off-by: Leonard Anderweit <[email protected]>

Add a list for all input files regardless of their destination. Use it to get the absolute path and size of all input files at once after config parsing instead of during writing. Signed-off-by: Leonard Anderweit <[email protected]>

Add a new function for config validation. For now only verifies checksums. Signed-off-by: Leonard Anderweit <[email protected]>

Checksum verification is done with the validate command. So remove it from the install command to save time when writing to a device. Update the docs accordingly. Signed-off-by: Leonard Anderweit <[email protected]>

These steps are not necessary for package installation so execute them only when validating a package. Also, update the tests accordingly. Signed-off-by: Leonard Anderweit <[email protected]>

Add an additional validation step to check if the input files of each partition fit into the corresponding partition. Signed-off-by: Leonard Anderweit <[email protected]>

Check if the fixed size partitions fit into the device. This does not check expanding partitions. Signed-off-by: Leonard Anderweit <[email protected]>

Signed-off-by: Leonard Anderweit <[email protected]>

Add a new function pu_parse_size() which accepts a size string and returns the size in bytes. Accepts strings with the same format as ped_parse_unit, but only with absolute units. Also, add a corresponding unit test. Signed-off-by: Leonard Anderweit <[email protected]>

The new command 'validate' validates a package either against a provided device or a new loop device. Signed-off-by: Leonard Anderweit <[email protected]>

Writing the package to the loop device is an additional validation step. This is similar to installing the package to a real device. Signed-off-by: Leonard Anderweit <[email protected]>

The generated image can be written with dd. Signed-off-by: Leonard Anderweit <[email protected]>

Signed-off-by: Leonard Anderweit <[email protected]>

landerweit-phytec added the enhancement New feature or request label Aug 2, 2023

landerweit-phytec added this to the v1.0.0 milestone Aug 2, 2023

landerweit-phytec self-assigned this Aug 2, 2023

landerweit-phytec force-pushed the wip-validate-command branch 4 times, most recently from 812ecaa to 92ddeb8 Compare August 14, 2023 15:03

landerweit-phytec force-pushed the wip-validate-command branch 10 times, most recently from 418f8e1 to 2458213 Compare August 17, 2023 09:47

landerweit-phytec requested a review from mschwan-phytec August 17, 2023 09:47

landerweit-phytec marked this pull request as ready for review August 17, 2023 11:02

landerweit-phytec force-pushed the wip-validate-command branch from 2458213 to 5cc4bbc Compare August 17, 2023 12:24

landerweit-phytec force-pushed the wip-validate-command branch 5 times, most recently from 319aac1 to ad3f15f Compare September 8, 2023 09:13

mschwan-phytec reviewed Sep 19, 2023

View reviewed changes

src/pu-utils.c Show resolved Hide resolved

mschwan-phytec requested changes Sep 22, 2023

View reviewed changes

landerweit-phytec force-pushed the wip-validate-command branch from ad3f15f to 794acc2 Compare September 22, 2023 13:29

landerweit-phytec force-pushed the wip-validate-command branch 2 times, most recently from 61ff635 to 64e332a Compare November 23, 2023 09:23

landerweit-phytec requested a review from mschwan-phytec November 23, 2023 09:25

landerweit-phytec force-pushed the wip-validate-command branch from 64e332a to 2289bdc Compare November 28, 2023 16:04

landerweit-phytec force-pushed the wip-validate-command branch from 2289bdc to e6dfcc1 Compare December 13, 2023 11:05

mschwan-phytec requested changes Jan 23, 2024

View reviewed changes

src/pu-utils.c Outdated Show resolved Hide resolved

doc/usage.rst Outdated Show resolved Hide resolved

landerweit-phytec force-pushed the wip-validate-command branch from e6dfcc1 to 8cc250f Compare January 23, 2024 10:43

landerweit-phytec requested a review from mschwan-phytec January 23, 2024 10:45

mschwan-phytec removed this from the v2.0.0 milestone Jan 30, 2024

landerweit-phytec added 19 commits January 31, 2024 09:19

src: utils: Add function to get the stdout output of a command

547b9bf

Add a new function pu_spwan_command_line_sync_result() which returns the output of stdout. Also, add a corresponding unit test. Signed-off-by: Leonard Anderweit <[email protected]>

src: utils: Allow copying to file destinations

4d7e856

Only append the source filename in case the destination is a directory. Signed-off-by: Leonard Anderweit <[email protected]>

src: Add loopdev functions

75c241d

Add functions to create and remove loop devices. Signed-off-by: Leonard Anderweit <[email protected]>

tests: Add test for loopdev

ec7a97b

Signed-off-by: Leonard Anderweit <[email protected]>

src: Make input files independent from emmc

8fd03ea

Separate input files from flash specific information as they only contain general information like filename, checksums and size on target. Signed-off-by: Leonard Anderweit <[email protected]>

tests: Add unit test for pu-input

b381b1d

Signed-off-by: Leonard Anderweit <[email protected]>

src: Add config validation function

4dbcb76

Add a new function for config validation. For now only verifies checksums. Signed-off-by: Leonard Anderweit <[email protected]>

src: Remove checksum verification from install command

c2b6c9f

Checksum verification is done with the validate command. So remove it from the install command to save time when writing to a device. Update the docs accordingly. Signed-off-by: Leonard Anderweit <[email protected]>

Check file size and raw overlap only in validate

c1dc6b8

These steps are not necessary for package installation so execute them only when validating a package. Also, update the tests accordingly. Signed-off-by: Leonard Anderweit <[email protected]>

src: emmc: Check if input files fit into partition

d46f7c8

Add an additional validation step to check if the input files of each partition fit into the corresponding partition. Signed-off-by: Leonard Anderweit <[email protected]>

src: emmc: Check if partitions fit into device

b5d6aa7

Check if the fixed size partitions fit into the device. This does not check expanding partitions. Signed-off-by: Leonard Anderweit <[email protected]>

tests: emmc: Add unit tests for validation

860574d

Signed-off-by: Leonard Anderweit <[email protected]>

src: main: Add validate command

4512cfa

The new command 'validate' validates a package either against a provided device or a new loop device. Signed-off-by: Leonard Anderweit <[email protected]>

src: main: Add option to write package to loop device

6a5da15

Writing the package to the loop device is an additional validation step. This is similar to installing the package to a real device. Signed-off-by: Leonard Anderweit <[email protected]>

src: main: Add option to build flashable image from package

d70fff1

The generated image can be written with dd. Signed-off-by: Leonard Anderweit <[email protected]>

doc: usage: Add validate command

e2ca74e

Signed-off-by: Leonard Anderweit <[email protected]>

github: workflows: system-tests: Add package validation step

b056f7f

Signed-off-by: Leonard Anderweit <[email protected]>

landerweit-phytec force-pushed the wip-validate-command branch from 8cc250f to b056f7f Compare January 31, 2024 09:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add package validation command #64

Add package validation command #64

landerweit-phytec commented Aug 2, 2023 •

edited

Loading

mschwan-phytec left a comment

mschwan-phytec commented Sep 22, 2023

mschwan-phytec Sep 22, 2023

mschwan-phytec Sep 22, 2023

mschwan-phytec left a comment

mschwan-phytec commented Jan 30, 2024

Add package validation command #64

Are you sure you want to change the base?

Add package validation command #64

Conversation

landerweit-phytec commented Aug 2, 2023 • edited Loading

mschwan-phytec left a comment

Choose a reason for hiding this comment

mschwan-phytec commented Sep 22, 2023

mschwan-phytec Sep 22, 2023

Choose a reason for hiding this comment

mschwan-phytec Sep 22, 2023

Choose a reason for hiding this comment

mschwan-phytec left a comment

Choose a reason for hiding this comment

mschwan-phytec commented Jan 30, 2024

landerweit-phytec commented Aug 2, 2023 •

edited

Loading