Adapt endpoint package to package storage v2

[kevinlog]

Change the endpoint package to adhere to the package storage v2 rquirements and to upload signed packages.

AC:

• Update the job that pushes the Endpoint package to package-storage.

Notes:

• Reference this ticket for more info: https://github.com/elastic/ingest-dev/issues/1119

[mtojek]

@kevinlog Could you please share a link/reference to your delivery pipeline? I may find some time to help you adjust it.

[kevinlog]

@mtojek thanks for reaching out - @pzl is looking at our Endpoint package right now and ensuring that we can ship a package in 8.4 given the new package storage and transform spec requirements.

@pzl can you follow up on this?

[mtojek]

No worries about shipping. We will make sure that you're onboarded before migration.

[mtojek]

Hi,

I reviewed the following files (thanks @pzl for sharing them):
https://github.com/elastic/endpoint-package/blob/master/scripts/release_manager/main.py
https://github.com/elastic/endpoint-package/blob/master/Makefile#L157-L158

and evaluated a list of changes to be applied:

1. Create a new CI pipeline in beats-ci to release endpoint-package triggered manually.
2. CI pipeline: build the endpoint-package.
3. CI pipeline: determine next release verson.
4. CI pipeline: call the Elastic signing pipeline to sign the Endpoint package.
5. CI pipeline: call the Package Storage publishing pipeline to publish the package.

Difference:

1. The publishing procedure will be performed on the CI host.
2. No need to open PRs against any repo. elastic/package-storage will be archived.
3. No more promotions. The package will be considered a prerelease based only on the semver.

Let me know what you think. I can start working on the setup.

[pzl]

prerelease based only on the semver.

FYI we currently use some illegal versioning. We haven't aligned our process to use the correct prerelease version strings to spec yet.

---

The pipeline sounds good. We have a Jenkinsfile and are set up with CI within our repo (and it is on the beats-ci cluster). It builds and checks on PRs/merge, etc. Happy to integrate or add the jobs in our repo's .ci/ so it can be in sync with repo build changes?
It would also be nice, if its simple while we're doing this, to publish pre-release versions on every commit (/merge) to our main branch. So we auto-deploy prereleases, but manual deploy job for release versions?

[mtojek]

It would also be nice, if its simple while we're doing this, to publish pre-release versions on every commit (/merge) to our main branch. So we auto-deploy prereleases, but manual deploy job for release versions?

Yes, we can go that way:

1. Publish a prerelease package on every commit.
2. Provide a pipeline parameter to publish a release (not prerelease).

[mtojek]

@pzl

These are the leftovers:

1. Remove the release script.
2. Update docs as there won't be promotions anymore.
3. Automate version bump (for development versions).
4. Enable full package validation, disable legacy_package mode.

Would you like me to create a new issue for those or do you prefer to keep this one?