Regenerating project files

[iocanel]

For a while now there have been discussions about the need for regenerating project files like:

• dockerfiles
• gitlab configuration
• github workflows
• tekton pipelines
• backstage catalog files (WIP)

The main need expressed in those discussions is that when users upgrade their version of Quarkus
these files often need upgrading. This is mostly due to

• references of static images
• files becoming obsolete
• new files being added

Sources of generated files

• Codestarts
• Quarkus Extensions:
  • Backstage (WIP)
  • Kubernetes
  • OpenShift
  • Helm

Potential ways of regenerating files

• CLI
• Build Tool Plugins (maven & gradle)
• DevUI (Optional)

CLI

The CLI could introduce a new subcommand generate which will act as an entry point to additonal commands and plugins

For example

  quarkus generate dockerfile
  quarkus generate tekton-pipeline
  quarkus generate backstage-catalog
  quarkus generate github-workflow
  quarkus generate gitlab-workflow

Use in non-java environments

As some of this commands may be invoked from environments that do not provide a java developer setup it would be good to
ensure that the commands above are standalone (don't require maven / gradle) and work as native binaries.

Build Tool Plugins

The CLI is not something all quarkus developers use, it would be good if this functionality is available in our build tools too.

Maven

For maven the usage could be:

  mvn quarkus:generate dockerfile
  mvn quarkus:generate tekton-pipeline
  mvn quarkus:generate backstage-catalog
  mvn quarkus:generate github-workflow
  mvn quarkus:generate gitlab-workflow

As the actual target is dynamic it should be passed as an agrgument

Gralde

In the same spirit for gradle:

  gradle quarkusGenerate dockerfile
  gradle quarkusGenerate tekton-pipeline
  gradle quarkusGenerate backstage-catalog
  gradle quarkusGenerate github-workflow
  gradle quarkusGenerate gitlab-workflow

Dev UI (Optional)

This is more of a nice to have. The added benefit of supporting this feature from the DevUI is the potential of showing a diff
that makes the changes/diff visible. Of course this can be achieved from the CLI, using a combination of commands and this is why
I am marking this as optional.

Generating files by source

As mentioned above there are two major sources the produce generated files that often require updating: codestarts and extensions.
Each case needs to be handled slightly differently.

Codestart

Detailed explanation of how codestarts work: https://quarkus.io/guides/extension-codestart

Each codestart provides a manifest and optionally qute templates. The generated files occur from rendering these templates with information
provided by the project model (e.g. project metadata etc) and the information available in the codestart etc. After each file is rendered there
are mutliple different strategies on how its added to the project (e.g. append, replace, ignore, forbidden).

Implementation proposal

Each tool that needs to re-generate files backed by codestarts can perform the following steps:

• identify the codestart(s) that produce the needed files
• identify the output strategy for each of the files
• identify additional parameters that need to be passed for rendering
• Invoke the codestart engine.

Related pull requests:

A pull request that uses an approach like this for regenerating dockerfiles is at: #36178

Quarkus extensions

Current situation

Currently most of our extensions that do generate project files via extension happen at build time. This means that files do get regenerated on each
build, which practically eliminates the possiblity of ending up with stale files. However, this approach suffers from a couple of limitations:

• Uneeded build time overhead
• Requires a build to get the files
  • Difficult to get them in non-java ready envioronments (e.g. in Backstage).
  • Chicken and egg probelm (Building the project in order to generate a pipeline that builds the project)
• There are tools and practices that suggest the presence of those files in version control (e.g. argocd)

Potential improvements

• Allow opting out of generating those files on each build.
  • Globally
  • Per case
• Provide a command that triggers the generation of those resources, without going through the build tool.
• Move the output directory of the generated files from build output directory to version control root (or a subdirectory of it).

Implementation ideas

As in most cases the generated code / resources make use of GeneratedFileSystemResourceBuildItem
We could possibly make use of that to control the file generation.

The existing build item has two fields:

• name: Relative path to the output directory. Indicates where the file will get created
• data: The actual bytes of the file.

So, any augmentation action that requires those build items will get them created, without actually generating them.
With them our tooling can produce the files anywhere we need.

Unfortunately, the only way to filter out the build items we need is by name (only applicable in case where files are generated in a sub-directory that can be used for filtering).
So maybe we should introduce a new field for optional grouping.

[cescoffier]

For the docker files, I believe multiple extensions should be able to participate in providing packages to install or unpack the native library in the right directory (thus, targetted architecture will be required).

[cescoffier]

Hello, @iocanel and I had a quick chat about it.

First, we would focus on docker file generation. By reducing the scope, we have more chances to make progress quickly while building the mechanism that would be used for the other generation.

In this context, the goal is to drop the Dockerfile generated by code.quarkus.io or quarkus create, and provide a quarkus dockerfile command (name to be discussed) that will create the docker file on demand (as a one-off task):

quarkus dockerfile native-micro dockerfile.native
quarkus dockerfile jvm dockerfile.jvm -Dfrom=quay.io/cescoffier/my-wonderful-base-image:latest

(Commands and parameters still under consideration)

The command will NOT update an existing docker file.

During the generation, extensions can contribute to the generated output by asking for a specific package to be installed and adding dockerfile clauses (mostly RUN and COPY).

[cescoffier]

Initial working group created: https://github.com/orgs/quarkusio/projects/27/views/1?pane=info

[iocanel]

Had a meeting earlier today with @cescoffier and @ia3andy. Here's a brief summary:

• Generate dockerfiles without codestarts (Introduce a new extension for generating Dockerfiles #42316)
• Use Qute.fmt instead of String format
• Generate on quarkus create (Figure out a way to tigger specific code generation operation on quarkus create #43589)
• Introduce quarkus:edit (Provide a quarkus edit command in the Quarkus CLI #43588)