feat: build binaries and Docker images in CI #24751

mgattozzi · 2024-03-11T16:43:15Z

feat: build binaries and Docker images in CI

For releases we need to have Docker images and binary images available for the
user to actually run influxdb3. These CI changes will build the binaries on a
release tag and the Docker image as well, test, sign, and publish them and make
them available for download.

Co-Authored-By: Brandon Pfeifer [email protected]

Closes #24773

mgattozzi · 2024-03-26T18:37:44Z

Okay this is open for review. Before merging I will need to set the CI to only run only on release tags. If you want to test the builds yourself let me know internally on slack and I can get you access to the docker builds and the binary builds.

mgattozzi · 2024-03-26T18:40:47Z

I'm having some issues with the defaults I've set for docker via env vars (something about permission denied for the fs) that I don't get just running the binary itself which is a docker issue with how I'm running it I think. Otherwise I think we're in a great spot.

hiltontj

Thanks @mgattozzi - this is a huge lift. I only had a couple questions/comments, but otherwise LGTM. Would probably be good to get @bnpfeife's approval as well.

.circleci/packages/influxdb3/control/post-uninstall

.circleci/scripts/package-validation/validate

bnpfeife · 2024-03-28T00:31:25Z

I'm having some issues with the defaults I've set for docker via env vars (something about permission denied for the fs) that I don't get just running the binary itself which is a docker issue with how I'm running it I think. Otherwise I think we're in a great spot.

Hey, I've created a PR that resolves the build issue. Though I'm out of time tonight to test it.

mgattozzi · 2024-03-28T20:18:44Z

I removed the files, included @bnpfeife's changes for making the directory, and I've exposed the right port as well as setting the proper default address to 0.0.0.0 so that docker containers and deployments of influxdb should work as expected. I tagged you for another rereview @hiltontj given there is a code change.

hiltontj

🚢

mgattozzi · 2024-04-02T17:41:34Z

With @hiltontj's changes to where heappy is located I'll need to make a few here in order to get the builds to work on windows again but it shouldn't be that hard. @bnpfeife if you think that the changes otherwise are good then I think we can merge once I fix the conflicts and turn on the release filters so CI isn't building artifacts all the time.

jdstrand · 2024-04-03T15:44:52Z

.circleci/packages/config.yaml

+    match: '^v[0-9]+.[0-9]+.[0-9]+'
+    value: '{{env.CIRCLE_TAG[1:]}}'
+  default:
+    value: '3.x-{{env.CIRCLE_SHA1[:8]}}'


This default value is difficult to work with. Consider:

$ dpkg --compare-versions 3.0.0 lt 3.0.1 && echo yes # good yes $ dpkg --compare-versions 3.0.0 lt 3.x-0123abcd && echo yes # maybe not good? yes $ dpkg --compare-versions 3.x-gfed9876 lt 3.x-0123abcd && echo yes # but what if 3.x-0123abcd was created before 3.x-gfed9876? [1]

Part of the reason why I'm commenting on this is that we are uploading to influxdb/releases/influxdb3, but these aren't releases, they're snapshots. Package managers are going to have trouble dealing with these versions. I would advocate that the influxdb/releases/influxdb3 only contain things that are considered a release and use something else for snapshots or nightlies.

Fyi, telegraf has telegraf/nightlies/telegraf_nightly_amd64.deb, which gets overwritten every night. Maybe that isn't what you want right now though (ie, perhaps it is useful to have different snapshots).

As a strawperson, I suggest that:

someone creates influxdb/snapshots/influxdb3 and move everything that was uploaded to influxdb/releases/influxdb3 there. New snapshots go to influxdb/snapshots/influxdb3 and use 3.0.0+snapshot-{{env.CIRCLE_SHA1[:8]}} as the default.

if one of the current uploads was an mvp, tag it with 3.0.0-0.alpha.0.0, rebuild it and put the rebuilt artifacts in influxdb/releases/influxdb3. For important internal pre-alpha releases, you can tag with 3.0.0-0.alpha.0.1, 3.0.0-0.alpha.0.2, etc. When official alpha 1 is released, tag with 3.0.0-0.alpha.1, then use 3.0.0-0.alpha.2 for 2nd official alpha, and so on. Continue with 3.0.0-0.beta.1, 3.0.0-0.rc.1 as desired. This versioning scheme works with both semver and packaging. Ie: 3.0.0-0.alpha.0.0 < 3.0.0-0.alpha.0.1 < 3.0.0-0.alpha.0.2 < 3.0.0-0.alpha.1 < 3.0.0-0.alpha.2 < 3.0.0-0.beta.1 < 3.0.0-0.rc.1. All of these tagged releases go to influxdb/releases/influxdb3

when 3.0.0 is released, tag as 3.0.0. semver is happy since 3.0.0-0.rc.1 < 3.0.0 and packaging is updated to use 3.0.0-1 such that 3.0.0-0.rc.1 < 3.0.0-1

In the above, we don't worry if 3.0.0+snapshot-{{env.CIRCLE_SHA1[:8]}} is higher or lower in packaging since people will be installing those manually.

This is definitely a good point. I may go ahead and update these for 1.X OSS/Enterprise and 2.X OSS. As you alluded to, the default version schemes weren't intended to be consumed by the general public. So I had just written them to make apt and yum happy. However, I definitely can foresee a future where someone makes these available to the general public (not here, but in 1.X and 2.X land).

bnpfeife · 2024-04-04T18:26:56Z

@mgattozzi

Hey, my apologies that I haven't looked at this for some time! I had left some comments but they hadn't appeared in the conversation part of the PR until I started a review. I updated/removed one of my review comments and added another so it aligns better with what @jdstrand suggested. Looking through it, I think everything is just minor changes.

Edit: Oh, my comments were "Pending". GitHub should really let you know that those aren't visible.
https://github.com/orgs/community/discussions/10369

bnpfeife · 2024-03-27T23:54:56Z

.circleci/config.yml

 commands:
+  install_rust:


Oh, I think that this can be removed.

.circleci/scripts/package-validation/validate

bnpfeife · 2024-04-04T18:07:04Z

.circleci/packages/config.yaml

+    match: '^v[0-9]+.[0-9]+.[0-9]+'
+    value: '{{env.CIRCLE_TAG[1:]}}'
+  default:
+    value: '3.x-{{env.CIRCLE_SHA1[:8]}}'


This is definitely a good point. I may go ahead and update these for 1.X OSS/Enterprise and 2.X OSS. As you alluded to, the default version schemes weren't intended to be consumed by the general public. So I had just written them to make apt and yum happy. However, I definitely can foresee a future where someone makes these available to the general public (not here, but in 1.X and 2.X land).

bnpfeife · 2024-04-04T18:23:17Z

.circleci/config.yml

+  publish-packages:
+    docker:
+      - image: cimg/python:3.6
+    steps:


Since we're publishing "snapshot" packages, let's change the destination depending on the release type:

publish-packages: docker: - image: cimg/python:3.6 + parameters: + # "destination" should be one of: + # - releases + # - nightlies + # - snapshots + destination: + - type: string steps: - attach_workspace: at: /tmp/workspace - aws-s3/sync: arguments: --acl public-read aws-region: RELEASE_AWS_REGION aws-access-key-id: RELEASE_AWS_ACCESS_KEY_ID aws-secret-access-key: RELEASE_AWS_SECRET_ACCESS_KEY from: /tmp/workspace/artifacts - to: s3://dl.influxdata.com/influxdb/releases + to: s3://dl.influxdata.com/influxdb/<< parameters.destination >>

The workflows will need to be modified to pass destination along to the jobs.

mgattozzi · 2024-04-05T19:57:03Z

I'll still need to address all of these. I did get the branch rebased to latest main and conflicts fixed. I'll get to these early next week when I'm back from the solar eclipse viewing.

mgattozzi · 2024-04-09T18:26:21Z

@jdstrand and @bnpfeife I made the requested changes. I also added the sed filtering you mentioned on slack @jdstrand to make using sha256sum --check work easier.

jdstrand

Thanks for these changes! See inline.

.circleci/config.yml

jdstrand · 2024-04-09T20:24:02Z

.circleci/packages/config.yaml

+    match: '^v[0-9]+.[0-9]+.[0-9]+'
+    value: '{{env.CIRCLE_TAG[1:]}}'
+  alpha:
+    match: '^v3.0.0.alpha.[0-9]+.[0-9]+'


This should be ^v3.0.0-0.alpha.[0-9]+.[0-9]+ (the -0 is important for package managers).

.circleci/packages/config.yaml

jdstrand · 2024-04-09T20:31:17Z

.circleci/packages/influxdb3/control/post-install

+    # We perform ownership change only for Debian-based systems.
+    # Moving these lines out of this if statement would make `rmp -V` fail after installation.
+    chown -R -L influxdb:influxdb $LOG_DIR
+    chown -R -L influxdb:influxdb $DATA_DIR


In pre-install you define:

USER=influxdb3 GROUP=influxdb3

It would be nice to do that here.

jdstrand · 2024-04-09T20:34:29Z

.circleci/packages/influxdb3/control/post-install

+if [[ -f /etc/debian_version ]]; then
+    # Ownership for RH-based platforms is set in build.py via the `rmp-attr` option.
+    # We perform ownership change only for Debian-based systems.
+    # Moving these lines out of this if statement would make `rmp -V` fail after installation.


AIUI, this is saying that on Debian-based systems, unconditionally chown and chmod the LOG_DIR and DATA_DIR. Does this happen on upgrades or just new installs? Why is this needed if post-install is already creating the directories correctly. This feels like something cargo-culted in from other packaging that was meant to fix a permissions problem for the other package that doesn't apply here, but maybe I'm missing something?

This is a straight copy from oss 2.x so maybe @bnpfeife would know more?

Yeah, unfortunately, this is something else that has existed before I started working here. My suspicion is that someone was having issues with directory permissions being too restrictive. They probably complained so the "solution" was to always reset the permissions in post-inst. I think since we're starting anew, we should remove this. Mostly because the packaging should attempt to respect the local configuration whenever possible.

jdstrand · 2024-04-09T20:47:59Z

Dockerfile

+
+# TODO: Make this and other env vars not specific to IOx
+ENV INFLUXDB_IOX_OBJECT_STORE=file
+ENV INFLUXDB_IOX_DB_DIR=/usr/local/share/influxdb3


Nitpick: /usr/local/share is an odd locations for database files. I would expect /var/lib/influxdb3 in the Dockerfile too. Why this path? Presumably this is a mount point for a writable dir, why not make it in a more standard location?

I can't remember, I think I wasn't sure or I was copying what we did in the other OSS bits. Happy to make it /var/lib.

mgattozzi · 2024-04-10T19:20:29Z

There are still some things that need addressing but am waiting on @bnpfeife's feedback to implement them correctly.

bnpfeife · 2024-04-18T17:28:16Z

Hey @jdstrand, I've been looking into updating ci-packager to support these changes:

don't append -1 to deb and rpm builds for alpha, beta and rc

update the rpm versions to use - instead of _ in the filename and internal version

I mentioned in out Slack that ci-packager executes fpm. fpm then modifies the version string before passing it to rpmbuild or dpkg-buildpackage. So, I experimented with calling rpmbuild and dpkg-buildpackage from ci-packager instead.

Ultimately, I ended up consulting both the debian and redhat versioning guidelines quite a bit. Both packaging formats consider the hyphen a separator between the "upstream version" and "package version". These components have slightly different names in the actual version specifications. However, the purpose of each component remains largely the same.

I can definitely make hyphens work, but I'm not sure that I like the implications. I interpret the "release version" ($CIRCLE_TAG) to correspond entirely to the "upstream version". Splitting the "release version" into an "upstream version" and "package version" indicates that multiple releases of the package have occurred when upstream remained unchanged.

It looks like 3.0.0-0.alpha.0 and the other pre-release versions have been crafted to sort before 3.0.0-1. I'm okay with this; ci-packager will just need to supply 1 as the "package version" when none could be parsed from the "release version".

With all of that being said, I'm wondering if it would be easier just to use ~ and +? In both packaging formats, ~ sorts before other components. Outside of that, neither impart special meaning or split the "release version".

# debian version comparisons
$ dpkg --compare-versions '1.0.0' gt '1.0.0~alpha0' # exit-code 0
$ dpkg --compare-versions '1.0.0' gt '1.0.0+snapshot0' # exit-code 1

# redhat version comparisons
$ rpmdev-vercmp '1.0.0' '1.0.0~alpha0'
1.0.0 > 1.0.0~alpha0
$ rpmdev-vercmp '1.0.0' '1.0.0+snapshot0'
1.0.0 < 1.0.0+snapshot0

jdstrand · 2024-04-18T20:05:12Z

Hey @jdstrand, I've been looking into updating ci-packager to support these changes:

don't append -1 to deb and rpm builds for alpha, beta and rc

update the rpm versions to use - instead of _ in the filename and internal version

I mentioned in out Slack that ci-packager executes fpm. fpm then modifies the version string before passing it to rpmbuild or dpkg-buildpackage. So, I experimented with calling rpmbuild and dpkg-buildpackage from ci-packager instead.

Ultimately, I ended up consulting both the debian and redhat versioning guidelines quite a bit. Both packaging formats consider the hyphen a separator between the "upstream version" and "package version". These components have slightly different names in the actual version specifications. However, the purpose of each component remains largely the same.

I can definitely make hyphens work, but I'm not sure that I like the implications. I interpret the "release version" ($CIRCLE_TAG) to correspond entirely to the "upstream version". Splitting the "release version" into an "upstream version" and "package version" indicates that multiple releases of the package have occurred when upstream remained unchanged.

Not having hyphens has packaging implications. In Debian, packages without hyphens are interpreted as Debian 'native' packages that come from Debian itself (eg, dpkg has version 1.21.1 in Debian whereas postgresql-14 has 14.2-1) and packaging is different between native and non-native packages. For releases of our software coming from git tags, we should use hyphens since we aren't a native package. One goal of the suggestion I gave is that 3.0.0-1 is intended to be 'the official release of 3.0.0' and '3.0.1-1' is "the official release of 3.0.1' whereas '3.0.1-2' is "the official release of 3.0.1 with a packaging change". This is fully inline with what you just said (but see below).

It looks like 3.0.0-0.alpha.0 and the other pre-release versions have been crafted to sort before 3.0.0-1. I'm okay with this; ci-packager will just need to supply 1 as the "package version" when none could be parsed from the "release version".

Indeed.

With all of that being said, I'm wondering if it would be easier just to use ~ and +? In both packaging formats, ~ sorts before other components. Outside of that, neither impart special meaning or split the "release version".
# debian version comparisons
$ dpkg --compare-versions '1.0.0' gt '1.0.0~alpha0' # exit-code 0
$ dpkg --compare-versions '1.0.0' gt '1.0.0+snapshot0' # exit-code 1

# redhat version comparisons
$ rpmdev-vercmp '1.0.0' '1.0.0~alpha0'
1.0.0 > 1.0.0~alpha0
$ rpmdev-vercmp '1.0.0' '1.0.0+snapshot0'
1.0.0 < 1.0.0+snapshot0

The problem is that in addition to deb and rpm packaging, I was trying to also reconcile semver versioning, and it doesn't allow a ~ or a + after the x.y.z, only -. I felt semver was important because this is coming from a git tag, git tags are shown in GitHub, more people are probably familiar with semver than esoteric Linux packaging version semantics and also we've seen Enterprise import influxdb in go.mod so I wanted good semver in case Community/Pro wanted to do something similar in Cargo.toml. I considered using 2.99.N for our alpha and beta builds, but I wanted the 3.0.0 to make it extremely clear that we are talking about InfluxDB 3, not any version of 2.x. Therefore, I took a common approach and used -0 to denote "this is unofficial" in the deb/rpm world with "this is prerelease" in the semver world, and then created the scheme I did that works for all of deb, rpm and semver for releases of our software that come from git tags:

use match: '^v[0-9]+.[0-9]+.[0-9]+(-[0-9]+.(alpha|beta|rc).[0-9]+(.[0-9]+)?)?$'

don't append -1 to deb and rpm builds for alpha, beta and rc (ed. since they already have -0...)

update the rpm versions to use - instead of _ in the filename and internal version

Yes, the alpha, beta and rc versions have rather verbose version, but it works from a packaging perspective and alpha, beta and rc builds are for internal use (eg, testing) so I wanted to also optimize for MAJOR.MINOR.MICRO-N versions of our software since by far most users will only ever use them. My goal for the alpha, beta and rc versioning is really just to be packaging friendly).

That leaves +snapshot where we agree: 3.0.0+snapshot-{{env.CIRCLE_SHA1[:8]}}. In this, I abandoned the -Nsomething since a) these aren't coming from git tags so semver doesn't matter and b) I wanted the to convey the idea of 'this is 3.0.0 plus stuff' (so it is greater than 3.0.0-1 but less than 3.0.1-1).

In the end, the package versioning I chose is a compromise that:

works with deb, rpm and semver package tooling with release versions coming from git tags which supports alpha, beta, rc and final release versions
uses recommended policy for deb and rpm for official final versions (ie, 3.0.0-1, 3.0.1-1, 3.0.1-2, etc)
uses established convention for deb and rpm for prerelease versions (ie, uses -0something)
follows semver policy for prerelease versions (ie, uses -0something)
use recommended policy for deb and rpm for unofficial snapshot releases that don't have a corresponding semver git tag (ie, 3.0.0+snapshot)

mgattozzi · 2024-04-23T17:14:20Z

@jdstrand and @bnpfeife I'm back from vacation and so trying to figure out what else needs to be done to get this over the line. Are there any actual changes to the versioning that need to be done here?

jdstrand · 2024-04-23T18:55:28Z

@jdstrand and @bnpfeife I'm back from vacation and so trying to figure out what else needs to be done to get this over the line. Are there any actual changes to the versioning that need to be done here?

From my perspective, these are the changes:

use match: '^v[0-9]+.[0-9]+.[0-9]+(-[0-9]+.(alpha|beta|rc).[0-9]+(.[0-9]+)?)?$'

config.yaml still has:

version:
  release:
    match: '^v[0-9]+.[0-9]+.[0-9]+'
    value: '{{env.CIRCLE_TAG[1:]}}'
  default:
    value: '3.0.0+snapshot-{{env.CIRCLE_SHA1[:8]}}'

This should be changed to:

version:
  release:
    match: '^v[0-9]+.[0-9]+.[0-9]+(-[0-9]+.(alpha|beta|rc).[0-9]+(.[0-9]+)?)?$'
    value: '{{env.CIRCLE_TAG[1:]}}'
  default:
    value: '3.0.0+snapshot-{{env.CIRCLE_SHA1[:8]}}'

don't append -1 to deb and rpm builds for alpha, beta and rc (ed. since they already have -0...)

update the rpm versions to use - instead of _ in the filename and internal version

I believe that @bnpfeife fixed these outside of this PR? @bnpfeife, can you comment?

mgattozzi · 2024-04-25T15:30:01Z

Okay I pushed those changes @jdstrand. Fixed the cargo deny issue and rebased off main to make sure CI passes. If this all looks good, then I'll add the correct filters so it won't run unless we want to do a release before we merge

jdstrand

For the portions I reviewed before, LGTM. I noticed a new file foo that seems like a mistake and have a question about the listening port inline. In the interest of time, I'm going to approve, but please remove the foo file and consider my question about the port.

Please have @bnpfeife also give it a once over.

jdstrand · 2024-04-25T18:45:36Z

foo

@@ -0,0 +1 @@
+38fd39f0659ac8406de0e8f83c64eb0a15ac0152a8198467407ddb56c2a6e684  /home/michael/1Password Emergency Kit A3-CP6WC8-team-influxdata.pdf


What's this file?

Leftover from me testing stuff with sha256sum. I thought I had deleted it, but I think I did that in the web UI and forgot to pull in those changes before a push 🤦

jdstrand · 2024-04-25T18:47:21Z

Dockerfile

@@ -54,7 +57,12 @@ ENV PACKAGE=$PACKAGE
 COPY --from=build "/root/$PACKAGE" "/usr/bin/$PACKAGE"
 COPY docker/entrypoint.sh /usr/bin/entrypoint.sh

-EXPOSE 8080 8082
+EXPOSE 8181


Should this be port 8086?

bnpfeife · 2024-04-26T22:19:17Z

Hey, @jdstrand @mgattozzi

I created a PR into this branch that uses a new version of ci-packager. It should support everything that was mentioned here.

mgattozzi · 2024-04-29T16:43:52Z

@bnpfeife I've merged it here and if we need any more changes as follow up I can do them here, but I think this will get us over the edge.

mgattozzi · 2024-05-02T20:59:23Z

Okay tomorrow I'm going to do some rebasing again, put the correct filters on, and then merge!

For releases we need to have Docker images and binary images available for the user to actually run influxdb3. These CI changes will build the binaries on a release tag and the Docker image as well, test, sign, and publish them and make them available for download. Co-Authored-By: Brandon Pfeifer <[email protected]>

* fix: use new packager to correct versions * chore: delete foo

mgattozzi force-pushed the mgattozzi/distributables branch from 4c4e18f to 5858411 Compare March 14, 2024 16:07

mgattozzi changed the title ~~feat: Add binary builds and homebrew formulas~~ feat: Add binary builds for releases Mar 14, 2024

mgattozzi force-pushed the mgattozzi/distributables branch 2 times, most recently from a3c905a to 2d78d18 Compare March 26, 2024 18:32

mgattozzi requested review from pauldix, hiltontj and bnpfeife and removed request for pauldix March 26, 2024 18:32

mgattozzi marked this pull request as ready for review March 26, 2024 18:33

mgattozzi changed the title ~~feat: Add binary builds for releases~~ feat: build binaries and Docker images in CI Mar 26, 2024

hiltontj approved these changes Mar 26, 2024

View reviewed changes

.circleci/packages/influxdb3/control/post-uninstall Outdated Show resolved Hide resolved

.circleci/scripts/package-validation/validate Outdated Show resolved Hide resolved

mgattozzi force-pushed the mgattozzi/distributables branch from 973c1ce to eb49cb4 Compare March 28, 2024 20:12

mgattozzi requested a review from hiltontj March 28, 2024 20:16

hiltontj approved these changes Apr 1, 2024

View reviewed changes

jdstrand requested changes Apr 3, 2024

View reviewed changes

bnpfeife requested changes Apr 4, 2024

View reviewed changes

mgattozzi force-pushed the mgattozzi/distributables branch from eb49cb4 to b2d4bce Compare April 5, 2024 19:54

mgattozzi requested review from bnpfeife and jdstrand April 9, 2024 18:25

jdstrand requested changes Apr 9, 2024

View reviewed changes

mgattozzi force-pushed the mgattozzi/distributables branch from 40e2e45 to 1d427c2 Compare April 25, 2024 15:28

mgattozzi requested a review from jdstrand April 25, 2024 15:28

mgattozzi force-pushed the mgattozzi/distributables branch from 1d427c2 to 1b7cd19 Compare April 25, 2024 15:33

jdstrand approved these changes Apr 25, 2024

View reviewed changes

bnpfeife approved these changes May 1, 2024

View reviewed changes

mgattozzi and others added 12 commits May 3, 2024 15:47

fix: Remove uneeded rust_install command

3674367

feat: Add destination parameter to publish-packages job

9e87866

fix: Add alpha tagging scheme and set better default tag

9cf6dcc

fix: Filter out paths in checksum files

180806d

fix: Implement some of the suggestions in the PR

67a3564

fix: remove unecessary code in post-install

9bf22e7

fix: change up packages/config.yaml

88982bb

fix: change versioning regex

3e9178e

fix: cargo-deny issues

901f455

feat: use new package builder (#24945)

d995f79

* fix: use new packager to correct versions * chore: delete foo

fix: Delete foo

5484819

mgattozzi force-pushed the mgattozzi/distributables branch from f31ac6e to 5484819 Compare May 3, 2024 19:48

mgattozzi added 2 commits May 3, 2024 15:49

feat: flip release filters on

b953c17

fix: cargo deny

5de68d2

mgattozzi merged commit c88cb5f into main May 3, 2024
13 checks passed

mgattozzi deleted the mgattozzi/distributables branch May 3, 2024 20:39

		@@ -0,0 +1 @@
		38fd39f0659ac8406de0e8f83c64eb0a15ac0152a8198467407ddb56c2a6e684 /home/michael/1Password Emergency Kit A3-CP6WC8-team-influxdata.pdf

feat: build binaries and Docker images in CI #24751

feat: build binaries and Docker images in CI #24751

Conversation

mgattozzi commented Mar 11, 2024 • edited Loading

mgattozzi commented Mar 26, 2024

mgattozzi commented Mar 26, 2024

hiltontj left a comment

Choose a reason for hiding this comment

bnpfeife commented Mar 28, 2024

mgattozzi commented Mar 28, 2024

hiltontj left a comment

Choose a reason for hiding this comment

mgattozzi commented Apr 2, 2024

jdstrand Apr 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bnpfeife commented Apr 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mgattozzi commented Apr 5, 2024

mgattozzi commented Apr 9, 2024

jdstrand left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mgattozzi commented Apr 10, 2024

bnpfeife commented Apr 18, 2024 • edited by jdstrand Loading

jdstrand commented Apr 18, 2024

mgattozzi commented Apr 23, 2024

jdstrand commented Apr 23, 2024

mgattozzi commented Apr 25, 2024 • edited Loading

jdstrand left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bnpfeife commented Apr 26, 2024 • edited Loading

mgattozzi commented Apr 29, 2024

mgattozzi commented May 2, 2024

mgattozzi commented Mar 11, 2024 •

edited

Loading

jdstrand Apr 3, 2024 •

edited

Loading

bnpfeife commented Apr 4, 2024 •

edited

Loading

bnpfeife commented Apr 18, 2024 •

edited by jdstrand

Loading

mgattozzi commented Apr 25, 2024 •

edited

Loading

bnpfeife commented Apr 26, 2024 •

edited

Loading