Move GCE config to google-guest-agent #150837

abbradar · 2021-12-15T12:14:58Z

Motivation for this change

google-compute-engine package is currently deprecated. Move us to a newer set of packages from Google.

Things done

Tested in production -- works, including both OS Login and fetched keys.

pkgs/tools/virtualization/google-guest-agent/default.nix

putchar · 2022-01-07T18:25:10Z

Hello

I tested this pr the following way

i used https://github.com/nix-community/nixos-generators + flake to create my image locally

{
  inputs = {
    nixpkgs.url = "github:abbradar/nixpkgs/google-guest-agent";
    nixos-generators = {
      url = "github:nix-community/nixos-generators";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };
  outputs = { self, nixpkgs, nixos-generators, ... }: {
    packages.x86_64-linux = {
      gce = nixos-generators.nixosGenerate {
        pkgs = nixpkgs.legacyPackages.x86_64-linux;
        format = "gce";
      };
    };
  };
}

then nix build .#gce

i pushed the tar.gz on a bucket. then made a vm image from it

both login from gcp console works (with metadata, with ssh-key)
the agent does push my public keys properly

also it works with
gcloud beta compute ssh --zone "europe-west1-b" "instance-1" --tunnel-through-iap --project "myproject"

flokli · 2022-01-08T19:10:42Z

@abbradar how is this related to #144761?

abbradar · 2022-01-08T20:32:14Z

@flokli I didn't see this one -- my patchset was done before #144761, but was used in our production privately. Thanks for the heads up!

After a brief comparison, I see these differences:

I moved google-compute-engine-oslogin to google-guest-oslogin, but I don't see any reason to do this now -- it just seems plain wrong. I'll investigate;
nixos/google-compute-config: fix module #144761 disables OS Login in google-guest-agent. I didn't see any issues with it back when implementing my patchset, though it was more than half a year ago and I might have missed something. OS Login seems to work now but I'll re-check in case any error messages appear;
I use upstream systemd units and as little extra configuration as possible. OTOH nixos/google-compute-config: fix module #144761 might have some interesting configuration options that I missed. Need to compare.

Overall both PRs do the same thing, so I'm okay with closing this one and helping with #144761. First we need to investigate why does my PR work correctly with OS Login (or at least used to work). I also see some interesting comments re:ZFS there.

picnoir · 2022-01-10T08:31:29Z

Overall both PRs do the same thing, so I'm okay with closing this one and helping with #144761. First we need to investigate why does my PR work correctly with OS Login (or at least used to work). I also see some interesting comments re:ZFS there.

As I was saying in 166761, I currently do not have any bandwidth to move it forward. Let's rather push your PR forward.

WRT. oslogin: the guest agent is going to try to setup the PAM module, NSS config & ssh config (see https://github.com/GoogleCloudPlatform/guest-agent/blob/main/google_guest_agent/oslogin.go#L99). It'll fail on NixOS (these paths are read-only).

Your PR is working fine because you're setting these using the NixOS module system. However, I think we should try to prevent the agent from trying to setup these as we do in #144761. We should upstream the patch allowing us to disable this config.

Apart from this, I don't see any major blocker in this PR. Ping me when you fix this and I'll properly review this PR.

abbradar · 2022-01-10T19:43:02Z

After more investigation I merged two PRs while addressing concerns above. Mainly we still fully support both legacy login via metadata (if users.mutableUsers = true) and OS Login. I added another patch for guest-agent, allowing to disable some parts of accounts daemon that produced errors. With that we now have clean logs without any errors.

While testing I discovered that OS Login test started to fail on newer unstable and sometimes on development machines I tested it on. Turns out we don't always start nscd daemon, which is required for successful OS Login. I split the fix into a separate PR coming soon.

Keep in mind that nixos.patch here is temporary; I intend to submit both patches to upstream in the coming days, but you can already start reviewing this new patchset.

During working on NixOS#150837 I discovered that `google-oslogin` test started failing, and so did some of my development machines. Turns out it was because nscd doesn't start by default; rather it's wanted by NSS lookup targets, which are not always fired up. To quote from section on systemd.special(7) on `nss-user-lookup.target`: > All services which provide parts of the user/group database should be > ordered before this target, and pull it in. Following this advice and comparing our unit to official `sssd.service` unit (which is a similar service), we now pull NSS lookup targets from the service, while starting it with `multi-user.target`.

abbradar · 2022-01-10T20:57:44Z

Opened GoogleCloudPlatform/guest-agent#152.

abbradar · 2022-01-11T17:20:23Z

I have replaced nixos.patch with links to patches submitted upstream, so no pending tasks remain for this PR before another round of reviews.

During working on #150837 I discovered that `google-oslogin` test started failing, and so did some of my development machines. Turns out it was because nscd doesn't start by default; rather it's wanted by NSS lookup targets, which are not always fired up. To quote from section on systemd.special(7) on `nss-user-lookup.target`: > All services which provide parts of the user/group database should be > ordered before this target, and pull it in. Following this advice and comparing our unit to official `sssd.service` unit (which is a similar service), we now pull NSS lookup targets from the service, while starting it with `multi-user.target`. (cherry picked from commit b451eca)

abbradar · 2022-01-12T20:33:30Z

Sadly upstream decided against including our changes. Given that, I propose to simplify the patch and just disable problematic features while changing minimal number of lines. This way the patch will be easier to maintain out-of-tree. @NinjaTrappeur what do you think?

abbradar · 2022-01-13T15:48:11Z

Minimized and moved our changes back to nixos.patch. Ready for review.

The python scripts we're using have been deprecated in favor of the guest agent, see more infos at: GoogleCloudPlatform/compute-image-packages@276e520#diff-267a2788071ee63df1443363c2ab882e7d321adc77ee53462a920229a962eabbL40 The guest agent onboard all the old python scripts as part of two new go binaries: google_guest_agent and google_metadata_script_runner. Both are using the same /etc/default-instance_configs.cfg configuration file. The configuration file we embed in this module mimicks the setup we had with the python scripts. We'll have two services: - The metadata script service: in charge of running the google compute startup/showdown scripts. - The guest agent: in charge of syncing the guest clock with the host and setup the network interfaces on boot. Co-authored-by: Mark Karpov <[email protected]>

andre-vspry · 2022-02-03T22:45:11Z

I'd like to see this merged. It would make upgrading to 21.11 much easier for GCP infra. Does it need more testing?

flokli

Some small styling nits to make this a bit more robust in case of upgrades.

Other than that, LGTM. I ran the tests and they still succeed.

pkgs/tools/virtualization/google-guest-configs/default.nix

pkgs/tools/virtualization/google-guest-agent/default.nix

flokli

Nice, thanks! Should we add some release notes for this - is there things in how this behaves differently from the previous release?

flokli

Hold on… the VM tests seem to fail now:

^[[2mserver # [    7.345607] python[856]: 169.254.169.254 - - [05/Feb/2022 20:18:47] "GET /computeMetadata/v1/oslogin/users?username=mockuser_nixos_org HTTP/1.1" 200 -^[[0m
^[[2mserver # [    7.346350] python[856]: b'{"loginProfiles": [{"name": "418156207295006328628", "posixAccounts": [{"primary": true, "username": "mockuser_nixos_org", "uid": "1009719690", "gid": "1009719690", "homeDirectory": "/home/mockuser_nixos_org", "operatingSystemType": "LINUX"}], "sshPublicKeys": {"fa80819b5035333b770ae3a7cf7f76ada71c2739e87419509064dc4106b91d48": {"key": "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBChdA2BmwcG49OrQN33f/sj+OHL5sJhwVl2Qim0vkUJQCry1zFpKTa9ZcDMiWaEhoAR6FGoaGI04ff7CS+1yybQ= snakeoil", "expirationTimeUsec": "1644092927255555.2", "fingerprint": "fa80819b5035333b770ae3a7cf7f76ada71c2739e87419509064dc4106b91d48"}}}]}'^[[0m
^[[2mserver # [    7.349218] python[856]: 169.254.169.254 - - [05/Feb/2022 20:18:47] "GET /computeMetadata/v1/oslogin/users?username=mockuser_nixos_org HTTP/1.1" 200 -^[[0m
^[[2mserver # [    7.350087] python[856]: b'{"loginProfiles": [{"name": "418156207295006328628", "posixAccounts": [{"primary": true, "username": "mockuser_nixos_org", "uid": "1009719690", "gid": "1009719690", "homeDirectory": "/home/mockuser_nixos_org", "operatingSystemType": "LINUX"}], "sshPublicKeys": {"fa80819b5035333b770ae3a7cf7f76ada71c2739e87419509064dc4106b91d48": {"key": "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBChdA2BmwcG49OrQN33f/sj+OHL5sJhwVl2Qim0vkUJQCry1zFpKTa9ZcDMiWaEhoAR6FGoaGI04ff7CS+1yybQ= snakeoil", "expirationTimeUsec": "1644092927256354.2", "fingerprint": "fa80819b5035333b770ae3a7cf7f76ada71c2739e87419509064dc4106b91d48"}}}]}'^[[0m
^[[2mserver # [    7.352931] python[856]: 169.254.169.254 - - [05/Feb/2022 20:18:47] "GET /computeMetadata/v1/oslogin/users?username=mockuser_nixos_org HTTP/1.1" 200 -^[[0m
^[[2mserver # [    7.353733] python[856]: b'{"loginProfiles": [{"name": "418156207295006328628", "posixAccounts": [{"primary": true, "username": "mockuser_nixos_org", "uid": "1009719690", "gid": "1009719690", "homeDirectory": "/home/mockuser_nixos_org", "operatingSystemType": "LINUX"}], "sshPublicKeys": {"fa80819b5035333b770ae3a7cf7f76ada71c2739e87419509064dc4106b91d48": {"key": "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBChdA2BmwcG49OrQN33f/sj+OHL5sJhwVl2Qim0vkUJQCry1zFpKTa9ZcDMiWaEhoAR6FGoaGI04ff7CS+1yybQ= snakeoil", "expirationTimeUsec": "1644092927299759.2", "fingerprint": "fa80819b5035333b770ae3a7cf7f76ada71c2739e87419509064dc4106b91d48"}}}]}'^[[0m
^[[2mserver # [    7.357394] python[856]: 169.254.169.254 - - [05/Feb/2022 20:18:47] "GET /computeMetadata/v1/oslogin/users?username=mockuser_nixos_org HTTP/1.1" 200 -^[[0m
client: output: 
cleanup
kill machine (pid 9)
^[[2mserver # [    7.358662] python[856]: b'{"loginProfiles": [{"name": "418156207295006328628", "posixAccounts": [{"primary": true, "username": "mockuser_nixos_org", "uid": "1009719690", "gid": "1009719690", "homeDirectory": "/home/mockuser_nixos_org", "operatingSystemType": "LINUX"}], "sshPublicKeys": {"fa80819b5035333b770ae3a7cf7f76ada71c2739e87419509064dc4106b91d48": {"key": "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBChdA2BmwcG49OrQN33f/sj+OHL5sJhwVl2Qim0vkUJQCry1zFpKTa9ZcDMiWaEhoAR6FGoaGI04ff7CS+1yybQ= snakeoil", "expirationTimeUsec": "1644092927307183.2", "fingerprint": "fa80819b5035333b770ae3a7cf7f76ada71c2739e87419509064dc4106b91d48"}}}]}'^[[0m
^[[2mserver # [    7.361550] python[856]: Unhandled path: /computeMetadata/v1/oslogin/groups?email=418156207295006328628^[[0m
^[[2mserver # [    7.362276] python[856]: 169.254.169.254 - - [05/Feb/2022 20:18:47] "GET /computeMetadata/v1/oslogin/groups?email=418156207295006328628 HTTP/1.1" 404 -^[[0m
^[[2mserver # [    7.363560] sshd[895]: Received disconnect from 192.168.1.1 port 58964:11: disconnected by user^[[0m
^[[2mclient # qemu-system-x86_64: terminating on signal 15 from pid 6 (/nix/store/dn4fwp0yx6nsa85cr20cwvdmg64xwmcy-python3-3.9.9/bin/python3.9)^[[0m
^[[2mserver # [    7.366536] sshd[895]: Disconnected from user mockuser_nixos_org 192.168.1.1 port 58964^[[0m
kill machine (pid 22)
^[[2mserver # qemu-system-x86_64: terminating on signal 15 from pid 6 (/nix/store/dn4fwp0yx6nsa85cr20cwvdmg64xwmcy-python3-3.9.9/bin/python3.9)^[[0m
(finished: cleanup, in 0.02 seconds)
Traceback (most recent call last):
  File "/nix/store/97nb219kaxqwj94cj596zcyzyw8d4ix7-nixos-test-driver-1.1/bin/.nixos-test-driver-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/97nb219kaxqwj94cj596zcyzyw8d4ix7-nixos-test-driver-1.1/lib/python3.9/site-packages/test_driver/__init__.py", line 86, in main
    driver.run_tests()
  File "/nix/store/97nb219kaxqwj94cj596zcyzyw8d4ix7-nixos-test-driver-1.1/lib/python3.9/site-packages/test_driver/driver.py", line 121, in run_tests
    self.test_script()
  File "/nix/store/97nb219kaxqwj94cj596zcyzyw8d4ix7-nixos-test-driver-1.1/lib/python3.9/site-packages/test_driver/driver.py", line 117, in test_script
    exec(self.tests, symbols, None)
  File "<string>", line 31, in <module>
  File "/nix/store/97nb219kaxqwj94cj596zcyzyw8d4ix7-nixos-test-driver-1.1/lib/python3.9/site-packages/test_driver/machine.py", line 559, in succeed
    raise Exception(
Exception: command `ssh mockuser_nixos_org@server 'true'` failed (exit code 254)
kill vlan (pid 8)
kill vlan (pid 7)

Full log

abbradar · 2022-02-05T20:26:47Z

Ugh. I'll check this, thanks for noticing.

re release notes: I'm honestly not sure if OS Login and old login path (via adding mutable users) worked well before. If they didn't, we effectively fixed a bug. Other than that -- should have no visible changes.

* Rename to google-guest-oslogin; * Fix systemd service; * Small cleanup.

abbradar · 2022-02-05T23:36:25Z

It works for me when I rebase atop unstable. I think it was caused by #157112.

flokli · 2022-02-06T10:27:23Z

RIght, with the rebase, the test succeeds again. Thanks!

andre-vspry · 2022-02-10T22:34:49Z

Is this suitable for backporting to 21.11 also?

flokli · 2022-02-10T23:48:41Z

This is a bit too scary to backport IMHO. People who want this before the next release can just stick to unstable for the time being.

jonringer · 2022-02-11T00:17:31Z

Agreed. If these changes expect a user to react, then it should be postponed. Stable should allow for users to determine "when" to eat the upgrade cost going between release branches, rather than some update.

andre-vspry · 2022-02-11T00:23:10Z

Wouldn't it only impact people running 21.11 on GCP? I'm stuck on 21.05 for GCP because the agent doesn't start and therefore doesn't get project provided ssh keys. If I build a custom image with keys then I can log in but the metadata service is borked. Unless I've made some error in deploying an image, I would say not many, if any, are running 21.11 on GCP with a working agent.

picnoir · 2022-02-11T07:14:06Z

Hmm, strange. The #148467 revert (backported to 21.11 via #148936) should have fixed the agent startup failure. It looks like we have another problem here. That's unrelated to this PR though. Could you open a new issue w/ the full error logs?

andre-vspry · 2022-02-14T00:37:01Z

I didn't realise that the google cloud console "Equivalent command line" button omits the --metadata switch. This lead to the instance not getting ssh keys from oslogin. Since the console log was all I could see and it showed the fetch ssh keys failing, I assumed this was the issue.

My mistake. I've opened an issue for the misleading service failure. #159921

abbradar requested review from FRidh and jonringer as code owners December 15, 2021 12:14

github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 6.topic: python 8.has: module (update) This PR changes an existing module in `nixos/` labels Dec 15, 2021

abbradar force-pushed the google-guest-agent branch 2 times, most recently from 34b2394 to 04caaa5 Compare December 15, 2021 12:24

github-actions bot removed the 6.topic: python label Dec 15, 2021

ofborg bot added 8.has: clean-up 8.has: package (new) This PR adds a new package labels Dec 15, 2021

ofborg bot requested a review from kalbasit December 15, 2021 12:41

ofborg bot added 11.by: package-maintainer This PR was created by the maintainer of the package it changes 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1-10 labels Dec 15, 2021

06kellyjac reviewed Jan 5, 2022

View reviewed changes

pkgs/tools/virtualization/google-guest-agent/default.nix Outdated Show resolved Hide resolved

abbradar force-pushed the google-guest-agent branch from 04caaa5 to 285fa69 Compare January 5, 2022 11:26

abbradar requested a review from flokli January 8, 2022 08:57

abbradar force-pushed the google-guest-agent branch from 285fa69 to 80dceec Compare January 10, 2022 19:37

abbradar mentioned this pull request Jan 10, 2022

nscd service: fix ordering and start automatically #154320

Merged

13 tasks

abbradar force-pushed the google-guest-agent branch from 80dceec to 7c76ca0 Compare January 11, 2022 17:18

abbradar force-pushed the google-guest-agent branch 3 times, most recently from 88bbe6c to e2b3f53 Compare January 13, 2022 15:44

flokli requested changes Feb 4, 2022

View reviewed changes

pkgs/tools/virtualization/google-guest-configs/default.nix Outdated Show resolved Hide resolved

pkgs/tools/virtualization/google-guest-agent/default.nix Outdated Show resolved Hide resolved

abbradar force-pushed the google-guest-agent branch from e2b3f53 to a1f273f Compare February 5, 2022 11:00

ofborg bot requested a review from flokli February 5, 2022 11:12

flokli approved these changes Feb 5, 2022

View reviewed changes

flokli requested changes Feb 5, 2022

View reviewed changes

abbradar added 4 commits February 5, 2022 23:33

google-guest-agent: init at 20220104.00

6c3dd63

google-guest-configs: init at 20211116.00

c3f278b

google-compute-engine-oslogin: 20210429.00 -> 20211213.00

077d052

* Rename to google-guest-oslogin; * Fix systemd service; * Small cleanup.

google-compute-config: update config

524aecf

abbradar force-pushed the google-guest-agent branch from a1f273f to 524aecf Compare February 5, 2022 23:36

ofborg bot requested a review from flokli February 6, 2022 01:03

flokli merged commit def482e into NixOS:master Feb 6, 2022

andre-vspry mentioned this pull request Feb 14, 2022

fetch-instance-ssh-keys.service on GCP with 21.11 shows failed when there is no metadata for sshKeys #159921

Open

c0bw3b mentioned this pull request Feb 24, 2022

google-compute-engine-oslogin: 20210429.00 -> 20210707.00 #131374

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move GCE config to google-guest-agent #150837

Move GCE config to google-guest-agent #150837

abbradar commented Dec 15, 2021

putchar commented Jan 7, 2022 •

edited

Loading

flokli commented Jan 8, 2022

abbradar commented Jan 8, 2022

picnoir commented Jan 10, 2022 •

edited

Loading

abbradar commented Jan 10, 2022 •

edited

Loading

abbradar commented Jan 10, 2022

abbradar commented Jan 11, 2022

abbradar commented Jan 12, 2022

abbradar commented Jan 13, 2022

andre-vspry commented Feb 3, 2022

flokli left a comment

flokli left a comment

flokli left a comment

abbradar commented Feb 5, 2022

abbradar commented Feb 5, 2022

flokli commented Feb 6, 2022

andre-vspry commented Feb 10, 2022

flokli commented Feb 10, 2022

jonringer commented Feb 11, 2022

andre-vspry commented Feb 11, 2022

picnoir commented Feb 11, 2022 via email

andre-vspry commented Feb 14, 2022 •

edited

Loading

Move GCE config to google-guest-agent #150837

Move GCE config to google-guest-agent #150837

Conversation

abbradar commented Dec 15, 2021

Motivation for this change

Things done

putchar commented Jan 7, 2022 • edited Loading

flokli commented Jan 8, 2022

abbradar commented Jan 8, 2022

picnoir commented Jan 10, 2022 • edited Loading

abbradar commented Jan 10, 2022 • edited Loading

abbradar commented Jan 10, 2022

abbradar commented Jan 11, 2022

abbradar commented Jan 12, 2022

abbradar commented Jan 13, 2022

andre-vspry commented Feb 3, 2022

flokli left a comment

Choose a reason for hiding this comment

flokli left a comment

Choose a reason for hiding this comment

flokli left a comment

Choose a reason for hiding this comment

abbradar commented Feb 5, 2022

abbradar commented Feb 5, 2022

flokli commented Feb 6, 2022

andre-vspry commented Feb 10, 2022

flokli commented Feb 10, 2022

jonringer commented Feb 11, 2022

andre-vspry commented Feb 11, 2022

picnoir commented Feb 11, 2022 via email

andre-vspry commented Feb 14, 2022 • edited Loading

putchar commented Jan 7, 2022 •

edited

Loading

picnoir commented Jan 10, 2022 •

edited

Loading

abbradar commented Jan 10, 2022 •

edited

Loading

andre-vspry commented Feb 14, 2022 •

edited

Loading