Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI often fails with "Could not resolve host: github.com" #549

Open
eu9ene opened this issue Apr 30, 2024 · 20 comments
Open

CI often fails with "Could not resolve host: github.com" #549

eu9ene opened this issue Apr 30, 2024 · 20 comments
Labels
bug Something is broken or not correct taskcluster Issues related to the Taskcluster implementation of the training pipeline

Comments

@eu9ene
Copy link
Collaborator

eu9ene commented Apr 30, 2024

If it's expected that it can fail we should add retries to all our steps:

[vcs 2024-04-30T01:09:53.281Z] fatal: unable to access 'https://github.com/mozilla/firefox-translations-training/': Could not resolve host: github.com

Here is an example task: https://firefox-ci-tc.services.mozilla.com/tasks/JdDA-zYDQnG4166Zbsqq6w
@eu9ene eu9ene added bug Something is broken or not correct taskcluster Issues related to the Taskcluster implementation of the training pipeline labels Apr 30, 2024
@bhearsum
Copy link
Collaborator

bhearsum commented May 1, 2024

I'm asking around other projects to see if they're seeing this as well.

@bhearsum
Copy link
Collaborator

bhearsum commented May 2, 2024

Haven't seen reports of this elsewhere.

@eu9ene - have you seen this on GPU workers only? Or also on the CPU workers?

@eu9ene
Copy link
Collaborator Author

eu9ene commented May 2, 2024

Haven't seen reports of this elsewhere.

@eu9ene - have you seen this on GPU workers only? Or also on the CPU workers?

I'm not sure but I feel like I've been seeing this in random places.

@gregtatum
Copy link
Member

Here's an instance on a GPU worker: https://firefox-ci-tc.services.mozilla.com/tasks/XRYrb2BxTOyJa209jrLQ7A/runs/0/logs/public/logs/live.log

@bhearsum
Copy link
Collaborator

Thanks; so it seems very unlikely to be related to specific worker images.

@aerickson - I don't suppose you have any idea what's going on here?

@eu9ene eu9ene mentioned this issue May 15, 2024
Merged
@aerickson
Copy link
Member

@bhearsum I'm not sure what's going on. Translations GPU workers on GCP should use GCP's DNS servers (provided by DHCP) and the Snakepit worker use our internal infoblox servers (configured in dnsmasq). It seems like a network blip or perhaps the DNS server could be overloaded for a second?

I haven't heard about any Github outages around DNS. I've never really heard of DNS outages (it's a pretty resilient service/protocol).

If we find a concentrated event or location let me know and I'll dig in some more.

@bhearsum
Copy link
Collaborator

We do see these fairly often - I would say maybe on 5-10% of the tasks run. I'll try to collect some data to help us analyze this better.

@bhearsum
Copy link
Collaborator

Here's failures by worker group:

defaultdict(<class 'int'>,
            {'us-central1': 7,
             'us-central1-a': 9,
             'us-central1-b': 7,
             'us-central1-c': 7,
             'us-central1-f': 9,
             'us-west1': 5,
             'us-west1-a': 5,
             'us-west1-b': 11})

And here's timestamps when we hit the failures:

['2024-01-19T21:20:34.616Z',
 '2024-02-02T20:02:49.997Z',
 '2024-02-22T16:07:57.737Z',
 '2024-02-26T23:13:59.353Z',
 '2024-02-28T00:36:35.015Z',
 '2024-02-28T17:48:06.906Z',
 '2024-02-29T15:03:54.666Z',
 '2024-03-06T14:32:56.066Z',
 '2024-03-06T18:10:27.186Z',
 '2024-03-21T15:03:54.430Z',
 '2024-03-21T20:29:42.584Z',
 '2024-03-26T13:29:20.434Z',
 '2024-03-30T22:11:50.010Z',
 '2024-04-01T14:10:30.644Z',
 '2024-04-01T15:18:22.619Z',
 '2024-04-09T10:47:13.166Z',
 '2024-04-18T12:41:01.952Z',
 '2024-04-22T19:58:27.752Z',
 '2024-04-23T19:46:39.366Z',
 '2024-04-24T17:57:31.559Z',
 '2024-04-25T07:33:55.024Z',
 '2024-04-29T17:55:15.937Z',
 '2024-04-30T00:38:45.847Z',
 '2024-04-30T01:08:46.769Z',
 '2024-04-30T14:06:18.417Z',
 '2024-04-30T18:56:31.177Z',
 '2024-05-01T23:21:11.540Z',
 '2024-05-02T00:22:43.542Z',
 '2024-05-06T14:40:00.719Z',
 '2024-05-06T16:41:19.051Z',
 '2024-05-07T00:01:30.432Z',
 '2024-05-07T21:23:47.145Z',
 '2024-05-07T21:36:54.706Z',
 '2024-05-09T19:01:17.356Z',
 '2024-05-09T22:10:58.358Z',
 '2024-05-09T22:11:47.850Z',
 '2024-05-10T18:42:00.163Z',
 '2024-05-10T22:14:23.709Z',
 '2024-05-13T23:01:23.656Z',
 '2024-05-13T23:24:46.266Z',
 '2024-05-14T14:49:27.875Z',
 '2024-05-14T15:08:43.843Z',
 '2024-05-14T23:18:39.852Z',
 '2024-05-14T23:43:02.154Z',
 '2024-05-15T01:01:44.678Z',
 '2024-05-15T14:30:49.732Z',
 '2024-05-15T23:20:08.322Z',
 '2024-05-16T21:56:58.285Z',
 '2024-05-17T11:38:30.465Z',
 '2024-05-17T14:02:32.250Z',
 '2024-05-17T16:18:16.690Z',
 '2024-05-17T17:51:07.523Z',
 '2024-05-17T20:37:44.360Z',
 '2024-05-17T20:49:01.813Z',
 '2024-05-17T20:49:18.471Z',
 '2024-05-17T21:36:37.947Z',
 '2024-05-17T22:05:41.152Z',
 '2024-05-17T22:06:42.275Z',
 '2024-05-17T23:11:32.583Z',
 '2024-05-20T15:29:29.312Z']

And by worker image:

defaultdict(<class 'int'>, {'gpu': 58, 'cpu': 2})

Clearly the most notable part here here is that we're seeing more issues on GPU images. And within that, we seem to have gotten more beginning in late April/early May. We added a test pool with a new image on April 22nd (that I was running a lot to test things) in https://phabricator.services.mozilla.com/D208202. That image went to production on May 8th in https://phabricator.services.mozilla.com/D209840.

mozilla-platform-ops/monopacker#140 was the PR related to this image, but I don't know how deterministic the other parts are? Eg: could we have picked up a change to a system package that is now causing problems?

@aerickson - do you have any thoughts? If we still have the old image, maybe we could poke around and compare the new to old one? (I'd be happy to do this if you want.)

@eu9ene eu9ene mentioned this issue May 22, 2024
Merged
@eu9ene
Copy link
Collaborator Author

eu9ene commented May 22, 2024

Ok, it happens every single pipeline run for me now and does not restart. Marking as blocker.

@eu9ene eu9ene added the blocker Very important issue that blocks training label May 22, 2024
bhearsum added a commit to bhearsum/firefox-translations-training that referenced this issue May 22, 2024
@bhearsum
Retry tasks on exit code 128 to work around intermittent issues with …
85fdc78 
…DNS failures

This should help make mozilla#549 less painful. I suggest we back it out once we get to the bottom of that.
@bhearsum bhearsum mentioned this issue May 22, 2024
Merged
bhearsum added a commit to bhearsum/firefox-translations-training that referenced this issue May 22, 2024
@bhearsum
Retry tasks on exit code 128 to work around intermittent issues with …
23551c0 
…DNS failures

This should help make mozilla#549 less painful. I suggest we back it out once we get to the bottom of that.
@bhearsum
Copy link
Collaborator

Poking at this a bit on an interactive instance, too. I sortof repro'ed (it seems to have retried with success though):

ubuntu@translations-1-b-linux-v100-gpu-ecnlewoqrtmrltrxrl8o3w:~/tasks/task_171642353933996$ host github.com
;; communications error to 127.0.0.53#53: timed out
github.com has address 140.82.116.3
github.com mail is handled by 5 alt2.aspmx.l.google.com.
github.com mail is handled by 1 aspmx.l.google.com.
github.com mail is handled by 5 alt1.aspmx.l.google.com.
github.com mail is handled by 10 alt3.aspmx.l.google.com.
github.com mail is handled by 10 alt4.aspmx.l.google.com.

Looking at the machine configuration, I see that it uses the standard resolved that we expect on Ubuntu:

ubuntu@translations-1-b-linux-v100-gpu-ecnlewoqrtmrltrxrl8o3w:~/tasks/task_171642353933996$ cat /etc/resolv.conf
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search c.fxci-production-level1-workers.internal google.internal
ubuntu@translations-1-b-linux-v100-gpu-ecnlewoqrtmrltrxrl8o3w:~/tasks/task_171642353933996$ resolvectl status
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (ens5)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 169.254.169.254
       DNS Servers: 169.254.169.254
        DNS Domain: c.fxci-production-level1-workers.internal google.internal

It claims to be timing out talking to 127.0.0.53 - but I don't know if that means it really couldn't talk to the local resolver, or if that's just the local resolver passing along a failure from the upstream. It seems more likely that it's the latter, but I can't say that with any certainty.

The upstream server is a reserved address, and I'm guessing it's something internal to GCP? I'm really not sure, to be honest - that's quite out of my depth.

I looked through syslogs and found nothing of note, just messages like this every time I perfomed a lookup:

May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Received dns UDP packet of size 28, ifindex=0, ttl=64, fragsize=0
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Got DNS stub UDP query packet for id 63009
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Looking up RR for github.com IN A.
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Cache miss for github.com IN A
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Firing regular transaction 18081 for <github.com IN A> scope dns on ens5/* (validate=yes).
q4cpec8nldnk-q systemd-resolved[445]: Received dns UDP packet of size 28, ifindex=0, ttl=64, fragsize=0
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Got DNS stub UDP query packet for id 7454
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Looking up RR for github.com IN MX.
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Cache miss for github.com IN MX
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Firing regular transaction 16268 for <github.com IN MX> scope dns on ens5/* (validate=yes).
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Using feature level UDP+EDNS0 for transaction 16268.
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Using DNS server 169.254.169.254 for transaction 16268.
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Announcing packet size 1432 in egress EDNS(0) packet.
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Emitting UDP, link MTU is 1460, socket MTU is 0, minimal MTU is 40
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Sending query packet with id 16268 of size 39.
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Processing query...
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Received dns UDP packet of size 154, ifindex=2, ttl=0, fragsize=0
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Processing incoming packet of size 154 on transaction 16268 (rcode=SUCCESS).
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Added positive unauthenticated non-confidential cache entry for github.com IN MX 3598s on ens5/INET/169.254.169.254
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: message repeated 4 times: [ Added positive unauthenticated non-confidential cache entry for github.com IN MX 3598s on ens5/INET/169.254.169.254]
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Regular transaction 16268 for <github.com IN MX> on scope dns on ens5/* now complete with <success> from network (unsigned; non-confidential).
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Sending response packet with id 7454 on interface 1/AF_INET of size 143.
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Freeing transaction 16268.

bhearsum added a commit to bhearsum/firefox-translations-training that referenced this issue May 23, 2024
@bhearsum
Retry tasks on exit code 128 to work around intermittent issues with …
3e74948 
…DNS failures

This should help make mozilla#549 less painful. I suggest we back it out once we get to the bottom of that.
bhearsum added a commit to bhearsum/firefox-translations-training that referenced this issue May 23, 2024
@bhearsum
Retry tasks on exit code 128 to work around intermittent issues with …
09b8802 
…DNS failures

This should help make mozilla#549 less painful. I suggest we back it out once we get to the bottom of that.
@bhearsum
Copy link
Collaborator

I was looking through worker logs of a worker that had a dns issue in production and found other things of interest.

In the task we had:

[vcs 2024-05-23T19:33:53.574Z] executing ['git', 'clone', 'https://github.com/mozilla/firefox-translations-training', '/home/ubuntu/tasks/task_171649283082993/checkouts/vcs']
[vcs 2024-05-23T19:33:53.582Z] Cloning into '/home/ubuntu/tasks/task_171649283082993/checkouts/vcs'...
[vcs 2024-05-23T19:34:13.971Z] fatal: unable to access 'https://github.com/mozilla/firefox-translations-training/': Could not resolve host: github.com

And in the syslogs I found:

May 23 19:34:32 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A.
2024-05-23 20:34:41.006
May 23 19:34:32 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Cache miss for github.com IN A
2024-05-23 20:34:41.006
May 23 19:34:32 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Firing regular transaction 65143 for <github.com IN A> scope dns on ens5/* (validate=yes).
2024-05-23 20:34:41.006
May 23 19:34:32 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN AAAA.
2024-05-23 20:34:41.006
May 23 19:34:32 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Cache miss for github.com IN AAAA
2024-05-23 20:34:41.006
May 23 19:34:32 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Firing regular transaction 23655 for <github.com IN AAAA> scope dns on ens5/* (validate=yes).
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Added positive unauthenticated non-confidential cache entry for github.com IN A 60s on ens5/INET/169.254.169.254
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Regular transaction 65143 for <github.com IN A> on scope dns on ens5/* now complete with <success> from network (unsigned; non-confidential).
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Not caching negative entry for: github.com IN AAAA, cache mode set to no-negative
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Regular transaction 23655 for <github.com IN AAAA> on scope dns on ens5/* now complete with <success> from network (unsigned; non-confidential).
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A.
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Positive cache hit for github.com IN A
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Regular transaction 7141 for <github.com IN A> on scope dns on ens5/* now complete with <success> from cache (unsigned; non-confidential).
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN AAAA.
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Cache miss for github.com IN AAAA
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Firing regular transaction 60167 for <github.com IN AAAA> scope dns on ens5/* (validate=yes).
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Not caching negative entry for: github.com IN AAAA, cache mode set to no-negative
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Regular transaction 60167 for <github.com IN AAAA> on scope dns on ens5/* now complete with <success> from network (unsigned; non-confidential).
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com.c.fxci-production-level1-workers.internal IN A.
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Cache miss for github.com.c.fxci-production-level1-workers.internal IN A
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Firing regular transaction 48751 for <github.com.c.fxci-production-level1-workers.internal IN A> scope dns on ens5/* (validate=yes).
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com.c.fxci-production-level1-workers.internal IN AAAA.
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Cache miss for github.com.c.fxci-production-level1-workers.internal IN AAAA
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Firing regular transaction 26112 for <github.com.c.fxci-production-level1-workers.internal IN AAAA> scope dns on ens5/* (validate=yes).
2024-05-23 20:34:48.739
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Cache miss for github.com.c.fxci-production-level1-workers.internal IN A
2024-05-23 20:34:48.739
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Firing regular transaction 48751 for <github.com.c.fxci-production-level1-workers.internal IN A> scope dns on ens5/* (validate=yes).
2024-05-23 20:34:48.739
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Cache miss for github.com.c.fxci-production-level1-workers.internal IN AAAA
2024-05-23 20:34:48.739
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Firing regular transaction 26112 for <github.com.c.fxci-production-level1-workers.internal IN AAAA> scope dns on ens5/* (validate=yes).
2024-05-23 20:34:48.739
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Not caching negative entry for: github.com.c.fxci-production-level1-workers.internal IN A, cache mode set to no-negative
2024-05-23 20:34:48.739
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Regular transaction 48751 for <github.com.c.fxci-production-level1-workers.internal IN A> on scope dns on ens5/* now complete with <rcode-failure> from network (unsigned; non-confidential).
2024-05-23 20:34:48.739
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Not caching negative entry for: github.com.c.fxci-production-level1-workers.internal IN AAAA, cache mode set to no-negative
2024-05-23 20:34:48.739
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Regular transaction 26112 for <github.com.c.fxci-production-level1-workers.internal IN AAAA> on scope dns on ens5/* now complete with <rcode-failure> from network (unsigned; non-confidential).
2024-05-23 20:34:52.098
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com.c.fxci-production-level1-workers.internal IN A.
2024-05-23 20:34:52.098
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Cache miss for github.com.c.fxci-production-level1-workers.internal IN A
2024-05-23 20:34:52.098

(For whatever reason, there seems to be a timestamp discrepancy between the task log and the system logs, for example we have messages like May 23 19:34:23 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw start-worker[572]: 2024/05/23 19:33:51 Environment: []string that show a 32 second difference.)

Within a very short time we see:

  • A lookup for github.com succeed
  • A lookup for github.com.c.fxci-production-level1-workers.internal fail

Maybe that second lookup is expected, maybe it's not - I'm really not sure. The github.com one succeeds, the .internal one does not (of course). I searched through all the logs and there are no failures to resolve the real github.com - the only failures are for .internal domains.

I'm still not sure what to make of this, just dropping more info at the moment.

bhearsum added a commit that referenced this issue May 23, 2024
@bhearsum
Retry tasks on exit code 128 to work around intermittent issues with …
207a773 
…DNS failures (#624)

This should help make #549 less painful. I suggest we back it out once we get to the bottom of that.
@bhearsum
Copy link
Collaborator

bhearsum commented May 23, 2024 •
edited
Loading

Another interesting thing is that we 12 attempts to look up the A record for github.com in the span of 12 seconds, while there were 3 attempts to clone over the span of ~40 seconds (the last one succeeding):

"May 23 19:34:32 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."
"May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."
"May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."
"May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."
"May 23 19:34:37 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."
"May 23 19:34:38 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."
"May 23 19:34:39 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."
"May 23 19:34:41 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."
"May 23 19:34:41 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."
"May 23 19:34:41 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."
"May 23 19:34:42 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."
"May 23 19:34:44 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."

[vcs 2024-05-23T19:33:53.574Z] executing ['git', 'clone', 'https://github.com/mozilla/firefox-translations-training', '/home/ubuntu/tasks/task_171649283082993/checkouts/vcs']
[vcs 2024-05-23T19:33:53.582Z] Cloning into '/home/ubuntu/tasks/task_171649283082993/checkouts/vcs'...
[vcs 2024-05-23T19:34:13.971Z] fatal: unable to access 'https://github.com/mozilla/firefox-translations-training/': Could not resolve host: github.com
[vcs 2024-05-23T19:34:15.976Z] executing ['git', 'clone', 'https://github.com/mozilla/firefox-translations-training', '/home/ubuntu/tasks/task_171649283082993/checkouts/vcs']
[vcs 2024-05-23T19:34:15.978Z] Cloning into '/home/ubuntu/tasks/task_171649283082993/checkouts/vcs'...
[vcs 2024-05-23T19:34:33.081Z] fatal: unable to access 'https://github.com/mozilla/firefox-translations-training/': Could not resolve host: github.com
[vcs 2024-05-23T19:34:37.087Z] executing ['git', 'clone', 'https://github.com/mozilla/firefox-translations-training', '/home/ubuntu/tasks/task_171649283082993/checkouts/vcs']
[vcs 2024-05-23T19:34:37.089Z] Cloning into '/home/ubuntu/tasks/task_171649283082993/checkouts/vcs'...

@eu9ene eu9ene mentioned this issue May 24, 2024
Closed
@eu9ene
Copy link
Collaborator Author

eu9ene commented Jun 25, 2024

Did this go away by itself?

@bhearsum
Copy link
Collaborator

I'm not aware of anything we did to fix it. So if it's not happening anymore...yes!

@bhearsum bhearsum mentioned this issue Sep 20, 2024
Open
@bhearsum
Copy link
Collaborator

We spoke too soon clearly, given #855.

Some suggestions for further investigation from the Taskcluster weekly meeting:

  • try changing the type of network card in the VM if possible
  • resolvconf/dhcpd/other things may be fighting for control over the network/dns config
  • when we upgraded from ubuntu 22.04 -> 24.04 elsewhere there were conflicts between systemd-networkd and something else (similar to the above)
  • we could consider trying a different instance; we've seen weird network stuff go away by changing instance types before
  • another thing to look at is netplan configuration

@aerickson
Copy link
Member

mozilla-platform-ops/monopacker#140 was the PR related to this image, but I don't know how deterministic the other parts are? Eg: could we have picked up a change to a system package that is now causing problems?

@aerickson - do you have any thoughts? If we still have the old image, maybe we could poke around and compare the new to old one? (I'd be happy to do this if you want.)

It's possible. We don't pin the base image (we accept patch level updates like 22.04.x) and the scripts run apt update && apt upgrade, so it's possible. We do have the older images if you want to investigate. We're rolling out SBOMs too, so it will be easier to compare in the future (no booting an instance, you'll be able to diff a markdown file).

@bhearsum
Copy link
Collaborator

bhearsum commented Oct 9, 2024

I had a chance to compare the most recent GPU image to the previous one today. As far as packages installed goes, there's a bunch of minor upgrades but nothing that really stands out. The closest things to a network related package that changed are bind9-host and a very minor kernel upgrade - neither of which seem terribly likely to cause intermittent DNS issues. The full list of packages changes is as follows:

@@ -7,26 +7,26 @@
 ii  adduser                          3.118ubuntu5                            all          add and remove users and groups
 ii  adwaita-icon-theme               41.0-1ubuntu1                           all          default icon theme of GNOME (small subset)
 ii  alsa-topology-conf               1.2.5.1-2                               all          ALSA topology configuration files
-ii  alsa-ucm-conf                    1.2.6.3-1ubuntu1.10                     all          ALSA Use Case Manager configuration files
+ii  alsa-ucm-conf                    1.2.6.3-1ubuntu1.11                     all          ALSA Use Case Manager configuration files
 ii  apparmor                         3.0.4-2ubuntu2.3                        amd64        user-space parser utility for AppArmor
 ii  apport                           2.20.11-0ubuntu82.5                     all          automatically generate crash reports for debugging
 ii  apport-symptoms                  0.24                                    all          symptom scripts for apport
 ii  apt                              2.4.11                                  amd64        commandline package manager
-ii  apt-transport-https              2.4.11                                  all          transitional package for https support
+ii  apt-transport-https              2.4.12                                  all          transitional package for https support
 ii  apt-utils                        2.4.11                                  amd64        package management related utility programs
 ii  at-spi2-core                     2.44.0-3                                amd64        Assistive Technology Service Provider Interface (dbus core)
-ii  base-files                       12ubuntu4.5                             amd64        Debian base system miscellaneous files
+ii  base-files                       12ubuntu4.6                             amd64        Debian base system miscellaneous files
 ii  base-passwd                      3.5.52build1                            amd64        Debian base system master password and group files
-ii  bash                             5.1-6ubuntu1                            amd64        GNU Bourne Again SHell
+ii  bash                             5.1-6ubuntu1.1                          amd64        GNU Bourne Again SHell
 ii  bash-completion                  1:2.11-5ubuntu1                         all          programmable completion for the bash shell
 ii  bc                               1.07.1-3build1                          amd64        GNU bc arbitrary precision calculator language
 ii  bcache-tools                     1.0.8-4ubuntu3                          amd64        bcache userspace tools
-ii  bind9-dnsutils                   1:9.18.18-0ubuntu0.22.04.1              amd64        Clients provided with BIND 9
-ii  bind9-host                       1:9.18.18-0ubuntu0.22.04.1              amd64        DNS Lookup Utility
-ii  bind9-libs:amd64                 1:9.18.18-0ubuntu0.22.04.1              amd64        Shared Libraries used by BIND 9
-ii  binutils                         2.38-4ubuntu2.5                         amd64        GNU assembler, linker and binary utilities
-ii  binutils-common:amd64            2.38-4ubuntu2.5                         amd64        Common files for the GNU assembler, linker and binary utilities
-ii  binutils-x86-64-linux-gnu        2.38-4ubuntu2.5                         amd64        GNU binary utilities, for x86-64-linux-gnu target
+ii  bind9-dnsutils                   1:9.18.18-0ubuntu0.22.04.2              amd64        Clients provided with BIND 9
+ii  bind9-host                       1:9.18.18-0ubuntu0.22.04.2              amd64        DNS Lookup Utility
+ii  bind9-libs:amd64                 1:9.18.18-0ubuntu0.22.04.2              amd64        Shared Libraries used by BIND 9
+ii  binutils                         2.38-4ubuntu2.6                         amd64        GNU assembler, linker and binary utilities
+ii  binutils-common:amd64            2.38-4ubuntu2.6                         amd64        Common files for the GNU assembler, linker and binary utilities
+ii  binutils-x86-64-linux-gnu        2.38-4ubuntu2.6                         amd64        GNU binary utilities, for x86-64-linux-gnu target
 ii  bsdextrautils                    2.37.2-4ubuntu3                         amd64        extra utilities from 4.4BSD-Lite
 ii  bsdutils                         1:2.37.2-4ubuntu3                       amd64        basic utilities from 4.4BSD-Lite
 ii  btrfs-progs                      5.16.2-1                                amd64        Checksumming Copy on Write Filesystem utilities
@@ -39,13 +39,13 @@
 ii  ca-certificates-java             20190909ubuntu1.2                       all          Common CA certificates (JKS keystore)
 ii  chrony                           4.2-2ubuntu2                            amd64        Versatile implementation of the Network Time Protocol
 ii  cloud-guest-utils                0.32-22-g45fe84a5-0ubuntu1              all          cloud guest utilities
-ii  cloud-init                       23.3.3-0ubuntu0~22.04.1                 all          initialization and customization tool for cloud instances
+ii  cloud-init                       23.4.4-0ubuntu0~22.04.1                 all          initialization and customization tool for cloud instances
 ii  cloud-initramfs-copymods         0.47ubuntu1                             all          copy initramfs modules into root filesystem for later use
 ii  cloud-initramfs-dyn-netconf      0.47ubuntu1                             all          write a network interface file in /run for BOOTIF
 ii  command-not-found                22.04.0                                 all          Suggest installation of packages in interactive bash sessions
 ii  console-setup                    1.205ubuntu3                            all          console font and keymap setup program
 ii  console-setup-linux              1.205ubuntu3                            all          Linux specific part of console-setup
-ii  coreutils                        8.32-4.1ubuntu1                         amd64        GNU core utilities
+ii  coreutils                        8.32-4.1ubuntu1.1                       amd64        GNU core utilities
 ii  cpio                             2.13+dfsg-7                             amd64        GNU cpio -- a program to manage archives of files
 ii  cpp                              4:11.2.0-1ubuntu1                       amd64        GNU C preprocessor (cpp)
 ii  cpp-11                           11.4.0-1ubuntu1~22.04                   amd64        GNU C preprocessor
@@ -54,52 +54,52 @@
 ii  cryptsetup                       2:2.4.3-1ubuntu1.2                      amd64        disk encryption support - startup scripts
 ii  cryptsetup-bin                   2:2.4.3-1ubuntu1.2                      amd64        disk encryption support - command line tools
 ii  cryptsetup-initramfs             2:2.4.3-1ubuntu1.2                      all          disk encryption support - initramfs integration
-ii  cuda                             12.3.2-1                                amd64        CUDA meta-package
-ii  cuda-12-3                        12.3.2-1                                amd64        CUDA 12.3 meta-package
-ii  cuda-cccl-12-3                   12.3.101-1                              amd64        CUDA CCCL
-ii  cuda-command-line-tools-12-3     12.3.2-1                                amd64        CUDA command-line tools
-ii  cuda-compiler-12-3               12.3.2-1                                amd64        CUDA compiler
-ii  cuda-crt-12-3                    12.3.107-1                              amd64        CUDA crt
-ii  cuda-cudart-12-3                 12.3.101-1                              amd64        CUDA Runtime native Libraries
-ii  cuda-cudart-dev-12-3             12.3.101-1                              amd64        CUDA Runtime native dev links, headers
-ii  cuda-cuobjdump-12-3              12.3.101-1                              amd64        CUDA cuobjdump
-ii  cuda-cupti-12-3                  12.3.101-1                              amd64        CUDA profiling tools runtime libs.
-ii  cuda-cupti-dev-12-3              12.3.101-1                              amd64        CUDA profiling tools interface.
-ii  cuda-cuxxfilt-12-3               12.3.101-1                              amd64        CUDA cuxxfilt
-ii  cuda-demo-suite-12-3             12.3.101-1                              amd64        Demo suite for CUDA
-ii  cuda-documentation-12-3          12.3.101-1                              amd64        CUDA documentation
-ii  cuda-driver-dev-12-3             12.3.101-1                              amd64        CUDA Driver native dev stub library
-ii  cuda-drivers                     545.23.08-1                             amd64        CUDA Driver meta-package, branch-agnostic
-ii  cuda-drivers-545                 545.23.08-1                             amd64        CUDA Driver meta-package, branch-specific
-ii  cuda-gdb-12-3                    12.3.101-1                              amd64        CUDA-GDB
+ii  cuda                             12.4.1-1                                amd64        CUDA meta-package
+ii  cuda-12-4                        12.4.1-1                                amd64        CUDA 12.4 meta-package
+ii  cuda-cccl-12-4                   12.4.127-1                              amd64        CUDA CCCL
+ii  cuda-command-line-tools-12-4     12.4.1-1                                amd64        CUDA command-line tools
+ii  cuda-compiler-12-4               12.4.1-1                                amd64        CUDA compiler
+ii  cuda-crt-12-4                    12.4.131-1                              amd64        CUDA crt
+ii  cuda-cudart-12-4                 12.4.127-1                              amd64        CUDA Runtime native Libraries
+ii  cuda-cudart-dev-12-4             12.4.127-1                              amd64        CUDA Runtime native dev links, headers
+ii  cuda-cuobjdump-12-4              12.4.127-1                              amd64        CUDA cuobjdump
+ii  cuda-cupti-12-4                  12.4.127-1                              amd64        CUDA profiling tools runtime libs.
+ii  cuda-cupti-dev-12-4              12.4.127-1                              amd64        CUDA profiling tools interface.
+ii  cuda-cuxxfilt-12-4               12.4.127-1                              amd64        CUDA cuxxfilt
+ii  cuda-demo-suite-12-4             12.4.127-1                              amd64        Demo suite for CUDA
+ii  cuda-documentation-12-4          12.4.127-1                              amd64        CUDA documentation
+ii  cuda-driver-dev-12-4             12.4.127-1                              amd64        CUDA Driver native dev stub library
+ii  cuda-drivers                     550.54.15-1                             amd64        CUDA Driver meta-package, branch-agnostic
+ii  cuda-drivers-550                 550.54.15-1                             amd64        CUDA Driver meta-package, branch-specific
+ii  cuda-gdb-12-4                    12.4.127-1                              amd64        CUDA-GDB
 ii  cuda-keyring                     1.0-1                                   all          GPG keyring for the CUDA repository
-ii  cuda-libraries-12-3              12.3.2-1                                amd64        CUDA Libraries 12.3 meta-package
-ii  cuda-libraries-dev-12-3          12.3.2-1                                amd64        CUDA Libraries 12.3 development meta-package
-ii  cuda-nsight-12-3                 12.3.101-1                              amd64        CUDA nsight
-ii  cuda-nsight-compute-12-3         12.3.2-1                                amd64        NVIDIA Nsight Compute
-ii  cuda-nsight-systems-12-3         12.3.2-1                                amd64        NVIDIA Nsight Systems
-ii  cuda-nvcc-12-3                   12.3.107-1                              amd64        CUDA nvcc
-ii  cuda-nvdisasm-12-3               12.3.101-1                              amd64        CUDA disassembler
-ii  cuda-nvml-dev-12-3               12.3.101-1                              amd64        NVML native dev links, headers
-ii  cuda-nvprof-12-3                 12.3.101-1                              amd64        CUDA Profiler tools
-ii  cuda-nvprune-12-3                12.3.101-1                              amd64        CUDA nvprune
-ii  cuda-nvrtc-12-3                  12.3.107-1                              amd64        NVRTC native runtime libraries
-ii  cuda-nvrtc-dev-12-3              12.3.107-1                              amd64        NVRTC native dev links, headers
-ii  cuda-nvtx-12-3                   12.3.101-1                              amd64        NVIDIA Tools Extension
-ii  cuda-nvvm-12-3                   12.3.107-1                              amd64        CUDA nvvm
-ii  cuda-nvvp-12-3                   12.3.101-1                              amd64        CUDA Profiler tools
-ii  cuda-opencl-12-3                 12.3.101-1                              amd64        CUDA OpenCL native Libraries
-ii  cuda-opencl-dev-12-3             12.3.101-1                              amd64        CUDA OpenCL native dev links, headers
-ii  cuda-profiler-api-12-3           12.3.101-1                              amd64        CUDA Profiler API
-ii  cuda-runtime-12-3                12.3.2-1                                amd64        CUDA Runtime 12.3 meta-package
-ii  cuda-sanitizer-12-3              12.3.101-1                              amd64        CUDA Sanitizer
-ii  cuda-toolkit-12-3                12.3.2-1                                amd64        CUDA Toolkit 12.3 meta-package
-ii  cuda-toolkit-12-3-config-common  12.3.101-1                              all          Common config package for CUDA Toolkit 12.3.
-ii  cuda-toolkit-12-config-common    12.3.101-1                              all          Common config package for CUDA Toolkit 12.
-ii  cuda-toolkit-config-common       12.3.101-1                              all          Common config package for CUDA Toolkit.
-ii  cuda-tools-12-3                  12.3.2-1                                amd64        CUDA Tools meta-package
-ii  cuda-visual-tools-12-3           12.3.2-1                                amd64        CUDA visual tools
-ii  curl                             7.81.0-1ubuntu1.15                      amd64        command line tool for transferring data with URL syntax
+ii  cuda-libraries-12-4              12.4.1-1                                amd64        CUDA Libraries 12.4 meta-package
+ii  cuda-libraries-dev-12-4          12.4.1-1                                amd64        CUDA Libraries 12.4 development meta-package
+ii  cuda-nsight-12-4                 12.4.127-1                              amd64        CUDA nsight
+ii  cuda-nsight-compute-12-4         12.4.1-1                                amd64        NVIDIA Nsight Compute
+ii  cuda-nsight-systems-12-4         12.4.1-1                                amd64        NVIDIA Nsight Systems
+ii  cuda-nvcc-12-4                   12.4.131-1                              amd64        CUDA nvcc
+ii  cuda-nvdisasm-12-4               12.4.127-1                              amd64        CUDA disassembler
+ii  cuda-nvml-dev-12-4               12.4.127-1                              amd64        NVML native dev links, headers
+ii  cuda-nvprof-12-4                 12.4.127-1                              amd64        CUDA Profiler tools
+ii  cuda-nvprune-12-4                12.4.127-1                              amd64        CUDA nvprune
+ii  cuda-nvrtc-12-4                  12.4.127-1                              amd64        NVRTC native runtime libraries
+ii  cuda-nvrtc-dev-12-4              12.4.127-1                              amd64        NVRTC native dev links, headers
+ii  cuda-nvtx-12-4                   12.4.127-1                              amd64        NVIDIA Tools Extension
+ii  cuda-nvvm-12-4                   12.4.131-1                              amd64        CUDA nvvm
+ii  cuda-nvvp-12-4                   12.4.127-1                              amd64        CUDA Profiler tools
+ii  cuda-opencl-12-4                 12.4.127-1                              amd64        CUDA OpenCL native Libraries
+ii  cuda-opencl-dev-12-4             12.4.127-1                              amd64        CUDA OpenCL native dev links, headers
+ii  cuda-profiler-api-12-4           12.4.127-1                              amd64        CUDA Profiler API
+ii  cuda-runtime-12-4                12.4.1-1                                amd64        CUDA Runtime 12.4 meta-package
+ii  cuda-sanitizer-12-4              12.4.127-1                              amd64        CUDA Sanitizer
+ii  cuda-toolkit-12-4                12.4.1-1                                amd64        CUDA Toolkit 12.4 meta-package
+ii  cuda-toolkit-12-4-config-common  12.4.127-1                              all          Common config package for CUDA Toolkit 12.4.
+ii  cuda-toolkit-12-config-common    12.4.127-1                              all          Common config package for CUDA Toolkit 12.
+ii  cuda-toolkit-config-common       12.4.127-1                              all          Common config package for CUDA Toolkit.
+ii  cuda-tools-12-4                  12.4.1-1                                amd64        CUDA Tools meta-package
+ii  cuda-visual-tools-12-4           12.4.1-1                                amd64        CUDA visual tools
+ii  curl                             7.81.0-1ubuntu1.16                      amd64        command line tool for transferring data with URL syntax
 ii  dash                             0.5.11+git20210903+057cd650a4ed-3build1 amd64        POSIX-compliant shell
 ii  dbus                             1.12.20-2ubuntu4.1                      amd64        simple interprocess messaging system (daemon and utilities)
 ii  dbus-user-session                1.12.20-2ubuntu4.1                      amd64        simple interprocess messaging system (systemd --user integration)
@@ -121,8 +121,8 @@
 ii  dmidecode                        3.3-3ubuntu0.1                          amd64        SMBIOS/DMI table decoder
 ii  dmsetup                          2:1.02.175-2.1ubuntu4                   amd64        Linux Kernel Device Mapper userspace library
 ii  dosfstools                       4.2-1build3                             amd64        utilities for making and checking MS-DOS FAT filesystems
-ii  dpkg                             1.21.1ubuntu2.2                         amd64        Debian package management system
-ii  dpkg-dev                         1.21.1ubuntu2.2                         all          Debian package development tools
+ii  dpkg                             1.21.1ubuntu2.3                         amd64        Debian package management system
+ii  dpkg-dev                         1.21.1ubuntu2.3                         all          Debian package development tools
 ii  e2fsprogs                        1.46.5-2ubuntu1.1                       amd64        ext2/ext3/ext4 file system utilities
 ii  eatmydata                        130-2build1                             all          Library and utilities designed to disable fsync and friends
 ii  ed                               1.18-1                                  amd64        classic UNIX line editor
@@ -152,7 +152,7 @@
 ii  gcc-12                           12.3.0-1ubuntu1~22.04                   amd64        GNU C compiler
 ii  gcc-12-base:amd64                12.3.0-1ubuntu1~22.04                   amd64        GCC, the GNU Compiler Collection (base package)
 ii  gdisk                            1.0.8-4build1                           amd64        GPT fdisk text-mode partitioning tool
-ii  gds-tools-12-3                   1.8.1.2-1                               amd64        Tools for GPU Direct Storage
+ii  gds-tools-12-4                   1.9.1.3-1                               amd64        Tools for GPU Direct Storage
 ii  gettext-base                     0.21-4ubuntu4                           amd64        GNU Internationalization utilities for the base system
 ii  gir1.2-glib-2.0:amd64            1.72.0-1                                amd64        Introspection data for GLib, GObject, Gio and GModule
 ii  gir1.2-packagekitglib-1.0        1.2.5-2ubuntu2                          amd64        GObject introspection data for the PackageKit GLib library
@@ -199,7 +199,7 @@
 ii  initramfs-tools-core             0.140ubuntu13.4                         all          generic modular initramfs generator (core tools)
 ii  install-info                     6.8-4build1                             amd64        Manage installed documentation in info format
 ii  iproute2                         5.15.0-1ubuntu2                         amd64        networking and traffic control tools
-ii  iptables                         1.8.7-1ubuntu5.1                        amd64        administration tools for packet filtering and NAT
+ii  iptables                         1.8.7-1ubuntu5.2                        amd64        administration tools for packet filtering and NAT
 ii  iptables-persistent              1.0.16                                  all          boot-time loader for netfilter rules, iptables plugin
 ii  iputils-ping                     3:20211215-1                            amd64        Tools to test the reachability of network hosts
 ii  iputils-tracepath                3:20211215-1                            amd64        Tools to trace the network path to a remote host
@@ -214,7 +214,7 @@
 ii  kmod                             29-1ubuntu1                             amd64        tools for managing Linux kernel modules
 ii  kpartx                           0.8.8-1ubuntu1.22.04.4                  amd64        create device mappings for partitions
 ii  landscape-common                 19.12-0ubuntu13                         amd64        Landscape administration system client - Common files
-ii  less                             590-1ubuntu0.22.04.1                    amd64        pager program similar to more
+ii  less                             590-1ubuntu0.22.04.2                    amd64        pager program similar to more
 ii  libacl1:amd64                    2.3.1-1                                 amd64        access control list - shared library
 ii  libaio1:amd64                    0.3.112-13build1                        amd64        Linux kernel AIO access library - shared library
 ii  libalgorithm-diff-perl           1.201-1                                 all          module to find differences between files
@@ -244,9 +244,9 @@
 ii  libavahi-client3:amd64           0.8-5ubuntu5.2                          amd64        Avahi client library
 ii  libavahi-common-data:amd64       0.8-5ubuntu5.2                          amd64        Avahi common data files
 ii  libavahi-common3:amd64           0.8-5ubuntu5.2                          amd64        Avahi common library
-ii  libbinutils:amd64                2.38-4ubuntu2.5                         amd64        GNU binary utilities (private shared library)
-ii  libblkid-dev:amd64               2.37.2-4ubuntu3                         amd64        block device ID library - headers
-ii  libblkid1:amd64                  2.37.2-4ubuntu3                         amd64        block device ID library
+ii  libbinutils:amd64                2.38-4ubuntu2.6                         amd64        GNU binary utilities (private shared library)
+ii  libblkid-dev:amd64               2.37.2-4ubuntu3.4                       amd64        block device ID library - headers
+ii  libblkid1:amd64                  2.37.2-4ubuntu3.4                       amd64        block device ID library
 ii  libblockdev-fs2:amd64            2.26-1                                  amd64        file system plugin for libblockdev
 ii  libblockdev-loop2:amd64          2.26-1                                  amd64        Loop device plugin for libblockdev
 ii  libblockdev-part-err2:amd64      2.26-1                                  amd64        Partition error utility functions for libblockdev
@@ -259,10 +259,10 @@
 ii  libbsd0:amd64                    0.11.5-1                                amd64        utility functions from BSD systems - shared library
 ii  libbz2-1.0:amd64                 1.0.8-5build1                           amd64        high-quality block-sorting file compressor library - runtime
 ii  libc-bin                         2.35-0ubuntu3.6                         amd64        GNU C Library: Binaries
-ii  libc-dev-bin                     2.35-0ubuntu3.6                         amd64        GNU C Library: Development binaries
-ii  libc-devtools                    2.35-0ubuntu3.6                         amd64        GNU C Library: Development tools
-ii  libc6:amd64                      2.35-0ubuntu3.6                         amd64        GNU C Library: Shared libraries
-ii  libc6-dev:amd64                  2.35-0ubuntu3.6                         amd64        GNU C Library: Development Libraries and Header Files
+ii  libc-dev-bin                     2.35-0ubuntu3.7                         amd64        GNU C Library: Development binaries
+ii  libc-devtools                    2.35-0ubuntu3.7                         amd64        GNU C Library: Development tools
+ii  libc6:amd64                      2.35-0ubuntu3.7                         amd64        GNU C Library: Shared libraries
+ii  libc6-dev:amd64                  2.35-0ubuntu3.7                         amd64        GNU C Library: Development Libraries and Header Files
 ii  libcairo-gobject2:amd64          1.16.0-5ubuntu2                         amd64        Cairo 2D vector graphics library (GObject library)
 ii  libcairo2:amd64                  1.16.0-5ubuntu2                         amd64        Cairo 2D vector graphics library
 ii  libcap-ng0:amd64                 0.7.9-2.2build3                         amd64        An alternate POSIX capabilities library
@@ -275,25 +275,25 @@
 ii  libcrypt-dev:amd64               1:4.4.27-1                              amd64        libcrypt development files
 ii  libcrypt1:amd64                  1:4.4.27-1                              amd64        libcrypt shared library
 ii  libcryptsetup12:amd64            2:2.4.3-1ubuntu1.2                      amd64        disk encryption support - shared library
-ii  libctf-nobfd0:amd64              2.38-4ubuntu2.5                         amd64        Compact C Type Format library (runtime, no BFD dependency)
-ii  libctf0:amd64                    2.38-4ubuntu2.5                         amd64        Compact C Type Format library (runtime, BFD dependency)
-ii  libcublas-12-3                   12.3.4.1-1                              amd64        CUBLAS native runtime libraries
-ii  libcublas-dev-12-3               12.3.4.1-1                              amd64        CUBLAS native dev links, headers
+ii  libctf-nobfd0:amd64              2.38-4ubuntu2.6                         amd64        Compact C Type Format library (runtime, no BFD dependency)
+ii  libctf0:amd64                    2.38-4ubuntu2.6                         amd64        Compact C Type Format library (runtime, BFD dependency)
+ii  libcublas-12-4                   12.4.5.8-1                              amd64        CUBLAS native runtime libraries
+ii  libcublas-dev-12-4               12.4.5.8-1                              amd64        CUBLAS native dev links, headers
 ii  libcudnn8                        8.8.1.3-1+cuda12.0                      amd64        cuDNN runtime libraries
 ii  libcudnn8-dev                    8.8.1.3-1+cuda12.0                      amd64        cuDNN development libraries and headers
-ii  libcufft-12-3                    11.0.12.1-1                             amd64        CUFFT native runtime libraries
-ii  libcufft-dev-12-3                11.0.12.1-1                             amd64        CUFFT native dev links, headers
-ii  libcufile-12-3                   1.8.1.2-1                               amd64        Library for GPU Direct Storage with CUDA 12.3
-ii  libcufile-dev-12-3               1.8.1.2-1                               amd64        Library for GPU Direct Storage native dev links, headers
-ii  libcups2:amd64                   2.4.1op1-1ubuntu4.7                     amd64        Common UNIX Printing System(tm) - Core library
-ii  libcurand-12-3                   10.3.4.107-1                            amd64        CURAND native runtime libraries
-ii  libcurand-dev-12-3               10.3.4.107-1                            amd64        CURAND native dev links, headers
+ii  libcufft-12-4                    11.2.1.3-1                              amd64        CUFFT native runtime libraries
+ii  libcufft-dev-12-4                11.2.1.3-1                              amd64        CUFFT native dev links, headers
+ii  libcufile-12-4                   1.9.1.3-1                               amd64        Library for GPU Direct Storage with CUDA 12.4
+ii  libcufile-dev-12-4               1.9.1.3-1                               amd64        Library for GPU Direct Storage native dev links, headers
+ii  libcups2:amd64                   2.4.1op1-1ubuntu4.8                     amd64        Common UNIX Printing System(tm) - Core library
+ii  libcurand-12-4                   10.3.5.147-1                            amd64        CURAND native runtime libraries
+ii  libcurand-dev-12-4               10.3.5.147-1                            amd64        CURAND native dev links, headers
 ii  libcurl3-gnutls:amd64            7.81.0-1ubuntu1.15                      amd64        easy-to-use client-side URL transfer library (GnuTLS flavour)
-ii  libcurl4:amd64                   7.81.0-1ubuntu1.15                      amd64        easy-to-use client-side URL transfer library (OpenSSL flavour)
-ii  libcusolver-12-3                 11.5.4.101-1                            amd64        CUDA solver native runtime libraries
-ii  libcusolver-dev-12-3             11.5.4.101-1                            amd64        CUDA solver native dev links, headers
-ii  libcusparse-12-3                 12.2.0.103-1                            amd64        CUSPARSE native runtime libraries
-ii  libcusparse-dev-12-3             12.2.0.103-1                            amd64        CUSPARSE native dev links, headers
+ii  libcurl4:amd64                   7.81.0-1ubuntu1.16                      amd64        easy-to-use client-side URL transfer library (OpenSSL flavour)
+ii  libcusolver-12-4                 11.6.1.9-1                              amd64        CUDA solver native runtime libraries
+ii  libcusolver-dev-12-4             11.6.1.9-1                              amd64        CUDA solver native dev links, headers
+ii  libcusparse-12-4                 12.3.1.170-1                            amd64        CUSPARSE native runtime libraries
+ii  libcusparse-dev-12-4             12.3.1.170-1                            amd64        CUSPARSE native dev links, headers
 ii  libdatrie1:amd64                 0.2.13-2                                amd64        Double-array trie library
 ii  libdb5.3:amd64                   5.3.28+dfsg1-0.8ubuntu3                 amd64        Berkeley v5.3 Database Libraries [runtime]
 ii  libdbus-1-3:amd64                1.12.20-2ubuntu4.1                      amd64        simple interprocess messaging system (library)
@@ -303,7 +303,7 @@
 ii  libdevmapper-event1.02.1:amd64   2:1.02.175-2.1ubuntu4                   amd64        Linux Kernel Device Mapper event support library
 ii  libdevmapper1.02.1:amd64         2:1.02.175-2.1ubuntu4                   amd64        Linux Kernel Device Mapper userspace library
 ii  libdns-export1110                1:9.11.19+dfsg-2.1ubuntu3               amd64        Exported DNS Shared Library
-ii  libdpkg-perl                     1.21.1ubuntu2.2                         all          Dpkg perl modules
+ii  libdpkg-perl                     1.21.1ubuntu2.3                         all          Dpkg perl modules
 ii  libdrm-amdgpu1:amd64             2.4.113-2~ubuntu0.22.04.1               amd64        Userspace interface to amdgpu-specific kernel DRM services -- runtime
 ii  libdrm-common                    2.4.113-2~ubuntu0.22.04.1               all          Userspace interface to kernel DRM services -- common files
 ii  libdrm-intel1:amd64              2.4.113-2~ubuntu0.22.04.1               amd64        Userspace interface to intel-specific kernel DRM services -- runtime
@@ -322,8 +322,8 @@
 ii  liberror-perl                    0.17029-1                               all          Perl module for error/exception handling in an OO-ish way
 ii  libestr0:amd64                   0.1.10-2.1build3                        amd64        Helper functions for handling strings (lib)
 ii  libevent-core-2.1-7:amd64        2.1.12-stable-1build3                   amd64        Asynchronous event notification library (core)
-ii  libexpat1:amd64                  2.4.7-1ubuntu0.2                        amd64        XML parsing C library - runtime library
-ii  libexpat1-dev:amd64              2.4.7-1ubuntu0.2                        amd64        XML parsing C library - development kit
+ii  libexpat1:amd64                  2.4.7-1ubuntu0.3                        amd64        XML parsing C library - runtime library
+ii  libexpat1-dev:amd64              2.4.7-1ubuntu0.3                        amd64        XML parsing C library - development kit
 ii  libext2fs2:amd64                 1.46.5-2ubuntu1.1                       amd64        ext2/ext3/ext4 file system libraries
 ii  libfakeroot:amd64                1.28-1ubuntu1                           amd64        tool for simulating superuser privileges - shared libraries
 ii  libfastjson4:amd64               0.99.9-1build2                          amd64        fast json library for C
@@ -356,7 +356,6 @@
 ii  libgl1-amber-dri:amd64           21.3.9-0ubuntu1~22.04.1                 amd64        free implementation of the OpenGL API -- Amber DRI modules
 ii  libgl1-mesa-dri:amd64            23.2.1-1ubuntu3.1~22.04.2               amd64        free implementation of the OpenGL API -- DRI modules
 ii  libglapi-mesa:amd64              23.2.1-1ubuntu3.1~22.04.2               amd64        free implementation of the GL API -- shared library
-ii  libgles2:amd64                   1.4.0-1                                 amd64        Vendor neutral GL dispatch library -- GLESv2 support
 ii  libglib2.0-0:amd64               2.72.4-0ubuntu2.2                       amd64        GLib library of C routines
 ii  libglib2.0-bin                   2.72.4-0ubuntu2.2                       amd64        Programs for the GLib library
 ii  libglib2.0-data                  2.72.4-0ubuntu2.2                       all          Common files for GLib library
@@ -386,8 +385,8 @@
 ii  libinih1:amd64                   53-1ubuntu3                             amd64        simple .INI file parser
 ii  libintl-perl                     1.26-3build2                            all          Uniforum message translations system compatible i18n library
 ii  libintl-xs-perl                  1.26-3build2                            amd64        XS Uniforum message translations system compatible i18n library
-ii  libip4tc2:amd64                  1.8.7-1ubuntu5.1                        amd64        netfilter libip4tc library
-ii  libip6tc2:amd64                  1.8.7-1ubuntu5.1                        amd64        netfilter libip6tc library
+ii  libip4tc2:amd64                  1.8.7-1ubuntu5.2                        amd64        netfilter libip4tc library
+ii  libip6tc2:amd64                  1.8.7-1ubuntu5.2                        amd64        netfilter libip6tc library
 ii  libisc-export1105:amd64          1:9.11.19+dfsg-2.1ubuntu3               amd64        Exported ISC Shared Library
 ii  libisl23:amd64                   0.24-2build1                            amd64        manipulating sets and relations of integer points bounded by linear constraints
 ii  libisns0:amd64                   0.101-0ubuntu2                          amd64        Internet Storage Name Service - shared libraries
@@ -408,8 +407,8 @@
 ii  libkrb5support0:amd64            1.19.2-2ubuntu0.3                       amd64        MIT Kerberos runtime libraries - Support library
 ii  libksba8:amd64                   1.6.0-2ubuntu0.2                        amd64        X.509 and CMS support library
 ii  liblcms2-2:amd64                 2.12~rc1-2build2                        amd64        Little CMS 2 color management library
-ii  libldap-2.5-0:amd64              2.5.16+dfsg-0ubuntu0.22.04.2            amd64        OpenLDAP libraries
-ii  libldap-common                   2.5.16+dfsg-0ubuntu0.22.04.2            all          OpenLDAP common files for libraries
+ii  libldap-2.5-0:amd64              2.5.17+dfsg-0ubuntu0.22.04.1            amd64        OpenLDAP libraries
+ii  libldap-common                   2.5.17+dfsg-0ubuntu0.22.04.1            all          OpenLDAP common files for libraries
 ii  libllvm15:amd64                  1:15.0.7-0ubuntu0.22.04.3               amd64        Modular compiler and toolchain technologies, runtime library
 ii  liblmdb0:amd64                   0.9.24-1build2                          amd64        Lightning Memory-Mapped Database shared library
 ii  liblocale-gettext-perl           1.07-4build3                            amd64        module using libc functions for internationalization in Perl
@@ -429,8 +428,8 @@
 ii  libmnl0:amd64                    1.0.4-3build2                           amd64        minimalistic Netlink communication library
 ii  libmodule-find-perl              0.15-1                                  all          module to find and use installed Perl modules
 ii  libmodule-scandeps-perl          1.31-1                                  all          module to recursively scan Perl code for dependencies
-ii  libmount-dev:amd64               2.37.2-4ubuntu3                         amd64        device mounting library - headers
-ii  libmount1:amd64                  2.37.2-4ubuntu3                         amd64        device mounting library
+ii  libmount-dev:amd64               2.37.2-4ubuntu3.4                       amd64        device mounting library - headers
+ii  libmount1:amd64                  2.37.2-4ubuntu3.4                       amd64        device mounting library
 ii  libmpc3:amd64                    1.2.1-2build1                           amd64        multiple precision complex floating-point library
 ii  libmpdec3:amd64                  2.5.1-2build2                           amd64        library for decimal floating point arithmetic (runtime library)
 ii  libmpfr6:amd64                   4.1.0-3build3                           amd64        multiple precision floating-point computation
@@ -447,8 +446,8 @@
 ii  libnghttp2-14:amd64              1.43.0-1ubuntu0.1                       amd64        library implementing HTTP/2 protocol (shared library)
 ii  libnl-3-200:amd64                3.5.0-0.1                               amd64        library for dealing with netlink sockets
 ii  libnl-genl-3-200:amd64           3.5.0-0.1                               amd64        library for dealing with netlink sockets - generic netlink
-ii  libnpp-12-3                      12.2.3.2-1                              amd64        NPP native runtime libraries
-ii  libnpp-dev-12-3                  12.2.3.2-1                              amd64        NPP native dev links, headers
+ii  libnpp-12-4                      12.2.5.30-1                             amd64        NPP native runtime libraries
+ii  libnpp-dev-12-4                  12.2.5.30-1                             amd64        NPP native dev links, headers
 ii  libnpth0:amd64                   1.6-3build2                             amd64        replacement for GNU Pth using system threads
 ii  libnsl-dev:amd64                 1.3.0-2build2                           amd64        libnsl development files
 ii  libnsl2:amd64                    1.3.0-2build2                           amd64        Public client interface for NIS(YP) and NIS+
@@ -457,18 +456,20 @@
 ii  libnss3:amd64                    2:3.68.2-0ubuntu1.2                     amd64        Network Security Service libraries
 ii  libntfs-3g89                     1:2021.8.22-3ubuntu1.2                  amd64        read/write NTFS driver for FUSE (runtime library)
 ii  libnuma1:amd64                   2.0.14-3ubuntu2                         amd64        Libraries for controlling NUMA policy
-ii  libnvidia-cfg1-545:amd64         545.23.08-0ubuntu1                      amd64        NVIDIA binary OpenGL/GLX configuration library
-ii  libnvidia-common-545             545.23.08-0ubuntu1                      all          Shared files used by the NVIDIA libraries
-ii  libnvidia-compute-545:amd64      545.23.08-0ubuntu1                      amd64        NVIDIA libcompute package
-ii  libnvidia-decode-545:amd64       545.23.08-0ubuntu1                      amd64        NVIDIA Video Decoding runtime libraries
-ii  libnvidia-encode-545:amd64       545.23.08-0ubuntu1                      amd64        NVENC Video Encoding runtime library
-ii  libnvidia-extra-545:amd64        545.23.08-0ubuntu1                      amd64        Extra libraries for the NVIDIA driver
-ii  libnvidia-fbc1-545:amd64         545.23.08-0ubuntu1                      amd64        NVIDIA OpenGL-based Framebuffer Capture runtime library
-ii  libnvidia-gl-545:amd64           545.23.08-0ubuntu1                      amd64        NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
-ii  libnvjitlink-12-3                12.3.101-1                              amd64        NVJIT link native runtime libraries
-ii  libnvjitlink-dev-12-3            12.3.101-1                              amd64        NVJIT native dev links, headers
-ii  libnvjpeg-12-3                   12.3.0.81-1                             amd64        NVJPEG native runtime libraries
-ii  libnvjpeg-dev-12-3               12.3.0.81-1                             amd64        NVJPEG native dev links, headers
+ii  libnvfatbin-12-4                 12.4.127-1                              amd64        nvfatbin link native runtime libraries
+ii  libnvfatbin-dev-12-4             12.4.127-1                              amd64        nvfatbin native dev links, headers
+ii  libnvidia-cfg1-550:amd64         550.54.15-0ubuntu1                      amd64        NVIDIA binary OpenGL/GLX configuration library
+ii  libnvidia-common-550             550.54.15-0ubuntu1                      all          Shared files used by the NVIDIA libraries
+ii  libnvidia-compute-550:amd64      550.54.15-0ubuntu1                      amd64        NVIDIA libcompute package
+ii  libnvidia-decode-550:amd64       550.54.15-0ubuntu1                      amd64        NVIDIA Video Decoding runtime libraries
+ii  libnvidia-encode-550:amd64       550.54.15-0ubuntu1                      amd64        NVENC Video Encoding runtime library
+ii  libnvidia-extra-550:amd64        550.54.15-0ubuntu1                      amd64        Extra libraries for the NVIDIA driver
+ii  libnvidia-fbc1-550:amd64         550.54.15-0ubuntu1                      amd64        NVIDIA OpenGL-based Framebuffer Capture runtime library
+ii  libnvidia-gl-550:amd64           550.54.15-0ubuntu1                      amd64        NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
+ii  libnvjitlink-12-4                12.4.127-1                              amd64        NVJIT link native runtime libraries
+ii  libnvjitlink-dev-12-4            12.4.127-1                              amd64        NVJIT native dev links, headers
+ii  libnvjpeg-12-4                   12.3.1.117-1                            amd64        NVJPEG native runtime libraries
+ii  libnvjpeg-dev-12-4               12.3.1.117-1                            amd64        NVJPEG native dev links, headers
 ii  libopengl0:amd64                 1.4.0-1                                 amd64        Vendor neutral GL dispatch library -- OpenGL support
 ii  libopeniscsiusr                  2.1.5-1ubuntu1                          amd64        iSCSI userspace library
 ii  libp11-kit0:amd64                0.24.0-6build1                          amd64        library for loading and coordinating access to PKCS#11 modules - runtime
@@ -545,7 +546,7 @@
 ii  libsqlite3-0:amd64               3.37.2-2ubuntu0.3                       amd64        SQLite 3 shared library
 ii  libss2:amd64                     1.46.5-2ubuntu1.1                       amd64        command-line interface parsing library
 ii  libssh-4:amd64                   0.9.6-2ubuntu0.22.04.3                  amd64        tiny C SSH library (OpenSSL flavor)
-ii  libssl3:amd64                    3.0.2-0ubuntu1.14                       amd64        Secure Sockets Layer toolkit - shared libraries
+ii  libssl3:amd64                    3.0.2-0ubuntu1.15                       amd64        Secure Sockets Layer toolkit - shared libraries
 ii  libstdc++-11-dev:amd64           11.4.0-1ubuntu1~22.04                   amd64        GNU Standard C++ Library v3 (development files)
 ii  libstdc++6:amd64                 12.3.0-1ubuntu1~22.04                   amd64        GNU Standard C++ Library v3
 ii  libstemmer0d:amd64               2.2.0-1build1                           amd64        Snowball stemming algorithms for use in Information Retrieval
@@ -558,7 +559,7 @@
 ii  libtext-wrapi18n-perl            0.06-9                                  all          internationalized substitute of Text::Wrap
 ii  libthai-data                     0.1.29-1build1                          all          Data files for Thai language support library
 ii  libthai0:amd64                   0.1.29-1build1                          amd64        Thai language support library
-ii  libtiff5:amd64                   4.3.0-6ubuntu0.7                        amd64        Tag Image File Format (TIFF) library
+ii  libtiff5:amd64                   4.3.0-6ubuntu0.8                        amd64        Tag Image File Format (TIFF) library
 ii  libtinfo5:amd64                  6.3-2ubuntu0.1                          amd64        shared low-level terminfo library (legacy version)
 ii  libtinfo6:amd64                  6.3-2ubuntu0.1                          amd64        shared low-level terminfo library for terminal handling
 ii  libtirpc-common                  1.3.2-2ubuntu0.1                        all          transport-independent RPC library - common files
@@ -575,8 +576,8 @@
 ii  liburcu8:amd64                   0.13.1-1                                amd64        userspace RCU (read-copy-update) library
 ii  libusb-1.0-0:amd64               2:1.0.25-1ubuntu2                       amd64        userspace USB programming library
 ii  libutempter0:amd64               1.2.1-2build2                           amd64        privileged helper for utmp/wtmp updates (runtime)
-ii  libuuid1:amd64                   2.37.2-4ubuntu3                         amd64        Universally Unique ID library
-ii  libuv1:amd64                     1.43.0-1                                amd64        asynchronous event notification library - runtime library
+ii  libuuid1:amd64                   2.37.2-4ubuntu3.4                       amd64        Universally Unique ID library
+ii  libuv1:amd64                     1.43.0-1ubuntu0.1                       amd64        asynchronous event notification library - runtime library
 ii  libvdpau1:amd64                  1.4-3build2                             amd64        Video Decode and Presentation API for Unix (libraries)
 ii  libwayland-client0:amd64         1.20.0-1ubuntu0.1                       amd64        wayland compositor infrastructure - client library
 ii  libwayland-cursor0:amd64         1.20.0-1ubuntu0.1                       amd64        wayland compositor infrastructure - cursor library
@@ -622,20 +623,20 @@
 ii  libxkbcommon-x11-0:amd64         1.4.0-1                                 amd64        library to create keymaps with the XKB X11 protocol
 ii  libxkbcommon0:amd64              1.4.0-1                                 amd64        library interface to the XKB compiler - shared library
 ii  libxkbfile1:amd64                1:1.1.0-1build3                         amd64        X11 keyboard file manipulation library
-ii  libxml2:amd64                    2.9.13+dfsg-1ubuntu0.3                  amd64        GNOME XML library
+ii  libxml2:amd64                    2.9.13+dfsg-1ubuntu0.4                  amd64        GNOME XML library
 ii  libxmlb2:amd64                   0.3.6-2build1                           amd64        Binary XML library
 ii  libxmlsec1:amd64                 1.2.33-1build2                          amd64        XML security library
 ii  libxmlsec1-openssl:amd64         1.2.33-1build2                          amd64        Openssl engine for the XML security library
 ii  libxmu6:amd64                    2:1.1.3-3                               amd64        X11 miscellaneous utility library
 ii  libxmuu1:amd64                   2:1.1.3-3                               amd64        X11 miscellaneous micro-utility library
-ii  libxnvctrl0:amd64                545.23.08-0ubuntu1                      amd64        NV-CONTROL X extension (runtime library)
+ii  libxnvctrl0:amd64                550.54.15-0ubuntu1                      amd64        NV-CONTROL X extension (runtime library)
 ii  libxpm4:amd64                    1:3.5.12-1ubuntu0.22.04.2               amd64        X11 pixmap library
 ii  libxrandr2:amd64                 2:1.5.2-1build1                         amd64        X11 RandR extension library
 ii  libxrender1:amd64                1:0.9.10-1build4                        amd64        X Rendering Extension client library
 ii  libxshmfence1:amd64              1.3-1build4                             amd64        X shared memory fences - shared library
 ii  libxslt1.1:amd64                 1.1.34-4ubuntu0.22.04.1                 amd64        XSLT 1.0 processing library - runtime library
 ii  libxt6:amd64                     1:1.2.1-1                               amd64        X11 toolkit intrinsics library
-ii  libxtables12:amd64               1.8.7-1ubuntu5.1                        amd64        netfilter xtables library
+ii  libxtables12:amd64               1.8.7-1ubuntu5.2                        amd64        netfilter xtables library
 ii  libxtst6:amd64                   2:1.2.3-1build4                         amd64        X11 Testing -- Record extension library
 ii  libxv1:amd64                     2:1.0.11-1build2                        amd64        X11 Video extension library
 ii  libxxf86dga1:amd64               2:1.1.5-0ubuntu3                        amd64        X11 Direct Graphics Access extension library
@@ -644,16 +645,20 @@
 ii  libyaml-0-2:amd64                0.2.2-1build2                           amd64        Fast YAML 1.1 parser and emitter library
 ii  libzstd1:amd64                   1.4.8+dfsg-3build1                      amd64        fast lossless compression algorithm
 ii  linux-base                       4.5ubuntu9                              all          Linux image base package
-ii  linux-gcp                        6.5.0.1013.13~22.04.1                   amd64        Complete Google Cloud Platform (GCP) Linux kernel and headers
-ii  linux-gcp-6.5-headers-6.5.0-1013 6.5.0-1013.13~22.04.1                   amd64        Header files related to Linux kernel version 6.5.0
-ii  linux-headers-6.5.0-1013-gcp     6.5.0-1013.13~22.04.1                   amd64        Linux kernel headers for version 6.5.0 on 64 bit x86 SMP
-ii  linux-headers-gcp                6.5.0.1013.13~22.04.1                   amd64        Google Cloud Platform (GCP) Linux kernel headers
-ii  linux-image-6.5.0-1013-gcp       6.5.0-1013.13~22.04.1                   amd64        Signed kernel image gcp
-ii  linux-image-gcp                  6.5.0.1013.13~22.04.1                   amd64        Google Cloud Platform (GCP) Linux kernel image
-ii  linux-libc-dev:amd64             5.15.0-94.104                           amd64        Linux Kernel Headers for development
-ii  linux-modules-6.5.0-1013-gcp     6.5.0-1013.13~22.04.1                   amd64        Linux kernel extra modules for version 6.5.0 on 64 bit x86 SMP
+ii  linux-gcp                        6.5.0.1018.18~22.04.1                   amd64        Complete Google Cloud Platform (GCP) Linux kernel and headers
+ii  linux-gcp-6.5-headers-6.5.0-1016 6.5.0-1016.16~22.04.1                   amd64        Header files related to Linux kernel version 6.5.0
+ii  linux-gcp-6.5-headers-6.5.0-1018 6.5.0-1018.18~22.04.1                   amd64        Header files related to Linux kernel version 6.5.0
+ii  linux-headers-6.5.0-1016-gcp     6.5.0-1016.16~22.04.1                   amd64        Linux kernel headers for version 6.5.0 on 64 bit x86 SMP
+ii  linux-headers-6.5.0-1018-gcp     6.5.0-1018.18~22.04.1                   amd64        Linux kernel headers for version 6.5.0 on 64 bit x86 SMP
+ii  linux-headers-gcp                6.5.0.1018.18~22.04.1                   amd64        Google Cloud Platform (GCP) Linux kernel headers
+ii  linux-image-6.5.0-1016-gcp       6.5.0-1016.16~22.04.1                   amd64        Signed kernel image gcp
+ii  linux-image-6.5.0-1018-gcp       6.5.0-1018.18~22.04.1                   amd64        Signed kernel image gcp
+ii  linux-image-gcp                  6.5.0.1018.18~22.04.1                   amd64        Google Cloud Platform (GCP) Linux kernel image
+ii  linux-libc-dev:amd64             5.15.0-105.115                          amd64        Linux Kernel Headers for development
+ii  linux-modules-6.5.0-1016-gcp     6.5.0-1016.16~22.04.1                   amd64        Linux kernel extra modules for version 6.5.0 on 64 bit x86 SMP
+ii  linux-modules-6.5.0-1018-gcp     6.5.0-1018.18~22.04.1                   amd64        Linux kernel extra modules for version 6.5.0 on 64 bit x86 SMP
 ii  locales                          2.35-0ubuntu3.6                         all          GNU C Library: National Language (locale) data [support]
-ii  login                            1:4.8.1-2ubuntu2.1                      amd64        system login tools
+ii  login                            1:4.8.1-2ubuntu2.2                      amd64        system login tools
 ii  logrotate                        3.19.0-1ubuntu1.1                       amd64        Log rotation utility
 ii  logsave                          1.46.5-2ubuntu1.1                       amd64        save the output of a command in a log file
 ii  lsb-base                         11.1.0ubuntu4                           all          Linux Standard Base init script functionality
@@ -674,7 +679,7 @@
 ii  mercurial-common                 6.1.1-1ubuntu1                          all          easy-to-use, scalable distributed version control system (common files)
 ii  mesa-vdpau-drivers:amd64         23.2.1-1ubuntu3.1~22.04.2               amd64        Mesa VDPAU video acceleration drivers
 ii  mokutil                          0.6.0-2~22.04.2                         amd64        tools for manipulating machine owner keys
-ii  motd-news-config                 12ubuntu4.5                             all          Configuration for motd-news shipped in base-files
+ii  motd-news-config                 12ubuntu4.6                             all          Configuration for motd-news shipped in base-files
 ii  mount                            2.37.2-4ubuntu3                         amd64        tools for mounting and manipulating filesystems
 ii  mtr-tiny                         0.95-1                                  amd64        Full screen ncurses traceroute tool
 ii  multipath-tools                  0.8.8-1ubuntu1.22.04.4                  amd64        maintain multipath block device access
@@ -690,34 +695,34 @@
 ii  netplan.io                       0.106.1-7ubuntu0.22.04.2                amd64        YAML network configuration abstraction for various backends
 ii  networkd-dispatcher              2.1-2ubuntu0.22.04.2                    all          Dispatcher service for systemd-networkd connection status changes
 ii  nftables                         1.0.2-1ubuntu3                          amd64        Program to control packet filtering rules by Netfilter project
-ii  nsight-compute-2023.3.1          2023.3.1.1-1                            amd64        NVIDIA Nsight Compute
-ii  nsight-systems-2023.3.3          2023.3.3.42-233333266658v0              amd64        Nsight Systems is a statistical sampling profiler with tracing features.
+ii  nsight-compute-2024.1.1          2024.1.1.4-1                            amd64        NVIDIA Nsight Compute
+ii  nsight-systems-2023.4.4          2023.4.4.54-234433681190v0              amd64        Nsight Systems is a statistical sampling profiler with tracing features.
 ii  ntfs-3g                          1:2021.8.22-3ubuntu1.2                  amd64        read/write NTFS driver for FUSE
-ii  nvidia-compute-utils-545         545.23.08-0ubuntu1                      amd64        NVIDIA compute utilities
-ii  nvidia-dkms-545                  545.23.08-0ubuntu1                      amd64        NVIDIA DKMS package
-ii  nvidia-driver-545                545.23.08-0ubuntu1                      amd64        NVIDIA driver metapackage
-ii  nvidia-kernel-common-545         545.23.08-0ubuntu1                      amd64        Shared files used with the kernel module
-ii  nvidia-kernel-source-545         545.23.08-0ubuntu1                      amd64        NVIDIA kernel source package
-ii  nvidia-modprobe                  545.23.08-0ubuntu1                      amd64        Load the NVIDIA kernel driver and create device files
+ii  nvidia-compute-utils-550         550.54.15-0ubuntu1                      amd64        NVIDIA compute utilities
+ii  nvidia-dkms-550                  550.54.15-0ubuntu1                      amd64        NVIDIA DKMS package
+ii  nvidia-driver-550                550.54.15-0ubuntu1                      amd64        NVIDIA driver metapackage
+ii  nvidia-firmware-550-550.54.15    550.54.15-0ubuntu1                      amd64        Firmware files used by the kernel module
+ii  nvidia-kernel-common-550         550.54.15-0ubuntu1                      amd64        Shared files used with the kernel module
+ii  nvidia-kernel-source-550         550.54.15-0ubuntu1                      amd64        NVIDIA kernel source package
 ii  nvidia-prime                     0.8.17.1                                all          Tools to enable NVIDIA's Prime
-ii  nvidia-settings                  545.23.08-0ubuntu1                      amd64        Tool for configuring the NVIDIA graphics driver
-ii  nvidia-utils-545                 545.23.08-0ubuntu1                      amd64        NVIDIA driver support binaries
+ii  nvidia-settings                  550.54.15-0ubuntu1                      amd64        Tool for configuring the NVIDIA graphics driver
+ii  nvidia-utils-550                 550.54.15-0ubuntu1                      amd64        NVIDIA driver support binaries
 ii  nvme-cli                         1.16-3ubuntu0.1                         amd64        NVMe management tool
 ii  open-iscsi                       2.1.5-1ubuntu1                          amd64        iSCSI initiator tools
-ii  open-vm-tools                    2:12.1.5-3~ubuntu0.22.04.4              amd64        Open VMware Tools for virtual machines hosted on VMware (CLI)
-ii  openjdk-11-jre:amd64             11.0.21+9-0ubuntu1~22.04                amd64        OpenJDK Java runtime, using Hotspot JIT
-ii  openjdk-11-jre-headless:amd64    11.0.21+9-0ubuntu1~22.04                amd64        OpenJDK Java runtime, using Hotspot JIT (headless)
+ii  open-vm-tools                    2:12.3.5-3~ubuntu0.22.04.1              amd64        Open VMware Tools for virtual machines hosted on VMware (CLI)
+ii  openjdk-11-jre:amd64             11.0.22+7-0ubuntu2~22.04.1              amd64        OpenJDK Java runtime, using Hotspot JIT
+ii  openjdk-11-jre-headless:amd64    11.0.22+7-0ubuntu2~22.04.1              amd64        OpenJDK Java runtime, using Hotspot JIT (headless)
 ii  openssh-client                   1:8.9p1-3ubuntu0.6                      amd64        secure shell (SSH) client, for secure access to remote machines
 ii  openssh-server                   1:8.9p1-3ubuntu0.6                      amd64        secure shell (SSH) server, for secure access from remote machines
 ii  openssh-sftp-server              1:8.9p1-3ubuntu0.6                      amd64        secure shell (SSH) sftp server module, for SFTP access from remote machines
-ii  openssl                          3.0.2-0ubuntu1.14                       amd64        Secure Sockets Layer toolkit - cryptographic utility
+ii  openssl                          3.0.2-0ubuntu1.15                       amd64        Secure Sockets Layer toolkit - cryptographic utility
 ii  os-prober                        1.79ubuntu2                             amd64        utility to detect other OSes on a set of drives
 ii  overlayroot                      0.47ubuntu1                             all          use an overlayfs on top of a read-only root filesystem
 ii  packagekit                       1.2.5-2ubuntu2                          amd64        Provides a package management service
 ii  packagekit-tools                 1.2.5-2ubuntu2                          amd64        Provides PackageKit command-line tools
 ii  parallel                         20210822+ds-2                           all          build and execute command lines from standard input in parallel
 ii  parted                           3.4-2build1                             amd64        disk partition manipulator
-ii  passwd                           1:4.8.1-2ubuntu2.1                      amd64        change and administer password and group data
+ii  passwd                           1:4.8.1-2ubuntu2.2                      amd64        change and administer password and group data
 ii  pastebinit                       1.5.1-1ubuntu1                          all          command-line pastebin client
 ii  patch                            2.7.6-7build2                           amd64        Apply a diff file to an original
 ii  pci.ids                          0.0~2022.01.22-1                        all          PCI ID Repository
@@ -737,11 +742,11 @@
 ii  procps                           2:3.3.17-6ubuntu2.1                     amd64        /proc file system utilities
 ii  psmisc                           23.4-2build3                            amd64        utilities that use the proc file system
 ii  publicsuffix                     20211207.1025-1                         all          accurate, machine-readable list of domain name suffixes
-ii  python-apt-common                2.4.0ubuntu2                            all          Python interface to libapt-pkg (locales)
+ii  python-apt-common                2.4.0ubuntu3                            all          Python interface to libapt-pkg (locales)
 ii  python-babel-localedata          2.8.0+dfsg.1-7                          all          tools for internationalizing Python applications - locale data files
 ii  python3                          3.10.6-1~22.04                          amd64        interactive high-level object-oriented language (default python3 version)
 ii  python3-apport                   2.20.11-0ubuntu82.5                     all          Python 3 library for Apport crash report handling
-ii  python3-apt                      2.4.0ubuntu2                            amd64        Python 3 interface to libapt-pkg
+ii  python3-apt                      2.4.0ubuntu3                            amd64        Python 3 interface to libapt-pkg
 ii  python3-attr                     21.2.0-1                                all          Attributes without boilerplate (Python 3)
 ii  python3-automat                  20.2.0-1                                all          Self-service finite-state machines for the programmer on the go
 ii  python3-babel                    2.8.0+dfsg.1-7                          all          tools for internationalizing Python applications - Python 3.x
@@ -755,14 +760,14 @@
 ii  python3-commandnotfound          22.04.0                                 all          Python 3 bindings for command-not-found.
 ii  python3-configobj                5.0.6-5                                 all          simple but powerful config file reader and writer for Python 3
 ii  python3-constantly               15.1.0-2                                all          Symbolic constants in Python
-ii  python3-cryptography             3.4.8-1ubuntu2.1                        amd64        Python library exposing cryptographic recipes and primitives (Python 3)
+ii  python3-cryptography             3.4.8-1ubuntu2.2                        amd64        Python library exposing cryptographic recipes and primitives (Python 3)
 ii  python3-dbus                     1.2.18-3build1                          amd64        simple interprocess messaging system (Python 3 interface)
 ii  python3-debconf                  1.5.79ubuntu1                           all          interact with debconf from Python 3
 ii  python3-debian                   0.1.43ubuntu1.1                         all          Python 3 modules to work with Debian-related data formats
 ii  python3-dev                      3.10.6-1~22.04                          amd64        header files and a static library for Python (default)
 ii  python3-distro                   1.7.0-1                                 all          Linux OS platform information API
 ii  python3-distro-info              1.1ubuntu0.2                            all          information about distributions' releases (Python 3 module)
-ii  python3-distupgrade              1:22.04.17                              all          manage release upgrades
+ii  python3-distupgrade              1:22.04.19                              all          manage release upgrades
 ii  python3-distutils                3.10.8-1~22.04                          all          distutils package for Python 3.x
 ii  python3-gdbm:amd64               3.10.8-1~22.04                          amd64        GNU dbm database support for Python 3.x
 ii  python3-gi                       3.42.1-0ubuntu1                         amd64        Python 3 bindings for gobject-introspection libraries
@@ -811,7 +816,7 @@
 ii  python3-systemd                  234-3ubuntu2                            amd64        Python 3 bindings for systemd
 ii  python3-twisted                  22.1.0-2ubuntu2.4                       all          Event-based framework for internet applications
 ii  python3-tz                       2022.1-1ubuntu0.22.04.1                 all          Python3 version of the Olson timezone database
-ii  python3-update-manager           1:22.04.18                              all          python 3.x module for update-manager
+ii  python3-update-manager           1:22.04.19                              all          python 3.x module for update-manager
 ii  python3-urllib3                  1.26.5-1~exp1ubuntu0.1                  all          HTTP library with thread-safe connection pooling for Python3
 ii  python3-wadllib                  1.3.6-1                                 all          Python 3 library for navigating WADL files
 ii  python3-wheel                    0.37.1-2ubuntu0.22.04.1                 all          built-package format for Python
@@ -857,19 +862,20 @@
 ii  tar                              1.34+dfsg-1ubuntu0.1.22.04.2            amd64        GNU version of the tar archiving utility
 ii  tcl                              8.6.11+1build2                          amd64        Tool Command Language (default version) - shell
 ii  tcl8.6                           8.6.12+dfsg-1build1                     amd64        Tcl (the Tool Command Language) v8.6 - shell
-ii  tcpdump                          4.99.1-3ubuntu0.1                       amd64        command-line network traffic analyzer
+ii  tcpdump                          4.99.1-3ubuntu0.2                       amd64        command-line network traffic analyzer
 ii  telnet                           0.17-44build1                           amd64        basic telnet client
 ii  thin-provisioning-tools          0.9.0-2ubuntu1                          amd64        Tools for handling thinly provisioned device-mapper meta-data
 ii  time                             1.9-0.1build2                           amd64        GNU time program for measuring CPU resource usage
 ii  tmux                             3.2a-4ubuntu0.2                         amd64        terminal multiplexer
 ii  tnftp                            20210827-4build1                        amd64        enhanced ftp client
-ii  tzdata                           2023d-0ubuntu0.22.04                    all          time zone and daylight-saving time data
-ii  ubuntu-advantage-tools           30~22.04                                amd64        management tools for Ubuntu Pro
+ii  tzdata                           2024a-0ubuntu0.22.04                    all          time zone and daylight-saving time data
+ii  ubuntu-advantage-tools           31.2~22.04                              all          transitional dummy package for ubuntu-pro-client
 ii  ubuntu-keyring                   2021.03.26                              all          GnuPG keys of the Ubuntu archive
 ii  ubuntu-minimal                   1.481.1                                 amd64        Minimal core of Ubuntu
 ii  ubuntu-mono                      20.10-0ubuntu2                          all          Ubuntu Mono Icon theme
-ii  ubuntu-pro-client-l10n           30~22.04                                amd64        Translations for Ubuntu Pro Client
-ii  ubuntu-release-upgrader-core     1:22.04.17                              all          manage release upgrades
+ii  ubuntu-pro-client                31.2~22.04                              amd64        Management tools for Ubuntu Pro
+ii  ubuntu-pro-client-l10n           31.2~22.04                              amd64        Translations for Ubuntu Pro Client
+ii  ubuntu-release-upgrader-core     1:22.04.19                              all          manage release upgrades
 ii  ubuntu-server                    1.481.1                                 amd64        The Ubuntu Server system
 ii  ubuntu-standard                  1.481.1                                 amd64        The Ubuntu standard system
 ii  ucf                              3.0043                                  all          Update Configuration File(s): preserve user changes to config files
@@ -877,7 +883,7 @@
 ii  ufw                              0.36.1-4ubuntu0.1                       all          program for managing a Netfilter firewall
 ii  uidmap                           1:4.8.1-2ubuntu2.2                      amd64        programs to help use subuids
 ii  unattended-upgrades              2.8ubuntu1                              all          automatic installation of security upgrades
-ii  update-manager-core              1:22.04.18                              all          manage release upgrades
+ii  update-manager-core              1:22.04.19                              all          manage release upgrades
 ii  update-notifier-common           3.192.54.6                              all          Files shared between update-notifier and other packages
 ii  usb-modeswitch                   2.6.1-3ubuntu2                          amd64        mode switching tool for controlling "flip flop" USB devices
 ii  usb-modeswitch-data              20191128-4                              all          mode switching data for usb-modeswitch
@@ -885,13 +891,13 @@
 ii  usbutils                         1:014-1build1                           amd64        Linux USB utilities
 ii  usrmerge                         25ubuntu2                               all          Convert the system to the merged /usr directories scheme
 ii  util-linux                       2.37.2-4ubuntu3                         amd64        miscellaneous system utilities
-ii  uuid-dev:amd64                   2.37.2-4ubuntu3                         amd64        Universally Unique ID library - headers and static libraries
+ii  uuid-dev:amd64                   2.37.2-4ubuntu3.4                       amd64        Universally Unique ID library - headers and static libraries
 ii  uuid-runtime                     2.37.2-4ubuntu3                         amd64        runtime components for the Universally Unique ID library
 ii  vdpau-driver-all:amd64           1.4-3build2                             amd64        Video Decode and Presentation API for Unix (driver metapackage)
-ii  vim                              2:8.2.3995-1ubuntu2.15                  amd64        Vi IMproved - enhanced vi editor
-ii  vim-common                       2:8.2.3995-1ubuntu2.15                  all          Vi IMproved - Common files
-ii  vim-runtime                      2:8.2.3995-1ubuntu2.15                  all          Vi IMproved - Runtime files
-ii  vim-tiny                         2:8.2.3995-1ubuntu2.15                  amd64        Vi IMproved - enhanced vi editor - compact version
+ii  vim                              2:8.2.3995-1ubuntu2.16                  amd64        Vi IMproved - enhanced vi editor
+ii  vim-common                       2:8.2.3995-1ubuntu2.16                  all          Vi IMproved - Common files
+ii  vim-runtime                      2:8.2.3995-1ubuntu2.16                  all          Vi IMproved - Runtime files
+ii  vim-tiny                         2:8.2.3995-1ubuntu2.16                  amd64        Vi IMproved - enhanced vi editor - compact version
 ii  wget                             1.21.2-2ubuntu1                         amd64        retrieves files from the web
 ii  whiptail                         0.52.21-5ubuntu2                        amd64        Displays user-friendly dialog boxes from shell scripts
 ii  x11-common                       1:7.7+23ubuntu2                         all          X Window System (X.Org) infrastructure
@@ -905,10 +911,10 @@
 ii  xfonts-utils                     1:7.7+6build2                           amd64        X Window System font utility programs
 ii  xfsprogs                         5.13.0-1ubuntu2                         amd64        Utilities for managing the XFS filesystem
 ii  xkb-data                         2.33-1                                  all          X Keyboard Extension (XKB) configuration data
-ii  xserver-common                   2:21.1.4-2ubuntu1.7~22.04.8             all          common files used by various X servers
-ii  xserver-xorg-core                2:21.1.4-2ubuntu1.7~22.04.8             amd64        Xorg X server - core server
-ii  xserver-xorg-video-nvidia-545    545.23.08-0ubuntu1                      amd64        NVIDIA binary Xorg driver
-ii  xxd                              2:8.2.3995-1ubuntu2.15                  amd64        tool to make (or reverse) a hex dump
+ii  xserver-common                   2:21.1.4-2ubuntu1.7~22.04.10            all          common files used by various X servers
+ii  xserver-xorg-core                2:21.1.4-2ubuntu1.7~22.04.10            amd64        Xorg X server - core server
+ii  xserver-xorg-video-nvidia-550    550.54.15-0ubuntu1                      amd64        NVIDIA binary Xorg driver
+ii  xxd                              2:8.2.3995-1ubuntu2.16                  amd64        tool to make (or reverse) a hex dump
 ii  xz-utils                         5.2.5-2ubuntu1                          amd64        XZ-format compression utilities
 ii  zerofree                         1.1.1-1build3                           amd64        zero free blocks from ext2, ext3 and ext4 file-systems
 ii  zlib1g:amd64                     1:1.2.11.dfsg-2ubuntu9.2                amd64        compression library - runtime

@bhearsum
Copy link
Collaborator

bhearsum commented Oct 9, 2024

Doing some more research on this. systemd/systemd#21123 is one of the first things that comes up. The solution that most people use is to switch to dnsmasq. @aerickson - is that something we can try here? (I can work up a monopacker PR for it - just want to check that it's a viable thing to try before doing so.)

@aerickson
Copy link
Member

Nice find. Yeah, definitely.

bhearsum added a commit to bhearsum/monopacker that referenced this issue Oct 10, 2024
@bhearsum
use dnsmasq for translations generic worker
409d286 
Hopefully a fix for mozilla/translations#549
@bhearsum bhearsum mentioned this issue Oct 10, 2024
Open
bhearsum added a commit to bhearsum/monopacker that referenced this issue Oct 10, 2024
@bhearsum
use dnsmasq for translations generic worker
3ef417a 
Hopefully a fix for mozilla/translations#549
@bhearsum bhearsum mentioned this issue Oct 11, 2024
Merged
@bhearsum
Copy link
Collaborator

bhearsum commented Dec 2, 2024

#949 is switching CPU tasks to a new image running Ubuntu 24.04. We'll be switching GPU tasks to that in the near future as well. I'm hoping that taking newer versions of systemd-resolved and/or other packages will help with this, but we'll see.

@eu9ene eu9ene removed the blocker Very important issue that blocks training label Dec 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something is broken or not correct taskcluster Issues related to the Taskcluster implementation of the training pipeline
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@aerickson @bhearsum @gregtatum @eu9ene