Automatic upgrade strategy

[countless-integers]

With AL2 I had a setup where yum-cron would apply security updates automatically. In AL23 this doesn't seem feasible as dnf-automatic (service for automated updates) doesn't pick up security updates as they come in new releases of AL23. I could add a timer+service to update release versions, but it feels a hacky (automatic package updates already felt a bit weird, but on AL2 it seem to have worked reasonably well). What would be a recommended way to tackle this problem on AL23?

[stewartsmith]

Have a look at the following two sections of the User Guide:

https://docs.aws.amazon.com/linux/al2023/ug/managing-repos-os-updates.html

https://docs.aws.amazon.com/linux/al2023/ug/deterministic-upgrades-usage.html

tl;dr: By passing latest as releasever, you can refer to whatever our latest release may be.

[countless-integers]

I don't think dnf-automatic can upgrade the release version on its own, only packages within the release. That's my problem -- it doesn't have an option to pass in the release version/tag. Anyway, I can drop dnf-automatic and just write a script for updates myself. Getting the latest version and running upgrade/update is easy enough, but it's really confusing to asses the state after I run the upgrade command. Take this example after running sudo dnf upgrade --security --assumeyes --releasever=2023.1.20230705 once:

[~]$ sudo dnf upgrade --security --assumeyes --releasever=2023.1.20230705
Last metadata expiration check: 0:15:24 ago on Mon Jul 10 09:49:15 2023.
Dependencies resolved.
Nothing to do.
Complete!
[~]$ dnf check-release-update --latest-only --version-only
2023.1.20230705
[~]$ dnf check-release-update
WARNING:
  A newer release of "Amazon Linux" is available.

  Available Versions:

  Version 2023.1.20230705:
    Run the following command to upgrade to 2023.1.20230705:

      dnf upgrade --releasever=2023.1.20230705

    Release notes:
     https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes.html

which one's output is correct? Is there some metadata caching layer somewhere that I'm not aware of? (I'm not very familiar with rhel/fedora distros tbh)

[celesteking]

Extremely frustrated with your choice of dumping enterprise users. Enterprise doesn't care for instance replacement, it cares for stability and longevity.
You should've offered a streamlined experience for users wanting to stay on a defined release AND wanting to receive security (0day) package updates, even if that would last 2 months. Reprovisioning an instance each day is absurd to point of stupidity.

dnf-automatic is a de-facto standard in RPM/DNF world and should be used to do the update procedure.
The documenation you've provided is confusing and contradicts itself.

[ozbenh]

dnf check-release-update will base its findings on the version of the system-release package that you have installed. As long as that package doesn't get updated, it will continue offerring a new release.

Note that we generally try to be careful with updating packages for security or grave bugs only. So we normally recommend normal upgrades. The main difference is that new releases might also offer new packages you can chose to install, but that's really about it.

[BansariSheth]

Can't agree more, did anyone find a way to automate the upgrades without having to rebuilt the servers every month and not update the servers manually?
I use deterministic upgrade process where /etc/dnf/vars/releasever is set to latest, but I see instance is not doing the system-release update

[xip-niekneuij]

In terraform, you can define data from AMI marketplace for the latest AL 2023.

Example for getting the latest AMI of Amazon Linux 2023 for ARM:

data "aws_ami" "amazon-linux-2023" {
  most_recent = true
  owners      = ["amazon"]
  filter {
    name   = "name"
    values = ["al2023-ami-2*arm64"]
  }
}

Then refer to the data in aws_instance.

Example of Amazon Linux 2023 from data source with instance type t4g.nano:

resource "aws_instance" "example" {
  ami                    = data.aws_ami.amazon-linux-2023.id
  instance_type          = "t4g.nano"
}

Which means, whenever you update your EC2 instance, you get the latest Amazon Linux 2023 release.

[seag-rvc]

Hi @stewartsmith,

You recommend replacing the instance, which is fair. However, the AMI you provide should be at the highest available version. That is not the case. For example at the time of writing (06 Nov 2024), the version of AMI you provide is al2023-ami-2023.6.20241010.0-kernel-6.1-x86_64 and once up and running I can see the available release versions are 2023.6.20241028 and 2023.6.20241031. It means the AMI you provide, is two release versions outdated.

So instead of getting the latest AMI while an instance is being built, you force us to run dnf upgrade --assumeyes --releasever=latest and possibly a reboot increasing the deployment time.

$date
Wed Nov  6 15:32:47 GMT 2024

$aws ec2 describe-images --owners amazon --filters "Name=root-device-type,Values=ebs" --filters "Name=name,Values=al2023-ami-2023.6.202410*-kernel-6.1-x86_64"
{
    "Images": [
        {
            "Architecture": "x86_64",
            "CreationDate": "2024-10-11T16:52:36.000Z",
            "ImageId": "ami-0acc77abdfc7ed5a6",
            "ImageLocation": "amazon/al2023-ami-2023.6.20241010.0-kernel-6.1-x86_64",
            "ImageType": "machine",
            "Public": true,
            "OwnerId": "137112412989",
            "PlatformDetails": "Linux/UNIX",
            "UsageOperation": "RunInstances",
            "State": "available",
            "BlockDeviceMappings": [
                {
                    "DeviceName": "/dev/xvda",
                    "Ebs": {
                        "DeleteOnTermination": true,
                        "Iops": 3000,
                        "SnapshotId": "snap-02e97e5e711a7d2ad",
                        "VolumeSize": 8,
                        "VolumeType": "gp3",
                        "Throughput": 125,
                        "Encrypted": false
                    }
                }
            ],
            "Description": "Amazon Linux 2023 AMI 2023.6.20241010.0 x86_64 HVM kernel-6.1",
            "EnaSupport": true,
            "Hypervisor": "xen",
            "ImageOwnerAlias": "amazon",
            "Name": "al2023-ami-2023.6.20241010.0-kernel-6.1-x86_64",
            "RootDeviceName": "/dev/xvda",
            "RootDeviceType": "ebs",
            "SriovNetSupport": "simple",
            "VirtualizationType": "hvm",
            "BootMode": "uefi-preferred",
            "DeprecationTime": "2025-01-09T16:53:00.000Z",
            "ImdsSupport": "v2.0"
        }
    ]
}

Last metadata expiration check: 2:52:16 ago on Wed Nov  6 12:17:53 2024.
=================================================================================================================================================================================================================
WARNING:
  A newer release of "Amazon Linux" is available.

  Available Versions:

  Version 2023.6.20241028:
    Run the following command to upgrade to 2023.6.20241028:

      dnf upgrade --releasever=2023.6.20241028

    Release notes:
     https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241028.html

  Version 2023.6.20241031:
    Run the following command to upgrade to 2023.6.20241031:

      dnf upgrade --releasever=2023.6.20241031

    Release notes:
     https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241031.html

[ozbenh]

Something is not right here, we release AMIs at the same time as we release repository releases (well, there's a very small delay but we are talking hours max). Our release process involves both the generation of repos, AMIs and release notes in one big pipeline.

I'll follow up internally

[ozbenh]

So normally AMIs are visiable around 24h after a repo release. We are working on making it completely simultaneous. What you observe above seems to be a case of accidentally hitting a combination of one-off issues unfortunately. I think we retired one of the AMIs due to an issue, and then delayed the release of the second one until after the events of Nov 5 (we generally delay releasing things around high-usage events). Mostly one-offs. The recommendation remains that using AMI replacement is a better update strategy.