Skip to content

Latest commit

 

History

History
334 lines (184 loc) · 14.7 KB

03.06-Upgrading a Lustre File System.md

File metadata and controls

334 lines (184 loc) · 14.7 KB
Raw

Upgrading a Lustre File System

This chapter describes interoperability between Lustre software releases. It also provides procedures for upgrading from Lustre software release 1.8 to Lustre software release 2.x , from a Lustre software release 2.x to a more recent Lustre software release 2.x (major release upgrade), and from a a Lustre software release 2.x.y to a more recent Lustre software release 2.x.y (minor release upgrade). It includes the following sections:

Release Interoperability and Upgrade Requirements

Lustre software release 2.x (major) upgrade:

  • All servers must be upgraded at the same time, while some or all clients may be upgraded independently of the servers.
  • All servers must be be upgraded to a Linux kernel supported by the Lustre software. See the Lustre Release Notes for your Lustre version for a list of tested Linux distributions.
  • Clients to be upgraded must be running a compatible Linux distribution as described in the Release Notes.

Lustre software release 2.x.y release (minor) upgrade:

  • All servers must be upgraded at the same time, while some or all clients may be upgraded.
  • Rolling upgrades are supported for minor releases allowing individual servers and clients to be upgraded without stopping the Lustre file system.

Upgrading to Lustre Software Release 2.x (Major Release)

The procedure for upgrading from a Lustre software release 2.x to a more recent 2.x release of the Lustre software is described in this section.

**Note **

This procedure can also be used to upgrade Lustre software release 1.8.6-wc1 or later to any Lustre software release 2.x. To upgrade other versions of Lustre software release 1.8.x, contact your support provider.

Note

In Lustre software release 2.2, a feature has been added for ldiskfs-based MDTs that allows striping a single file across up to 2000 OSTs. By default, this "wide striping" feature is disabled. It is activated by setting the ea_inode option on the MDT using either mkfs.lustre or tune2fs. For example after upgrading an existing file system to Lustre software release 2.2 or later, wide striping can be enabled by running the following command on the MDT device before mounting it:

tune2fs -O large_xattr

Once the wide striping feature is enabled and in use on the MDT, it is not possible to directly downgrade the MDT file system to an earlier version of the Lustre software that does not support wide striping. To disable wide striping:

  1. Delete all wide-striped files, OR use lfs_migrate -c 160 (or fewer stripes) to migrate the files to use fewer OSTs. This does not affect the total number of OSTs that the whole filesystem can access.

  2. Unmount the MDT.

  3. Run the following command to turn off the large_xattr option:

    tune2fs -O ^large_xattr

Using either mkfs.lustre or tune2fs with large_xattr or ea_inode option reseults in ea_inode in the file system feature list.

**Note **

To generate a list of all files with more than 160 stripes use lfs find with the --stripe-count option:

lfs find ${mountpoint} --stripe-count=+160

Introduced in Lustre 2.4

Note

In Lustre software release 2.4, a new feature allows using multiple MDTs, which can each serve one or more remote sub-directories in the file system. The root directory is always located on MDT0.

Note that clients running a release prior to the Lustre software release 2.4 can only see the namespace hosted by MDT0 and will return an IO error if an attempt is made to access a directory on another MDT.

To upgrade a Lustre software release 2.x to a more recent major release, complete these steps:

  1. Create a complete, restorable file system backup.

    Caution

    Before installing the Lustre software, back up ALL data. The Lustre software contains kernel modifications that interact with storage devices and may introduce security issues and data loss if not installed, configured, or administered properly. If a full backup of the file system is not practical, a device-level backup of the MDT file system is recommended. See Backing Up and Restoring a File System for a procedure.

  2. Shut down the entire filesystem by following the section called “ Stopping the Filesystem”

  3. Upgrade the Linux operating system on all servers to a compatible (tested) Linux distribution and reboot.

  4. Upgrade the Linux operating system on all clients to Red Hat Enterprise Linux 6 or other compatible (tested) distribution and reboot.

  5. Download the Lustre server RPMs for your platform from the Lustre Releases repository. See Table 5, “Packages Installed on Lustre Servers” for a list of required packages.

  6. Install the Lustre server packages on all Lustre servers (MGS, MDSs, and OSSs).

    1. Log onto a Lustre server as the root user

    2. Use the yum command to install the packages:

      # yum --nogpgcheck install pkg1.rpm pkg2.rpm ...

    3. Verify the packages are installed correctly:

      rpm -qa|egrep "lustre|wc"

    4. Repeat these steps on each Lustre server.

  7. Download the Lustre client RPMs for your platform from the Lustre Releases repository. See Table 6, “Packages Installed on Lustre Clients” for a list of required packages.

    Note

    The version of the kernel running on a Lustre client must be the same as the version of the lustre-client-modules- verpackage being installed. If not, a compatible kernel must be installed on the client before the Lustre client packages are installed.

  8. Install the Lustre client packages on each of the Lustre clients to be upgraded.

    1. Log onto a Lustre client as the root user.

    2. Use the yum command to install the packages:

      # yum --nogpgcheck install pkg1.rpm pkg2.rpm ...

    3. Verify the packages were installed correctly:

      # rpm -qa|egrep "lustre|kernel"

    4. Repeat these steps on each Lustre client.

  9. (Optional) For upgrades to Lustre software release 2.2 or higher, to enable wide striping on an existing MDT, run the following command on the MDT:

    tune2fs -O ea_inode /dev/mdtdev

    For more information about wide striping, see the section called “Lustre Striping Internals”.

  10. (Optional) For upgrades to Lustre software release 2.4 or higher, to format an additional MDT, complete these steps:

    1. Determine the index used for the first MDT (each MDT must have unique index). Enter:

      client$ lctl dl | grep mdc
36 UP mdc lustre-MDT0000-mdc-ffff88004edf3c00 
      4c8be054-144f-9359-b063-8477566eb84e 5

      In this example, the next available index is 1.

    2. Add the new block device as a new MDT at the next available index by entering (on one line):

      mds# mkfs.lustre --reformat --fsname=filesystem_name --mdt \
    --mgsnode=mgsnode --index 1 
/dev/mdt1_device

  11. (Optional) If you are upgrading to Lustre software release 2.3 or higher from Lustre software release 2.2 or earlier and want to enable the quota feature, complete these steps:

    1. Before setting up the file system, enter on both the MDS and OSTs:

      tunefs.lustre --quota

    2. (Optional) If you are upgrading before Lustre software release 2.10, to enable the project quota feature enter the following on every ldiskfs backend target:

      tune2fs –O project /dev/dev

      Note

      Enabling the project feature will prevent the filesystem from being used by older versions of ldiskfs, so it should only be enabled if the project quota feature is required and/or after it is known that the upgraded release does not need to be downgraded.

    3. When setting up the file system, enter:

      conf_param $FSNAME.quota.mdt=$QUOTA_TYPE
conf_param $FSNAME.quota.ost=$QUOTA_TYPE

  12. (Optional) If you are upgrading from Lustre software release 1.8, you must manually enable the FID-in-dirent feature. On the MDS, enter:

    tune2fs –O dirdata /dev/mdtdev

    Warning

    This step is not reversible. Do not complete this step until you are sure you will not be downgrading the Lustre software.

    Introduced in Lustre 2.4This step only enables FID-in-dirent for newly created files. If you are upgrading to Lustre software release 2.4, you can use namespace LFSCK to enable FID-in-dirent for the existing files. For the case of upgrading from Lustre software release 1.8, it is important to note that if you do NOT enable dirdata via the tune2fs command above, the namespace LFSCK will NOT generate FID-in-dirent for the existing files. For more information about FID-in-dirent and related functionalities in LFSCK, see the section called “ Lustre File System Storage and I/O”.

  13. Start the Lustre file system by starting the components in the order shown in the following steps:

    1. Mount the MGT. On the MGS, run

      mgs# mount -a -t lustre

    2. Mount the MDT(s). On each MDT, run:

      mds# mount -a -t lustre

    3. Mount all the OSTs. On each OSS node, run:

      oss# mount -a -t lustre

      Note

      This command assumes that all the OSTs are listed in the /etc/fstab file. OSTs that are not listed in the /etc/fstab file, must be mounted individually by running the mount command:

      mount -t lustre /dev/block_device/mount_point

    4. Mount the file system on the clients. On each client node, run:

      client# mount -a -t lustre

**Note **

The mounting order described in the steps above must be followed for the initial mount and registration of a Lustre file system after an upgrade. For a normal start of a Lustre file system, the mounting order is MGT, OSTs, MDT(s), clients.

If you have a problem upgrading a Lustre file system, see the section called “Reporting a Lustre File System Bug” for ways to get help.

Upgrading to Lustre Software Release 2.x.y (Minor Release)

Rolling upgrades are supported for upgrading from any Lustre software release 2.x.y to a more recent Lustre software release 2.X.y. This allows the Lustre file system to continue to run while individual servers (or their failover partners) and clients are upgraded one at a time. The procedure for upgrading a Lustre software release 2.x.y to a more recent minor release is described in this section.

To upgrade Lustre software release 2.x.y to a more recent minor release, complete these steps:

  1. Create a complete, restorable file system backup.

    Caution

    Before installing the Lustre software, back up ALL data. The Lustre software contains kernel modifications that interact with storage devices and may introduce security issues and data loss if not installed, configured, or administered properly. If a full backup of the file system is not practical, a device-level backup of the MDT file system is recommended. See Backing Up and Restoring a File System for a procedure.

  2. Download the Lustre server RPMs for your platform from the Lustre Releases repository. See Table 5, “Packages Installed on Lustre Servers” for a list of required packages.

  3. For a rolling upgrade, complete any procedures required to keep the Lustre file system running while the server to be upgraded is offline, such as failing over a primary server to its secondary partner.

  4. Unmount the Lustre server to be upgraded (MGS, MDS, or OSS)

  5. Install the Lustre server packages on the Lustre server.

    1. Log onto the Lustre server as the root user

    2. Use the yum command to install the packages:

      # yum --nogpgcheck install pkg1.rpm pkg2.rpm ...

    3. Verify the packages are installed correctly:

      rpm -qa|egrep "lustre|wc"

    4. Mount the Lustre server to restart the Lustre software on the server:

      server# mount -a -t lustre

    5. Repeat these steps on each Lustre server.

  6. Download the Lustre client RPMs for your platform from the Lustre Releases repository. See Table 6, “Packages Installed on Lustre Clients” for a list of required packages.

  7. Install the Lustre client packages on each of the Lustre clients to be upgraded.

    1. Log onto a Lustre client as the root user.

    2. Use the yum command to install the packages:

      # yum --nogpgcheck install pkg1.rpm pkg2.rpm ...

    3. Verify the packages were installed correctly:

      # rpm -qa|egrep "lustre|kernel"

    4. Mount the Lustre client to restart the Lustre software on the client:

      client# mount -a -t lustre

    5. Repeat these steps on each Lustre client.

If you have a problem upgrading a Lustre file system, see the section called “Reporting a Lustre File System Bug” for some suggestions for how to get help.