Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to restore thin pool LVs and thin LV via vgcfgrestore? #126

Open
stevenshiau opened this issue Dec 19, 2019 · 28 comments
Open

How to restore thin pool LVs and thin LV via vgcfgrestore? #126

stevenshiau opened this issue Dec 19, 2019 · 28 comments

Comments

@stevenshiau
Copy link

In the lvmthin manul, it says:
lvremove of thin pool LVs, thin LVs and snapshots cannot be reversed with vgcfgrestore.
vgcfgbackup does not back up thin pool metadata.

I tried to use vgcfgbackup to save the LVM layout which includes thin pool LVs and thin LVs. When restore it by "vgcfgrestore --force", the thin pool LVs and thin LVs are restored. However, I just can not make it active..
The following is the issue I have:
root@debian:~# lvscan
inactive '/dev/testVG/test-pool' [2.00 GiB] inherit
inactive '/dev/testVG/test-thinLV' [1.00 GiB] inherit

root@debian:~# lvchange -ay -K -v /dev/testVG/test-pool
Activating logical volume testVG/test-pool.
activation/volume_list configuration setting not defined: Checking only host tags for testVG/test-pool.
Creating testVG-test--pool_tmeta
Loading table for testVG-test--pool_tmeta (254:0).
Resuming testVG-test--pool_tmeta (254:0).
Creating testVG-test--pool_tdata
Loading table for testVG-test--pool_tdata (254:1).
Resuming testVG-test--pool_tdata (254:1).
Executing: /usr/sbin/thin_check -q /dev/mapper/testVG-test--pool_tmeta
Creating testVG-test--pool-tpool
Loading table for testVG-test--pool-tpool (254:2).
Resuming testVG-test--pool-tpool (254:2).
Thin pool testVG-test--pool-tpool (254:2) transaction_id is 2, while expected 1.
Removing testVG-test--pool-tpool (254:2)
Removing testVG-test--pool_tdata (254:1)
Removing testVG-test--pool_tmeta (254:0)

So my question, how can I backup the LVM config, including thin provisioning, then restore it on a blank device?
Thank you very much.

Steven

@jthornber
Copy link
Owner

The thin pool metadata holds all the mappings between the thin devices and the the 'data device' (which is typically a hidden LV). These mappings are changed by the kernel when blocks are provisioned, or writes to snapshots occur. The metadata changes far too quickly for it to be part of the LVM metadata. Because it's constantly changing we cannot back it up in any meaningful way unless the pool is never used between the backup and restore.

Could you tell me some more about what you're trying to do please? eg, if you're moving the pool to a new machine you just need to activate the metadata and data devices (but not the pool), and copy them across.

@zkabelac
Copy link

vgcfgrestore --force is there purely for the purpose of 'repairing' metadata when user is well aware of the context - --force is not meant to be used in random way here (aka. let's try and see).

When thin-pool metadata are repaired with external tool (thin_repair) - they need to be then matching lvm2 user space metadata and transaction id stored in them - this is currently non-automated manual work, where skilled user may need to update lvm2 metadata if some volumes/transactions got lost during kernel metadata repair operation.

To be more explicit here - lvm2 metadata store information about LV size, name, dates and which segment of a PV keeps info for segments in LV (typical granularity 4M) (only occasionally changed)
Thin-pool kernel metadata store info about which chunk in thin-pool represent which chunk in thin LV (granilarity >= 64K) (frequently changed).

Currently there is not much point in support of storing backups of kernel metadata - since they are becoming invalid relatively quickly (i.e. bTrees are no longer addressing same data blocks) - there is only 1 limited case - if thin-pool has been used for 1 thin-pool device without discard - in that case 'allocated' kernel blocks would be more or less persistent - but why would anyone used thin-pool in this case?? when plain Linear LV does this job in way more elegant way...

@bryancooper989
Copy link

@jthornber We are trying to clone or image the file systems on LV, including thin provisioning, as requested here:
https://sourceforge.net/p/clonezilla/bugs/330/
The file system on LV will be in unmounted status, so maybe there is some method we can take an image or clone it?
Thank you very much.

@bryancooper989
Copy link

@zkabelac Thank you very much. As mentioned previously, we'd like to take an image for the whole file systems on the partitions and LV, including thin provisioning. Since the file system is in unmounted status, hence the data on it won't be changed, there should be some way we can image it.

@jthornber
Copy link
Owner

jthornber commented Dec 19, 2019 via email

@stevenshiau
Copy link
Author

Yes, we do. So what's the best way to do this? Thank you so much.

Steven

@zkabelac
Copy link

At this moment - you can 'activate' thin-pool's data & metadata device in so-called 'component' activation mode - so both subLVs can be activated in read-only mode - you can copy these LVs
(via 'dd') and then you need to 'copy' from vgcfgbackup respective portion of related LVs - which is not completely 'trival' but not super hard either.

If you are copying 'whole' PV - you then get whole copy of lvm2 metadata with all content of the PV itself - but that also mean 100% copy of whole device (even unused portions of it)

ATM there is no other 'cloning' mechanism available - but it could be possibly interesting RFE...

It just need to be clear 'vgcfgbackup/vgcfgrestore' can't be see as 'cloning' mechanism - it expects underlying content of device is matching restored lvm2 metadata - lvm2 is not solving this.

@stevenshiau
Copy link
Author

Thank you very much. Could you please let me know how to "'activate' thin-pool's data & metadata device in so-called 'component' activation mode"?
Actually for the moment the key is the block device can not be created and activated by "vgcfgrestore --force" and lvchange commands.
Once the block device is ready, we can use Partclone or dd to write the data to the thin LV device, like /dev/testVG/test-thinLV.

Steven

@zkabelac
Copy link

zkabelac commented Dec 20, 2019

Component activation is supported from version >= 2.02.178.
If the thin-pool and all thin LVs from this pool are inactive - you can activate vg/pool_tdata and vg/pool_tmeta as standalone LVs (in read-only mode) via 'standard' lvchange -ay command.

@stevenshiau
Copy link
Author

Thanks for your reply. I am running Debian Sid, so the lvm2 is 2.03:
root@debian:~# dpkg -l lvm2
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-=================================
ii lvm2 2.03.02-3 amd64 Linux Logical Volume Manager

When I tried to active the component, it failed as the following:
root@debian:~# lvscan
inactive '/dev/testVG/test-pool' [2.00 GiB] inherit
inactive '/dev/testVG/test-thinLV' [1.00 GiB] inherit

root@debian:~# lvchange -ay -K -v /dev/testVG/testVG-test--pool_tmeta
Do you want to activate component LV in read-only mode? [y/n]: y
Accepted input: [y]
Allowing activation of component LV.
Failed to find logical volume "testVG/testVG-test--pool_tmeta"

root@debian:~# lvchange -ay -v /dev/testVG/testVG-test--pool_tmeta
Do you want to activate component LV in read-only mode? [y/n]: y
Accepted input: [y]
Allowing activation of component LV.
Failed to find logical volume "testVG/testVG-test--pool_tmeta"

root@debian:~# lvchange -ay -K -v /dev/testVG/testVG-test--pool_tdata
Do you want to activate component LV in read-only mode? [y/n]: y
Accepted input: [y]
Allowing activation of component LV.
Failed to find logical volume "testVG/testVG-test--pool_tdata"

root@debian:~# lvchange -ay -v /dev/testVG/testVG-test--pool_tdata
Do you want to activate component LV in read-only mode? [y/n]: y
Accepted input: [y]
Allowing activation of component LV.
Failed to find logical volume "testVG/testVG-test--pool_tdata"

Looks like something more I have to do so that the components can be activated?
Thank you very much.

Steven

@zkabelac
Copy link

zkabelac commented Dec 23, 2019

Please always use VG/LV name. Assuming your VG name is: testVG-test
and your LV name is: pool_tdata & pool_tmeta. Avoid using any other prefixes & paths.

So in your case:

lvchange -ay testVG-test/pool_tdata
lvchange -ay testVG-test/pool_tmeta

@stevenshiau
Copy link
Author

stevenshiau commented Jun 5, 2020

Sorry for the late response. Now I am still having problem to active the thin volume.
The following is the whole steps I have done, still I can not make it. Where am I wrong? Thank you very much.

root@debian:# fdisk /dev/sda

Welcome to fdisk (util-linux 2.35.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): n
Partition type
p primary (1 primary, 0 extended, 3 free)
e extended (container for logical partitions)
Select (default p): p
Partition number (2-4, default 2):
First sector (16386048-16777215, default 16386048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (16386048-16777215, default 16777215):

Created a new partition 2 of type 'Linux' and of size 191 MiB.

Command (m for help): wq
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

root@debian:# cat lvm_testVG.conf
# Generated by LVM2 version 2.03.07(2) (2019-11-30): Fri Jun 5 06:43:56 2020

contents = "Text Format Volume Group"
version = 1

description = "vgcfgbackup -f /tmp/vgcfg_tmp.5KB91l testVG"

creation_host = "debian" # Linux debian 5.6.0-2-amd64 #1 SMP Debian 5.6.14-1 (2020-05-23) x86_64
creation_time = 1591339436 # Fri Jun 5 06:43:56 2020

testVG {
id = "ZshH9l-cWgM-GfIO-EYEq-zb3f-QFEF-j6nKUO"
seqno = 6
format = "lvm2" # informational
status = ["RESIZEABLE", "READ", "WRITE"]
flags = []
extent_size = 8192 # 4 Megabytes
max_lv = 0
max_pv = 0
metadata_copies = 0

    physical_volumes {

            pv0 {
                    id = "R1piNO-a61j-6sq5-byW5-0Z9w-jY4x-v4F1h3"
                    device = "/dev/sda1"    # Hint only

                    status = ["ALLOCATABLE"]
                    flags = []
                    dev_size = 16384000     # 7.8125 Gigabytes
                    pe_start = 2048
                    pe_count = 1999 # 7.80859 Gigabytes
            }
    }

    logical_volumes {

            test-pool {
                    id = "o6VNOt-Tef2-sKqE-r2DN-63Ke-xAxE-UOC4nS"
                    status = ["READ", "WRITE", "VISIBLE"]
                    flags = []
                    creation_time = 1591339380      # 2020-06-05 06:43:00 +0000
                    creation_host = "debian"
                    segment_count = 1

                    segment1 {
                            start_extent = 0
                            extent_count = 512      # 2 Gigabytes

                            type = "thin-pool"
                            metadata = "test-pool_tmeta"
                            pool = "test-pool_tdata"
                            transaction_id = 1
                            chunk_size = 128        # 64 Kilobytes
                            discards = "passdown"
                            zero_new_blocks = 1
                    }
            }

            test-thinLV {
                    id = "IKUL4H-lFZe-9J27-hAXi-hgRM-0uZn-eVNHea"
                    status = ["READ", "WRITE", "VISIBLE"]
                    flags = []
                    creation_time = 1591339380      # 2020-06-05 06:43:00 +0000
                    creation_host = "debian"
                    segment_count = 1

                    segment1 {
                            start_extent = 0
                            extent_count = 256      # 1024 Megabytes

                            type = "thin"
                            thin_pool = "test-pool"
                            transaction_id = 0
                            device_id = 1
                    }
            }

            lvol0_pmspare {
                    id = "VNf9tT-jbzt-vkkx-TLb8-4dvR-M0bE-btQlhN"
                    status = ["READ", "WRITE"]
                    flags = []
                    creation_time = 1591339380      # 2020-06-05 06:43:00 +0000
                    creation_host = "debian"
                    segment_count = 1

                    segment1 {
                            start_extent = 0
                            extent_count = 1        # 4 Megabytes

                            type = "striped"
                            stripe_count = 1        # linear

                            stripes = [
                                    "pv0", 0
                            ]
                    }
            }

            test-pool_tmeta {
                    id = "h0HBaY-guml-nrz5-qCUH-f2j2-Qsu9-xkLD4X"
                    status = ["READ", "WRITE"]
                    flags = []
                    creation_time = 1591339380      # 2020-06-05 06:43:00 +0000
                    creation_host = "debian"
                    segment_count = 1

                    segment1 {
                            start_extent = 0
                            extent_count = 1        # 4 Megabytes

                            type = "striped"
                            stripe_count = 1        # linear

                            stripes = [
                                    "pv0", 513
                            ]
                    }
            }

            test-pool_tdata {
                    id = "haDzV8-RUHe-kcYL-6HtW-LX4R-dkds-fuOcOr"
                    status = ["READ", "WRITE"]
                    flags = []
                    creation_time = 1591339380      # 2020-06-05 06:43:00 +0000
                    creation_host = "debian"
                    segment_count = 1

                    segment1 {
                            start_extent = 0
                            extent_count = 512      # 2 Gigabytes

                            type = "striped"
                            stripe_count = 1        # linear

                            stripes = [
                                    "pv0", 1
                            ]
                    }
            }
    }

}
root@debian:# pvcreate -ff --yes --uuid R1piNO-a61j-6sq5-byW5-0Z9w-jY4x-v4F1h3 --zero y --restorefile lvm_testVG.conf /dev/sda1
WARNING: Couldn't find device with uuid R1piNO-a61j-6sq5-byW5-0Z9w-jY4x-v4F1h3.
Physical volume "/dev/sda1" successfully created.

root@debian:# pvscan
PV /dev/sda1 lvm2 [7.81 GiB]
Total: 1 [7.81 GiB] / in use: 0 [0 ] / in no VG: 1 [7.81 GiB]

root@debian:# vgcfgrestore --force -f lvm_testVG.conf testVG
WARNING: Forced restore of Volume Group testVG with thin volumes.
Restored volume group testVG.

root@debian:# vgscan
Found volume group "testVG" using metadata type lvm2

root@debian:# lvscan
inactive '/dev/testVG/test-pool' [2.00 GiB] inherit
inactive '/dev/testVG/test-thinLV' [1.00 GiB] inherit

root@debian:# lvchange -ay testVG-test/pool_tdata
Do you want to activate component LV in read-only mode? [y/n]: y
Allowing activation of component LV.
Volume group "testVG-test" not found
Cannot process volume group testVG-test

root@debian:# lvchange -ay testVG/test-pool
Thin pool testVG-test--pool-tpool (254:2) transaction_id is 2, while expected 1.

root@debian:# lvchange -ay testVG/test-pool_tdata
Do you want to activate component LV in read-only mode? [y/n]: y
Allowing activation of component LV.

root@debian:# lvchange -ay testVG/test-pool_tmeta
Do you want to activate component LV in read-only mode? [y/n]: y
Allowing activation of component LV.

root@debian:# lvscan
inactive '/dev/testVG/test-pool' [2.00 GiB] inherit
inactive '/dev/testVG/test-thinLV' [1.00 GiB] inherit

root@debian:# ls -l /dev/testVG/
total 0
lrwxrwxrwx 1 root root 7 Jun 5 11:39 test-pool_tdata -> ../dm-0
lrwxrwxrwx 1 root root 7 Jun 5 11:39 test-pool_tmeta -> ../dm-1

root@debian:# lvchange -ay testVG
Activation of logical volume testVG/test-pool is prohibited while logical volume testVG/test-pool_tmeta is active.
Activation of logical volume testVG/test-thinLV is prohibited while logical volume testVG/test-pool_tmeta is active.

root@debian:~# lvscan
inactive '/dev/testVG/test-pool' [2.00 GiB] inherit
inactive '/dev/testVG/test-thinLV' [1.00 GiB] inherit

@stevenshiau stevenshiau changed the title How to restore thin pool LVs and thin LV vig vgcfgrestore? How to restore thin pool LVs and thin LV via vgcfgrestore? Apr 25, 2021
@mailinglists35
Copy link

hi, is there a roadmap for this? clonezilla fails to clone block devices containing vg with thin pools, they mention this issue in their support mailing lists.

@jthornber
Copy link
Owner

jthornber commented Jun 30, 2023 via email

@mailinglists35
Copy link

what they do is activate all LV's and dump their contents one by one using partclone/partimage/dd, and also they do some vgcfgbackup/vgcfgrestore I believe. I haven't examined the logs in detail, but the bottom line is they fail to duplicate VGs that contain thin LVs, and invoke particularly this github issue as the root cause for the inability to "clone".

@mailinglists35
Copy link

this is the referenced clonezilla discussion

https://sourceforge.net/p/clonezilla/bugs/330/

@jthornber
Copy link
Owner

jthornber commented Jun 30, 2023 via email

@mailinglists35
Copy link

Sounds like you just want to copy the metadata and data devices then.

I dd copied them, but how do I restore them, since after doing a vgcfgrestore it is only allowed to activate them readonly?
does the blk-archive script allow also restoring to a clean new block device after doing a vgcfgrestore?

@zkabelac
Copy link

For a 'dd' - you just 'lvcreate LV1' for data, then lvcreate LV2' for metadata, and then you make a thin-pool from both LVs and join them into a thin-pool - with parameter you can specify - getting the the info from 'vgcfgrestore',
There is no direct support for 'cloning' of a thin-pool - that would likely need ATM some awk/sed/perl/python work to adapt 'newly' created vg - to match the old with all thins on top of thin-pool. There are likely few 'road to Rome' here..

We already have some similar request for 'empty' VG - but currently it's not supported by upstream lvm2.

@zkabelac
Copy link

As a 'secondary' idea - you could just 'reload' tables for 'read-only' activated volumes with 'dmsetup load' and make devices 'writable' - currently this kind of operations is not supported by lvm2 - eventually we could introduce some 'read-write' component activation - which would be tracked by lvm2 metadata - as the risk of data corruption for user when data are 'writeable' is very high...

@mingnus
Copy link
Collaborator

mingnus commented Jun 30, 2023

does the blk-archive script allow also restoring to a clean new block device after doing a vgcfgrestore?

Yes, blk-archive helps imaging thin devices, and you can restore archived thin-devices to another thin-pool. However, in the case of disk cloning, this way might not be faster than simply copying thin-pool components (metadata & data devices), since an intermediate archiving process is introduced, and extra storage is required.

Right now, a simple 'dd' of component LVs might be the easiest way for disk migration of thin-pools. Further optimizations could be made by copying only metadata/data blocks in-use, like what the Partclone does, yet there's no similar feature in thin-provisioning-tools.

@mailinglists35
Copy link

mailinglists35 commented Jul 2, 2023

hey @stevenshiau what do you think of this approach for clonezilla?
looks quite interesting (vgcfgbackup, dd dumping the tmeta and tdata devices, then vgcfgrestore, then activating tmeta and tdata readonly, then placing their device mapper tables into writeable mode with dmsetup, then restoring the dd image on them)

@stevenshiau
Copy link
Author

"dd dumping the tmeta and tdata devices" -> If you can be more specific, especially with some of examples, we can do more tests to see if it works well.

@mailinglists35
Copy link

"dd dumping the tmeta and tdata devices" -> If you can be more specific, especially with some of examples, we can do more tests to see if it works well.

see lvmthin(7)

A thin pool LV is created by combining two standard LVs: a large data LV that will hold blocks for thin LVs, and a metadata LV that will hold metadata. The metadata tracks which data blocks belong to each thin LV.

in your previous comments, these are the components of the thin pool

root@debian:# ls -l /dev/testVG/
total 0
lrwxrwxrwx 1 root root 7 Jun 5 11:39 test-pool_tdata -> ../dm-0
lrwxrwxrwx 1 root root 7 Jun 5 11:39 test-pool_tmeta -> ../dm-1

you can not activate at the same time the components and the thin LVs. you either activate thin LVs normally OR activate only the thin pool components. the second case is what you have to do to in order clone the whole thin pool (NOT individual thin LVs!).

to enumerate thin pool metadata components:
/usr/sbin/lvm lvs -a --select 'lv_attr=~^e.*' --noheadings -o vg_name,lv_name,DMPath

thin pool data components:
/usr/sbin/lvm lvs -a --select 'lv_attr=~^T.*' --noheadings -o vg_name,lv_name,DMPath

@mailinglists35
Copy link

@stevenshiau for the restoring part: after doing vgcfgrestore, activate only the thin components in read only mode, then hack the device mapper tables to reopen the components devices in read-write mode, as suggested earlier in this discussion, then dd dump back from the previously saved file

for converting the device mapper table from ro to rw:

dmsetup table | grep the component name
this string will contain "ro" , sed it to "rw"
dmsetup suspend the table
feed the new string to dmsetup reload
dmsetup resume the table

after finishing the dd restore, lvchange -an the components

@mingnus
Copy link
Collaborator

mingnus commented Jul 9, 2023

Be aware that the component LVs might be virtual targets (targets that are not map to PV directly), e.g., cache, writecache, or raid. You should handle those target types with care. For example, given a cached thin-pool where the cache-pool is located in other drives, it's better to uncache the thin-pool (lvconver --splitcache) before copying the component LVs, or do migrating the cache beforehand if the user wants to do so. Otherwise, the source and destination thin-pool will share the same cache, which is unacceptable.

@stevenshiau
Copy link
Author

@stevenshiau for the restoring part: after doing vgcfgrestore, activate only the thin components in read only mode, then hack the device mapper tables to reopen the components devices in read-write mode, as suggested earlier in this discussion, then dd dump back from the previously saved file

for converting the device mapper table from ro to rw:

dmsetup table | grep the component name this string will contain "ro" , sed it to "rw" dmsetup suspend the table feed the new string to dmsetup reload dmsetup resume the table

after finishing the dd restore, lvchange -an the components

@mailinglists35 Today I finally got some time to give it a try. However, here I do not find anything after doing vgcfgrestore. dmsetup table shows nothing.
Maybe you can follow my previous post to show all the related commands, based on the LV/VG/PV device names? It will be clearer to me, since I am not really familiar with LVM thin provisioning.
Thank you very much.

@mailinglists35
Copy link

I will try to setup a simulation environment and show what I do step by step

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants