Pool migration ignores rdonly status of target

[onnozweers]

Dear dCache devs,

I have a strong impression that the "migration move" function of a pool ignores the rdonly (read only) status of a target pool.

This is a bit impractical when you have two pools that you want to drain simultaneously with the poolgroup as a target. They keep on writing files to each other, so it's difficult to get them to a stage where they are empty and can be decommissioned.

So, what I do is set the source pools to rdonly, in the PoolManager:

psu set pool shark11_atlasdisk rdonly
psu set pool shark12_atlasdisk rdonly

Then I drain them both like this:

migration move -concurrency=6 -target=pgroup atlas_writediskpools

And then they are writing files to many pools, including each other. I wouldn't expect that, since they are rdonly. It's actually worse: because they are getting emptier, they are becoming increasingly attractive target pools for each other's migrations.

I work around this by setting the max diskspace to 10 (bytes) repeatedly, so that they don't have any available space to write into. But that's a bit clumsy.

Cheers,
Onno

[paulmillar]

I believe you are right: the migration command does ignore any read-only status of the target pool.

One could argue this is a bug (the pool is read-only, so should accept any more data), but also one could argue this isn't a bug (the admin told me to do this, so I'm overriding the pool's settings).

For now, I'm marking this request an enhancement, but somebody should decide whether this behaviour is actually a bug.

If it isn't a bug then one possible solution would be to add an option to the migration command so that, by default it honours the pools settings but there's an option (something like -f --force) that overrides the pool settings. One could even imagine that option taking an argument (e.g., a list of pools) so that only those pools are overridden, or that the command only overrides the status of destination pools, or ..

[ageorget]

Hi Onno,

I'm also facing this when I have to take out servers for replacement.
The easier work around I found is to use the migration options -include to target only the new installed servers or -exclude to avoid migration between take out servers. And of course set them in readonly mode to avoid new files written by clients.

Cheers
Adrien

[kofemann]

The main misunderstanding comes from the fact, that psu set pool shark11_atlasdisk rdonly sets the pool ONLY read only for PoolManager, e.g only for user-initiated requests. The pool itself stays read-write. Thus, all request that doesn't use pool manager and directly talks to pool can write to it. So, if you need pool to operate in read-only mode, then you should do so on the pool pool disable -rdonly

[kofemann]

I would say: works as expected. But it looks like that too many admins run into this situation. So maybe we need to update psu to disable the pool unless an extra --soft option is specified.

[paulmillar]

Good catch. I missed that it was a pool-manager psu set pool command that was disabling the pool.

So, I agree with @kofemann .

Just to add my 2c-worth, another option might be to update pools so they have an extra state (or extra states) where the pool would be disabled for client-triggered requests (upload/download/stage), but would still allow sys-admin triggered requests. This would have broadly the same effect as the current psu set pool command, but the pool would be the authoritative source of information (pool-manager could cache this info, but that would be only a cache).

That would be a bigger change, but (I think) would bring clarity to a somewhat confusing situation.

[onnozweers]

Thanks guys, I wasn't aware that the poolmanager rdonly was something different than the pool rdonly. Thanks for clearing that up.

I like @kofemann's idea to have the poolmanager's psu set pool shark11_atlasdisk rdonly command update also the pool status (unless specified otherwise). For now I guess I can use the pool disable -rdonly as workaround.

[onnozweers]

Now that I'm using pool disable -rdonly, I can see that a migration goes less fluent.

...
14:40:04 [202318] 000017D69E45D1E64D7482CFC99273057049: Pool [badger4_atlasdisk@local] failed (Pool is disabled)
14:40:22 [202336] 00004C3F96CD18554D24831FD77BFCF5EF28: Pool [badger4_atlasdisk@local] failed (Pool is disabled)
14:40:33 [202349] 0000EDADD9EE44EF4B7081CA3676F0E71492: Pool [badger4_atlasdisk@local] failed (Pool is disabled)

This regularly puts the migration in "sleeping" mode. Wouldn't it be better if the migration move -target=pgroup operation used the PoolManager to choose targets?

[onnozweers]

Just to add my 2c-worth, another option might be to update pools so they have an extra state (or extra states) where the pool would be disabled for client-triggered requests (upload/download/stage), but would still allow sys-admin triggered requests.

Disabling downloads from a pool while draining it with a migration move might be a bit impractical. I prefer the users to be able to access their data while it is waiting to be migrated. In a case where you'd really want the user to back off while draining the pool (e.g. for performance reasons) you can always set the number of movers to a low value or even zero.

[kofemann]

this issues looks like a good candidate to test github discussions :). So, I here am I. 😎