log: list the number of pending cmds for each device

[lxbsz]

Currently we have hit the problem that tcmu-runner couldn't get any
response from the gluster backend for many times in the product, and
there is no any explict evidence from the logs.

This patch will add one timer for each cmd and will record the number
pending cmds for each tcmu device in 5/10/15/20/25/30+ seconds.

[lxbsz]

@mikechristie @pkalever Please review, thanks.

[mikechristie]

@lxbsz just to make sure I am on the same page. This patch will be used instead of the other timer patch? Will you need the functionality in the other timer patch in the future?

[lxbsz]

@lxbsz just to make sure I am on the same page. This patch will be used instead of the other timer patch? Will you need the functionality in the other timer patch in the future?

Currently in gluster side there still has some problems to add the timeout & clean the ringbuffer patch for each command here.

And this listing the pending cmds patch will be helpful before that patch is done, and this patch should also be useful for the other handlers, which do not have the timer in the backend stoarge sevice.

IMO, this two patches( the timeout & clean patch and the listing the pending cmds patch) should be coexist. The timeout & clean patch could help recovery the iscsi device or multipath device and keep it working well and the second one could give us some clue to troubleshooting.

Thanks,

[lxbsz]

@mikechristie Hi Mike, what's the status of this PR ?
We are hitting the stuck issue many times, so we need this patch to get some useful logs, or we must keep reproducing it again and again to confirm it.

Thanks.

[mikechristie]

Sorry. I pinged Richard for you the other week. He is looking into the proper file heading for the timer code.

However, I was looking for possible existing timer libs that could be used. For example, I was thinking we could maybe use the uv timer lib from the azure PR

#540

Maybe move that timer code to the core libtcmu or tcmu-runner code and add a handler call back so we can do what you needed to do to log the cmd and then also allow handlers to cancel cmds like how azure wants to.

[lxbsz]

Sorry. I pinged Richard for you the other week. He is looking into the proper file heading for the timer code.

However, I was looking for possible existing timer libs that could be used. For example, I was thinking we could maybe use the uv timer lib from the azure PR

#540

Maybe move that timer code to the core libtcmu or tcmu-runner code and add a handler call back so we can do what you needed to do to log the cmd and then also allow handlers to cancel cmds like how azure wants to.

Cool, that will be very nice if we can use the uv timer lib here. And it seems [1] could work for us by using a higher precision.

[1] http://docs.libuv.org/en/v1.x/timer.html

[lxbsz]

@mikechristie
Switch to libuv now.

@cavery
For the azblk handler, will this be helpful for you ?

@pkalever Please review, thanks.

[lxbsz]

Rebased to the latest and changed the STEP from 5 second to 30.

[mikechristie]

Closing this PR. Implemented here

#568