#evaluation metrics of Summe and TVSum datasets τ and ρ

[Pwoer-zy]

hello, how does the evaluation metrics of Summe and TVSum datasets τ and ρ calculate? I only see the calculation method of fscore in the code.
Looking forward to getting your reply, thank you very much!

[boheumd]

Hello. You can refer to the following code to calculate rho and tau results.
https://github.com/e-apostolidis/CA-SUM/blob/main/evaluation/choose_best_model.py#L67
https://gitlab.uni-hannover.de/hussainkanafani/unsupervised-video-summarization/-/blob/master/src/evaluation/BaseEvaluator.py#L7
https://github.com/TIBHannover/MSVA/blob/master/train.py#L430

[Pwoer-zy]

Hello, thank you for your reply

[rodosingh]

Hi, thanks for this good work and such clean code.

I don't think the computation of rho and tau is consistent throughout all the links you mentioned.
Can you provide your source code for this? That would be very much helpful.

Thanks in advance.

[Pwoer-zy]

Hello. You can refer to the following code to calculate rho and tau results. https://github.com/e-apostolidis/CA-SUM/blob/main/evaluation/choose_best_model.py#L67 https://gitlab.uni-hannover.de/hussainkanafani/unsupervised-video-summarization/-/blob/master/src/evaluation/BaseEvaluator.py#L7 https://github.com/TIBHannover/MSVA/blob/master/train.py#L430

I tried according to what you said, but the result was not good. Can you provide your source code for your code? That would be very much helpful.

Thanks in advance.

[joserochh]

Hello,

My question here would be. Are you using original user scores for both SumMe and TVSum? or are you averaging user summaries and taking that as user scores? Do you do it differently for SumMe and TVSum? I have not seen yet user annotations for SumMe dataset.

Thank you!