Evaluation Results - ALL SQ and Stuff SQ Lower than Reported in Paper

[YuanZhao42]

Hi,
When I reproduced the results for Section 4.2.2 (Evaluation) using both the pretrained checkpoint and my own trained checkpoint, I noticed that the ALL SQ and Stuff SQ scores are consistently 3 to 4 points lower than those reported in the paper.

Are there any known factors or configuration details that could lead to this discrepancy?

Thank you in advance for your assistance!

[anhquancao]

Hi,

In section 4.2.2 of the Readme, I showed the results I got with the released model when I released the code and models. I copy paste it here. The mIoU is 22.07 and All PQ is 28.43.

method, P, R, IoU, mIoU, All PQ dagger, All PQ, All SQ, All RQ, Thing PQ, Thing SQ, Thing RQ, Stuff PQ, Stuff SQ, Stuff RQ
subnet 0, 59.27, 68.86, 46.74, 20.39, 20.78, 10.96, 55.53, 17.42, 4.47, 46.48, 7.89, 14.20, 60.05, 22.19
subnet 1, 57.90, 69.95, 46.37, 20.14, 20.35, 10.57, 58.42, 16.77, 4.02, 56.09, 7.03, 13.84, 59.58, 21.65
ensemble, 62.66, 65.70, 47.22, 22.07, 28.43, 11.04, 52.86, 17.34, 5.09, 47.10, 8.93, 14.02, 55.74, 21.55
=====================================
==> pq
method, car, bicycle, motorcycle, truck, other-vehicle, person, road, parking, sidewalk, other-ground, building, fence, vegetation, terrain, pole, traffic-sign, other-structure, other-object
subnet 0, 14.98, 0.00, 1.20, 7.34, 1.53, 1.79, 70.77, 4.22, 34.66, 0.25, 24.68, 0.62, 10.31, 6.42, 3.31, 12.67, 0.32, 2.19
subnet 1, 14.12, 0.18, 0.92, 5.76, 1.57, 1.55, 70.25, 4.13, 33.80, 0.29, 24.80, 0.73, 9.59, 6.28, 2.50, 12.09, 0.13, 1.55
ensemble, 16.57, 0.00, 1.59, 8.89, 1.92, 1.58, 71.71, 4.08, 36.25, 0.00, 23.52, 0.61, 8.91, 4.66, 3.27, 13.23, 0.11, 1.82
==> sq
method, car, bicycle, motorcycle, truck, other-vehicle, person, road, parking, sidewalk, other-ground, building, fence, vegetation, terrain, pole, traffic-sign, other-structure, other-object
subnet 0, 58.22, 0.00, 52.01, 54.43, 54.62, 59.60, 74.87, 58.31, 57.89, 62.00, 55.82, 57.20, 54.99, 56.37, 56.55, 66.91, 54.67, 65.05
subnet 1, 57.65, 52.41, 52.77, 56.32, 54.96, 62.41, 74.45, 58.30, 57.92, 58.69, 55.82, 60.48, 54.95, 56.47, 55.98, 67.00, 53.16, 61.78
ensemble, 58.39, 0.00, 54.00, 55.12, 54.42, 60.70, 75.73, 57.44, 59.22, 0.00, 55.90, 63.81, 55.49, 58.33, 55.86, 67.74, 55.85, 63.47
==> rq
method, car, bicycle, motorcycle, truck, other-vehicle, person, road, parking, sidewalk, other-ground, building, fence, vegetation, terrain, pole, traffic-sign, other-structure, other-object
subnet 0, 25.72, 0.00, 2.31, 13.49, 2.81, 3.01, 94.52, 7.24, 59.87, 0.41, 44.21, 1.09, 18.74, 11.39, 5.86, 18.94, 0.59, 3.37
subnet 1, 24.49, 0.35, 1.74, 10.23, 2.85, 2.49, 94.36, 7.08, 58.36, 0.49, 44.43, 1.20, 17.45, 11.12, 4.46, 18.04, 0.25, 2.51
ensemble, 28.38, 0.00, 2.95, 16.13, 3.53, 2.61, 94.70, 7.11, 61.22, 0.00, 42.09, 0.96, 16.06, 7.99, 5.85, 19.53, 0.19, 2.87
[2.1090309619903564, 1.5366406440734863, 1.534111738204956, 1.6147339344024658, 1.208867073059082, 1.52060866355896]
inference time:  1.3322471255736077
[0.02621603012084961, 0.02443861961364746, 0.027606964111328125, 0.02429676055908203, 0.024660348892211914, 0.0240786075592041]
ensemble time:  0.024390966416286672
Uncertainty threshold:  0.5
=====================================
method, ins ece, ins nll, ssc nonempty ece, ssc empty ece, ssc nonempty nll, ssc empty nll,  count, inference time
subnet 0,  0.7668, 4.9833, 0.1879, 0.1252, 1.1738, 2.6776, 35104, 0.00
subnet 1,  0.7735, 5.1097, 0.1990, 0.1225, 1.1935, 2.6386, 35466, 0.00
ensemble,  0.5899, 3.8083, 0.1616, 0.1068, 1.1075, 2.1397, 18990, 0.00
allocated 17902.39387990762