Two camera calibration orientation (green screen example)

[tostercx]

Describe the bug

I'm not sure if this is a bug or not, but perhaps someone can clarify. I need to calibrate two cameras that may not be in the same orientation. In the green screen two camera calibration there's a long paragraph about why the orientation is assumed to always be down and that the algorithm behind it isn't supposed to be able to tell which side is which if the image is flipped by 180 degrees.

Azure-Kinect-Sensor-SDK/examples/green_screen/main.cpp

Lines 395 to 418 in 61951da

	// Before we go on, there's a quick problem with calibration to address. Because the chessboard looks the same when
	// rotated 180 degrees, it is possible that the chessboard corner finder may find the correct points, but in the
	// wrong order.
	// A visual:
	// Image 1 Image 2
	// ..................... .....................
	// ..................... .....................
	// .........xxxxx2...... .....xxxxx1..........
	// .........xxxxxx...... .....xxxxxx..........
	// .........xxxxxx...... .....xxxxxx..........
	// .........1xxxxx...... .....2xxxxx..........
	// ..................... .....................
	// ..................... .....................
	// The problem occurs when this case happens: the find_chessboard() function correctly identifies the points on the
	// chessboard (shown as 'x's) but the order of those points differs between images taken by the two cameras.
	// Specifically, the first point in the list of points found for the first image (1) is the *last* point in the list
	// of points found for the second image (2), though they correspond to the same physical point on the chessboard.
	// To avoid this problem, we can make the assumption that both of the cameras will be oriented in a similar manner
	// (e.g. turning one of the cameras upside down will break this assumption) and enforce that the vector between the
	// first and last points found in pixel space (which will be at opposite ends of the chessboard) are pointing the
	// same direction- so, the dot product of the two vectors is positive.

However while looking for a solution some comments about how OpenCV works suggested that using a non-symmetrical chessboard image solves the problem (can't find source atm). And the chessboard image that comes with the example isn't symmetrical! (One side starts with black squares in the corners and the other one with white ones.)

After disabling the lines responsible for flipping the detected corner coordinates and testing for while surprisingly it works perfectly in any orientation as far as I can tell.

I expected that the example's author simply made a mistake by flipping the coordinates (in which case this would be a bug) but looking trough OpenCV's docs I can't seem to find anything related to this problem. I started looking trough the source but it's a bit too over my head to be honest. So I still don't know for sure if OpenCV checks the colors of the nearby squares to detect the orientation or if my results are a fluke. Or perhaps the detection is there and the example's author had tested this with an older version of OpenCV or a symmetrical chessboard image? The code and comments seem to be well thought out so I'm not sure.

To Reproduce

• Set up two cameras, one flipped upside down
• Run the example
• ---> orientation is not detected correctly
• Disable the following lines:
  

  Azure-Kinect-Sensor-SDK/examples/green_screen/main.cpp

  Lines 420 to 425 in 61951da

  	cv::Vec2f main_image_corners_vec = main_chessboard_corners.back() - main_chessboard_corners.front();
  	cv::Vec2f secondary_image_corners_vec = secondary_chessboard_corners.back() - secondary_chessboard_corners.front();
  	if (main_image_corners_vec.dot(secondary_image_corners_vec) <= 0.0)
  	{
  	std::reverse(secondary_chessboard_corners.begin(), secondary_chessboard_corners.end());
  	}

• recompile, run the example
• ---> as far as I can tell the orientation seems to be detected perfectly

Expected behavior

If OpenCV is built to correctly detect the orientation with non-symmetrical images, these lines should be removed and the example would work even with an upside-down camera. Or perhaps I'm missing some vital information.

Screenshots

Current behavior (corners detected in wrong orientation):

Flipping disabled (corners detected correctly):

edit

And here's an example with the same patch (flipping points disabled) with a symmetrical image (last column covered, starts and ends with a black square) --> corners not detected correctly. The algorithm does seem to take their color into consideration.

[qm13]

@tostercx please also look at #1420.

There may be other bugs in the sample code. You are welcome to submit fixes.