Refactor test suite to support starting components asynchronously

[mszulcz-mitre]

This pull request addresses Issue #127.

[mszulcz-mitre]

@HalosGhost So far, the only commit in this PR addresses the specific issue we previously discussed: the inability of the atomizer to wait to sync with watchtowers in atomizer_end_to_end_test.cpp. On my machine, at least, this commit makes the previous call to sleep between initializing the watchtowers and atomizer in the test unnecessary (line 50):

opencbdc-tx/tests/integration/atomizer_end_to_end_test.cpp

Lines 46 to 52 in 9daab81

	std::thread w_init_thread([&]() {
	ASSERT_TRUE(m_ctl_watchtower->init());
	});
	std::this_thread::sleep_for(std::chrono::milliseconds(100));
	ASSERT_TRUE(m_ctl_atomizer->init());

Is this sufficient to fix this test? I doubt it--but I don't have any way of knowing for certain right now. Unfortunately, I still can't reproduce the failures described by Issues #119 and #127. I've followed the instructions in #119:

• The log level is already set to trace in two_phase_end_to_end_test.cpp.
• I launch the tests though a shell in emacs.

I've also tried the following:

• Running the tests in a docker container to slow things down (launched via emacs).
• Running the tests in a docker container that's restricted to only use 1 CPU core to slow things down even more (launched via emacs).
• Commenting out all the calls to std::this_thread::sleep_for(std::chrono::milliseconds(100)); (lines 38, 42, 43, 45):
  

  opencbdc-tx/tests/integration/two_phase_end_to_end_test.cpp

  Lines 33 to 50 in 9daab81

  	m_ctl_sentinel
  	= std::make_unique<cbdc::sentinel_2pc::controller>(0,
  	m_opts,
  	m_logger);
  	std::this_thread::sleep_for(std::chrono::milliseconds(100));
  	ASSERT_TRUE(m_ctl_shard->init());
  	std::this_thread::sleep_for(std::chrono::milliseconds(100));
  	ASSERT_TRUE(m_ctl_coordinator->init());
  	std::this_thread::sleep_for(std::chrono::milliseconds(100));
  	ASSERT_TRUE(m_ctl_sentinel->init());
  	std::this_thread::sleep_for(std::chrono::milliseconds(100));
  	reload_sender();
  	reload_receiver();
  	std::this_thread::sleep_for(m_wait_interval);

I recall that you may have mentioned some other way to try to trigger the problems. Could you please remind me what you said?