Fix flaky tests

[decentral1se]

How to help

• Pick a test from the list below 🗒️
• Run the test over and over again until you see a failure (see below for HOWTO) 🏃‍♀️
• If it fails, open up an issue to document your digging (e.g. TestMigrate test flaky #257) 🤓
• Try to submit a fix 😄
• If it doesn't fail after $a_lot_of_runs, let's re-enable it (e.g. Re-enable TestStatusCalls test #256) 🌞

Good Luck 🧑‍🚒 👷‍♀️

Testing the tests

cd graph
LIBRARIAN_WRITEALL=0 go test -v ./ -run TestBadger | tee -a test.log # wait $a_while
cat test.log | grep FAIL # check for failures
cat test.log | grep PASS$ | wc -l # count passes

TODO

• TestStatusCalls Re-enable TestStatusCalls test #256
• TestPublishUnicode test: TestPublishUnicode works again #259
• TestPersistence test: TestPersistence works again #263
• TestMigrate Fix TestMigrate #264
• TestDontStartTwiceOnTheSameRepo test: TestDontStartTwiceOnTheSameRepo is back #265
• TestIndexFixtures TestIndexFixtures test is failing #279 Fix TestIndexFixtures #280 Probably ought to update TestIndexFixtures #281 Update TestIndexFixtures to ssb-fixtures 3.0.2 #283
• TestSpec TestSpec is not really flaky #275 Re-enable TestSpec #276
• TestInvalidFuzzed TestInvalidFuzzed is flaky #284 Fix TestInvalidFuzzed #285 Fix TestInvalidFuzzed (Simplified) #286
• TestLegacyInviteJSAccept Fix TestLegacyInviteJSAccept #295
• TestNames TestNames test flaky #250 Attempting to fix TestNames #255 Fix multiple index concurrency issues #308
• TestAboutNames TestAboutNames is flaky #282 Fix TestAboutNames #309
• TestSubsetQueryPlanExecution TestSubsetQueryPlanExecution is flaky #277 Fix TestSubsetQueryPlanExecution #310
• TestSignMessages TestSignMessages is flaky #289 Fix multiple index concurrency issues #308
• TestBadger TestBadger still flaking #266 Fix TestBadger #313
• TestStartup TestStartup is flaky #261 Fix TestStartup #315
• TestMetafeedIndexes TestMetafeedIndexes is flaky #299 Fix TestMetafeedIndexes #316
• TestPeople TestPeople is really, really broken #274 Fix TestPeople #322
• TestLegacyInviteJSCreate TestLegacyInviteJSCreate failed #249
• TestReplicate TestReplicate is flaky #260
• TestNullFeed TestNullFeed is flaky #268 Attempting to fix TestNullFeed #269
• TestBlocking TestBlocking is flaky #273
• TestFSCK TestFSCK flaking #296
• TestNullFetched TestNullFetched is flaky #298
• TestPrivateGroupsManualDecrypt TestPrivateGroupsManualDecrypt is flaky #300
• TestPrivMsgsFromGo TestPrivMsgsFromGo failing #328

[KyleMaas]

That's a much better format.

[KyleMaas]

Of note: if the tests are logging ANSI escape sequences, see this for how to filter them out:

https://superuser.com/questions/380772/removing-ansi-color-codes-from-text-stream

[KyleMaas]

Example command to run a test over and over again while filtering ANSI escapes:

watch 'LIBRARIAN_WRITEALL=0 go test -v ./ -run TestNullFeed | sed -e '"'"'s/\x1b\[[0-9;]*m//g'"'"' | tee -a ~/testlog.txt'

[KyleMaas]

Quick note: The TestInvalidFuzzed item is pointing to the draft pull request I closed and not the merged one.

[decentral1se]

OK, that's every failure mapped out into an issue 😅