Broken or Flaky test: Several broken or flaky tests in main branch

[ctubbsii]

Test name(s)

• org.apache.accumulo.test.functional.SplitIT.bulkImportThatCantSplitHangsCompaction (seems to fail consistently)
• org.apache.accumulo.test.functional.SimpleBalancerFairnessIT.simpleBalancerFairness (flaky, sometimes passes)
• org.apache.accumulo.test.VerifySerialRecoveryIT.testSerializedRecovery (flaky, sometimes passes)

Describe the failure observed

For SplitIT:

org.opentest4j.AssertionFailedError: expected: <0> but was: <1>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:145)
	at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:531)
	at org.apache.accumulo.test.functional.SplitIT.bulkImportThatCantSplitHangsCompaction(SplitIT.java:515)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

For SimpleBalancerFairnessIT:

org.opentest4j.AssertionFailedError: Expected difference in tablets to be less than or equal to 2 but was 3. Counts [4, 1] ==> expected: <true> but was: <false>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
	at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
	at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:214)
	at org.apache.accumulo.test.functional.SimpleBalancerFairnessIT.simpleBalancerFairness(SimpleBalancerFairnessIT.java:107)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

For VerifySerialRecoveryIT:

org.opentest4j.AssertionFailedError: expected: <false> but was: <true>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertFalse.failNotFalse(AssertFalse.java:63)
	at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:36)
	at org.junit.jupiter.api.AssertFalse.assertFalse(AssertFalse.java:31)
	at org.junit.jupiter.api.Assertions.assertFalse(Assertions.java:231)
	at org.apache.accumulo.test.VerifySerialRecoveryIT.testSerializedRecovery(VerifySerialRecoveryIT.java:135)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Testing Environment:

• Version of this project: 4.0.0-SNAPSHOT
• First commit known to fail (or current commit): 99f8863 (unlikely related to that commit, but that was the HEAD when I tested)
• Environment:

Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae)
Maven home: /var/lib/jenkins/tools/hudson.tasks.Maven_MavenInstallation/Maven_3.9.6
Java version: 17.0.13, vendor: Red Hat, Inc., runtime: /usr/lib/jvm/java-17-openjdk-17.0.13.0.11-3.fc41.x86_64
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "6.11.8-300.fc41.x86_64", arch: "amd64", family: "unix"

What have you tried already?
I have a local Jenkins job that reruns failing tests. Each test was rerun if it failed the first time.

Additional context

In addition, the following tests seem to time out easily:

• org.apache.accumulo.test.functional.BulkSplitOptimizationIT.testBulkSplitOptimization (times out consistently for me after 2 minutes)
• org.apache.accumulo.test.compaction.ExternalCompactionProgressIT.testProgressViaMetrics (the timeout in this results in a test error, rather than a test failure: 155 » IllegalState Entries read and written metrics values did not match expected values. Timeout exceeded; the test finishes with an error after about 69 seconds)
• org.apache.accumulo.test.compaction.FlakyExternalCompaction2IT.* (several tests in this class time out after 5 minutes, but sometimes pass; these include testDeleteTableCancelsExternalCompaction, testDeleteTableCancelsUserExternalCompaction, testSplitCancelsExternalCompaction, and testUserCompactionCancellation; for me, they all failed the first attempt and passed the second; given the name of the test, my guess is that this is a known problem)

[DomGarguilo]

#5148 addressed the flaky SplitIT