Make tablet servers respond more aggressively to failed minor compaction.

[keith-turner]

Is your feature request related to a problem? Please describe.

Sometimes when a minor compaction fails with an exception it will retry. In cases where it does not retry because its not known if its safe to retry then it may be best to take some action to prevent future interaction with the tablet. Currently the only action taken is an exception is logged and the tablet is left in a half functional state where reads/writes could still attempt to run against it. No minor compaction will ever run again on the tablet.

Describe the solution you'd like

There are a few possible actions that could be taken

• Halt the tablet server
• Make all operations against the tablet fail, like all read and write RPCs would fail after the minor compaction failed.
• Try to increase the ability to retry, however will not be able to always retry. If a NPE, ArrayOutofBoundsException, etc happen during a minor compaction then the state of the tablet in the tablet server is unknown at that point and retrying may not lead to correct result.

[dlmarion]

If #5145 is merged, then this could call ServiceLock.verifyLockAtSource() and halt the VM if it returns false. If it returns true, maybe retry.

[keith-turner]

If #5145 is merged, then this could call ServiceLock.verifyLockAtSource() and halt the VM if it returns false. If it returns true, maybe retry.

That sounds good. Also opened #5146 about checking after walog failures.