Investigate ghost processes in tproxy and JDC

[jbesraa]

While working on #1207 it was noticed that some of the spawned tasks are not exiting upon dropping the main task. This results in hanging test as the future of those spawned tasks is never resolved and the test never ends.

@Shourya742 @plebhash tagging you as you have done some work here already

edit by @plebhash: it's important to get to the bottom of this, because it seems it's become a blocker to Integration Test issues such as #1207

[plebhash]

one of the main findings around this is the fact that this usually happens with JDC and tProxy, and both are using a task_collector pattern in very similar ways:

• tProxy: task_collector: Arc<Mutex<Vec<(AbortHandle, String)>>>
• JDC: task_collector: Arc<Mutex<Vec<AbortHandle>>>

we don't observe this behavior in any other role, and none of them have the task_collector pattern

[plebhash]

here's some findings from manual tests

they were observed without any Integration Tests or tokio-console

the text below might seem very dense, but we can actually infer some patterns from these observations

---

The first setup had the following:

• TP on testnet4
• Pool
• JDS
• JDC
• tProxy (connected to JDC)

I tried killing JDC and observing tProxy in two different ways:

• wait for tProxy to establish the connection with JDC before killing JDC: tProxy becomes unresponsive to ctrl+C and needs to be killed manually
• don't wait for tProxy to open an extended channel with JDC before killing JDC: tProxy is still responsive to ctrl+C

then I tried killing tProxy and observing JDC, and the same patterns was observed:

• wait for tProxy to establish the connection with JDC before killing tProxy: JDC becomes unresponsive to ctrl+C and needs to be killed manually
• don't wait for tProxy to establish the connection with JDC before killing tProxy: tProxy is still responsive to ctrl+C

then I tried killing Pool and observing JDC, but the same pattern was not observed:

• wait for JDC to establish connection with Pool before killing Pool: JDC is still responsive to ctrl+C
• don't wait for JDC to establish connection with Pool before killing Pool: JDC is still responsive to ctrl+C

then I tried killing JDS and observing JDC, but the same pattern was not observed:

• wait for JDC to establish connection with JDS before killing JDS: JDC is still responsive to ctrl+C
• don't wait for JDC to establish connection with JDS before killing JDS: JDC is still responsive to ctrl+C

---

The second setup was the following:

• TP on testnet4
• Pool
• tProxy (connected to Pool)

I tried killing Pool and observing tProxy in two different ways:

• wait for tProxy to establish the connection with Pool before killing Pool: tProxy becomes unresponsive to ctrl+C and needs to be killed manually
• don't wait for tProxy to establish the connection with Pool before killing Pool: tProxy is still responsive to ctrl+C

then I tried killing tProxy and observing Pool, but the same pattern was not observed:

• wait for tProxy to establish connection with Pool before killing tProxy: Pool is still responsive to ctrl+C
• don't wait for tProxy to establish connection with Pool before killing tProxy: Pool is still responsive to ctrl+C

[plebhash]

another relevant observation on JD-less setup:

I tried killing Pool and observing tProxy in two different ways:

• wait for tProxy to establish the connection with Pool before killing Pool: tProxy becomes unresponsive to ctrl+C > and needs to be killed manually
• don't wait for tProxy to establish the connection with Pool before killing Pool: tProxy is still responsive to ctrl+C

after killing Pool and making tProxy unresponsive to Ctrl+C, I avoided trying to force-kill tProxy (via pkill or kill -9), and simply re-launched the Pool

then tProxy re-connected to Pool, and went back to responding to Ctrl+C normally again

---

this should provide meaningful insight wrt task_collector