Microsoft-ServiceFabric.Trinity.StatefulService does not pass Chaos Testing

[TaviTruman]

As part of our integration testing suite, we have found that Graph Engine Stateful server implementation does not pass the recommended Chaos Testing!

Once we start to randomly inject breakage into the app then the Stateful sever seems to recover. If we power-reset the cluster or the machine once the machine and cluster come back up the GE/SF services never get to a good state.

Please let me know what data, information or environment requirement you think you need to assist; again, if the GE Team is not able to respond to this issue, please kindly let me know and we will commit our resources to the issue.

{
"Timestamp": "2020-04-10T09:29:04.5950419-07:00",
"ProviderName": "Microsoft-ServiceFabric.Trinity.StatefulService",
"Id": 3,
"Message": "[P1-Champion of the Universe]\tClientSocket: Incorrect server handshake ACK message. Thread Id = 39164, Socket = 4544",
"ProcessId": 30080,
"Level": "Error",
"Keywords": "0x0000F00000000001",
"EventName": "GraphEngineLogErr",
"ActivityID": null,
"RelatedActivityID": null,
"Payload": {
"message": "[P1-Champion of the Universe]\tClientSocket: Incorrect server handshake ACK message. Thread Id = 39164, Socket = 4544"
}
}