Calibration not happening

[Surajj4837-th]

I have a fish eye camera and I'm trying to get the intrinsic parameters of the same. I used the cameraCalibrationWithUndistortion.py file. The checkerboard I used is a 9x9 grid. But I'm getting the following kind of output in the end:

One thing to note is that the connecting colourful lines plotted in the intermediate output change orientation in every image.

As shown in the above image the red line is at the top of the checkerboard, in some cases, it will be on the left or right side as well. This may or may not be the root cause of the issue.
What can be the root cause?