Model inference fails with torchserve

[cotrane]

Hi,
I have fine-tuned a tf_efficientdet_d1 model on a custom dataset and would like to use torchserve for inference.
I can run validate.py using the same checkpoint and get a correct result. When trying to use torchserve I get a RuntimeError though:

2021-11-24 10:11:15,854 [INFO ] W-9007-effdet-d1_1.0-stdout MODEL_LOG - Traceback of TorchScript, original code (most recent call last):
2021-11-24 10:11:15,854 [INFO ] W-9007-effdet-d1_1.0-stdout MODEL_LOG -   File "/efficientdet-pytorch/effdet/efficientdet.py", line 34, in forward
2021-11-24 10:11:15,854 [INFO ] W-9007-effdet-d1_1.0-stdout MODEL_LOG -     def forward(self, x: List[torch.Tensor]) -> List[torch.Tensor]:
2021-11-24 10:11:15,854 [INFO ] W-9007-effdet-d1_1.0-stdout MODEL_LOG -         for module in self:
2021-11-24 10:11:15,854 [INFO ] W-9007-effdet-d1_1.0-stdout MODEL_LOG -             x = module(x)
2021-11-24 10:11:15,854 [INFO ] W-9007-effdet-d1_1.0-stdout MODEL_LOG -                 ~~~~~~ <--- HERE
2021-11-24 10:11:15,854 [INFO ] W-9007-effdet-d1_1.0-stdout MODEL_LOG -         return x
2021-11-24 10:11:15,854 [INFO ] W-9007-effdet-d1_1.0-stdout MODEL_LOG -   File "/efficientdet-pytorch/effdet/efficientdet.py", line 295, in forward
2021-11-24 10:11:15,854 [INFO ] W-9007-effdet-d1_1.0-stdout MODEL_LOG -     def forward(self, x: List[torch.Tensor]):
2021-11-24 10:11:15,854 [INFO ] W-9007-effdet-d1_1.0-stdout MODEL_LOG -         for fn in self.fnode:
2021-11-24 10:11:15,854 [INFO ] W-9007-effdet-d1_1.0-stdout MODEL_LOG -             x.append(fn(x))
2021-11-24 10:11:15,854 [INFO ] W-9007-effdet-d1_1.0-stdout MODEL_LOG -                      ~~ <--- HERE
2021-11-24 10:11:15,854 [INFO ] W-9007-effdet-d1_1.0-stdout MODEL_LOG -         return x[-self.num_levels::]
2021-11-24 10:11:15,855 [INFO ] W-9007-effdet-d1_1.0-stdout MODEL_LOG -   File "/efficientdet-pytorch/effdet/efficientdet.py", line 234, in forward
2021-11-24 10:11:15,855 [INFO ] W-9007-effdet-d1_1.0-stdout MODEL_LOG -             edge_weights = nn.functional.relu(self.edge_weights.to(dtype=dtype))
2021-11-24 10:11:15,855 [INFO ] W-9007-effdet-d1_1.0-stdout MODEL_LOG -             weights_sum = torch.sum(edge_weights)
2021-11-24 10:11:15,855 [INFO ] W-9007-effdet-d1_1.0-stdout MODEL_LOG -             out = torch.stack(
2021-11-24 10:11:15,855 [INFO ] W-9007-effdet-d1_1.0-stdout MODEL_LOG -                   ~~~~~~~~~~~ <--- HERE
2021-11-24 10:11:15,855 [INFO ] W-9007-effdet-d1_1.0-stdout MODEL_LOG -                 [(nodes[i] * edge_weights[i]) / (weights_sum + 0.0001) for i in range(len(nodes))], dim=-1)
2021-11-24 10:11:15,855 [INFO ] W-9007-effdet-d1_1.0-stdout MODEL_LOG -         elif self.weight_method == 'sum':
2021-11-24 10:11:15,855 [INFO ] W-9007-effdet-d1_1.0-stdout MODEL_LOG - RuntimeError: stack expects each tensor to be equal size, but got [1, 88, 35, 27] at entry 0 and [1, 88, 36, 28] at entry 1

I have created a model archive using:
torch-model-archiver --model-name effdet-d1 --version 1.0 --serialized-file ./model_best.pth.tar --model-file ./effdet/efficientdet.py --extra-files ./index_to_name.json --requirements-file ./requirements.txt --handler object_detector

copied the file effdet-d1.mar into folder ./model_store and ran:
torchserve --start --ncs --ts-config inference/config.properties --model-store ./model_store --models effdet-d1.mar

It seems to start and load the model fine but when I submit a request:
curl http://127.0.0.1:8080/predictions/effdet-d1 -T ./test-image.jpg

I get above error.
The same image works fine when adjusting validate.py to do inference on it. I also converted the model file to torchscript using torch.jit.script and ran an inference without torchserve successfully. I also tried using that separately built torchscript file to load into torchserve but get the same error.

I'm running torch==1.10.0, torchvision==0.11.1, torchserve==0.5.0, torch-model-archiver==0.5.0 on a Mac.

Any hint about how to fix this is highly appreciated! Thanks for the great repo!

[rwightman]

@cotrane hmm, I haven't gone through that process with these models, that error means the feature maps aren't lined up across resolution levels, which is usually due to padding (SAME) or interpolation size targets not being correct. This can happen if you don't use the appropriate sized inputs (matching model config) or if the padding is wrong (which can happen if you trace the model with the wrong size, or again trace with one size and use the model with a different size input).

The model is definitely being exported via scripting and not tracing yes?

[cotrane]

Thank you for your response. You were right - it was simply that the default handler for object detection I was using didn't resize the image. So obviously it didn't work. I had to write my own custom handler to make it work. Thanks again!