Failed to call service with turtlesim/TeleportRelative_Request message requires a float for field linear, but got a <class 'str'>

[MarissaCampa]

Hi, I tried to run this demo with the following prompt:

python3 main.py \
    --prompt "\
        Move the turtle left by 2, then rotate 180 degrees, and move back to (5, 5).\
        Finally, spawn a turtle named turtle2 at (10, 10) and remove turtle1.\
    " \
    --key your_key

And I got this error message:

Generating API calls. This may take some time...
Done.
Getting required service. This might take some time...
Done.
Calling service /turtle1/teleport_relative of type turtlesim/TeleportRelative with args {'linear': '-2', 'angular': '0'}
Press Enter to continue...
Failed to call service with turtlesim/TeleportRelative_Request message requires a float for field linear, but got a <class 'str'>.
Getting required service. This might take some time...
Done.
Calling service /turtle1/teleport_relative of type turtlesim/TeleportRelative with args {'linear': '0', 'angular': '3.14'}
Press Enter to continue...
Failed to call service with turtlesim/TeleportRelative_Request message requires a float for field linear, but got a <class 'str'>.
Getting required service. This might take some time...
Done.
Calling service /turtle1/teleport_absolute of type turtlesim/TeleportAbsolute with args {'x': '5', 'y': '5', 'theta': '0'}
Press Enter to continue...

Is there another way of parsing the arguments?

Thanks

[mhubii]

Hi, thank you for the feedback!

The issue is that ChatGPT doesn't always follow the schema in turtlesim_api.json.

I have tried to "condition" ChatGPT on the schema here
https://github.com/mhubii/chatgpt_turtlesim/blob/25c06f459021e77e90cecec212cd871abda8617c/prompt/openai_interface.py#LL11C14-L11C14 (similar to autogpt), but this does not always work reliably, since ChatGPT output is not deterministic. One might tune the temperature https://github.com/mhubii/chatgpt_turtlesim/blob/25c06f459021e77e90cecec212cd871abda8617c/prompt/openai_interface.py#L48.

Please keep in mind this is all very experimental, but maybe you can find a way to have ChatGPT create reliable responses in the expected format or to parse the generated response properly. There is now a simple chat history in this repo so with proper error handling one might also try to ask ChatGPT to correct its previous respone.

Please find below some useful links:

• Microsoft blog post: https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/chatgpt?pivots=programming-language-chat-completions
• AutoGPT prompts: https://github.com/Significant-Gravitas/Auto-GPT/blob/master/autogpt/prompts/default_prompts.py

[mhubii]

it would also be interesting to see how one could extend this demo to the entire ROS ecosystem in an autogpt like approach. Maybe someone could create an organization on github and work on this

[MarissaCampa]

I called it again multiple times, and the majority of the time works perfectly, like you said. And my turtle moved around and spawned another one.

python3 main.py --key my_key

Generating API calls. This may take some time...
Done.
Getting required service. This might take some time...
Done.
Calling service /turtle1/teleport_relative of type turtlesim/TeleportRelative with args {'linear': -2.0, 'angular': 0.0}
Press Enter to continue...
Getting required service. This might take some time...
Done.
Calling service /turtle1/teleport_relative of type turtlesim/TeleportRelative with args {'linear': 0.0, 'angular': 3.14}
Press Enter to continue...
Getting required service. This might take some time...
Done.
Calling service /turtle1/teleport_absolute of type turtlesim/TeleportAbsolute with args {'x': 5.0, 'y': 5.0, 'theta': 0.0}
Press Enter to continue...
Getting required service. This might take some time...
Done.
Calling service /spawn of type turtlesim/Spawn with args {'x': 10.0, 'y': 10.0, 'theta': 0.0, 'name': 'turtle2'}
Press Enter to continue...
Getting required service. This might take some time...
Done.
Calling service /kill of type turtlesim/Kill with args {'name': 'turtle1'}
Press Enter to continue...

At some point I got this error from the ros_bridge:

[rosbridge_websocket-1] [ERROR] [1685015724.990080832] [rosbridge_websocket]: [Client xxx] [id: call_service:/turtle1/teleport_relative:2] call_service FieldTypeMismatchException: turtlesim/TeleportRelative_Request message requires a float for field linear, but got a <class 'str'>

And yes, this must happen because ChatGPT doesn't follow the schema sometimes. But I agree that this is experimental and also ChatGPT responses are not deterministic.

After that I tweaked the prompts to not kill the first turtle, and then moved around the two turtles like this:

python3 main.py \
    --prompt "\
        Move the turtle left by 2, then rotate 180 degrees, and move back to (5, 5).\
        Finally, spawn a turtle named turtle2 at (10, 10).\
    " \
    --key my_key

python3 main.py \
    --prompt "\
        Move the first turtle down by 2, then rotate 90 degrees, and move right by 5).\
        Finally, move the second turtle back by 10.\
    " \
    --key my_key
    
python3 main.py \
    --prompt "\
        Move the first turtle to the center.\
        Finally, move the second turtle to the center plus 1 to the left\
    " \
    --key my_key

Thank you for the links, they are very useful.
I will say, this is pretty cool, and I'll keep looking around. Great work!

[mhubii]

great you found a solution, lets keep this issue open to improve awareness of these shortcomings