Error occurred when rocon_gateway is shutdown.

[dwlee]

This log is from waiterbot.
The following error is happened when gateway is shutdown.

[INFO] [WallTime: 1433148471.084004] Gateway : lost connection with hub, attempting to disconnect...
Traceback (most recent call last):
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/scripts/gateway.py", line 22, in <module>
    gateway.spin()
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_node.py", line 81, in spin
    self._gateway.spin()
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway.py", line 78, in spin
    self.watcher_thread.start()
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/watcher_thread.py", line 75, in start
    self._gateway.update_flipped_interface(connections, remote_gateway_hub_index)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway.py", line 184, in update_flipped_interface
    status = hub.get_flip_request_status(flip.remote_rule)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_hub.py", line 717, in get_flip_request_status
    status = self.get_multiple_flip_request_status([remote_rule])
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_hub.py", line 748, in get_multiple_flip_request_status
    rule_status, source, connection_list = utils.deserialize_request(flip)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/utils.py", line 218, in deserialize_request
    deserialized_list = deserialize(request_str)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/utils.py", line 182, in deserialize
    return pickle.loads(str_msg)
ValueError: insecure string pickle

Exception in thread Thread-39:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_hub.py", line 60, in run
    self._hub_connection_lost_hook()
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_hub.py", line 156, in _hub_connection_lost_hook
    self._hub_connection_lost_gateway_hook(self)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_node.py", line 189, in _disengage_hub
    self._gateway.disengage_hub(hub)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway.py", line 106, in disengage_hub
    self.hub_manager.disengage_hub(hub)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/hub_manager.py", line 240, in disengage_hub
    hub_to_be_disengaged.disconnect()  # necessary to kill failing socket receives
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_hub_client/src/rocon_hub_client/hub_client.py", line 134, in disconnect
    self._redis_server.connection_pool.disconnect()
  File "/home/yujin/ros/turtlebot/src/rocon_tools/rocon_python_redis/src/rocon_python_redis/connection.py", line 416, in disconnect
    for connection in all_conns:
RuntimeError: Set changed size during iteration

[jihoonl]

Insecure pickle error is related to #280

[jihoonl]

Second error seems to happen when gateway get disengaged with hub. We may add log in connection.py to get more information.

It seems like the size of all_conn list is changed while iteration.

[stonier]

Ok, following the crumbs...shutdown and hub connection checker thread callbacks.

Here the situation is probably not a dead connection, so it is more likely that the thread is getting terminated. In this case, the hub manager is calling gateway_hub.unregister_gateway() which pipes a few commands off to the hub. This can modify the connection pool. Question is when is the thread going down?

I added some code to gracefully terminate the thread in c72cb44 without engaging the destructive connection pool disconnect from a parallel thread.

This should fix it. I haven't been able to reproduce in any way, but gracefully terminating the thread is good behaviour anyway.

If it does happen again, reopen this issue thread.