Service fails when database connection expires

[zfi]

The cloud session service fails if the database connection expires. This can happen if there are no queries to the backing store for 8 hours in the current implementation. Once this event occurs, the service does not make any attempt to reconnect to the backing store. All subsequent calls to the service that require any interaction with the backing store will fail until the service is restarted.

This is a sample stack trace of the issue:

[2019-03-01 17:55:29,143] ERROR in app: Exception on /authenticate/local [POST]
Traceback (most recent call last):
  File "/home/developer/projects/cs-server/venv/local/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/developer/projects/cs-server/venv/local/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/developer/projects/cs-server/venv/local/lib/python2.7/site-packages/flask_restful/__init__.py", line 480, in wrapper
    resp = resource(*args, **kwargs)
  File "/home/developer/projects/cs-server/venv/local/lib/python2.7/site-packages/flask/views.py", line 84, in view
    return self.dispatch_request(*args, **kwargs)
  File "/home/developer/projects/cs-server/venv/local/lib/python2.7/site-packages/flask_restful/__init__.py", line 595, in dispatch_request
    resp = meth(*args, **kwargs)
  File "./app/Authenticate/controllers.py", line 40, in post
    user = user_services.get_user_by_email(email)
  File "./app/User/services.py", line 31, in get_user_by_email
    return User.query.filter_by(email=email).first()
  File "/home/developer/projects/cs-server/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first
    ret = list(self[0:1])
  File "/home/developer/projects/cs-server/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__
    return list(res)
  File "/home/developer/projects/cs-server/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__
    return self._execute_and_instances(context)
  File "/home/developer/projects/cs-server/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2901, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/home/developer/projects/cs-server/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/home/developer/projects/cs-server/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/developer/projects/cs-server/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/developer/projects/cs-server/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/home/developer/projects/cs-server/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/home/developer/projects/cs-server/venv/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/developer/projects/cs-server/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/home/developer/projects/cs-server/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
  File "/home/developer/projects/cs-server/venv/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line 250, in execute
    self.errorhandler(self, exc, value)
  File "/home/developer/projects/cs-server/venv/local/lib/python2.7/site-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler
    raise errorvalue
OperationalError: (_mysql_exceptions.OperationalError) (2006, 'MySQL server has gone away') [SQL: u'SELECT user.id AS user_id, user.email AS user_email, user.password AS user_password, user.salt AS user_salt, user.auth_source AS user_auth_source, user.locale AS user_locale, user.blocked AS user_blocked, user.confirmed AS user_confirmed, user.screen_name AS user_screen_name, user.birth_month AS user_birth_month, user.birth_year AS user_birth_year, user.parent_email AS user_parent_email, user.parent_email_source AS user_parent_email_source \nFROM user \nWHERE user.email = %s \n LIMIT %s'] [parameters: ('[email protected]', 1)] (Background on this error at: http://sqlalche.me/e/e3q8)
[pid: 6995|app: 0|req: 10/23] 127.0.0.1 () {40 vars in 568 bytes} [Fri Mar  1 17:55:28 2019] POST /authenticate/local => generated 37 bytes in 324 msecs (HTTP/1.1 500) 2 headers in 90 bytes (1 switches on core 1)