Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_logCompactionRegressionTest1 randomly sees a silent EOFError #86

Open
JustAnotherArchivist opened this issue Jul 31, 2018 · 1 comment

Comments

@JustAnotherArchivist
Copy link
Contributor

JustAnotherArchivist commented Jul 31, 2018

test_logCompactionRegressionTest1 sometimes produces an EOFError in the pickle module, but this error does not have any influence on the test result, i.e. the test still passes even when the error occurs. The error is therefore only visible when running pytest with the -s option to disable output capturing. Here's the test output for pytest -v -s -k 'test_logCompactionRegressionTest1' test_syncobj.py when the test fails (paths redacted):

 test_syncobj.py::test_logCompactionRegressionTest1 [2018-07-31 12:48:20,997 syncobj.py:1325 ERROR]  failed to load full dump
 Traceback (most recent call last):
   File ".../pysyncobj/pickle.py", line 54, in load
     return pickle.load(file)
 EOFError: Ran out of input
 
 During handling of the above exception, another exception occurred:
 
 Traceback (most recent call last):
   File ".../pysyncobj/syncobj.py", line 1295, in __loadDumpFile
     data = self.__serializer.deserialize()
   File ".../pysyncobj/serializer.py", line 106, in deserialize
     return pickle.load(g)
   File ".../pysyncobj/pickle.py", line 58, in load
     return pickle._load(file)
   File ".../lib/python3.6/pickle.py", line 1551, in _load
     encoding=encoding, errors=errors).load()
   File ".../lib/python3.6/pickle.py", line 1048, in load
     raise EOFError
 EOFError
 PASSED

Looking at the test code, I noticed that the test compares the raftLog object before and after deserialisation. However, raftLog is not the actual log data but just a MemoryJournal object. Since that class doesn't implement an __eq__ method, the comparison is done based on the object identity. And as raftLog refers to the same object before and after deserialisation, this will always be true, even if the contents of the journal were modified inbetween. In this case, that doesn't happen: o1._SyncObj__raftLog[:] is identical before and after deserialisation ([(b'\x01', 1, 0), (b'\x01', 2, 2)]). So there must be something else going on. But the test should still be fixed to compare the data before and after rather than the object against itself (by assigning o1._SyncObj__raftLog[:] instead of o1._syncObj__raftLog to logAfterCompaction and logAfterDeserialize).

This may be related to #85 as it's seemingly also caused by the serialiser and only happens occasionally.

@JustAnotherArchivist
Copy link
Contributor Author

By the way, the same issue of comparing raftLog objects instead of the log contents is also present in test_randomTest1, but it's only used for a debugging print there, so that's not as important. (It will always print False though because each SyncObj instance has its own journal object.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant