Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using cachetools v3.x breaks lib/join (dpp 1.7.1) #155

Open
brew opened this issue Nov 15, 2018 · 0 comments
Open

Using cachetools v3.x breaks lib/join (dpp 1.7.1) #155

brew opened this issue Nov 15, 2018 · 0 comments

Comments

@brew
Copy link
Contributor

brew commented Nov 15, 2018

This is with datapackage-pipelines v1.7.1.

Unpinned cachetools is declared as a dependency in setup. And used by utilities/kvstore.py. If cachetools==3.0.0 (latest version) is used, my pipeline fails with this error:

[./denormalized_flow:T_0] >>> INFO    :join: Traceback (most recent call last):
[./denormalized_flow:T_0] >>> INFO    :join:   File "/Users/brew/virtualenvs/os-api/lib/python3.6/site-packages/datapackage_pipelines/specs/../lib/join.py", line 297, in <module>
[./denormalized_flow:T_0] >>> INFO    :join:     new_resource_iterator(resource_iterator))
[./denormalized_flow:T_0] >>> INFO    :join:   File "/Users/brew/virtualenvs/os-api/lib/python3.6/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 64, in spew
[./denormalized_flow:T_0] >>> INFO    :join:     for res in resources_iterator:
[./denormalized_flow:T_0] >>> INFO    :join:   File "/Users/brew/virtualenvs/os-api/lib/python3.6/site-packages/datapackage_pipelines/specs/../lib/join.py", line 213, in new_resource_iterator
[./denormalized_flow:T_0] >>> INFO    :join:     collections.deque(indexer(resource), maxlen=0)
[./denormalized_flow:T_0] >>> INFO    :join:   File "/Users/brew/virtualenvs/os-api/lib/python3.6/site-packages/datapackage_pipelines/specs/../lib/join.py", line 168, in indexer
[./denormalized_flow:T_0] >>> INFO    :join:     db[key] = current
[./denormalized_flow:T_0] >>> INFO    :join:   File "/Users/brew/virtualenvs/os-api/lib/python3.6/site-packages/cachetools/lru.py", line 21, in __setitem__
[./denormalized_flow:T_0] >>> INFO    :join:     cache_setitem(self, key, value)
[./denormalized_flow:T_0] >>> INFO    :join:   File "/Users/brew/virtualenvs/os-api/lib/python3.6/site-packages/cachetools/cache.py", line 47, in __setitem__
[./denormalized_flow:T_0] >>> INFO    :join:     size = self.getsizeof(value)
[./denormalized_flow:T_0] >>> INFO    :join:   File "/Users/brew/virtualenvs/os-api/lib/python3.6/site-packages/datapackage_pipelines/utilities/kvstore.py", line 100, in _dbget
[./denormalized_flow:T_0] >>> INFO    :join:     value = self.db.get(key)
[./denormalized_flow:T_0] >>> INFO    :join:   File "/Users/brew/virtualenvs/os-api/lib/python3.6/site-packages/datapackage_pipelines/utilities/kvstore.py", line 65, in get
[./denormalized_flow:T_0] >>> INFO    :join:     ret = self.db.get(key.encode('utf8'))
[./denormalized_flow:T_0] >>> INFO    :join: AttributeError: 'dict' object has no attribute 'encode'

If I explicitly install an older version (cachetools==2.1.0), the pipeline runs successfully.

@brew brew changed the title Using cachetools==3.0.0 seems to break lib/join Using cachetools v3.x breaks lib/join (dpp 1.7.1) Nov 15, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant