Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chunk-manager not check disk chunk_cache correctness #158

Closed
lizhihongTest opened this issue Apr 3, 2023 · 2 comments
Closed

chunk-manager not check disk chunk_cache correctness #158

lizhihongTest opened this issue Apr 3, 2023 · 2 comments
Assignees
Labels
bug Something isn't working CYFS Stack This is CYFS Stack

Comments

@lizhihongTest
Copy link
Collaborator

Describe the bug
If I modify a chunk file in /cyfs/data/chunk-cache , thenI get the chunk by stack.ndn_service().get_data(), chunk-manager not check the correctness of the file hash in the disk

To Reproduce

  1. Modify a chunk file in /cyfs/data/chunk-cache
  2. Get the chunk through stack.ndn_service().get_data()

Expected behavior

  1. The chunk-manager should check the correctness of the chunk, because the chunk content does not match the chunk_id, get_data should return an error
  2. The chunk_id is calculated through the hash of the full text of the chunk content, and inconsistencies should be prohibited

System information
Linux/Windows

@lizhihongTest lizhihongTest added the bug Something isn't working label Apr 3, 2023
@lurenpluto lurenpluto added the CYFS Stack This is CYFS Stack label Apr 3, 2023
@lurenpluto
Copy link
Member

The chunks data currently managed by cyfs-stack mainly includes the following two types:

  • Chunks managed by the chunk-manager

These chunks will be added to the {cyfs}/data/chunk-cache directory and are responsible for read and write operations by the chunk-manager.

  • External files

These are usually external files and directories added directly through the trans.add_file function, which are managed by the stack using ndc and tracker to manage the corresponding files and internal chunks data.

Therefore, from this perspective, the "external files" may not be very reliable, as they are files on the user's device, such as OOD or PC, which may be moved, deleted, modifyed, or even replaced. So when the stack needs to read the corresponding chunk, it may no longer exist, or the content of the chunk may have changed. To prevent reading the wrong chunk, the stack adds an extra level of validation when reading chunks from these "external files," which checks if the hash of the chunk's data matches the chunk-id.

However, for the files inside chunk-cache, it is clear that they are managed by the chunk-manager of the stack, and they should not be modified externally on the OOD. Users should not modify or replace these files themselves. Therefore, adding a hash validation when reading from here might be a performance waste.

Another challenge is that chunk-manager supports memory-mapped file mode for reading and managing chunk-cache, so adding hash validation functionality may also be challenging.

@lurenpluto
Copy link
Member

If chunk hash mentioned in #204 detection is introduced on top of this, it would be a good mechanism for chunk-manager managed chunks and should solve the problem

@lurenpluto lurenpluto moved this from 💬To Discuss to 🅿️Blocked/Parked in CYFS-Stack & Services Apr 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working CYFS Stack This is CYFS Stack
Projects
Status: 🅿️Blocked/Parked
Development

No branches or pull requests

2 participants