Skip to content
This repository has been archived by the owner on Dec 9, 2024. It is now read-only.

Fix Payload Duplication Bug in build_payloads Function by Copying Metadata for Each Chunk #89

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

suryadevarapranav
Copy link

This pull request resolves the Issue #88.

Description

Overview: This pull request addresses the issue of duplicated text in the payloads generated by the build_payloads function. Previously, each payload incorrectly contained the same text due to a shared reference to doc.metadata, which caused in-place updates across all payloads.

Changes Made:

Updated build_payloads to create a shallow copy of doc.metadata for each chunk. This ensures each payload dictionary is independent, containing the correct text for each specific chunk.
Added a comment for clarity.

Testing: Tested the function with multiple chunks, confirming that each payload now contains the expected text and metadata without interference.

Example:

News document from Alpaca.

image

Payloads uploaded to Qdrant.

image

Qdrant Query used to verify the issue fix.

POST collections/alpaca_news/points/scroll
{
  "filter": {
    "must": [
      {
        "key": "date",
        "match": {
          "text": "2024-01-01T13:15:32+00:00"
        }
      }
    ]
  }
}

Fix: Ensure unique payloads for each document chunk by copying metadata in 'build_payloads' function
@suryadevarapranav
Copy link
Author

Hello @iusztinpaul @Paulescu @Joywalker, Can you please take a look at my pull request, when you find some time?

Thanks,
Deva.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant