[Bug][Jira] Jira Issues Data Collection is deleting old registers

[caioq]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

The problem occurs during subsequent executions of a blueprint after the initial or full-refresh data collection has been completed successfully. Initially, all data is collected perfectly, and there are no errors or unusual behavior logged during these runs. However, in some cases, during subsequent runs of the same blueprint, old issues seem to be deleted, leaving only issues that were created or updated after the last pipeline execution.

What do you expect to happen

Records should be incrementally added to the issues table.

How to reproduce

This started happening 2 weeks ago in the production environment. I don't know if the JIRA incident on November 5-6 may have impacted this collection process.

Anything else

Upon further investigation, I found that the _raw_jira_api_issues table still contains 73,145 records, while the _tool_jira_issues and issues tables only hold 1,115 records. This suggests there might be an issue with the extractor or data transformation during these runs.

Version

v1.0.1-beta8@e195107 or v1.0.2-beta4

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dosubot]

I found a similar closed issue that might be related to your problem: [Bug][jira] Jira issue extractor produces records less than expected](#4709). This issue was closed, but it might provide some insights into the behavior you're experiencing ^[1].

Regarding your specific issue, it seems like the incremental data update process in Apache DevLake's Jira plugin might be affecting the records in the issues table. The process involves using a StatefulApiExtractor that, when in incremental mode, deletes existing records related to the issue in the JiraIssueLabel and JiraIssueRelationship tables before processing new data. This is intended to ensure that any changes to labels or relationships are accurately reflected in the database ^[2].

Given the discrepancy between the _raw_jira_api_issues and the other tables, it might be worth investigating whether the deletion process is affecting more records than intended. Additionally, consider checking if there are any specific configurations or settings that might be influencing this behavior.

To continue talking to Dosu, mention @dosu.

[klesh]

Any new project was created recently?
Was there any parallel pipeline running at the same time?
Is there any shared Jira board between projects?

Try executing the "Retransform Data" and see if the issues reappear.

[caioq]

Any new project was created recently? Was there any parallel pipeline running at the same time? Is there any shared Jira board between projects?

Try executing the "Retransform Data" and see if the issues reappear.

Hi @klesh !
When this problem arose, no project/configuration was created or changed.

Fortunately, when I retransform the data, the issues reappear 🙏🏼

Unfortunately, this past weekend this problem happened again. It's not something that happens often. I'm still investigating what could have happened differently.

[klesh]

I wonder if this issue is related to #7826