Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Component behavior violates at-least-once platform behavior #35

Open
zubairov opened this issue Dec 2, 2019 · 1 comment
Open

Component behavior violates at-least-once platform behavior #35

zubairov opened this issue Dec 2, 2019 · 1 comment
Assignees

Comments

@zubairov
Copy link
Contributor

zubairov commented Dec 2, 2019

When reading files from SFTP Server, failure in component execution do lead to retry

Steps to reproduce

  1. Upload 10+ files of 50MB to SFTP folder
  2. Build an integration flow with a single SFTP trigger reading from the folder
  3. Launch it

Expected result

  1. Flow suspended, however number of files in .elasticio_processed directory equal to the number of successfully generated messages (seen as green in log)

Actual result

  1. Flow suspended however .elasticio_processed folder has more files than successful messages in the log

Reasoning behind

SFTP component first moves the file to processed directory and after that reads the file, this behavior is beneficial from concurrency perspective when multiple pollers poll the same folder, however in case of container failures (e.g. OOM) file is not rep-processed / re-uploaded again.

@jhorbulyk
Copy link
Contributor

This concern should be solved by #31

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants