embulk-output-redshift

[sureshinforma]

Hi Team - Need your input for one scenario I want to incorporate:
I want to process data incrementally to redshift from TD. I am reading delta data from TD into a table. I am using insert mode and want to check and delete the already present data records in target table based on a key column. This key column is not unique and can have multiple records. If I use merge mode I am ending up with exact duplicates where we have multiple records for that key column.
So I want to delete the already present records for that key column and then insert the upcoming records. How can I achieve this if you can share some guidance.

[hiroyuki-sato]

Hello, @sureshinforma If you are Treasure Data customer, I recommend you to ask your question to TD support.
It seems you need a create intermediate table and update the table using Workflow Tool. ( dbt, digdag, Airflow or something)
Or use before_load / after_load in embulk-output-redshift.

[dmikurube]

We see no response for 6 months. Let us close this discussion.