Question about generate-table-partitions for Oracle to PG validation

[helensilva14]

Currently, I'm validating data between Oracle and PostgresSql, I have a table of big data of about 500M.

How to generate a file yaml to validate 10k newest data? I checked the document but I didn't find the solution to handle my case.

Please support me.
Thank you so much.

Originally posted by @TienNguyenVanDev in #1336

[helensilva14]

Hi @TienNguyenVanDev! Here's the corresponding question issue for your case.

I think @nj1973 might be able to provide information for you.

[nj1973]

Hi @TienNguyenVanDev, if you have a column you can use to identify recent data then you can use that in a filter. For example:

data-validation validate row -sc=ora -tc=pg \
-tbls=acme.big_table --primary-keys=id \
--filters="create_date > DATE'2024-11-20'" \
--hash="*" \
-c /tmp/big_table_recent.yaml

But that would rely on you knowing how many records are newer than the date value in the --filters option.

We don't have specific functionality to pick 10k records. Perhaps if your table has a numeric primary key you can take the max and subtract 10,000 and use a filter, for example:

--filters="id > 987654321" \

[helensilva14]

Hi @TienNguyenVanDev! Were you able to check the comment added by Neil? Thanks!