how to Hazelcast jet JDBC continues source get data when new record insert or existing record updated in a database table?

[nuriozalp]

please urgent help,about how to Hazelcast jet JDBC continues source get data when new record insert or existing record updated in a database table?

[viliam-durina]

These are batch sources, they read records in the table and quit. Read operations in SQL databases typically read one consistent snapshot of the table.

[nuriozalp]

you are all the time say realtime elt and you can create connector but this is the one of the most important data source and jdbc has already listeners and you can easly implement listeners and you can activate windows sliding. i really want to use jet in big project but because of this reason it will create some diffecult for most of elt operation. i will get data from service bus now. if you have plan to add listeners also to database tables, please let me know about your plan?

…

On Fri, Feb 1, 2019, 15:26 Viliam Durina ***@***.*** wrote: These are batch sources, they read records in the table and quit. Read operations in SQL databases typically read one consistent snapshot of the table. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#170 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGwfMMjH2K_X9anL7OzpOktERCgFgM84ks5vJE6OgaJpZM4aePy_> .

[viliam-durina]

JDBC doesn't provide a standardized way for listeners. There are many possible workarounds, such as using triggers, polling, db-specific ways. Therefore it's not easy to provide one-size-fits-all out-of-the-box solution. You can use SourceBuilder or, if you need feature that's not covered by it, implement a Processor directly.

Also, Jet absolutely is very close to real-time inside. But it also supports batch processing, which is the case for the bundled JDBC connector.

[nuriozalp]

yes but there are a few most common database and each of them can be seperated by db name. and in framework can decide whiich one should it to use. i wanna know how can i write any task in pipeline and acording to my windows sliding i will ger my source sql query (new record and only updated record between specific time.) i want to use same structure of jet pipeline and source and join and sink structure but somehow i need to trigger this job with new input arguments? can show help for it ? maybe some code sample?

[cangencer]

@nuriozalp you could use a custom source / SourceBuilder to achieve this. A code sample which polls a HTTP endpoint is available here: https://github.com/hazelcast/hazelcast-jet-code-samples/tree/0.7-maintenance/source-builder

Instead of the HTTP endpoint you would be polling your SQL database.