Operation part 1

[samdash]

Hi,

Do you happend to have codebase for part 1 where you collect data and push it to HIVE.
This codebase assumes you have data already in HIVE.
Thanks
Samar

[tspannhw]

pushing to hive is trivial in NiFi
https://community.hortonworks.com/articles/108718/ingesting-rdbms-data-as-new-tables-arrive-automagi.html

https://community.hortonworks.com/articles/45706/using-the-new-hiveql-processors-in-apache-nifi-070.html

PutHDFS / autogen table
https://community.hortonworks.com/articles/64122/incrementally-streaming-rdbms-data-to-your-hadoop.html

https://community.hortonworks.com/articles/189514/converting-csv-files-to-apache-hive-tables-with-ap.html

https://community.hortonworks.com/articles/92496/qadcdc-our-how-to-ingest-some-database-tables-to-h.html

https://community.hortonworks.com/articles/155527/ingesting-golden-gate-records-from-apache-kafka-an.html

[samdash]

Thanks, i am looking into pushing the mertics data to our intenal hosting system.
what processors were used to capture below data shown in the link for status/metrics/failure? or do we have to write any custom reporting task using Prometheus?

https://dzone.com/articles/building-a-custom-apache-nifi-operations-dashboard

[tspannhw]

https://community.hortonworks.com/articles/224554/building-a-custom-apache-nifi-operations-dashboard.html

As seen in the top you can use the default reporting tasks push them to nifi and have it send to your system
https://community.hortonworks.com/articles/224554/building-a-custom-apache-nifi-operations-dashboard.html

There also a prometheus reporter

https://github.com/mkjoerg/nifi-prometheus-reporter

You can also read these values from rest and have your system call those rest apis
Or could have nifi read its own rest apis and push them to your system

[samdash]

Thank you, i will explore these 6 reporting tasks.

[tspannhw]

https://nifi.apache.org/docs/nifi-docs/rest-api/index.html

https://community.hortonworks.com/questions/69004/nifi-monitoring-processor-and-nifi-service.html

[tspannhw]

https://www.datainmotion.dev/2019/04/monitoring-number-of-of-flow-files.html

https://medium.com/@julienlaurenceau/monitoring-nifi-with-graylog-22ef1493310a

https://stackoverflow.com/questions/53806197/any-way-to-monitor-nifi-processors-any-utility-dashboard

https://www.slideshare.net/Hadoop_Summit/best-practices-and-lessons-learnt-from-running-apache-nifi-at-renault