This repository is deprecated as of 08/05/2020.
The code and infrastructure has moved to https://github.com/usdot-jpo-sdc-projects/sdc-dot-waze-pipeline
Please refer to the waze pipeline repo for all future development.
This lambda function is responsible for polling the batches to start the persistence of data in Redshift.
TO BE UPDATED
There are two primary functions that this lambda function serves:
- poll_for_batches - gets the messages from the data persistence queue to start the persistence in Redshift
- publish_message_to_sns - publishes the message to the SNS topic indicating that the data persistence has started for a particular batch Id.
The following instructions describe the procedure to build and deploy the lambda.
- N/A
*N/A
*N/A
*Python 3.6
Below are the environment variable needed :-
BATCH_NOTIFICATION_SNS - {arn_of_the_sns_topic_to_send_the_notification}
persistence_sqs - {name_of_the_sqs_queue_to_poll_messages_from}
Step 1: Setup virtual environment on your system by foloowing below link https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example-deployment-pkg.html#with-s3-example-deployment-pkg-python
Step 2: Create a script with below contents e.g(sdc-dot-waze-poll-for-batches-to-process.sh)
cd sdc-dot-waze-poll-for-batches-to-process
zipFileName="sdc-dot-waze-poll-for-batches-to-process.zip"
zip -r9 $zipFileName common/*
zip -r9 $zipFileName lambdas/*
zip -r9 $zipFileName README.md
zip -r9 $zipFileName poll_for_batches_to_process_handler_main.py
zip -r9 $zipFileName root.py
Step 3: Change the permission of the script file
chmod u+x sdc-dot-waze-poll-for-batches-to-process.sh
Step 4 Run the script file ./sdc-dot-waze-poll-for-batches-to-process.sh
Step 5: Upload the sdc-dot-waze-poll-for-batches-to-process.zip generated from Step 4 to a lambda function via aws console.
TO BE UPDATED