README.md

This repository is DEPRECATED!

This repository is deprecated as of 08/05/2020.

The code and infrastructure has moved to https://github.com/usdot-jpo-sdc-projects/sdc-dot-waze-pipeline

Please refer to the waze pipeline repo for all future development.

sdc-dot-waze-poll-for-batches-to-process

This lambda function is responsible for polling the batches to start the persistence of data in Redshift.

Table of Contents

I. Release Notes

II. Overview

III. Design Diagram

IV. Getting Started

V. Unit Tests

VI. Support

---

I. Release Notes

TO BE UPDATED

II. Overview

There are two primary functions that this lambda function serves:

• poll_for_batches - gets the messages from the data persistence queue to start the persistence in Redshift
• publish_message_to_sns - publishes the message to the SNS topic indicating that the data persistence has started for a particular batch Id.

III. Design Diagram

IV. Getting Started

The following instructions describe the procedure to build and deploy the lambda.

Prerequisites

• N/A

---

ThirdParty library

*N/A

Licensed softwares

*N/A

Programming tool versions

*Python 3.6

---

Build and Deploy the Lambda

Environment Variables

Below are the environment variable needed :-

BATCH_NOTIFICATION_SNS - {arn_of_the_sns_topic_to_send_the_notification}

persistence_sqs - {name_of_the_sqs_queue_to_poll_messages_from}

Build Process

Step 1: Setup virtual environment on your system by foloowing below link https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example-deployment-pkg.html#with-s3-example-deployment-pkg-python

Step 2: Create a script with below contents e.g(sdc-dot-waze-poll-for-batches-to-process.sh)

cd sdc-dot-waze-poll-for-batches-to-process
zipFileName="sdc-dot-waze-poll-for-batches-to-process.zip"

zip -r9 $zipFileName common/*
zip -r9 $zipFileName lambdas/*
zip -r9 $zipFileName README.md
zip -r9 $zipFileName poll_for_batches_to_process_handler_main.py
zip -r9 $zipFileName root.py

Step 3: Change the permission of the script file

chmod u+x sdc-dot-waze-poll-for-batches-to-process.sh

Step 4 Run the script file ./sdc-dot-waze-poll-for-batches-to-process.sh

Step 5: Upload the sdc-dot-waze-poll-for-batches-to-process.zip generated from Step 4 to a lambda function via aws console.

Back to top

---

V. Unit Tests

TO BE UPDATED

---

VI. Support

For any queries you can reach to [email protected]

Back to top