Skip to content

Commit

Permalink
docs: add documentation to README
Browse files Browse the repository at this point in the history
  • Loading branch information
DCajiao committed Aug 12, 2024
1 parent c0d9164 commit 7a2881d
Showing 1 changed file with 49 additions and 1 deletion.
50 changes: 49 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1 +1,49 @@
# PipeLogger
<h1 align="center">PipeLogger Library</h1>

### What does the Pipelogger library do?
Pipelogger is a library created to establish a standard format for execution logs in pipelines mainly related to data ingestion. The standard form is this:

```json
{
"PipelineLogs": {
"PipelineID": "Pipeline-Example",
"Timestamp": "MM-DD-YY-THH:MM:SS",
"Status": "Success",
"Message": "Data uploaded successfully",
"ExecutionTime": 20.5075738430023
},
"BigQueryLogs": [
{
"BigQueryID": "project.pipeline-example.table_1",
"Size": 1555
},
{
"BigQueryID": "project.pipeline-example.table_2",
"Size": 3596
}
],
"Details": [
{
"additional_info": [
"Data downloaded successfully",
"Data processed successfully",
"Data uploaded successfully"
]
}
]
}
```

### What should you consider when implementing the PipeLogger in your pipeline?

* The pipeline must be deployed in GCP, either as Cloud Function or Cloud Run.
* The pipeline must feed Big Query tables.
* The pipeline needs a bucket to store the logs.


###

### How to implement it in any Pipeline?
> [Take a look at the Official PipeLogger Documentation](https://github.com/Romboost-Repo/PipeLogger/blob/main/docs/implementation.ipynb)
---

0 comments on commit 7a2881d

Please sign in to comment.