Develop a log ingestor system that can efficiently handle vast volumes of log data, and offer a simple interface for querying this data using full-text search or specific field filters. Both the systems (the log ingestor and the query interface) can be built using any programming language of your choice. The logs should be ingested (in the log ingestor) over HTTP, on port `3000`.
Explore the docs »
View Demo
·
Report Bug
·
Request Feature
Table of Contents
The logs to be ingested will be sent in this format.
{
"level": "error",
"message": "Failed to connect to DB",
"resourceId": "server-1234",
"timestamp": "2023-09-15T08:00:00Z",
"traceId": "abc-xyz-123",
"spanId": "span-456",
"commit": "5e5342f",
"metadata": {
"parentResourceId": "server-0987"
}
}
The requirements for the log ingestor and the query interface are specified below.
- Develop a mechanism to ingest logs in the provided format.
- Ensure scalability to handle high volumes of logs efficiently.
- Mitigate potential bottlenecks such as I/O operations, database write speeds, etc.
- Make sure that the logs are ingested via an HTTP server, which runs on port
3000
by default.
- Offer a user interface (Web UI or CLI) for full-text search across logs.
- Include filters based on:
- level
- message
- resourceId
- timestamp
- traceId
- spanId
- commit
- metadata.parentResourceId
- Aim for efficient and quick search results.
These features aren’t compulsory to implement, however, adding them might increase the chances of your submission being accepted.
- Implement search within specific date ranges.
- Utilize regular expressions for search.
- Allow combining multiple filters.
- Provide real-time log ingestion and searching capabilities.
- Implement role-based access to the query interface.
The Log Ingestor uses Flask as an HTTP server for log ingestion. It stores logs in SQLite for structured data and Elasticsearch for efficient search capabilities.
The Query Interface is a command-line interface (CLI) built with Click. It allows users to perform searches and apply filters on logs stored by the Log Ingestor.
Log Ingestor:
- Ingests logs over HTTP on port 3000.
- Stores logs in both SQLite and Elasticsearch for scalability.
Query Interface:
- CLI for user interaction.
- Search logs based on various parameters (level, message, resourceId, etc.).
- Additional features: Date range search, regular expression search, and combined filters.
Advanced Features (Bonus):
- Date range search in Query Interface.
- Regular expression search in Query Interface.
This project consists of a Log Ingestor and a Query Interface. The Log Ingestor ingests logs over HTTP and stores them, while the Query Interface allows users to search and filter logs.
-
Install the necessary dependencies:
pip install flask pip install sqlite3 pip install elasticsearch
-
Run the Log Ingestor:
python log_ingestor.py
The Log Ingestor will run on http://localhost:3000 by default.
-
Install the necessary dependencies:
pip install click
-
Run the Query Interface:
python query_interface_cli.py query_logs
This will display options for filtering logs. Use the available options to query logs.
-The error handling in both Log Ingestor and Query Interface is basic. Further refinement is needed.
- This example doesn't cover security aspects. In a production environment, consider securing the API endpoints and implementing authentication.
- The implementation lacks real-time log ingestion and instant searching. This could be achieved with additional tools or frameworks.
Distributed under the MIT License. See LICENSE.txt
for more information.
Your Name - @linkedin_profile - [email protected]
Project Link: https://github.com/dyte-submissions/november-2023-hiring-sarkar-sayan