Log Ingestor and Query Interface

Develop a log ingestor system that can efficiently handle vast volumes of log data, and offer a simple interface for querying this data using full-text search or specific field filters. Both the systems (the log ingestor and the query interface) can be built using any programming language of your choice. The logs should be ingested (in the log ingestor) over HTTP, on port `3000`.
Explore the docs »

View Demo · Report Bug · Request Feature

Table of Contents

1. About The Project
   • System Design
2. Getting Started/ README
3. Identified Issues
4. License
5. Contact
6. Acknowledgments

About The Project

Sample Log Data Format:

The logs to be ingested will be sent in this format.

{
	"level": "error",
	"message": "Failed to connect to DB",
  "resourceId": "server-1234",
	"timestamp": "2023-09-15T08:00:00Z",
	"traceId": "abc-xyz-123",
  "spanId": "span-456",
  "commit": "5e5342f",
  "metadata": {
      "parentResourceId": "server-0987"
    }
}

Requirements

The requirements for the log ingestor and the query interface are specified below.

Log Ingestor:

• Develop a mechanism to ingest logs in the provided format.
• Ensure scalability to handle high volumes of logs efficiently.
• Mitigate potential bottlenecks such as I/O operations, database write speeds, etc.
• Make sure that the logs are ingested via an HTTP server, which runs on port 3000 by default.

Query Interface:

• Offer a user interface (Web UI or CLI) for full-text search across logs.
• Include filters based on:
  • level
  • message
  • resourceId
  • timestamp
  • traceId
  • spanId
  • commit
  • metadata.parentResourceId
• Aim for efficient and quick search results.

Advanced Features (Bonus):

These features aren’t compulsory to implement, however, adding them might increase the chances of your submission being accepted.

• Implement search within specific date ranges.
• Utilize regular expressions for search.
• Allow combining multiple filters.
• Provide real-time log ingestion and searching capabilities.
• Implement role-based access to the query interface.

(back to top)

System Design

Log Ingestor

The Log Ingestor uses Flask as an HTTP server for log ingestion. It stores logs in SQLite for structured data and Elasticsearch for efficient search capabilities.

Query Interface

The Query Interface is a command-line interface (CLI) built with Click. It allows users to perform searches and apply filters on logs stored by the Log Ingestor.

Features Implemented

Log Ingestor:

• Ingests logs over HTTP on port 3000.
• Stores logs in both SQLite and Elasticsearch for scalability.

Query Interface:

• CLI for user interaction.
• Search logs based on various parameters (level, message, resourceId, etc.).
• Additional features: Date range search, regular expression search, and combined filters.

Advanced Features (Bonus):

• Date range search in Query Interface.
• Regular expression search in Query Interface.

(back to top)

Getting Started/ README

Log Ingestor and Query Interface

This project consists of a Log Ingestor and a Query Interface. The Log Ingestor ingests logs over HTTP and stores them, while the Query Interface allows users to search and filter logs.

Running the Project

Log Ingestor

1. Install the necessary dependencies:

   pip install flask
   pip install sqlite3
   pip install elasticsearch

2. Run the Log Ingestor:

   python log_ingestor.py

The Log Ingestor will run on http://localhost:3000 by default.

Query Interface

1. Install the necessary dependencies:

   pip install click

2. Run the Query Interface:

   python query_interface_cli.py query_logs
   

This will display options for filtering logs. Use the available options to query logs.

(back to top)

Identified Issues

Limited Error Handling:

-The error handling in both Log Ingestor and Query Interface is basic. Further refinement is needed.

Security Considerations:

• This example doesn't cover security aspects. In a production environment, consider securing the API endpoints and implementing authentication.

Real-time Ingestion and Searching:

• The implementation lacks real-time log ingestion and instant searching. This could be achieved with additional tools or frameworks.

(back to top)

License

Distributed under the MIT License. See LICENSE.txt for more information.

(back to top)

Contact

Your Name - @linkedin_profile - [email protected]

Project Link: https://github.com/dyte-submissions/november-2023-hiring-sarkar-sayan

(back to top)

Acknowledgments

(back to top)