qsearch - quick search log by time

A scirpt tool for searching huge file quickly .

Requirement

• perl must be installed
• log file shold has timestamp filed.

Introduction

When the log size exceeds 10G, the speed of the grep command is very slow and will consumes a lot of machine resources. qsearch is a tool that uses the time information to compute the offset of the file and then retrieves parts of the file quickly.

The qsearch command executes very quickly, and consumes very little resources and almost does not increase rapidly as the file size increases.

Typical application scenario: 1) each request has a unique traceid to identify 2) traceid is generated by the server and contains timestamp information. 3) Traceid returns to the client

Features

• very Fast
• Automatically detects the possible file name of the target log (if log file are cut per hour,like xxx.2017051703)

Install

no need install.

Useage

perl qsearch.pl fileName command strTime [searchFileType searchLogRange logTimePatten ] 
      help:
        fileName       : search file name 
        command        : exec cmd . 
        strTime        : search time ,time\'s fmt is "YYYY-MM-DD HH:II:SS.xxx"
        searchFileType : 0（defautl）:auto detect maby file ; otherwise will not .
        searchLogRange : search log range size 
                         default : 20971520
        logTimePatten  : match the time in the log file .
                         time fmt must is "YYYY-MM-DD HH:II:SS.xxx"
      usage:
         perl qsearch.pl /home/log.log "grep  0abcd55c591c1d4b0000221f7d9f511d" "2017-05-03 09:01:01.023"
         perl qsearch.pl /home/log.log "grep NOTICE|wc -l"  "2017-11-03 09:01:01.023" 1 20971520