Skip to content

Demonstrate org-mode logging with a Github scraper

Notifications You must be signed in to change notification settings

mengwong/git-scrape

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

8 Commits
 
 
 
 
 
 

Repository files navigation

HOWTO Log To Org-Mode

What if your logs looked like this?

./emacs.screenshot.png

If you sprinkle printf statements throughout your scripts, prefix output lines with asterisks. This adds lightweight structure to your log levels, and makes your log files suitable for opening in emacs org-mode.

Audience

Programmers who use Emacs and Org-Mode.

Example: Scraping Github

I wrote a simple Github scraper in Perl.

Given a username, the script first queries the Github API for the URLs of their repositories.

Then the script runs git clone for each repository, or git pull if the repository already exists.

If you use “printf debugging” style, you can guess that this is the sort of script that could produce reams of logging output.

How to manage the mess of logs?

Use org-mode as a logging target!

Just add asterisks, judiciously, to the start of your print output.

Tee your STDOUT/STDERR to a log.org output file.

Then open the log file in Emacs and use TAB to fold and unfold.

It’s magic.

Usage

$ mkdir gitusername
$ cd    gitusername
$ git-scrape 2>&1 | tee gitusername.org

Note: the .org at the end is what tells Emacs to use org-mode. You may need to M-x org-mode.

It turns this kind of noise …

20200620-14:36:49 mengwong@venice4:~/src% mkdir jhpyle
20200620-14:36:51 mengwong@venice4:~/src% cd jhpyle
20200620-14:36:53 mengwong@venice4:~/src/jhpyle% git-scrape 2>&1 | tee log.org
* git-scrape jhpyle
already in jhpyle/
* what repositories does jhpyle have?
page 1 returned 29 urls
page 2 returned 0 urls
** jhpyle has 29 repositories
- https://github.com/jhpyle/charts.git
- https://github.com/jhpyle/docacon-dot-com.git
- https://github.com/jhpyle/docassemble.git
- https://github.com/jhpyle/docassemble-appeal.git
- https://github.com/jhpyle/docassemble-empty.git
- https://github.com/jhpyle/docassemble-income.git
- https://github.com/jhpyle/docassemble-LSLettersEmails.git
- https://github.com/jhpyle/docassemble-monitor.git
- https://github.com/jhpyle/docassemble-MotionTutorial.git
- https://github.com/jhpyle/docassemble-os.git
- https://github.com/jhpyle/docassemble-profileme.git
- https://github.com/jhpyle/docassemble-rpi.git
- https://github.com/jhpyle/docassemble-spot.git
- https://github.com/jhpyle/docassemble-ssareportchangesletter.git
- https://github.com/jhpyle/docassemble-suffolk.git
- https://github.com/jhpyle/docassemble-testpackage.git
- https://github.com/jhpyle/docassemble-testproj.git
- https://github.com/jhpyle/flask-kvsession.git
- https://github.com/jhpyle/lawyersongithub.git
- https://github.com/jhpyle/lsreportapi.git
- https://github.com/jhpyle/lstexttwilio.git
- https://github.com/jhpyle/mediawiki-pla.git
- https://github.com/jhpyle/mTCP.git
- https://github.com/jhpyle/p5-census-us.git
- https://github.com/jhpyle/pdfminer.git
- https://github.com/jhpyle/pennsylvaniacao.git
- https://github.com/jhpyle/python-docx-template.git
- https://github.com/jhpyle/testxsendfile.git
- https://github.com/jhpyle/textstat.git
* cloning 29 repositories
** https://github.com/jhpyle/charts.git
Cloning into 'charts'...
remote: Enumerating objects: 278, done.
remote: Counting objects: 100% (278/278), done.
remote: Compressing objects: 100% (175/175), done.
remote: Total 278 (delta 155), reused 216 (delta 95), pack-reused 0
Receiving objects: 100% (278/278), 297.19 KiB | 369.00 KiB/s, done.
Resolving deltas: 100% (155/155), done.

… into this colour-coded structured log

with a hierarchy which you can TAB to fold/unfold

./emacs.screenshot.png

You can watch the logs grow in realtime if you turn on M-x auto-revert-mode and move your cursor to the end of the file.

to auto-open a particular section upon opening the file, try:

print(“:PROPERTIES:\n:VISIBILITY: all\n:END:”);

See Also

This is a featherweight form of structured logging, sort of a substitute for log levels. For more on real structured logging (think JSON output instead of throwaway mutterings) see, e.g., https://engineering.grab.com/structured-logging

About

Demonstrate org-mode logging with a Github scraper

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages