Python module monitoring user-level file system events like Creation, Modification, Move, Copy, and Deletion of files and folders. Lazydog tries to aggregate low-level events between them in order to emit a minimum number of high-level events (actually one event per user action). Lazydog uses python Watchdog API to detect low-level events.
The easiest way:
$ pip3 install lazydogWhere the watchdog module would throw dozen of events after each user event, lazydog only throws one. For example, ask lazidog to watch any existing directory:
$ lazydog /the/directory/you/want/to/watchAnd just move a file in the watched directory (here from /watched/directory/move_test.txt to /watched/directory/move_test_2.txt), and wait 2 seconds. You will get something like this in the console:
INFO -
INFO - LIST OF THE LAST EVENTS:
INFO - moved: '/move_test.txt' to '/move_test_2.txt' mtime[1512151173.0] size[5]
INFO -Try to copy the same file, and you will get somthiing like this:
INFO -
INFO - LIST OF THE LAST EVENTS:
INFO - copied: '/move_test_2.txt' to '/move_test_2 - Copie.txt' mtime[1512151173.0] size[5]
INFO -Only one event per user action. You can try it with other type of action (Deletion, Creation, Modification), and also with directories.
Below is an example on how to rapidly initialize the high-level lazydog event handler,
and log every new event in the console (using logging module).
The watched directory is the current one (using os.getcwd()).
Please note that once installed, using the $ lazydog command in the console
does just the same.
import logging
import os
from lazydog.handlers import HighlevelEventHandler
# LOG
# create logger
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
# create console handler with a higher log level
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.DEBUG)
# create formatter and add it to the handlers
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
console_handler.setFormatter(formatter)
# add the handlers to the logger
logger.addHandler(console_handler)
# INITIALIZE
# get dir in parameter else current dir
watched_dir = directory if len(directory) > 1 else os.getcwd()
# initializing a new HighlevelEventHandler
highlevel_handler = HighlevelEventHandler.get_instance(watched_dir)
# starting it (since it is a thread)
highlevel_handler.start()
# log first message
logging.info('LISTENING EVENTS IN DIR: \'%s\'' % watched_dir)
# OPERATING
try:
while True:
# The following loop check every 1 second if any new event.
time.sleep(1)
local_events = highlevel_handler.get_available_events()
# If any, it logs it directly in the console.
for e in local_events:
logging.info(e)
# Keyboard <CTRL+C> interrupts the loop
except KeyboardInterrupt:
highlevel_handler.stop()Please find full code documentation in an HTML format on ReadTheDocs.org: http://lazydog.readthedocs.io/
Watchdog uses inotify by default on Linux to monitor directories for changes. It's not uncommon to encounter a system limit on the number of files you can monitor (for example 8192 directories). You can get your current inotify file watch limit by executing:
$ cat /proc/sys/fs/inotify/max_user_watches
8192When this limit is not enough to monitor all files inside a directory, the limit must be increased for Lazydog to work properly. You can set a new limit temporary with:
$ sudo sysctl fs.inotify.max_user_watches=524288
$ sudo sysctl -pIf you like to make your limit permanent, use:
$ echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
$ sudo sysctl -pThese instructions will get you a copy of the project up and running on your local machine for development and testing purposes.
Main dependency of lazydog, is the python watchdog API. You can install it using the following command:
$ pip3 install watchdogPlease read the official documentation for any question about this project: https://pypi.org/project/watchdog/
Just clone the repository in your local working directory (or fork it).
$ git clone https://github.com/warniiiz/LazydogIn order to contribute, you will need pytest for testing purpose (or refer to the pytest documentation ).
$ pip3 install pytestYou will also need Sphinx package for documentation purpose (or refer to the Sphinx documentation ).
$ apt-get install python-sphinxThe different python module are in the /lazydog directory. Each of them has attached test functions, that are in the /lazydog/test directory. You can launch tests unitary like this (for example for testing the events module):
$ pytest lazydog/test/test_events.pyKind of results:
================================= test session starts =================================
platform linux -- Python 3.4.2, pytest-3.5.0, py-1.5.3, pluggy-0.6.0
rootdir: /media/maxtor/media/Python/Lazydog, inifile:
plugins: cov-2.5.1
collected 16 items
lazydog/test/test_events.py ................ [100%]
============================== 16 passed in 0.51 seconds ==============================You can also test the whole package (assuming you are in the developpement directory):
$ pytestCheck the test coverage:
$ py.test --cov lazydogTest coverage is > 90%. The metric is not very relevant about the test quality, but at least you will be reasssured there are some tests ;)
========================== test session starts ===========================
platform linux -- Python 3.4.2, pytest-3.5.0, py-1.5.3, pluggy-0.6.0
rootdir: /media/maxtor/media/Python/Lazydog, inifile:
plugins: cov-2.5.1
collected 58 items
lazydog/test/test_events.py ................ [ 27%]
lazydog/test/test_handlers.py ...................... [ 65%]
lazydog/test/test_queues.py .. [ 68%]
lazydog/test/test_states.py .................. [100%]
----------- coverage: platform linux, python 3.4.2-final-0 -----------
Name Stmts Miss Cover
--------------------------------------------------------------------------
lazydog/__init__.py 0 0 100%
lazydog/dropbox_content_hasher.py 66 14 79%
lazydog/events.py 249 7 97%
lazydog/handlers.py 214 29 86%
lazydog/lazydog.py 39 39 0%
lazydog/queues.py 18 0 100%
lazydog/revised_watchdog/__init__.py 0 0 100%
lazydog/revised_watchdog/events.py 31 1 97%
lazydog/revised_watchdog/observers/__init__.py 0 0 100%
lazydog/revised_watchdog/observers/inotify.py 49 6 88%
lazydog/revised_watchdog/observers/inotify_buffer.py 12 0 100%
lazydog/revised_watchdog/observers/inotify_c.py 72 22 69%
lazydog/states.py 109 1 99%
lazydog/test/test_events.py 261 2 99%
lazydog/test/test_handlers.py 355 3 99%
lazydog/test/test_queues.py 31 0 100%
lazydog/test/test_states.py 172 0 100%
--------------------------------------------------------------------------
TOTAL 1678 124 93%
====================== 58 passed in 30.15 seconds ========================Please find full code documentation in an HTML format on ReadTheDocs.org: http://lazydog.readthedocs.io/
This documentation is automatically updated each time an update is made no GitHub.
Please document each change. If you want to check the result before publishing, you can run the following after each documentation modification:
$ cd docs # first go in the /docs subdirectory.
$ make html # recompute the sphinx documentationThe resulted documentation is then in the local relative folder /docs/_build/html/index.html.
Note that if you did not modify local file from /docs subdirectory, the changes will not be taken... you can use the following command to force recomputing all the changes:
$ touch autodoc.rst; make html Last thing. If you modified the main README.md, and you want the changes to appear in the documentation (and not only on github), you have to convert the .md file to a .rst one. You can use the pandoc app to do thiss conversion, using the following command (after installing Pandoc, please refer to Pandoc documentation for more information):
pandoc --from=markdown --to=rst --output=README.rst ../README.md # Assuming you are in the /docs subdirectory.Then don't forget to run the previous command again to recompute the whole documentation.
For lazydog to be a truly great project, third-party code contributions are important. If you want to enhance lazydog, spot bugs or fix them, or just ask for new enhancements, you are so much welcome! Below is a list of things that might help you in contributing to lazydog.
The list of the current bugs, issues, new enhancement proposals, etc. are all grouped on GitHub Issues' tab:
For more information about GitHub, please check the followings:
To get involved in code enhancement:
- Make sure you have a GitHub account
- Get the latest version, by either way cloning of forking this repository (depending on what you want to do)
- Install the requirements via pip:
pip install -r requirements.txt - Submit an issue directly on GitHub:
- For bugs, clearly describe the issue including steps to reproduce
- For enhancement proposals, be sure to indicate if you're willing to work on implementing the enhancement
If you do not have GitHub account and you just want to notify for a new bug, please report me by e-mail.
lazydogdoes not use any git Workflow until now. This will remains until the volume of changes and contribution needs a clearer workflow.- Make commits of logical units.
- Check for unnecessary whitespace with
git diff --checkbefore committing. - Make sure you have added the necessary tests for your changes.
- Run
python setup.py testto make sure your tests pass - Run
coverage run --source=lazydog setup.py testif you have thecoveragepackage installed to generate coverage data - Check your coverage by running
coverage report - Please correctly document the code you wrote, and ensure it is readable once HTML generated
- Update main documentation files (README.md, etc.) when necessary.
- Push your changes to the feature branch in your fork of the repository.
- Submit a pull request to the main repository
We use SemVer for versioning. Please read RELEASE-NOTES.md for details about each releases.
- Clément Warneys - Initial work - warniiiz
This project is licensed under the Apache License Version 2.0. Please see the LICENSE.md file for details.
Thanks to Jeff Knupp for this general guidelines for open sourcing a python project (which helped me a lot since it is my first open source project I deliver):