Local efficient Data Platform

A data platform running with the less infrastructure as possible.

Project requirements

Pyenv and Python 3.11.6

• Install pyenv to manage your Python versions and virtual environments:

  curl -sSL https://pyenv.run | bash

  • If you are on MacOS and experiencing errors on python install with pyenv, follow this comment
  • Add these lines to your ~/.bashrc or ~/.zshrc to be able to activate pyenv virtualenv:

    eval "$(pyenv init -)"
    eval "$(pyenv virtualenv-init -)"
    eval "$(pyenv init --path)"

  • Restart your shell

• Install the right version of Python with pyenv:

  pyenv install 3.11.6

Poetry

• Install Poetry to manage your dependencies and tooling configs:

  curl -sSL https://install.python-poetry.org | python - --version 1.7.0

  If you have not previously installed any Python version, you may need to set your global Python version before installing Poetry:

  pyenv global 3.11.6

Installation

Python virtual environment and dependencies

1. Create a pyenv virtual environment and link it to your project folder:

   pyenv virtualenv 3.11.6 local-efficient-data-platform
   pyenv local local-efficient-data-platform

   Now, every time you are in your project directory your virtualenv will be activated!

2. Install dependencies with Poetry:

   poetry install --no-root

Steps 1. and 2. can also be done in one command:

make install

Install git hooks (running before commit and push commands)

poetry run pre-commit install

Testing

To run unit tests, run pytest with:

pytest tests --cov src

or

make test

Formatting and static analysis

Code formatting with ruff

To check code formatting, run ruff format with:

ruff format --check .

or

make format-check

You can also integrate it to your IDE to reformat your code each time you save a file.

Static analysis with ruff

To run static analysis, run ruff with:

ruff check src tests

or

make lint-check

To run static analysis and to apply auto-fixes, run ruff with:

make lint-fix

Type checking with mypy

To type check your code, run mypy with:

mypy src --explicit-package-bases --namespace-packages

or

make type-check