Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Add support for downloading encrypted files. #8

Merged
merged 4 commits into from
Apr 15, 2024
Merged

feat: Add support for downloading encrypted files. #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0754829
feat: Add support for downloading encrypted files.
bvanelli Apr 14, 2024
b5b614d
feat: Implement upload with encryption and improve documentation.
bvanelli Apr 14, 2024
cbf8999
refactor: Move encrypt to other method.
bvanelli Apr 15, 2024
19ea246
fix: Raise proper error if decryption fails.
bvanelli Apr 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 46 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Python API implementation for Actual server.

[Actual Budget](https://actualbudget.org/) is a super fast and privacy-focused app for managing your finances.
[Actual Budget](https://actualbudget.org/) is a superfast and privacy-focused app for managing your finances.

> **WARNING:** The [Javascript API](https://actualbudget.org/docs/api/) to interact with Actual server already exists,
> and is battle-tested as it is the core of the Actual frontend libraries. If you intend to use a reliable and well
Expand All @@ -29,6 +29,7 @@ from actual.queries import get_transactions
with Actual(
base_url="http://localhost:5006", # Url of the Actual Server
password="<your_password>", # Password for authentication
encryption_password=None, # Optional: Password for the file encryption. Will not use it if set to None.
file="<file_id_or_name>", # Set the file to work with. Can be either the file id or file name, if name is unique
data_dir="<path_to_data_directory>" # Optional: Directory to store downloaded files. Will use a temporary if not provided
) as actual:
Expand Down Expand Up @@ -62,11 +63,13 @@ You will then have a freshly created new budget to use:

![created-budget](./docs/static/new-budget.png)

If the `encryption_password` is set, the budget will additionally also be encrypted on the upload step to the server.

# Adding new transactions

After you created your first budget (or when updating an existing budget), you can add new transactions by adding them
using the `actual.session.add()` method. You cannot use the SQLAlchemy session directly because that adds the entries to your
local database, but will not sync the results back to the server (that is only possible when reuploading the file).
local database, but will not sync the results back to the server (that is only possible when re-uploading the file).

The method will make sure the local database is updated, but will also send a SYNC request with the added data so that
it will be immediately available on the frontend:
Expand All @@ -75,19 +78,56 @@ it will be immediately available on the frontend:
import decimal
import datetime
from actual import Actual
from actual.queries import get_accounts, create_transaction_from_ids
from actual.queries import create_transaction, create_account

with Actual(base_url="http://localhost:5006", password="mypass", file="My budget") as actual:
act = get_accounts(actual.session)[0] # get first account
t = create_transaction_from_ids(
actual.session, datetime.date.today(), act.id, None, notes="My first transaction", amount=decimal.Decimal(10.5)
act = create_account(actual.session, "My account")
t = create_transaction(
actual.session,
datetime.date.today(),
act,
"My payee",
notes="My first transaction",
amount=decimal.Decimal(-10.5),
)
actual.session.add(t)
actual.commit() # use the actual.commit() instead of session.commit()!
```

![added-transaction](./docs/static/added-transaction.png)

# Understanding how Actual handles changes

The Actual budget is stored in a sqlite database hosted on the user's browser. This means all your data is fully local
and can be encrypted with a local key, so that not even the server can read your statements.

The Actual Server is a way of only hosting files and changes. Since re-uploading the full database on every single
change is too heavy, Actual only stores one state of the database and everything added by the user via frontend
or via the APIs are individual changes above the base stored database on the server, stored via separate endpoint.
This means that on every change, done locally, a SYNC request is sent to the server with a list of the following string
parameters:

- `dataset`: the name of the table where the change happened.
- `row`: the row identifier for the entry that was added/update. This would be the primary key of the row (a uuid value)
- `column`: the column that had the value changed
- `value`: the new value. Since it's a string, the values are either prefixed by `S:` to denote a string, `N:` to denote
a numeric value and `0:` to denote a null value.

All individual column changes are computed on an insert, serialized with protobuf and sent to the server to be stored.
New clients can use this individual changes to then sync their local copies and add the changes executed on other users.

But this also means that new users need to download a long list of changes, possibly making the initialization slow.
Thankfully, user is also allowed to reset the sync. This would make sure all changes are actually stored in the
database. This is done on the frontend under *Settings > Reset sync*, and causes the current file to be reset (removed
from the server) and re-uploaded again, with all changes already in place. In this case, the sync list of changes is
reset because the server already has the latest version of the database.

This means that, when using this library to operate changes on the database, you have to make sure that either:

- do a sync request is made using the `actual.commit()` method. This only handles pending operations that haven't yet
been committed, generates a change list with them and posts them on the sync endpoint.
- do a full re-upload of the database is done.

# Contributing

The goal is to have more features implemented and tested on the Actual API. If you have ideas, comments, bug fixes or
Expand Down
126 changes: 97 additions & 29 deletions actual/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
from __future__ import annotations

import base64
import datetime
import io
import json
Expand All @@ -9,12 +10,13 @@
import tempfile
import uuid
import zipfile
from typing import TYPE_CHECKING, Union
from typing import Union

import sqlalchemy
import sqlalchemy.orm

from actual.api import ActualServer, RemoteFileListDTO
from actual.crypto import create_key_buffer, decrypt_from_meta, encrypt, make_salt
from actual.database import (
MessagesClock,
get_attribute_by_table_name,
Expand All @@ -28,9 +30,6 @@
)
from actual.protobuf_models import HULC_Client, Message, SyncRequest

if TYPE_CHECKING:
pass


class Actual(ActualServer):
def __init__(
Expand All @@ -39,6 +38,7 @@ def __init__(
token: str = None,
password: str = None,
file: str = None,
encryption_password: str = None,
data_dir: Union[str, pathlib.Path] = None,
bootstrap: bool = False,
):
Expand All @@ -53,6 +53,7 @@ def __init__(
:param token: the token for authentication, if this is available (optional)
:param password: the password for authentication. It will be used on the .login() method to retrieve the token.
:param file: the name or id of the file to be set
:param encryption_password: password used to configure encryption, if existing
:param data_dir: where to store the downloaded files from the server. If not specified, a temporary folder will
be created instead.
:param bootstrap: if the server is not bootstrapped, bootstrap it with the password.
Expand All @@ -66,11 +67,13 @@ def __init__(
# set the correct file
if file:
self.set_file(file)
self._encryption_password = encryption_password
self._master_key = None
self._in_context = False

def __enter__(self) -> Actual:
if self._file:
self.download_budget()
self.download_budget(self._encryption_password)
self._session = self.session_maker()
self._in_context = True
return self
Expand Down Expand Up @@ -126,8 +129,9 @@ def run_migrations(self, migration_files: list[str]):
conn.commit()
conn.close()

def create_budget(self, budget_name: str):
"""Creates a budget using the remote server default database and migrations."""
def create_budget(self, budget_name: str, encryption_password: str = None):
"""Creates a budget using the remote server default database and migrations. If password is provided, the
budget will be encrypted."""
migration_files = self.data_file_index()
# create folder for the files
if not self._data_dir:
Expand All @@ -138,16 +142,14 @@ def create_budget(self, budget_name: str):
# also write the metadata file with default fields
random_id = str(uuid.uuid4()).replace("-", "")[:7]
file_id = str(uuid.uuid4())
(self._data_dir / "metadata.json").write_text(
json.dumps(
{
"id": f"My-Finances-{random_id}",
"budgetName": budget_name,
"userId": self._token,
"cloudFileId": file_id,
"resetClock": True,
}
)
self.update_metadata(
{
"id": f"My-Finances-{random_id}",
"budgetName": budget_name,
"userId": self._token,
"cloudFileId": file_id,
"resetClock": True,
}
)
self._file = RemoteFileListDTO(name=budget_name, fileId=file_id, groupId=None, deleted=0, encryptKeyId=None)
# create engine for downloaded database and run migrations
Expand All @@ -160,6 +162,40 @@ def create_budget(self, budget_name: str):
# create a clock
self.load_clock()

def _gen_zip(self) -> bytes:
binary_data = io.BytesIO()
with zipfile.ZipFile(binary_data, "a", zipfile.ZIP_DEFLATED, False) as z:
z.write(self._data_dir / "db.sqlite", "db.sqlite")
z.write(self._data_dir / "metadata.json", "metadata.json")
return binary_data.getvalue()

def encrypt(self, encryption_password: str):
"""Encrypts the local database using a new key, and re-uploads to the server.

WARNING: this resets the file on the server. Make sure you have a copy of the database before attempting this
operation.
"""
if encryption_password and not self._file.encrypt_key_id:
# password was provided, but encryption key not, create one
key_id = str(uuid.uuid4())
salt = make_salt()
self.user_create_key(self._file.file_id, key_id, encryption_password, salt)
self.update_metadata({"encryptKeyId": key_id})
self._file.encrypt_key_id = key_id
elif self._file.encrypt_key_id:
key_info = self.user_get_key(self._file.file_id)
salt = key_info.data.salt
else:
raise ActualError("Budget is encrypted but password was not provided")
self._master_key = create_key_buffer(encryption_password, salt)
# encrypt binary data with
encrypted = encrypt(self._file.encrypt_key_id, self._master_key, self._gen_zip())
binary_data = io.BytesIO(base64.b64decode(encrypted["value"]))
encryption_meta = encrypted["meta"]
self.reset_user_file(self._file.file_id)
self.upload_user_file(binary_data.getvalue(), self._file.file_id, self._file.name, encryption_meta)
self.set_file(self._file.file_id)

def upload_budget(self):
"""Uploads the current file to the Actual server."""
if not self._data_dir:
Expand All @@ -168,8 +204,11 @@ def upload_budget(self):
with zipfile.ZipFile(binary_data, "a", zipfile.ZIP_DEFLATED, False) as z:
z.write(self._data_dir / "db.sqlite", "db.sqlite")
z.write(self._data_dir / "metadata.json", "metadata.json")
binary_data.seek(0)
return self.upload_user_file(binary_data.getvalue(), self._file.file_id, self._file.name)
# we have to first upload the user file so the reference id can be used to generate a new encryption key
self.upload_user_file(binary_data.getvalue(), self._file.file_id, self._file.name)
# encrypt the file and re-upload
if self._encryption_password or self._master_key or self._file.encrypt_key_id:
self.encrypt(self._encryption_password)

def reupload_budget(self):
self.reset_user_file(self._file.file_id)
Expand All @@ -183,9 +222,7 @@ def apply_changes(self, messages: list[Message]):
for message in messages:
if message.dataset == "prefs":
# write it to metadata.json instead
config = json.loads((self._data_dir / "metadata.json").read_text() or "{}")
config[message.row] = message.get_value()
(self._data_dir / "metadata.json").write_text(json.dumps(config))
self.update_metadata({message.row: message.get_value()})
continue
table = get_class_by_table_name(message.dataset)
column = get_attribute_by_table_name(message.dataset, message.column)
Expand All @@ -198,16 +235,38 @@ def apply_changes(self, messages: list[Message]):
s.flush()
s.commit()

def download_budget(self):
def update_metadata(self, patch: dict):
"""Updates the metadata.json from the Actual file with the patch fields. The patch is a dictionary that will
then be merged on the metadata and written again to a file."""
config = json.loads((self._data_dir / "metadata.json").read_text() or "{}") | patch
(self._data_dir / "metadata.json").write_text(json.dumps(config))

def download_budget(self, encryption_password: str = None):
"""Downloads the budget file from the remote. After the file is downloaded, the sync endpoint is queries
for the list of pending changes. The changes are individual row updates, that are then applied on by one to
the downloaded database state."""
the downloaded database state.

If the budget is password protected, the password needs to be present to download the budget, otherwise it will
fail.
"""
file_bytes = self.download_user_file(self._file.file_id)

if self._file.encrypt_key_id and encryption_password is None:
raise ActualError("File is encrypted but no encryption password provided.")
if encryption_password is not None:
file_info = self.get_user_file_info(self._file.file_id)
key_info = self.user_get_key(self._file.file_id)
self._master_key = create_key_buffer(encryption_password, key_info.data.salt)
# decrypt file bytes
file_bytes = decrypt_from_meta(self._master_key, file_bytes, file_info.data.encrypt_meta)
self._load_zip(file_bytes)

def _load_zip(self, file_bytes: bytes):
f = io.BytesIO(file_bytes)
try:
zip_file = zipfile.ZipFile(f)
except zipfile.BadZipfile as e:
raise InvalidZipFile(f"Invalid zip file: {e}")
raise InvalidZipFile(f"Invalid zip file: {e}") from None
if not self._data_dir:
self._data_dir = pathlib.Path(tempfile.mkdtemp())
# this should extract 'db.sqlite' and 'metadata.json' to the folder
Expand All @@ -219,10 +278,17 @@ def download_budget(self):
# load the client id
self.load_clock()
# after downloading the budget, some pending transactions still need to be retrieved using sync
request = SyncRequest({"messages": [], "fileId": self._file.file_id, "groupId": self._file.group_id})
request = SyncRequest(
{
"messages": [],
"fileId": self._file.file_id,
"groupId": self._file.group_id,
"keyId": self._file.encrypt_key_id,
}
)
request.set_null_timestamp(client_id=self._client.client_id) # using 0 timestamp to retrieve all changes
changes = self.sync(request)
self.apply_changes(changes.get_messages())
self.apply_changes(changes.get_messages(self._master_key))
if changes.messages:
self._client = HULC_Client.from_timestamp(changes.messages[-1].timestamp)

Expand Down Expand Up @@ -257,14 +323,16 @@ def commit(self):
)
# create sync request based on the session reference that is tracked
req = SyncRequest({"fileId": self._file.file_id, "groupId": self._file.group_id})
if self._file.encrypt_key_id:
req.keyId = self._file.encrypt_key_id
req.set_null_timestamp(client_id=self._client.client_id)
# first we add all new entries and modify is required
for model in self._session.new:
req.set_messages(model.convert(is_new=True), self._client)
req.set_messages(model.convert(is_new=True), self._client, master_key=self._master_key)
# modify if required
for model in self._session.dirty:
if self._session.is_modified(model):
req.set_messages(model.convert(is_new=False), self._client)
req.set_messages(model.convert(is_new=False), self._client, master_key=self._master_key)
# make sure changes are valid before syncing
self._session.commit()
# sync all changes to the server
Expand Down
Loading
Loading