LTFS-VOF.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "be9ff49e",
   "metadata": {
    "tags": []
   },
   "source": [
    "# LTFS Versioned Object Format\n",
    "\n",
    "The LTFS Versioned Object Format (LTFS-VOF) is a format for saving object data to tapes. This document describes the format so that others may create compatible systems or tools. Python code is included which implements decoding of LTFS-VOF data and metadata. You can download and use this Jupyter notebook interactively to decode your own data.\n",
    "\n",
    "This specification overlays the Linear Tape File System (LTFS). An\n",
    "implementation of the LTFS-VOF will want to refer to the [LTFS\n",
    "specification][ltfs] or use a pre-built LTFS driver. LTFS provides a\n",
    "format for storing files on tape with a POSIX programming interface.\n",
    "The Versioned Object Format layers on top of one or more LTFS tapes to provide:\n",
    "\n",
    "[ltfs]: https://www.snia.org/tech_activities/standards/curr_standards/ltfs\n",
    "\n",
    "1. Efficient object and metadata packing. LTFS-VOF uses large files on\n",
    "   LTFS, which both minimizes the overhead of LTFS metadata and tape\n",
    "   load/unload time.\n",
    "\n",
    "2. Support for very small and very large object sizes. Very small\n",
    "   objects will be packed into large LTFS files, allowing rapid\n",
    "   transfer of many small objects to/from tape. Very large objects may\n",
    "   span tapes.\n",
    "\n",
    "3. S3-compliant object versioning. Unlike POSIX, objects in LTFS-VOF may\n",
    "   have multiple versions, including delete markers.\n",
    "\n",
    "4. S3-compliant object names and metadata. POSIX and S3 have different\n",
    "   naming restrictions and differences in the format of metadata. LTFS-VOF\n",
    "   captures object metadata in LTFS files, similar to object data.\n",
    "\n",
    "5. Modern compression, encryption, and hash codes. LTFS-VOF uses Zstandard\n",
    "   compression, which allows users a great deal of flexibility to\n",
    "   trade off speed vs. compression efficiency. AES-256 encryption is\n",
    "   used, with flexible AES key identifiers that may reference various\n",
    "   encryption key managers. Data integrity is assured with modern hash codes\n",
    "   such as XXHash.\n",
    "\n",
    "6. Support for tape-set parity. In configurations with multiple tape\n",
    "   libraries, parity packs may be stored to maximize data availability\n",
    "   in the event that a library is down or a set of tapes is damaged.\n",
    "   (This feature is not documented in this version of the LTFS-VOF format,\n",
    "   but will be included in a future version.)\n",
    "\n",
    "In addition to LTFS, a LTFS-VOF implementation uses [MessagePack][msgpack] to encode various structures. MessagePack implementations are available for most programming languages.\n",
    "\n",
    "[msgpack]: https://msgpack.org/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e3c13c6",
   "metadata": {},
   "source": [
    "# Object System Concepts\n",
    "\n",
    "S3-compatible object stores have different concepts and terminology than file systems. This section provides an overview.\n",
    "\n",
    "![Buckets, Objects, Versions](figures/concepts1.svg)\n",
    "\n",
    "### Buckets\n",
    "\n",
    "A *bucket* is the outermost container for objects; it is most similar\n",
    "to a file system. Buckets tend to have high-level policies that apply\n",
    "to all objects within them, for example lifecycle policies that\n",
    "control where data is placed and when tape copies are mode.\n",
    "\n",
    "### Objects and Versions\n",
    "\n",
    "Buckets contain *objects*. Each object has a name (or key) and other\n",
    "metadata associated with it, and a map of data blocks. Objects have\n",
    "one or more *versions*. The last version for a given name is called\n",
    "the *current version*. The object may be thought of as a pointer to\n",
    "the current version. To avoid confusion, this document will almost\n",
    "exclusively discuss versions instead of objects.\n",
    "\n",
    "In addition to the object data, a version has metadata about the version record itself, for example the version's ID, ETag, and creation time. The version metadata is also saved to tape so that the tape set will contain both the data and metadata for all objects.\n",
    "\n",
    "![Blocks and Packs](figures/concepts2.svg)\n",
    "\n",
    "### Blocks\n",
    "\n",
    "A version's data is stored in one or more *blocks*. Each block is a\n",
    "slice of object data, typically 10MB in size before compression. An\n",
    "object whose length is less than the block length will simply be\n",
    "composed of one short block. A larger object will be composed of\n",
    "multiple blocks with a short block at the end. Each block is\n",
    "compressed and encrypted individually so that a S3 client performing\n",
    "range reads may be answered by decoding only the blocks containing the\n",
    "range the client is asking for.\n",
    "\n",
    "A set of blocks are stored together in *packs*. Packs are typically\n",
    "large files, where the optimal pack size is determined by the type of\n",
    "storage media. For tape, packs will be multiple gigabytes in size, and\n",
    "will contain hundreds of blocks stored end-to-end. Packs may contain\n",
    "blocks belonging to many versions.\n",
    "\n",
    "Packs will also contain metadata in the form of pack lists and version records, not just blocks. This metadata allows a tape set to be fully self-describing."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15fa079b",
   "metadata": {},
   "source": [
    "# Tape Format Overview\n",
    "\n",
    "The LTFS Versioned Object Format is composed of several levels. At each level is\n",
    "an encoding scheme that is straightforward to implement, while\n",
    "providing high runtime efficiency. This section presents an overview\n",
    "of each level, with detailed descriptions in following sections.\n",
    "\n",
    "![Levels](figures/levels.svg)\n",
    "\n",
    "\n",
    "### Level 0: LTFS\n",
    "\n",
    "The base (zero) level in the system is LTFS, as specified by the LTFS\n",
    "Format Specification version 2 or later. Any LTFS-compliant driver may\n",
    "be used. The LTFS-VOF should be implemented in a user-space\n",
    "process, using standard POSIX file system calls to manipulate the\n",
    "files on a LTFS tape. LTFS uses a combination of data blocks, index\n",
    "blocks, and file marks to lay out data on a tape. This is entirely\n",
    "transparent to the Vail Tape Format, however.\n",
    "\n",
    "### Level 1: Packs\n",
    "\n",
    "The first level is *packs*, which are LTFS files that store encoded\n",
    "data or metadata. Each pack stores either object data in the form of\n",
    "*blocks*, or metadata in the form of *versions*. Both are described in\n",
    "later sections.\n",
    "\n",
    "LTFS may use one or more extents to store a file, and each extent is a\n",
    "series of data blocks followed by an index block describing the\n",
    "extent. LTFS also stores the index block in a separate index partition\n",
    "which is read when a tape is mounted. Level one is also mostly\n",
    "transparent to a LTFS-VOF implementation.\n",
    "\n",
    "### Level 2: TLV\n",
    "\n",
    "The second level is an end-to-end stacking of records within packs.\n",
    "Each record is encoded with a tag-length-value (TLV) format that uses a\n",
    "fixed-size header and variable-length value. This header contains the\n",
    "minimal information required to identify the type of data stored, its\n",
    "length, and hash codes to validate both the header's integrity and the\n",
    "value's integrity. Any TLV may be read and decoded by simply reading\n",
    "its range of bytes within the pack.\n",
    "\n",
    "### Level 3: Value Encoding\n",
    "\n",
    "The third level uses a variable-size header that provides details on\n",
    "compression, encryption, and the application version used to encode\n",
    "the value. MessagePack is used to decode the header first, then the\n",
    "value contents must be decrypted and decompressed, then those\n",
    "block/version bytes are decoded.\n",
    "\n",
    "### Level 4: Data and Metadata\n",
    "\n",
    "The final level is the LTFS-VOF data and metadata objects. These are the\n",
    "version records, pack lists, and data blocks which comprise all the data\n",
    "stored in the system."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57057254",
   "metadata": {},
   "source": [
    "Pack Files\n",
    "----------\n",
    "\n",
    "Packs are named with a [ULID][ulid] followed by the extension `.blk`\n",
    "for packs containing blocks, or `.ver` for packs containing versions.\n",
    "Blocks and versions are stored separately so that a system importing a\n",
    "set of tapes need only scan the `.ver` packs to build its database of\n",
    "metadata.\n",
    "\n",
    "[ulid]: https://github.com/oklog/ulid\n",
    "\n",
    "ULID is similar to a UUID, however it contains an embedded timestamp\n",
    "and sorts lexicographically from oldest time to newest time. Pack IDs\n",
    "should use the current time (in UTC) when the pack is first written to.\n",
    "\n",
    "Pack files should be stored in the root directory of a LTFS tape.\n",
    "\n",
    "It is important for read performance that pack files be stored as long\n",
    "runs of contiguous LTFS data extents. While LTFS supports interleaving\n",
    "of file data--which may happen if multiple files are written\n",
    "concurrently--this is not very efficient, as reading such a file will\n",
    "require LTFS to perform many seeks. Therefore it is strongly\n",
    "recommended that a LTFS-VOF writer sequentially write full packs to tape,\n",
    "one pack at a time, so that packs may be composed of large LTFS\n",
    "extents without on-tape interleaving."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f69b0811",
   "metadata": {},
   "source": [
    "# TLV Encoding\n",
    "\n",
    "![TLV Header](figures/tlv_header.svg)\n",
    "\n",
    "The tag-length-value (TLV) format is used to store many records into a\n",
    "pack file. Its role is to provide just enough information for an\n",
    "application to scan through a pack file, hop from one TLV to the next,\n",
    "and ensure that records are valid before attempting to decode them.\n",
    "\n",
    "TLV uses a 32 byte header and variable-length value. A TLV reader\n",
    "should read the header and validate it using these steps, as illustrated\n",
    "in the code which follows.\n",
    "\n",
    "1. The header \"magic\" is correct. The sequence should match\n",
    "   `0x89`, `T`, `L`, `V`, `0x0d`, `0x0a`, `0x1a`, `0x0a`. This\n",
    "   sequence identifies the TLV and allows early detection of certain\n",
    "   types of corruption, for example end-of-line mangling if the TLV\n",
    "   was accidentally treated as text. (The header magic is borrowed\n",
    "   from the PNG file format. Further detail on its rationale may be\n",
    "   found in the [PNG file format specification][png]).\n",
    "\n",
    "2. The version at byte 24 indicates the version of the\n",
    "   TLV format used when this TLV was created. This is currently 0. Decoding\n",
    "   should stop if an unknown version number is seen.\n",
    "\n",
    "3. At this point the header hash should be calculated and verified.\n",
    "   The hash type is stored at byte 27, and header hash value is stored at bytes 30..31.\n",
    "   Hash type 8, XXHash64, is standard. This is validated by\n",
    "   calculating a [XXHash64][xxhash], keeping only the lower 16\n",
    "   bits, and comparing those to the stored value. Decoding should stop\n",
    "   if the header hash code does not match.\n",
    "\n",
    "The following code demonstrates reading and unpacking the TLV header.\n",
    "\n",
    "[png]: http://www.libpng.org/pub/png/spec/1.2/PNG-Rationale.html#R.PNG-file-signature\n",
    "\n",
    "[xxhash]: http://cyan4973.github.io/xxHash/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ddb46e90",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install necessary Python modules for the code in this notebook.\n",
    "# Uncomment following two lines if you have import failures:\n",
    "# import sys\n",
    "# !{sys.executable} -m pip install xxhash msgpack zstd cryptography\n",
    "\n",
    "# Import modules needed for sample code below\n",
    "from __future__ import annotations\n",
    "import base64, io, msgpack, typing, unittest, zstd\n",
    "from pprint import pprint\n",
    "from ulid import ULID\n",
    "from struct import unpack\n",
    "from typing import Optional\n",
    "from xxhash import xxh64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f1501527",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TlvHeader(tag b'C!', dlen 14)\n"
     ]
    }
   ],
   "source": [
    "class TlvHeader:\n",
    "    def __init__(self, f: typing.BinaryIO):\n",
    "        buf = f.read(32)\n",
    "\n",
    "        if len(buf) == 0:\n",
    "            raise EOFError\n",
    "\n",
    "        if len(buf) < 32:\n",
    "            raise RuntimeError(f'TLV header too short; need 32 bytes, got {len(buf)}')\n",
    "\n",
    "        self.magic, self.dlen, self.dhash, self.version, \\\n",
    "            self.tag, self.hashtype, self.hhash = unpack(\"!8sQQB2sBxxH\", buf)\n",
    "\n",
    "        if self.magic != b'\\x89TLV\\r\\n\\x1a\\n':\n",
    "            raise 'invalid TLV header magic'\n",
    "\n",
    "        if self.version != 0:\n",
    "            raise f'unknown version {self.version}; can only handle TLV version 0'\n",
    "\n",
    "        if self.hashtype != 8:\n",
    "            raise f'invalid hash type {self.hashtype}; can only handle 8 (xxhash64)'\n",
    "\n",
    "        if self.hhash != (xxh64(buf[0:30]).intdigest() % 2 ** 16):\n",
    "            raise 'TLV header hash mismatch'\n",
    "\n",
    "    def __repr__(self):\n",
    "        return f'TlvHeader(tag {self.tag}, dlen {self.dlen})'\n",
    "\n",
    "\n",
    "# Small sample TLV, base64-encoded\n",
    "with io.BytesIO(base64.b64decode(\"iVRMVg0KGgoAAAAAAAAADuM9tfSfjss2AEMhCAAAuxRkYXRhIGRhdGEgZGF0YQ==\")) as f:\n",
    "    header = TlvHeader(f)\n",
    "    pprint(header)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db9e69c0",
   "metadata": {},
   "source": [
    "Once the header has been unpacked and verified, its remaining fields may be considered trustworthy. The tag at bytes 25..26 is used to identify the data type of the value. The data length at bytes 8..15 is stored in network byte order (big-endian). Finally, The data buffer integrity should then be validated using the hash code stored at bytes 16..23. This uses the same hash type as in step 3 above. (The full 64 bits are used this time, instead of 16, as used for header validation.)\n",
    "\n",
    "To consume the data portion of the TLV, simply read the number of data bytes indicated in the header, and validate using the data hash code from the header.\n",
    "\n",
    "The following code illustrates a simple TLV reader which reads and validates both the header and value. This code does no further decoding on the value; that is covered in the next section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1161e9b2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TLV (tag b'C!', dlen 14) data b'data data data'\n"
     ]
    }
   ],
   "source": [
    "class TlvSimple:\n",
    "    \"\"\"\n",
    "    Simple form of TLV that consumes a TLV from a stream, validates its integrity, but\n",
    "    does not do any decoding on the value.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, f: typing.BinaryIO):\n",
    "        \"\"\"\n",
    "        Creates a TLV from input stream f, leaving the stream positioned at the start of the next TLV.\n",
    "        :param f: any file-like stream.\n",
    "        \"\"\"\n",
    "        self.header = TlvHeader(f)\n",
    "        self.data = f.read(self.header.dlen)\n",
    "\n",
    "        if len(self.data) < self.header.dlen:\n",
    "            raise f'short data read: expected {self.header.dlen} bytes, got {len(self.data)}'\n",
    "\n",
    "        if self.header.dhash != xxh64(self.data).intdigest():\n",
    "            raise \"TLV data hash mismatch\"\n",
    "\n",
    "    def __repr__(self):\n",
    "        return f'TLV (tag {self.header.tag}, dlen {len(self.data)})'\n",
    "\n",
    "    \n",
    "# Small sample TLV, base64-encoded\n",
    "with io.BytesIO(base64.b64decode(\"iVRMVg0KGgoAAAAAAAAADuM9tfSfjss2AEMhCAAAuxRkYXRhIGRhdGEgZGF0YQ==\")) as f:\n",
    "    tlv = TlvSimple(f)\n",
    "    print(f'{tlv} data {tlv.data}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "854dc0ab",
   "metadata": {},
   "source": [
    "Multiple TLVs may be stored end-to-end within a file. The example file `3simple.tlv` has three TLVs with simple strings for values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c77f6d5e-226d-49d5-b7b5-2582f30cc8ad",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TLV (tag b'bk', dlen 6) data b'data 1'\n",
      "TLV (tag b'bk', dlen 6) data b'data 2'\n",
      "TLV (tag b'bk', dlen 6) data b'data 3'\n"
     ]
    }
   ],
   "source": [
    "with open('sample_data/3simple.tlv', 'rb') as f:\n",
    "    for i in range(3):\n",
    "        tlv = TlvSimple(f)\n",
    "        print(f'{tlv} data {tlv.data}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06e111fa",
   "metadata": {},
   "source": [
    "# Value Encoding\n",
    "\n",
    "TLV values are encoded using a separate, second level called *value\n",
    "encoding*. This is separate from TLV because:\n",
    "\n",
    "1. TLV allows many items to be stored together and validated without decoding the values. This allows copying TLVs from one system to another, or migrating from one storage medium to another, even if the encryption keys are not accessible.\n",
    "\n",
    "2. Applications may scan through many TLVs looking for specific types (identified by tag) or a specific instance (identified by data hash) and then decode only the TLVs it needs.\n",
    "\n",
    "3. TLV requires a fixed-size header by design. The value encoder uses a variable-size header because the encoding may have several stages, and the parameters of each stage will be specified in the header. For example, if the value is encrypted, the header will include crypt-specific details. If not encrypted, these fields will not be present.\n",
    "\n",
    "Value encoding provides the following features:\n",
    "\n",
    "1. Compression, by default using the Zstandard algorithm. If the data is not compressible, then compression may be skipped.\n",
    "   \n",
    "2. Encryption with AES-256.\n",
    "\n",
    "3. Data format versioning. If the application changes its saved data format, it will increment the version number for saved values. Decoding should inspect the version number and decode accordingly.\n",
    "   \n",
    "4. Flexibility for future features. Fields may be added to the header as necessary.\n",
    "\n",
    "[MessagePack][msgpack] is used to serialize both the header and the application-defined data. High-quality implementations of MessagePack are available for most programming languages. MessagePack is a fast, compact, binary encoding format.\n",
    "\n",
    "[msgpack]: https://msgpack.org/index.html\n",
    "\n",
    "![Value Encoding](figures/value_encoding.svg)\n",
    "\n",
    "An encoded value has a manadatory first part describing the encoding and containing any structured data, called the primary part. It may also have an optional secondary part. The secondary part, if present, will be raw bytes which are usually compressed and/or encrypted in the same manner as the primary part.\n",
    "\n",
    "## Primary Part Decode\n",
    "\n",
    "The following code shows how to decode a value. In a nutshell, the process is:\n",
    "\n",
    "1. MessagePack decode the value, using the provided structure\n",
    "   definitions below. This first pass decodes the header; the\n",
    "   value's primary part remains encoded in the `e` field. In\n",
    "   following steps, the term *primary part* will refer to bytes\n",
    "   that are initially stored in `e` field and get passed through\n",
    "   the various decoding steps below.\n",
    "   \n",
    "2. If key `z` (crypt info) is provided, then the primary part\n",
    "   must be decrypted. Details on cipher setup and key identification follow\n",
    "   in later sections.\n",
    "   \n",
    "3. If key `c` (compression type) is provided, then use the appropriate\n",
    "   algorithm to decompress the primary part. Zstandard is the\n",
    "   default algorithm.\n",
    "\n",
    "4. The key `v` (version) is provided, it will indicate what version of structure is stored. Currently all LTFS-VOF structures are version zero so key `v` will not be present. If breaking changes are made to the format, this number will indicate which version is stored.\n",
    "   \n",
    "5. Now that the primary part's type is known (from TLV decode) and the\n",
    "   version is known (from prior step), use MessagePack to decode the\n",
    "   primary part into the appropriate data structure.\n",
    "   \n",
    "## Secondary Part Decode\n",
    "\n",
    "The secondary part has its own encoding parameters. Note that this field is a list to allow for multiple secondary parts. In its current form LTFS-VOF will only use one secondary part in a value.\n",
    "\n",
    "If key `s` is provided for secondary encoding parameters, the mandatory sub-key `l` will specify the encoded length of the secondary part. For all other secondary encoding parameters, they should be assumed to match the encoding of the primary part unless explicitly overridden in the secondary encoding structure.\n",
    "\n",
    "Whereas the primary part is always encoded with MessagePack, the secondary part will be raw bytes. In LTFS-VOF only data blocks will have a secondary part.\n",
    "\n",
    "## Encryption\n",
    "\n",
    "Values may be encrypted. If key `z` is present the parameters needed for decryption will be provided. These will include the algorithm (generally AES-256), nonce, and a data field which is used to identify the key. The format of the data field will vary depending on the key manager in use, but it will generally provide whatever metadata is required to retreive the key needed for decryption.\n",
    "\n",
    "## Compression\n",
    "\n",
    "Either or both parts may be compressed. If key `c` is present and non-zero, use Zstandard to decompress."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "742593ec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'value 1 header'\n",
      "b'value 1 data'\n",
      "b'value 2 header'\n",
      "b'value 2 data'\n",
      "b'value 3 header'\n",
      "b'value 3 data'\n"
     ]
    }
   ],
   "source": [
    "class TLV:\n",
    "    \"\"\"\n",
    "    TLV reader that reads a TLV from a stream, validates its integrity, and decodes the value.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, f: typing.BinaryIO):\n",
    "        \"\"\"\n",
    "        Creates a TLV from input stream f, leaving the stream positioned at the start of the next TLV.\n",
    "        :param f: any file-like stream.\n",
    "        \"\"\"\n",
    "        self.header = TlvHeader(f)\n",
    "        data = f.read(self.header.dlen)\n",
    "\n",
    "        if len(data) < self.header.dlen:\n",
    "            raise f'short data read: expected {self.header.dlen} bytes, got {len(data)}'\n",
    "\n",
    "        if self.header.dhash != xxh64(data).intdigest():\n",
    "            raise \"TLV data hash mismatch\"\n",
    "\n",
    "        self.__decode_value(data)\n",
    "\n",
    "    def __decode_value(self, data: bytes):\n",
    "        \"\"\"\n",
    "        Decode a value from self.data, setting the primary part as self.value\n",
    "        and the secondary part (if present) as self.secondary. self.value\n",
    "        will be a dict and self.secondary will be bytes or None.\n",
    "        \"\"\"\n",
    "        # Use streaming unpacker because extra data may be present\n",
    "        unpacker = msgpack.Unpacker(io.BytesIO(data), use_list=False)\n",
    "        val: dict = unpacker.unpack()\n",
    "\n",
    "        if 'z' in val:\n",
    "            # TODO: take apart z to obtain key properties, consult KMS for key, decrypt\n",
    "            raise NotImplementedError('encrypted values not supported')\n",
    "\n",
    "        # Decompress encoded value if compressed\n",
    "        if val.get('c') == 1:\n",
    "            val['e'] = zstd.decompress(val['e'])\n",
    "\n",
    "        self.value: dict = msgpack.unpackb(val['e'], use_list=False)\n",
    "        self.secondary: bytes = bytes(0)\n",
    "\n",
    "        try:\n",
    "            sec_enc = val['s'][0]  # Encoding specifier of secondary part\n",
    "            sec_len = sec_enc['l']  # Length of secondary part\n",
    "            self.secondary = data[len(data) - sec_len:]\n",
    "\n",
    "            # If secondary encoding specifies compression, or that key is missing and primary encoding specifies\n",
    "            # compression, then decompress the secondary value.\n",
    "            if sec_enc.get('c', val.get('c')) == 1:\n",
    "                self.secondary = zstd.decompress(self.secondary)\n",
    "        except IndexError:\n",
    "            pass  # no secondary part\n",
    "        except KeyError:\n",
    "            pass  # no secondary part\n",
    "\n",
    "    def __repr__(self):\n",
    "        return f'TLV (tag {self.header.tag})'\n",
    "\n",
    "\n",
    "with open('sample_data/3values.tlv', 'rb') as f:\n",
    "    for i in range(3):\n",
    "        tlv = TLV(f)\n",
    "        pprint(tlv.value)\n",
    "        pprint(tlv.secondary)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1174cd07",
   "metadata": {},
   "source": [
    "# Blocks and Pack Lists\n",
    "\n",
    "The values in LTFS Versioned Object Format may be of one of three types:\n",
    "\n",
    "1. Blocks, which store data.\n",
    "\n",
    "2. Pack list metadata, which describe how blocks are arranged into versions.\n",
    "\n",
    "3. Version metadata, which are described in a later section.\n",
    "\n",
    "This section describes blocks and pack lists. As data enters a LTFS-VOF system, it is split into slices (default 10MB in size) and each slice becomes a block. A version will have one or more blocks. In addition, a pack list is created which shows which ranges of a version are mapped to which ranges of each block.\n",
    "\n",
    "## Blocks\n",
    "\n",
    "Each stored block will have a primary part and secondary part. The primary part will only have key `I` which specifies the composite version ID this block belongs to. The secondary part will contain the raw bytes for the block.\n",
    "\n",
    "### Composite Version IDs\n",
    "\n",
    "A composite Version ID is a combination of the bucket name, object name, and a unique ULID for the version. The format is:\n",
    "\n",
    "    [26 byte version ULID]:bucketname/objectname\n",
    "\n",
    "The `bucketname` will be a bucket name that complies with AWS S3 bucket naming conventions. The `objectname` will be any S3 compliant object name. Note, when parsing, that an object name may contain slashes.\n",
    "\n",
    "The following sample code demonstrates parsing a composite version ID."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "faab0594",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "VersionID(bucket, object/name.txt, 7YGGZJ4YSFMYW6BQVHFKD5KKTV)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class VersionID:\n",
    "    \"\"\"\n",
    "    VersionID represents a LTFS-VOF composite version identifier.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, bucket: str, object: str, version: str):\n",
    "        self.bucket: str = bucket\n",
    "        self.object: str = object\n",
    "        self.version: str = version\n",
    "\n",
    "    def __repr__(self):\n",
    "        return f'VersionID({self.bucket}, {self.object}, {self.version})'\n",
    "\n",
    "    @classmethod\n",
    "    def from_str(cls, v_str: str) -> VersionID:\n",
    "        \"\"\"\n",
    "        Parse a version string into a VersionID.\n",
    "        \"\"\"\n",
    "        # First 26 characters is a ULID specifying the version\n",
    "        version = ULID.from_str(v_str[:26])\n",
    "        # Remaining characters (after a ':' separator) are bucket/object name\n",
    "        bucket, object = v_str[27:].split('/', maxsplit=1)\n",
    "        # Together these form the complete version identifier\n",
    "        return cls(bucket=bucket, object=object, version=version)\n",
    "\n",
    "    @classmethod\n",
    "    def from_dict(cls, v_dict: dict) -> VersionID:\n",
    "        \"\"\"\n",
    "        Convert dict form of version to VersionID.\n",
    "        \"\"\"\n",
    "        return cls(v_dict['b'], v_dict['o'], ULID.from_str(v_dict['v']))\n",
    "\n",
    "VersionID.from_str('7YGGZJ4YSFMYW6BQVHFKD5KKTV:bucket/object/name.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "131c65bb",
   "metadata": {},
   "source": [
    "### Block Parsing\n",
    "\n",
    "The rest of the block parsing is straightforward:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "86f9484e",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Block:\n",
    "    \"\"\"\n",
    "    Block represents a single block of object data.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, tlv: TLV):\n",
    "        self.versionid: VersionID = VersionID.from_str(tlv.value['I'])\n",
    "        self.data: bytes = tlv.secondary\n",
    "\n",
    "    def __repr__(self):\n",
    "        return f'Block({self.versionid}, {len(self.data)} bytes)'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b9a1531",
   "metadata": {},
   "source": [
    "## Pack Lists\n",
    "\n",
    "In the following figure, consider a version which is split into five blocks, and those blocks are stored across two packs. Version `V1` would require a pack list with two entries, the first referring to a range of `Pack A` which contains its first two blocks, the second referring to `Pack B` which contains the other three blocks.\n",
    "\n",
    "![Blocks and Pack Lists](figures/pack_list.svg)\n",
    "\n",
    "### Source and Pack Ranges\n",
    "\n",
    "Each entry contains a source range which refers to the version's data without any compression or encryption. It also contains a pack range which refers to where that data is stored, including compression and encryption. When reassembling a version from pack files, the pack range will include the TLV and value header. Thus the decode process should seek to the start of the pack range and expect to decode one or more blocks.\n",
    "\n",
    "### Block and Source Length Lists\n",
    "\n",
    "Each pack list entry also contains two optional lists for block lengths and source lengths.\n",
    "The block lengths list specifies the stored length of each block except the last one; the length of the last block may be inferred from the pack length range minus the other stored block lengths. The source lengths list will usually be empty. If the version was created with a regular stride (e.g. 10MB in this example) then all blocks will have the same source length, except the last which may be shorter. If blocks were created on an irregular stride, then the source lengths list will contain deltas from the version's normal stride. (Again, this is usually zero.)\n",
    "\n",
    "When reassembling whole versions, the block and source lengths lists do not need to be used. They are only required for efficient recall of partial objects.\n",
    "\n",
    "### Pack List Parsing\n",
    "\n",
    "The following code shows how to handle encoded pack lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4968d2d7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Block(VersionID(bucket, object, 7YF1QJW74PNYV552JB3YPAJJX1), 12 bytes)\n",
      "Block(VersionID(bucket, object, 7YF1QJW74PNYV552JB3YPAJJX1), 12 bytes)\n",
      "Block(VersionID(bucket, object, 7YF1QJW74PNYV552JB3YPAJJX1), 12 bytes)\n",
      "PackList(VersionID(bucket, object, 7YF1QJW74PNYV552JB3YPAJJX1), 1 PackEntry)\n"
     ]
    }
   ],
   "source": [
    "class Range:\n",
    "    \"\"\"\n",
    "    Range simply stores a start offset and length.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, r_dict: dict):\n",
    "        self.start: int = r_dict.get('s', 0)\n",
    "        self.len: int = r_dict.get('l', 0)\n",
    "\n",
    "    def __repr__(self):\n",
    "        return f'Range(start {self.start}, len {self.len})'\n",
    "\n",
    "\n",
    "class PackEntry:\n",
    "    def __init__(self, pe_dict: dict):\n",
    "        self.packid: ULID = ULID.from_str(pe_dict['p'])\n",
    "        self.sourcerange: Range = Range(pe_dict['o'])\n",
    "        self.packrange: Range = Range(pe_dict['t'])\n",
    "        self.blocklens: list[int] = pe_dict.get('E', [])\n",
    "        self.sourcelens: list[int] = pe_dict.get('N', [])\n",
    "\n",
    "    def __repr__(self):\n",
    "        return f'PackEntry({self.packid}, src {self.sourcerange}, pack {self.packrange})'\n",
    "\n",
    "\n",
    "class Packs(list):\n",
    "    \"\"\"\n",
    "    List type defined so that can easily tell the difference between a pack reference and a list[PackEntry].\n",
    "    \"\"\"\n",
    "\n",
    "    def __repr__(self):\n",
    "        return f'Packs({super().__repr__()})'\n",
    "\n",
    "\n",
    "class PackList:\n",
    "    \"\"\"\n",
    "    PackList is a stored map of one version's list of packs storing that version's data.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, tlv: TLV):\n",
    "        self.versionid: VersionID = VersionID.from_str(tlv.value['I'])\n",
    "        self.uploadid: str = tlv.value.get('U', '')\n",
    "        self.packs: Packs = Packs([PackEntry(pe_dict) for pe_dict in tlv.value.get('P', [])])\n",
    "\n",
    "    def __repr__(self):\n",
    "        return f'PackList({self.versionid}, {len(self.packs)} PackEntry)'\n",
    "\n",
    "\n",
    "def data_pack_reader(f: typing.BinaryIO):\n",
    "    \"\"\"\n",
    "    Scan a data pack file, yielding each entry as a Block or PackList tuple.\n",
    "    :param f: TLV-encoded data pack file\n",
    "    \"\"\"\n",
    "    handlers = {b'bk': Block, b'ol': PackList}\n",
    "\n",
    "    while True:\n",
    "        try:\n",
    "            tlv = TLV(f)\n",
    "            if tlv.header.tag not in handlers:\n",
    "                raise RuntimeError(f'unknown tag {tlv.header.tag}; no handler registered')\n",
    "\n",
    "            yield handlers[tlv.header.tag](tlv)\n",
    "        except EOFError:\n",
    "            break\n",
    "\n",
    "\n",
    "with open('sample_data/3blocks.blk', 'rb') as f:\n",
    "    for entry in data_pack_reader(f):\n",
    "        pprint(entry)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54dac783-a262-4a6c-a685-1b7e9e12df29",
   "metadata": {},
   "source": [
    "# Versions\n",
    "\n",
    "The last top-level structure in LTFS-VOF is the version record. As mentioned earlier, a version ID is a composite of a ULID, the bucket name, and object name. The version ULIDs sort in order of their creation time with millisecond resolution. Version records will be stored in their own pack files, so the complete history of a bucket can be assembled by reading all the version packs.\n",
    "\n",
    "When reading the version packs, each unique combination of bucket name and object name (that is, each object) will have one or more version records. They should be sorted by version ID to get the correct order of events. The following S3 event types will be represented:\n",
    "\n",
    "1. _Put object_ and _complete multipart upload_ will both create version records with most of their fields filled in. The _delete_ flag will be absent.\n",
    "\n",
    "2. _Delete object_ will create a version with the _delete_ flag set. This is called a delete marker.\n",
    "\n",
    "3. _Delete object_ with a version ID specified will create a special _version delete_ object. This is different from a delete marker, which indicates that the object has been removed. A version delete indicates deletion of a specific version only. The version delete object is required because pack files are append-only and immutable once finalized.\n",
    "\n",
    "The TLV tag for a version record is `vr`. The tag for a version delete record is `vd`.\n",
    "\n",
    "## Version Clones & Data References\n",
    "\n",
    "A copy of a version's data is called a clone. The version records will contain a list of clones under key `p`. The clone structure includes the pool identifier--one of which will be the tape pool--and an encoded data field. The data field encoding may take multiple forms depending on the type of pool. For a tape-based pool, MessagePack is used to encode a structure with the pack list (if small) or a reference to where the pack list is stored (if large). The included code handles both scenarios.\n",
    "\n",
    "## Embedded Version Data\n",
    "\n",
    "If the data of a version is very small (hundreds of bytes) it will be embedded in the version record directly and no data packs or pack lists will be created. The clones list may be empty in this scenario."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9a5d626-77c5-449d-8656-d0e31674e195",
   "metadata": {},
   "source": [
    "## Version Decoding Code\n",
    "\n",
    "Code for parsing version records follows. In addition, the generic `ltfsvof_reader` function can be used with any stream of LTFS-VOF encoded data, both data and version packs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "792e5462-78b7-43e1-83d4-e0f2aa160d6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ACL:\n",
    "    \"\"\"\n",
    "    ACL represents a single ACL entry.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, acl_dict: dict):\n",
    "        self.idtype: int = acl_dict['t']  # 0: user, 1: group\n",
    "        self.id: str = acl_dict['i']  # user/group ID\n",
    "        self.permissions: int = acl_dict['p']  # 1: read, 2: write, 4: read acl, 8: write acl\n",
    "\n",
    "    def __repr__(self):\n",
    "        return f'ACL({self.idtype}, {self.id}, {self.permissions})'\n",
    "\n",
    "\n",
    "class CryptData:\n",
    "    \"\"\"\n",
    "    CryptData represents the encryption metadata for an object.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, cd_dict: dict):\n",
    "        self.type: int = cd_dict['x']  # 0: none, 1: customer managed key, 2: S3 managed key\n",
    "        self.datakey: bytes = cd_dict['k']  # encrypted data key or MD5 of customer key\n",
    "        self.extra: bytes = cd_dict['e']  # extra string data\n",
    "\n",
    "    def __repr__(self):\n",
    "        return f'CryptData({self.type}, {self.datakey}, {self.extra})'\n",
    "\n",
    "\n",
    "class PackReference:\n",
    "    \"\"\"\n",
    "    PackReference represents a reference to a pack.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, pr_dict: dict):\n",
    "        self.pack: str = pr_dict['k']\n",
    "        self.packrange: Range = Range(pr_dict['r'])\n",
    "\n",
    "    def __repr__(self):\n",
    "        return f'PackReference({self.pack}, {self.packrange})'\n",
    "\n",
    "\n",
    "class Clone:\n",
    "    \"\"\"\n",
    "    Clone represents a single clone of an object.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, clone_dict: dict):\n",
    "        data = clone_dict['l']\n",
    "        try:\n",
    "            # see if this is a pack list\n",
    "            ref = msgpack.unpackb(data)\n",
    "            if 'p' in ref:\n",
    "                data = Packs([PackEntry(p) for p in ref['p']])\n",
    "            elif 'R' in ref:\n",
    "                data = PackReference(ref['R'])\n",
    "        except msgpack.FormatError:\n",
    "            # must not be msgpack, so leave data field as-is\n",
    "            pass\n",
    "\n",
    "        self.pool: str = clone_dict['p']\n",
    "        self.data: Packs | PackReference | bytes = data\n",
    "        self.flags: int = clone_dict.get('f', 0)\n",
    "        self.blocklen: int = clone_dict['B']\n",
    "        self.len: int = clone_dict['s']\n",
    "\n",
    "    def __repr__(self):\n",
    "        return f'Clone({self.pool}, {self.data}, {self.flags}, {self.blocklen}, {self.len})'\n",
    "\n",
    "\n",
    "class Version:\n",
    "    \"\"\"\n",
    "    Version represents a single version of an object.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, tlv: TLV):\n",
    "        val = tlv.value\n",
    "        self.versionid: VersionID = VersionID.from_dict(val)\n",
    "        self.owner: str = val.get('w', '')\n",
    "        self.acls: list[ACL] = [ACL(a) for a in val.get('A', [])]\n",
    "        self.len: int = val.get('l', 0)\n",
    "        self.etag: str = val.get('e', '')\n",
    "        self.deletemarker: bool = val.get('d', False)\n",
    "        self.nullversion: bool = val.get('N', False)\n",
    "        self.crypt: Optional[CryptData] = CryptData(val['c']) if 'c' in val else None\n",
    "        self.clones: list[Clone] = [Clone(c) for c in val.get('p', [])]\n",
    "        self.metadata: dict[str, str] = val.get('s', {})\n",
    "        self.usermetadata: dict[str, str] = val.get('m', {})\n",
    "        self.legalhold: bool = val.get('h', False)\n",
    "        self.data: bytes = val.get('D', b'')\n",
    "\n",
    "    def __repr__(self):\n",
    "        return f'Version(id {self.versionid})'\n",
    "\n",
    "\n",
    "class VersionDelete:\n",
    "    \"\"\"\n",
    "    VersionDelete represents the deletion of a single verison.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, tlv: TLV):\n",
    "        # TODO: update for version delete structure once tags are known\n",
    "        self.versionid: VersionID = VersionID.from_dict(tlv.value)\n",
    "        self.deleteid: VersionID = VersionID.from_str(tlv.value['???'])\n",
    "\n",
    "    def __repr__(self):\n",
    "        return f'VersionDelete(id {self.versionid}, deleteid {self.deleteid})'\n",
    "\n",
    "\n",
    "def ltfsvof_reader(f: typing.BinaryIO):\n",
    "    \"\"\"\n",
    "    Scan any LTFS-VOF file, yielding each entry as tuple of the appropriate type.\n",
    "    :param f: file-like stream with TLV-encoded blocks or versions\n",
    "    \"\"\"\n",
    "    handlers = {b'bk': Block, b'ol': PackList, b'vm': Version, b'vd': VersionDelete}\n",
    "\n",
    "    while True:\n",
    "        try:\n",
    "            tlv = TLV(f)\n",
    "            if tlv.header.tag not in handlers:\n",
    "                raise RuntimeError(f'unknown tag {tlv.header.tag}; no handler registered')\n",
    "\n",
    "            yield handlers[tlv.header.tag](tlv)\n",
    "        except EOFError:\n",
    "            break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d56a777c",
   "metadata": {},
   "source": [
    "# Appendix: Tests\n",
    "\n",
    "This section is implements basic tests for the code in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "70477a4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "class TlvTests(unittest.TestCase):\n",
    "    def setUp(self) -> None:\n",
    "        self.tlv_data = base64.b64decode(\"iVRMVg0KGgoAAAAAAAAADuM9tfSfjss2AEMhCAAAuxRkYXRhIGRhdGEgZGF0YQ==\")\n",
    "\n",
    "    def test_read_tlv_header(self):\n",
    "        with io.BytesIO(self.tlv_data) as f:\n",
    "            header = TlvHeader(f)\n",
    "            self.assertEqual(header.magic, b'\\x89TLV\\x0d\\x0a\\x1a\\x0a')\n",
    "            self.assertEqual(header.dlen, 14)\n",
    "            self.assertEqual(header.hashtype, 8)\n",
    "            self.assertEqual(header.version, 0)\n",
    "            self.assertEqual(header.tag, b'C!')\n",
    "\n",
    "    def test_read_tlv(self):\n",
    "        with io.BytesIO(self.tlv_data) as f:\n",
    "            tlv = TlvSimple(f)\n",
    "            self.assertEqual(tlv.data, b'data data data')\n",
    "\n",
    "    def test_3tlv(self):\n",
    "        with open('sample_data/3simple.tlv', 'rb') as f:\n",
    "            tlv = TlvSimple(f)\n",
    "            self.assertEqual(tlv.header.tag, b'bk')\n",
    "            self.assertEqual(tlv.data, b'data 1')\n",
    "            tlv = TlvSimple(f)\n",
    "            self.assertEqual(tlv.header.tag, b'bk')\n",
    "            self.assertEqual(tlv.data, b'data 2')\n",
    "            tlv = TlvSimple(f)\n",
    "            self.assertEqual(tlv.header.tag, b'bk')\n",
    "            self.assertEqual(tlv.data, b'data 3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "09a2d51c",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ValueTests(unittest.TestCase):\n",
    "    def test_value_decode(self):\n",
    "        with open('sample_data/3values.tlv', 'rb') as f:\n",
    "            for i in range(3):\n",
    "                tlv = TLV(f)\n",
    "                self.assertEqual(tlv.value, bytes(f'value {i + 1} header', 'utf-8'))\n",
    "                self.assertEqual(tlv.secondary, bytes(f'value {i + 1} data', 'utf-8'))\n",
    "\n",
    "    def test_compressed_value_decode(self):\n",
    "        with open('sample_data/compressed_value.tlv', 'rb') as f:\n",
    "            tlv = TLV(f)\n",
    "            self.assertEqual(tlv.value, b'header header header header header header header header')\n",
    "            self.assertEqual(tlv.secondary, b'data data data data data data data data data data data')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5ea6706e-2fd4-4ddb-b3bd-cbf14beaa9cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "class BlockTests(unittest.TestCase):\n",
    "    def test_read_block(self):\n",
    "        blocks = []\n",
    "        packlist = None\n",
    "\n",
    "        # Sample file contains 3 simple blocks and a packlist\n",
    "        with open('sample_data/3blocks.blk', 'rb') as f:\n",
    "            for entry in data_pack_reader(f):\n",
    "                pprint(entry)\n",
    "                if isinstance(entry, Block):\n",
    "                    blocks.append(entry)\n",
    "                else:\n",
    "                    packlist = entry\n",
    "\n",
    "        self.assertEqual(len(blocks), 3)\n",
    "        self.assertEqual(blocks[0].data, b'block 1 data')\n",
    "        self.assertEqual(blocks[1].data, b'block 2 data')\n",
    "        self.assertEqual(blocks[2].data, b'block 3 data')\n",
    "        self.assertEqual(1, len(packlist.packs))\n",
    "        self.assertEqual(0, packlist.packs[0].sourcerange.start)\n",
    "        self.assertEqual(3 * len(b'block x data'), packlist.packs[0].sourcerange.len)\n",
    "\n",
    "        # Furthermore, we can read individual blocks based on the packlist\n",
    "        with open('sample_data/3blocks.blk', 'rb') as f:\n",
    "            packentry = packlist.packs[0]\n",
    "\n",
    "            # Block 1 will be at the start of the pack range\n",
    "            f.seek(packentry.sourcerange.start)\n",
    "            block1 = Block(TLV(f))\n",
    "            self.assertEqual(block1.data, b'block 1 data')\n",
    "\n",
    "            # Block 2 will be start of pack range plus the first blocklen\n",
    "            f.seek(packentry.sourcerange.start + packentry.blocklens[0])\n",
    "            block2 = Block(TLV(f))\n",
    "            self.assertEqual(block2.data, b'block 2 data')\n",
    "\n",
    "            # Block 3 will be start of pack range plus the first two blocklens\n",
    "            f.seek(packentry.sourcerange.start + packentry.blocklens[0] + packentry.blocklens[1])\n",
    "            block3 = Block(TLV(f))\n",
    "            self.assertEqual(block3.data, b'block 3 data')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "2655ad60-011d-49e7-92b2-70b55bfdf0df",
   "metadata": {},
   "outputs": [],
   "source": [
    "class VersionTests(unittest.TestCase):\n",
    "    def test_read_version(self):\n",
    "        with open('sample_data/7YF1JH4PP45BYWK21Y7H0YHFYN.ver', 'rb') as f:\n",
    "            for entry in ltfsvof_reader(f):\n",
    "                # pprint(entry)\n",
    "\n",
    "                if isinstance(entry, Version):\n",
    "                    v: Version = entry\n",
    "                    if isinstance(v.clones[0].data, Packs):\n",
    "                        print('version has embedded packlist')\n",
    "                        pprint(v.clones[0].data)\n",
    "                    elif isinstance(v.clones[0].data, PackReference):\n",
    "                        pr: PackReference = v.clones[0].data\n",
    "                        print(f'need to load packlist from {pr.pack}')\n",
    "                        with open(f'sample_data/{pr.pack}.blk', 'rb') as f2:\n",
    "                            f2.seek(pr.packrange.start)\n",
    "                            packlist = PackList(TLV(f2))\n",
    "                            pprint(packlist.packs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a1e0a135",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "test_read_block (__main__.BlockTests) ... ok\n",
      "test_3tlv (__main__.TlvTests) ... ok\n",
      "test_read_tlv (__main__.TlvTests) ... ok\n",
      "test_read_tlv_header (__main__.TlvTests) ... ok\n",
      "test_compressed_value_decode (__main__.ValueTests) ... ok\n",
      "test_value_decode (__main__.ValueTests) ... ok\n",
      "test_read_version (__main__.VersionTests) ... ok\n",
      "\n",
      "----------------------------------------------------------------------\n",
      "Ran 7 tests in 0.005s\n",
      "\n",
      "OK\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Block(VersionID(bucket, object, 7YF1QJW74PNYV552JB3YPAJJX1), 12 bytes)\n",
      "Block(VersionID(bucket, object, 7YF1QJW74PNYV552JB3YPAJJX1), 12 bytes)\n",
      "Block(VersionID(bucket, object, 7YF1QJW74PNYV552JB3YPAJJX1), 12 bytes)\n",
      "PackList(VersionID(bucket, object, 7YF1QJW74PNYV552JB3YPAJJX1), 1 PackEntry)\n",
      "version has embedded packlist\n",
      "Packs([PackEntry(7YF1JH4PP45BYWK21Y7H4QPHAT, src Range(start 0, len 36), pack Range(start 0, len 303))])\n",
      "need to load packlist from 7YF1JH4PP45BYWK21Y7H4QPHAT\n",
      "Packs([PackEntry(7YF1JH4PP45BYWK21Y7H4QPHAT, src Range(start 0, len 36), pack Range(start 0, len 303))])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<unittest.main.TestProgram at 0x103b8ae60>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unittest.main(argv=[''], verbosity=2, exit=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e552170a-0b10-4e30-a269-1534b150d5dd",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "toc-autonumbering": false,
  "toc-showmarkdowntxt": false,
  "toc-showtags": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}