Working with Serialised Data

The Bitcoin protocol is formalised at the byte level, where public keys, transactions and network messages follow specific serialisation formats in order to be communicated over the wire.

A brief overview of how serialised bytes are handled in Libbitcoin may be helpful for beginners during the study of Libbitcoin.

Fixed-length Byte Container

We will use the Bitcoin public key point to illustrate a fixed-length byte container in Libbitcoin. The following compressed public key is of fixed length 33-Bytes: The first byte (02/03) indicates whether the associated y-coordinate is even or odd. The following 32-Bytes represent the x-coordinate of the public key.

Example: Compressed Public Key

// 33-Bytes: Compressed public key.
0228026f91e1c97db3f6453262484ef5f69f71d89474f10926aae24d3c3eeb5f00

// 1-Byte: Even or Odd y-coordinate.
02
// 32-Bytes: x-coordinate.
28026f91e1c97db3f6453262484ef5f69f71d89474f10926aae24d3c3eeb5f00

Since a compressed public key is always of length 33-bytes, a standard fixed-length byte container is used to represent it.

// Declare a compressed public key.
ec_compressed my_pubkey;
// It will have a length of 33 single-byte elements.
std::cout << my_pubkey.size() << std::endl; // Prints 33.

The compressed public key class is a type alias for a byte array, which in turn is generated by the standard array class template with a single byte unsigned integer parameter.

// A compressed public key is a byte array of 33 bytes.
constexpr size_t ec_compressed_size = 33u;
using ec_compressed = byte_array<ec_compressed_size>;

// A byte array is an array of single-byte unsigned integers.
template <size_t Size>
using byte_array = std::array<uint8_t, Size>;

// For the 33-byte compressed public keys, this compiles to an standard array
// of single-byte unsigned integers with 33 elements.
template<size_t 33u>
using byte_array = std::array<uint8_t, 33u>;

Other fixed-length classes in Libbitcoin such as private keys, signatures and hash digests for example are also based on the byte array type.

Dynamic-length Byte Container

There are also many data formats in Bitcoin which can vary in length. In such a case, the Libbitcoin data chunk type comes is used.

For example, there are a number of valid entropy sizes which can be represented by a mnemonic word list .

Example: Valid Entropy Sizes for Mnemonic Word Lists

// 128 bits
4c05bddb9a3b8347c23830e67e97a299

// 160 bits
4c05bddb9a3b8347c23830e67e97a29935c524c5

// 192 bits
4c05bddb9a3b8347c23830e67e97a29935c524c5a4fd9397

// 224 bits
4c05bddb9a3b8347c23830e67e97a29935c524c5a4fd939713eff634

// 256bits
4c05bddb9a3b8347c23830e67e97a29935c524c5a4fd939713eff6344bd1bff8

To create the entropy examples above we simply instantiate a data chunk of a certain byte length, before we fill it with a random value.

// Data chunks can be instantiated with arbitrary sizes
data_chunk my_entropy_16(16);      //128bits allocated
data_chunk my_entropy_32(32);      //256bits allocated

// Fill data chunk object with entropy
pseudo_random_fill(my_entropy_16); //128bit entropy
pseudo_random_fill(my_entropy_32); //256bit entropy

Looking a little closer, the data chunk is simply an alias of a standard byte vector.

using data_chunk = std::vector<uint8_t>;

Transferring Data between Byte Arrays and Byte Vectors

There are helper functions in Libbitcoin to copy byte data from a fixed-length byte array to a dynamic-length byte vector and back. For example, we may need to transfer bytes from a byte array to a data chunk so we can pass it to a function which expects a data chunk argument.

// Declare a byte array of length 16
constexpr size_t my_array_size = 16u;
byte_array<my_array_size> my_array;

// The pseudo random fill function expects a data_chunk type input.
// We cannot pass a parameter of type byte array.
pseudo_random_fill(my_array); // not ok.

// So we first copy the byte data into a data chunk
// ...with the to_chunk helper function.
data_chunk my_chunk = to_chunk(my_array);
pseudo_random_fill(my_chunk); // ok.

// We can copy the data back into a byte array with the to_array helper function.
byte_array<my_array_size> my_array2;
my_array2 = to_array<my_array_size>(my_chunk);

Passing Byte Containers as Arguments:

There are functions in Libbitcoin which accept both fixed and dynamic length byte containers as arguments.

However, as we observed in the previous example, there is no implicit type conversion between types byte arrays and data chunks, just as there is no implicit type conversion between standard arrays and vectors.

So Libbitcoin provides a convenient wrapper type for iterable byte containers: Both byte arrays and data chunks can be implicitly converted to the array slice class during compile time. A function which accepts a data slice type argument can now accept both fixed and dynamic byte containers. The hash160 implementation in Libbitcoin is such an example.

// Hash160 function signature.
short_hash bitcoin_short_hash(data_slice data);

byte_array<4u> my_array {{0, 1, 2, 3}};
data_chunk my_chunk {0, 1, 2, 3};

auto first_hash = bitcoin_short_hash(my_array);  //ok
auto second_hash = bitcoin_short_hash(my_chunk); //ok

std::cout << (first_hash == second_hash) << std::endl; //True

A closer look at the data slice reveals that it is a type alias of array_slice<uint8_t>, which has a single argument constructor used by the compiler for implicit conversion from other iterable byte containers.

Implicit conversion of a byte container to array slice:

// Array slice constructor.
// Container can be any iterable byte container type.
template <typename Container>
array_slice(const Container& container);

// For a byte_array of length Size, this compiles to:
array_slice(const byte_array<Size>& container);

// For a data_chunk, this compiles to:
array_slice(const data_chunk& container);

You can find the entire example code from this chapter here.