| URL: | http://www.pytables.org/ |
|---|
PyTables is a package for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data.
It is built on top of the HDF5 library and the NumPy package. It features an object-oriented interface that, combined with C extensions for the performance-critical parts of the code (generated using Cython), makes it a fast, yet extremely easy to use tool for interactively save and retrieve very large amounts of data. One important feature of PyTables is that it optimizes memory and disk resources so that they take much less space (between a factor 3 to 5, and more if the data is compressible) than other solutions, like for example, relational or object oriented databases.
PyTables is not designed to work as a relational database replacement, but rather as a teammate. If you want to work with large datasets of multidimensional data (for example, for multidimensional analysis), or just provide a categorized structure for some portions of your cluttered RDBS, then give PyTables a try. It works well for storing data from data acquisition systems (DAS), simulation software, network data monitoring systems (for example, traffic measurements of IP packets on routers), or as a centralized repository for system logs, to name only a few possible uses.
A table is defined as a collection of records whose values are stored in fixed-length fields. All records have the same structure and all values in each field have the same data type. The terms "fixed-length" and strict "data types" seems to be quite a strange requirement for an interpreted language like Python, but they serve a useful function if the goal is to save very large quantities of data (such as is generated by many scientific applications, for example) in an efficient manner that reduces demand on CPU time and I/O.
There are other useful objects like arrays, enlargeable arrays or
variable length arrays that can cope with different missions on your
project. Also, quite a bit of effort has been invested to make
browsing the hierarchical data structure a pleasant
experience. PyTables implements a few easy-to-use methods for
browsing. See the documentation (located in the doc/ directory)
for more details.
One of the principal objectives of PyTables is to be user-friendly. To that end, special Python features like generators, slots and metaclasses in new-brand classes have been used. In addition, iterators has been implemented were context was appropriate so as to enable the interactive work to be as productive as possible. For these reasons, you will need to use Python 2.6 or higher to take advantage of PyTables.
We are using Linux on top of Intel32 and Intel64 boxes as the main development platforms, but PyTables should be easy to compile/install on other UNIX or Windows machines. Nonetheless, caveat emptor: more testing is needed to achieve complete portability, we'd appreciate input on how it compiles and installs on your platform.
To compile PyTables you will need, at least, a recent version of HDF5 (C flavor) library, the Zlib compression library and the NumPy and Numexpr packages. Besides, if you want to take advantage of the LZO and bzip2 compression libraries support you will also need recent versions of them. LZO and bzip2 compression libraries are, however, optional.
Make sure you have HDF5 version 1.8.4 or above.
On OSX you can install HDF5 using Homebrew:
$ brew tap homebrew/science $ brew install hdf5
On ubuntu:
$ sudo apt-get install libhdf5-serial-dev
If you have the HDF5 library in some non-standard location (that is: where the compiler and the linker can't find it) you can use the environment variable HDF5_DIR to specify its location. See the manual for more details.
Make sure your python installation is in good health, that is you have the package installer pip and it works ok. Check the Python Packaging User Guide for further instructions.
Optionally, consider to install the LZO compression library and/or the bzip2 compression library.
Install!:
$ pip install tables
To run the test suite run:
$ python -m tables.tests.test_all
If there is some test that does not pass, please send the complete output for tests back to us.
Enjoy data! -- The PyTables Team