-
Notifications
You must be signed in to change notification settings - Fork 50
Developer Guide
<<toc>>
One of the most powerful aspects of pypes is its extensibility. The system was designed from the ground up, to be completely extendable through well defined entry points into the system. At the same time, we've worked hard to make the process painless and free from obfuscation. To that extent, pypes provides project templates that can be used to generate the directory structure, build scripts, and boiler-plate code necessary in creating your own components. This allows developers to focus explicitly on their custom functionality rather than the semantics of interfacing with the system.
A component operates on data objects referred to as "packets". These data objects are essentially JSON structures and pypes provides an interface to help abstract away the idiosyncrasies. Packets model data through key/value pairs. Each key or packet "field" provides a named attribute where users can store information about the data being carried by the packet. Packets can contain multi-valued fields (i.e., list of values) and values can themselves, be complex structures depending on your needs.
Packets also support meta-data at both the packet and field level. Users can attach meta-data to entire packets or to individual fields. This provides a simple mechanism for passing information about packets or fields to other components operating on the data. For instance, a packet field may contain information in a language that differs from that of the other packet fields. Using meta-data, users can attach language specific attributes to a field as a means of providing internal knowledge to other components.
Modeling packets as JSON objects gives the system performance advantages over XML based approaches. JSON is much more efficient in terms of accessing and manipulating content while still providing a standards based data exchange format. This makes interacting with other web services extremely easy.
Components themselves implement an interface. The interface abstracts the semantics of Stackless Python or more specifically, Stackless tasklets. A tasklet is a tiny, lightweight thread of execution often referred to as a "micro-thread". Under the hood, each component represents a stackless tasklet. We recommend reading Grant Oslon's Introduction to Concurrent Programming with Stackless Python for those interested in a detailed introduction to the benefits of Stackless Python.
Aside from providing an abstraction layer to Stackless tasklets, the component API provides methods that allow developers to expose runtime configuration parameters to their users. As a developer, this allows you to retrieve configuration data at runtime giving you the ability to better generalize your component to address a broader audience of users.
The following sections provide a gentle introduction to developing your own components. We think you'll find the process to be intuative and simple yet powerful and expressive.
todo
- Reverse Field - Simple example of a component that reverses the content stored in a configurable field
The Packet interface defines the set of operations available on the packet data objects. Below is a comprehensive list of methods along with descriptions and examples pertaining to each individual method.
Sets the attribute only if it does not already exist. If multi is True and value is a list, then the attribute will be set multivalued and each item in value will be a value for the attribute.
attr is the attribute to add, value is the value to assign the attribute and multi defines if the value should be treated as multiple values (defaults to False).
This method returns no value.
#!python
def example:
p = Packet()
p.add("firstname", "Matt")
# the next call will silently fail because firstname already exists
p.add("firstname", "Eric")Appends a value to an attribute. If the attribute is not already multivalued then it becomes multivaliued. If extend is True and value is a list, then each item in value becomes a value in the attribute.
attr is the attribute to append to, value is the value to append to the attribute, and extend defines if we extend the existing items , defaults to False
This method returns no value.
#!python
def example:
p = Packet()
# authors field does not exist, it will be created
p.append("authors", "Matt Weber")
# authors will become multivalued
p.append("authors", "Eric Gaumer")
# Shannon and Katie will be added as authors
p.append("authors", ["Shannon", "Katie"], extend=True)Creates a copy of the packet. If metas is True then meta information is also copied.
metas determines If we include metadata in the clone (defaults to True)
Returns a copy of the packet object with or without metadata.
#!python
def example:
p = Packet()
# clone without meta
p2 = p.clone(metas=False)
# clone with meta
p3 = p.clone()Delete an attribute. All metadata related to the attribute will be lost.
attr is the attribute to delete.
Returns True if success otherwise False.
#!python
def example:
p = Packet()
p.delete("authors")Deletes the specified metadata value.
meta is the meta key and attr is the attribute to delete metadata from (defaults to None)
Returns True on success otherwise False
#!python
def example:
p = Packet()
# delete packet level metadata
p.delete("index")
# delete attribute level metadata alternamte_name from the author attribute
p.delete("alternate_name", "author")Returns the attribute or the default value if it does not exist.
attr specifies the attribute to get and default specifies the default value to return (defaults to None). The method returns a string if single value, list if multivalue, default if the attribute does not exist.
#!python
def example:
p = Packet()
# gets the title attribute, returns None is it does not exist
p.get("title")
# gets the author attribute, returns an empty string if it does not exit
p.get("author", "")Used to retrieve the entire set of attributes for a packet.
Returns the packet attributes as a dictionary
#!python
def example:
p = Packet()
attrs = p.get_attributes()
Used to retrieve all attribute names for the packet
Returns the attribute names as a list
#!python
def example:
p = Packet()
for attr in p.get_attribute_names():
print "%s: %s" % (attr, p.get(attr, ""))
Retrieves the metadata value if it exists, otherwise the default value is returned.
meta is the meta key, attr is the attribute to get metadata from (defaults to None), and default is the default value if meta does not exist (defaults to None).
Returns metadata value if it exists, otherwise the default value is returned.
#!python
def example:
p = Packet()
# get metadata from the packet, return None if it does not exist
p.get_meta("index")
# get metadata from an attribute, return an empty string if it does not exist
p.get_meta("alternate_name", attr="author", default="")Used to retrieve all the meta-data for a given packet.
Returns tuple of metadata. The first item is the packet level metadata as a dictionary and the second item is the attribtue level metadata as a dictionary.
#!python
def example:
p = Packet()
packetmeta, attrmeta = p.get_metas()Used to retrieve the meta-data namespace for a given packet.
Returns tuple of metadata names. The first item is the packet metadata names as a list and the second item is the attribtue metadata names in a dictionary with the attribute name as the key, and a list as the value.
#!python
def example:
p = Packet()
packet_meta_names, attribute_meta_names = p.get_meta_names()
for packet_meta_name in packet_meta_names:
print "Packet metadata: %s" % packet_meta_name
for attribute_meta_name in attribute_meta_names:
print "Attribtue %s metadata: %s" % (attribute_meta_name,
[x for x in attribute_meta_names[attribute_meta_name]])Checks whether or not the packet contains the given attribute.
attr is the attribute to check for
True if the attribute exists otherwise False.
#!python
def example:
p = Packet()
if p.has("authors"):
# do something interesting
else:
passChecks if a given metadata value exists.
meta is the meta key to check for and attr is the attribute to check for the metadata (defaults to None)
Returns True if meta exists otherwise False.
#!python
def example:
p = Packet()
# check for packet level metadata
if not p.has_meta("index"):
# check for attribute level metadata
if p.has_meta("alternate_name", "author")
# do something interesting
else:
passTests a packet attribute to see whether or not it is multivalued.
attr is the attribute to check is multivalued.
Returns True if the attribute is multivalued otherwise False
#!python
def example:
p = Packet()
if p.is_multivalued("authors"):
print “authors has more than one value”Merges this packet with another packet. If metas is False, meta information is not merged.
other is the other packet to be merge with and metas determines whether or not to merge meta-data (defaults to False).
This method returns no value.
#!python
def example:
p = Packet()
p2 = Packet()
p3 = Packet()
# merge the packets not include metadata
p.merge(p2)
# merge including metadata
p.merge(p3, metas=True)Remove an item from the attribute. If the attribute is not multivalued, this is equilavent to a delete. If the attribute is multivalued, the item at the specified index will be removed.
attr is the attribute to remove an item from and index is the integer index of the item to remove (defaults to 0 which is the first item).
Returns True if success otherwise failed.
#!python
def example:
p = Packet()
p.set("authors", ["Matt Weber", "Eric Gaumer", "Katie", "Shannon"], multi=True)
# remove Katie
p.remove("authors", 2)
# remove Matt
p.remove("authors")Replaces an item in the attribute. If the attribute is not multivalued, this is equilavent to a set. If the attribute is multivalued, the item at the specified index will be replaced. The index position must already exist, or the call will fail.
attr is the attribute to replace an item from, value is the replacement value and index is the index of the item to replace (default 0, i.e., first item).
Returns True if successful otherwise False.
#!python
def example:
p = Packet()
p.set("authors", ["Matt Weber", "Eric Gaumer", "Katie", "Shannon"], multi=True)
# replace Katie
p.replace("authors", "Wendy", 2)
# replace Matt
p.replace("authors", "Thomas")Sets the attribute, overwriting any existing value(s). If multi is True and value is a list, the attribute will be set to multivalued and each item in value will be a value for the attribute. If meta is false, all metadata for the attribute will be lost.
attr specifies the attribute to set, value is the value to assign to the attribute and multi defines if the value should be treated as multivalued (defaults to False). keep_meta defines whether related metadata should be deleted or not (defaults to True).
This method returns no value.
#!python
def example
p = Packet()
# sets author, keeps any existing metadata attached to the author field
p.set("author", "Matt Weber")
# sets multiple authors
p.set("authors", ["Matt Weber", "Eric Gaumer"])
# sets author, erasing any existing metadata attached to the author field
p.set("author", "Eric Gaumer", keep_meta=False)Set a metadata value. If the metadata value already exists, it will be overwritten. To set metadata on an attribute, supply the optional attr parameter. This method will return False if the supplied attribute does not exist.
meta is the meta key, value is the meta value, and attr is the attribute to set metadata for (defaults to None)
Returns True if successful otherwise False.
#!python
def example:
p = Packet()
p.set("author", "Matt Weber")
# add packets level metadata
p.set_meta("index", False)
# add metadata to the author attribute
p.set_meta("alternate_name", "Matthew Weber", "author")Packet == operator
rhs is the other packet to compare
Returns True if the packets are equal otherwise False
#!python
def example:
p = Packet()
p2 = Packet()
if p == p2:
print "packets are equal"Packet iterator
Returns pairs of tuples.
#!python
def example:
p = Packet()
for attr, vals in p:
# do something interesting Packet != operator
rhs is the packet on the right hand side of the operator
Returns True if the packets are not equal otherwise False
#!python
def example:
p = Packet()
p2 = Packet()
if p != p2:
print "packets are not equal"