INTRODUCTION
============

This project contain extension steps for Calabash
(http://xmlcalabash.com/) that uses Daffodil
(https://opensource.ncsa.illinois.edu/confluence/display/DFDL/Daffodil%3A+Open+Source+DFDL)
to parse a file with a DFDL schema.


NEW XPROC STEPS
===============

This project introduces two new steps: dfdl:parse-file and dfdl:parse.


dfdl:parse-file
---------------

dfdl:parse-file parses a file into an XML Infoset, using Daffodil. The XProc 
signature for dfdl:parse-file looks like this:
    
   <declare-step type="dfdl:parse-file">
      <output port="result" />
      <option name="file" required="true" />   
      <option name="schema" required="true" />
      <option name="root" />          <!-- (QName) -->
   </declare-step>

The root option, if present, must be a QName that is passed to the
Daffodil parser as the distinguished root node of the DFDL Schema to
use when parsing.  The distinguished root node specifies how to start
parsing the input when there are multiple top-level elements in the
DFDL Schema.  When the root option is absent, Daffodil uses the first
top-level element as the distinguished root node (roughly equivalent
to the starting non-terminal of a grammar).


dfdl:parse
----------

dfdl:parse uses an input port instead of the file attribute.  

   <declare-step type="dfdl:parse">
      <input port="source" />   
      <output port="result" />
      <option name="schema" required="true" />
      <option name="root" />          <!-- (QName) -->
   </declare-step>

dfdl:parse expects XML on the input port as produced by the p:data
construct in XProc: an XML wrapper around text data with optional
'encoding', 'content-type', and 'charset' attributes. The text content
of the root element of the input is the input for the parser.
         
If the root element has an 'encoding' attribute with a value of
'base64', then the input is base64-decoded before being passed to
Daffodil.
         
Otherwise, if there is a 'charset' attribute or a charset parameter in
a 'content-type' attribute, then the text is encoded using that
character set before being passed to parser.
         
Otherwise, if no valid character set is specified, the text is encoded
to UTF-8 before being passed to the parser.


Differences between dfdl:parse-file and dfdl:parse
--------------------------------------------------
         
dfdl:parse-file is more efficient than dfdl:parse because the input
file is passed directly to Daffodil; in contrast, for dfdl:parse the
input is converted to XML to get it into the XProc pipeline and then
back to a binary stream of data for Daffodil.  For binary input, the
situation is even worse: the data is base64-encoded to get it into the
XProc pipeline, then base64-decoded before being passed to Daffodil.

On the other hand, dfdl:parse can be combined with other steps more 
easily in an XProc pipeline, as other XProc steps can preprocess the
input before dfdl:parse.


RUNNING FROM COMMAND-LINE
=========================

The daffodil-calabash extension currently only works with scala 2.10,
since that's what daffodil 0.13.0 is built with.  Scala 2.10 must be
be downloaded and installed on the PATH before the daffodil-calabash
extension can be used. See http://scala-lang.org/download/all.html to
download Scala 2.10.

The examples directory contains example XProc pipelines for CSV and
PCAP. The PCAP DFDL examples are from https://github.com/DFDLSchemas/PCAP.
Here's a taste:

<p:declare-step xmlns:p="http://www.w3.org/ns/xproc"
                xmlns:dfdl="urn:daffodil:calabash"
                version="1.0">
    <p:output port="result"/>
    <p:import href="../../etc/daffodil-library.xpl"/>
    <dfdl:parse-file file="simpleCSV" schema="csv.dfdl.xsd" 
        root="ex:file" xmlns:ex="http://example.com"/>
</p:declare-step>

Of course, the magic isn't in simply running Daffodil; the magic is in
manipulating the parsed data using XProc.

Running Calabash with the Daffodil extension is as easy as running
Calabash from the command-line with specific arguments.  The examples
below assume that Scala 2.10 is available on the PATH.

From Unix/Linux:
$ ./calabash-daffodil.sh examples/csv/csv-file-pipeline.xpl
(or, for a prettier version of the output)
$ ./calabash-daffodil.sh examples/csv/csv-file-pipeline.xpl | xmllint --format -

The PCAP example can also be run (generates fairly large output)
$ ./calabash-daffodil.sh examples/PCAP/pcap-file-pipeline.xpl

From Windows:
> scala -cp lib\*;lib_managed\*;target\scala-2.10\daffodil-calabash-extension_2.10-0.5.0.jar com.xmlcalabash.drivers.Main -c etc\calabash-config.xml examples\csv\csv-file-pipeline.xpl

OR like this for troubleshooting (Unix/Linux example):
$ scala -cp lib/*:lib_managed/*:target/scala-2.10/daffodil-calabash-extension_2.10-0.5.0.jar \
-Djava.util.logging.config.file=etc/logging.properties \
com.xmlcalabash.drivers.Main -D -c etc/calabash-config.xml \
examples/csv/csv-file-pipeline.xpl

Important things to notice in the command-line: 
* This command-line executes Calabash with a custom classpath and
  configuration. 
* The extension and all its dependencies are added to the classpath.
* A configuration file is used to tie the dfdl:parse XProc step to the
  extension code. 


ACKNOWLEDGEMENTS
Thanks for Florent Georges for his very helpful blog post at 
http://fgeorges.blogspot.com/2011/09/writing-extension-step-for-calabash-to.html.


NOTICE 
This software was produced for the U. S. Government 
under Contract No. W15P7T-13-C-A802, and is 
subject to the Rights in Noncommercial Computer Software
and Noncommercial Computer Software Documentation 
Clause 252.227-7014 (FEB 2012)

Copyright (C) 2013-2014 The MITRE Corporation, All Rights Reserved.

Permission is hereby granted, free of charge, to any person
obtaining a copy of this software and associated documentation
files (the "Software"), to deal with the Software without
restriction, including without limitation the rights to use, copy,
modify, merge, publish, distribute, sublicense, and/or sell copies
of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

 1. Redistributions of source code must retain the above copyright
    notice, this list of conditions and the following disclaimers.

 2. Redistributions in binary form must reproduce the above
    copyright notice, this list of conditions and the following
    disclaimers in the documentation and/or other materials
    provided with the distribution.

 3. Neither the names of The MITRE Corporation, nor the names of
    its contributors may be used to endorse or promote products
    derived from this Software without specific prior written
    permission.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE CONTRIBUTORS OR COPYRIGHT
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
DEALINGS WITH THE SOFTWARE.


See the NOTICE.txt file in src/main/resources/META-INF for a list of 
libraries that are distributed with their own license terms.