<?xml version="1.0" encoding="iso-8859-1" standalone="no" ?> 
<!DOCTYPE document SYSTEM "rml_1_0.dtd"> 
<document filename="PyRXP_Documentation.pdf"> 
<!-- Last Modified:             $Date: 2002/04/29 13:54:15 $ -->
<!-- Document Version Number:   $Revision: 1.1 $            -->

<template pageSize="(595, 842)" 
        leftMargin="72" 
        showBoundary="0"
        title="PyRXP User Documentation"
        author="Reportlab Inc"
        >

    <pageTemplate id="cover">
    <pageGraphics>
        <setFont name="Times-Bold" size="48"/>
        <drawCentredString x="297" y="516">PyRXP 0.7</drawCentredString>
        <setFont name="Times-Bold" size="12"/>
        <drawCentredString x="297" y="414">User Documentation</drawCentredString>

        <lines>
        72 132 523 132
        </lines>

        <image file="replogo.gif" x="155" y="626"/>

        <setFont name="Times-Roman" size="10"/>
        <drawString x="72" y="110">Lombard Business Park</drawString>
        <drawString x="72" y="98">8 Lombard Road</drawString>
        <drawString x="72" y="86">Wimbledon</drawString>
        <drawString x="72" y="74">London, ENGLAND SW19 3TZ</drawString>

        <drawRightString x="523" y="110">103 Bayard Street</drawRightString>
        <drawRightString x="523" y="98">New Brunswick</drawRightString>
        <drawRightString x="523" y="86">New Jersey, 08901</drawRightString>
        <drawRightString x="523" y="74">USA</drawRightString>
    </pageGraphics>
    <frame id="first" x1="72" y1="72" width="451" height="698"/>
    </pageTemplate>

    <pageTemplate id="contents">
    <pageGraphics>
    </pageGraphics>
    <frame id="first" x1="72" y1="72" width="451" height="698"/>
    </pageTemplate>

    <pageTemplate id="inside">
    <pageGraphics>
        <lines>
        72 796 523 796
        </lines>
        <setFont name="Times-Roman" size="10"/>
        <drawString x="72" y="808">PyRXP User Documentation</drawString>
        <drawRightString x="523" y="808">$Revision: 1.1 $</drawRightString>
        <drawCentredString x="297.5" y="60">Page <pageNumber/></drawCentredString>
    </pageGraphics>
    <frame id="first" x1="72" y1="72" width="451" height="698"/>
    </pageTemplate>

</template>

<stylesheet>
    <initialize>
            <alias id="style.normal" value="style.Normal"/>
    </initialize>

    <paraStyle name="style.h1"
    fontName="Times-Bold"
    fontSize="20"
    leading="36"
    />

    <paraStyle name="style.h2"
    fontName="Times-Bold"
    fontSize="20"
    leading="36"
    />

    <paraStyle name="style.h3"
    fontName="Times-Bold"
    fontSize="12"
    leading="15"
    />

    <paraStyle name="contents"
    fontName="Times-Bold"
    fontSize="12"
    />

    <paraStyle name="contents2"
    fontName="Times-Roman"
    fontSize="12"
    leftIndent="20"
    />

    <paraStyle name="contents2a"
    fontName="Times-Roman"
    fontSize="12"
    leftIndent="20"
    spaceAfter="12"
    />

    <paraStyle name="contents2h"
    fontName="Times-Bold"
    fontSize="12"
    leftIndent="20"
    />
    
    <paraStyle name="contents3"
    fontName="Times-Roman"
    fontSize="12"
    leftIndent="40"
    />

    <paraStyle name="contents3a"
    fontName="Times-Roman"
    fontSize="12"
    leftIndent="40"
    spaceAfter="12"
    />

    <paraStyle name="contents3h"
    fontName="Times-Bold"
    fontSize="12"
    leftIndent="40"
    />

    <paraStyle name="style.Normal"
    fontName="Times-Roman"
    fontSize="10"
    leading="12"
    spaceAfter="12"
    firstLineIndent="0"
    />

    <paraStyle name="programListing"
    parent="style.Normal"
    fontName="Courier-Bold"
    fontSize="8"
    leading="10"
    spaceBefore="2"
    spaceAfter="10"
    textColor="darkblue"
    leftIndent="1.2cm"
    />

    <paraStyle name="programListingConventionLabel"
    parent="programListing"
    spaceAfter="0"
    leftIndent="0"
    />
    
    <paraStyle name="bulletList" 
    parent="style.Normal" 
    bulletFontName="ZapfDingbats"
    bulletFontSize="5"
    bulletIndent="20"
    leftIndent="35"
    spaceAfter="6"
    />

    <paraStyle name="bulletNospace" 
    parent="bulletList" 
    spaceAfter="0"
    />
    
    <paraStyle name="nospace" 
    spaceAfter="0"
    parent="style.Normal" 
    />
    
    <paraStyle name="figureCaption" 
    spaceBefore="0"
    spaceAfter="15"
    alignment="CENTER"
    fontSize="8"
    fontName="Times-Italic"
    />
    
    <paraStyle name="paddingTableStyle"
    fontName="Helvetica-BoldOblique"
    fontSize="10"
    textColor="white"
    alignment="CENTER"
    />

    <blockTableStyle id="plainTable">
        <blockFont name="Times-Bold" size="10" start="0,0" stop="-1,0"/>
        <blockFont name="Courier" size="8" start="0,1" stop="-1,-1"/>
        <blockTextColor colorName="black"/>
        <blockAlignment value="CENTER"/>
        <blockValign value="MIDDLE" start="0,0" stop="0,0"/>
        <blockAlignment value="LEFT" start="0,0" stop="0,-1"/>
        <blockValign value="BOTTOM" start="0,0" stop="-1,-1"/>
        <lineStyle kind="GRID" colorName="silver"/>
    </blockTableStyle>

    <blockTableStyle id="contents">
        <blockFont name="Times-Roman" size="10" start="0,0" stop="0,-1"/>
        <blockAlignment value="LEFT" start="0,0" stop="0,-1"/>
        <blockFont name="Times-Bold" size="10" start="1,0" stop="1,-1"/>
        <blockAlignment value="RIGHT" start="1,0" stop="1,-1"/>
    </blockTableStyle>
    
    
</stylesheet>


<story>


<outlineAdd level="0">PyRXP User Documentation</outlineAdd>
<setNextTemplate name="inside"/>
<nextFrame/>



<h1>PyRXP Documentation</h1>

<h2>
Contents
</h2>

<blockTable style="contents" colWidths="300,50">
    <tr><td>1. Introduction</td><td>3</td></tr>
    <tr><td>1.1    Who is this document aimed at?</td><td>3</td></tr>
    <tr><td>1.2    What is PyRXP?</td><td>3</td></tr>
    <tr><td>1.3    License terms</td><td>3</td></tr>
    <tr><td>1.4    Why another XML toolkit?</td><td>3</td></tr>
    <tr><td>1.5    Design Goals</td><td>4</td></tr>
    <tr><td>1.6    Design non-goals</td><td>4</td></tr>
    <tr><td>1.7    How fast is it?</td><td>5</td></tr>
    <tr><td>1.8    The Tuple Tree structure</td><td>5</td></tr>
    <tr><td>1.9    Can I get involved?</td><td>6</td></tr>
    <tr><td>2.  Installation and Setup</td><td>7</td></tr>
    <tr><td>2.1    Windows binary - pyRXP.pyd</td><td>7</td></tr>
    <tr><td>2.2    Source Code installation</td><td>7</td></tr>
    <tr><td>2.3    Examples</td><td>7</td></tr>
    <tr><td>3.  Using pyRXP</td><td>8</td></tr>
    <tr><td>3.1    Simple use without validation</td><td>8</td></tr>
    <tr><td>3.2    Validating against a DTD</td><td>10</td></tr>
    <tr><td>3.3    Interface Summary</td><td>12</td></tr>
    <tr><td>3.4    Parser Object Attributes and Methods</td><td>12</td></tr>
    <tr><td>3.5    List of Flags</td><td>14</td></tr>
    <tr><td>3.6    Flag explanations and examples</td><td>15</td></tr>
    <tr><td>4. The examples and utilities</td><td>21</td></tr>
    <tr><td>4.1    Benchmarking</td><td>21</td></tr>
    <tr><td>4.2    xmlutils and the TagWrapper</td><td>21</td></tr>
    <tr><td>5.  Future Directions</td><td>23</td></tr>
    <tr><td>5.1    Test Suite</td><td>23</td></tr>
    <tr><td>5.2    Standardize the wrapper</td><td>23</td></tr>
    <tr><td>5.3    Other parsers</td><td>23</td></tr>
    <tr><td>5.4    Better benchmark suite</td><td>23</td></tr>
    <tr><td>5.5    Type Conversion Utility</td><td>23</td></tr>
    <tr><td>5.6    Source file references</td><td>23</td></tr>
    <tr><td>5.7    (longer term and debatable) Richer tuple tree structure</td><td>24</td></tr></blockTable>


<nextFrame/>

<h1>1. Introduction</h1>
<outlineAdd level="1">1. Introduction</outlineAdd>
<namedString id="ref_1">1. Introduction</namedString>

<h2>1.1  Who is this document aimed at?</h2>
<outlineAdd level="2">1.1 Who is this document aimed at?</outlineAdd>

<para>
This document is aimed at anyone who wants to know how to use the pyRXP parser extension from
Python. It’s assumed that you know how to use the Python programming
language and understand its terminology. We make no attempt to teach XML in this
document, so you should already know the basics (what a DTD is, some of
the syntax etc.)
</para>


<h2>1.2    What is PyRXP?</h2>
<outlineAdd level="2">1.2 What is PyRXP?</outlineAdd>

<para>
PyRXP is a Python language wrapper around the excellent RXP parser.
RXP is a validating namespace-aware XML parser written in C. Together,
they provide the fastest XML-parsing framework available to Python
programmers today.
</para>

<para>
RXP was written by Richard Tobin at the Language Technology Group, 
Human Communication Research Centre, University of Edinburgh.
PyRXP was written by Robin Becker at ReportLab.
</para>

<para>
This documentation describes pyRXP-0.7 being used with RXP 1.2.5, as
well as ReportLab's emerging XML toolkit which uses it.
</para>

<h2>1.3    License terms</h2>
<outlineAdd level="2">1.3 License terms</outlineAdd>

<para>
Edinburgh University have released RXP under the GPL.  This is
generally fine for in-house or open-source use.  But if you want to
use it in a closed-source commercial product, you may need to
negotiate a separate license with them.  By contrast, most Python
software uses a less restrictive license; Python has its own license,
and ReportLab uses the FreeBSD license for our PDF Toolkit, which
means you CAN use it in commercial products.
</para>

<para>
We have therefore no choice but to do the same.  We licensed RXP for
our commercial products, but are releasing pyRXP under the GPL.  If
you want to use pyRXP for a commercial product, you need to negotiate
with ReportLab and with Edinburgh University.  We fully encourage
commercial enquiries and will act as a single point of contact. 
Any commercial licensing deals will help to support both development
teams involved.
</para>

<para>
However, the XML framework ReportLab is using and building will be
under our own license.  It predates pyRXP and can be made to work off any 
XML parser (such as expat), and we hope to produce something which can go into
the Python distribution one day.
</para>




<h2>1.4 Why another XML toolkit?</h2>
<outlineAdd level="2">1.4 Why another XML toolkit?</outlineAdd>

<para>
This grew out of real world needs which others in the Python community
may share.  ReportLab make tools which read in some kind of data and
make PDF reports.  One common input format these days is XML.  It’s
very convenient to express the interface to a system as an XML file.
Some other system might send us some XML with tags like &lt;invoice&gt; and
&lt;customer&gt;, and we turn these into nice looking invoices.
</para>

<para>
Also, we have a commercial product called Report Markup Language –
we sell a converter to turn RML files into PDF.  This has to parse
XML, and do it fast and accurately.
</para>

<para>
Typically we want to get this XML into memory as fast as possible.
And, if the performance penalties are not too great, we’d like the
option to validate it as well.  Validation is useful because we can
stop bad data at the point of input; if someone else sends our system
an XML ‘invoice packet’ which is not valid according to the agreed
DTD, and gets a validation error, they will know what’s going on.
This is a lot more helpful than getting a strange and unrelated-sounding
error during the formatting stage.
</para>

<para>
We tried to use all the parsers we could find.  We found that almost all
of them were constructing large object models in Python code, which took
a long time and a lot of memory.  Even the fastest C-based parser, expat
(which was not yet a standard part of Python at the time), calls back
into Python code on every start and end tag, which defeats most of
the benefit.   Aaron Watters of ReportLab sat down for a couple of days and produced
his own parser, rparsexml, which uses string.find and got pretty much
the same speed as pyexpat.  We also evolved our own representation of
a tree in memory, which became the cornerstone of our approach; and
when we found RXP we found it easy to make a wrapper around it to
produce the "tuple tree".
</para>

<para>
We have now reached the point in our internal bag-of-tools where XML
parsing is a standard component, running entirely at C-like speeds,
and we don't even think much about it any more.  Which means we must
be doing something right and it's time to share it :-)
</para>




<h2>1.5    Design Goals</h2>
<outlineAdd level="2">1.5 Design Goals</outlineAdd>

<para>
This is part of an XML framework which we will polish up and release
as soon as it can all be documented.  The general components are:</para>
<para style="bulletList" bulletText="l" spaceAfter="0">A standard in-memory representation of an XML document (the <i>tuple tree</i> below)</para>
<para style="bulletList" bulletText="l" spaceAfter="0">Various parsers which can produce this – principally pyRXP</para>
<para style="bulletList" bulletText="l" spaceAfter="0">A ‘lazy wrapper’ around this which gives a very friendly Pythonic interface for navigating the tree</para>
<para style="bulletList" bulletText="l" spaceAfter="6">A lightweight transformation tool which does a lot of what XSLT can do, but again with Pythonic syntax</para>

<para>In general we want to get the whole structure of an XML document into 
memory as soon as possible.  Having done so, we're going to traverse through
it and move the data into our own object model anyway; so we don't
really care what kind of "node objects" we're dealing with and
whether they are DOM-compliant. Our goals for the whole framework are:</para>

<para style="bulletList" bulletText="l" spaceAfter="0">Fast - XML parsing should not be an overhead for a program</para>
<para style="bulletList" bulletText="l" spaceAfter="0">Validate when needed, with little or no performance penalty</para>
<para style="bulletList" bulletText="l" spaceAfter="6">Construct a complete tree in memory which is easy and natural to access</para>

<para>Note that pyRXP is just the main parsing component and not the framework itself.</para>


<h2>1.6    Design non-goals</h2>
<outlineAdd level="2">1.6 Design non-goals</outlineAdd>

<para>
It's often much more helpful to spell out what a system or component will NOT do.
Most of all we are NOT trying to produce a standards-compliant parser.</para>
<para style="bulletList" bulletText="l" spaceAfter="0">Not a SAX parser</para>
<para style="bulletList" bulletText="l" spaceAfter="0">Not a DOM parser</para>
<para style="bulletList" bulletText="l">Does not capture full XML structure</para>
<para>Why not?  Aren't standards good?</para>
<para>
It's great that Python has support for SAX and DOM, but these are basically
Java (or at least cross-platform) APIs.  If you're doing Python, it's possible
to make things simpler, and we've tried.  Let's imagine you have some XML
containing an <i>invoice</i> tag, that this in turn contains <i>lineItems</i>
tags, and each of these has some text content and an <i>amount</i> attribute.
Wouldn't it be nice if you could write some Python code this simple?
</para>

<pre style="programListing">
invoice = pyRXP.Parser().parse(myInvoiceText)
for lineItem in invoice:
    print invoice.amount
</pre>

<para>Likewise, if a node is known to contain text, it would be really handy
to just treat it as a string.  We have a preprocessor we use to insert data
into HTML and RML files which lets us put Python expressions in curly braces, 
and we often do things like</para>

<pre style="programListing">
<![CDATA[
<html><head><title>My web page</title></head>
<body>
<h1>Statement for {{xml.customer.DisplayName}}</h1>
<!-- etc etc -->
</body>
</html>
<h1></h1>
]]>
</pre>

<para>
Try to write the equivalent in Java and you'll have loads of method calls to
getFirstElement(), getNextElement() and so on.  Python has beautifully compact
and readable syntax, and we'd rather use it.  So we're not bothering with
SAX and DOM support ourselves.  (Although if other people want to contribute full DOM
and SAX wrappers for pyRXP, we'll accept the patches).
</para>


<h2>1.7 How fast is it?</h2>
<outlineAdd level="2">1.7 How fast is it?</outlineAdd>

<para>The examples file includes a crude benchmarking script.  It measures speed
and memory allocation of a number of different parsers and frameworks. This
is documented later on.  Suffice to say that we can parse hamlet in 0.15
seconds with full validation on a P500 laptop.  Doing the same with the
<i>minidom</i> in the Python distribution takes 33 times as long and 
allocates 8 times as much memory, and does not validate.  It also 
appears to have a significant edge on Microsoft's XML parser and on
FourThought's cDomlette.   Using pyRXP means that XML parsing will typically
take a tiny amount of time compared to whatever your Python program will
do with the data later.</para>


<h2>1.8  The Tuple Tree structure</h2><outlineAdd level="2">1.8 The Tuple Tree structure</outlineAdd>

<para>
Most ‘tree parsers’ such as DOM create ‘node objects’ of some
sort.  The DOM gives one consensus of what such an object should look
like.  The problem is that "objects" means "class instances in
Python", and the moment you start to use such beasts, you move away
from fast C code to slower interpreted code.  Furthermore, the nodes
tend to have magic attribute names like "parent" or "children",
which one day will collide with structural names.
</para>

<para>
So, we defined the simplest structure we could which captured the
structure of an XML document.  Each tag is represented as a tuple of
</para>
<para style="programListing">
(tagName, dict_of_attributes, list_of_children, spare)
</para>
<para>
The dict_of_attributes can be None (meaning no attributes) or a
dictionary mapping attribute names to values. The list_of_children may
either be None (meaning a singleton tag) or a list with elements that
are 4-tuples or plain strings.
</para>
<para>A great advantage of this representation - which only uses built-in
types in Python - is that you can marshal it (and then zip or encrypt the 
results) with one line of Python code.  Another is that one can write
fast C code to do things with the structure. And it does not require any 
classes installed on the client machine, which is very useful when
moving xml-derived data around a network.</para>

<para>
This does not capture the full structure of XML.  We make decisions
before parsing about whether to expand entities and CDATA nodes, and
the parser deals with it; after parsing we have most of the XML
file’s content, but we can’t get back to the original in 100% of
cases.  For example both of the following will (with default settings)
return the string "Smith &amp; Jones", and you can't tell from the
tuple tree which one was in the file:
</para>

<pre style="programListing">
<![CDATA[
<provider>Smith &amp; Jones<provider>
]]>
</pre>
<para>
Alternatively one can use
</para>
<pre style="programListing">
<![CDATA[
<provider><[CDATA[Smith & Jones]]>]<![CDATA[]><provider>
]]>
</pre>

<para>
So if you want a tool to edit and rewrite XML files with perfect
fidelity, our model is not rich enough.  However, note that RXP itself
DOES provide all the hooks and could be the basis for such a parser.
</para>

<nextFrame/>



<h2>1.9    Can I get involved?</h2>
<outlineAdd level="2">1.9 Can I get involved?</outlineAdd>

<para>
Sure!  Join us on the Reportlab-users mailing list (<i>www.egroups.com/group/reportlab-users</i>), 
and feel free to contribute patches.  The final section of this manual has a brief
"wish list".</para>


<para>Because the Reportlab Toolkit is used in many mission
critical applications and because tiny changes in parsers can have unintended
consequences, we will keep checkin rights on sourceforge to a trusted
few developers; but we will do our best to consider and process patches.
</para>

<nextFrame/>

<h1>2.  Installation and Setup</h1>
<outlineAdd level="1">2. Installation and Setup</outlineAdd>

<para>
We make available pre-built Windows binaries.  On other platforms you can build it 
from source using distutils.  pyRXP is a single extension module with no other
dependencies outside Python itself.
</para>


<h2>2.1 Windows binary - pyRXP.pyd</h2>
<outlineAdd level="2">2.1 Windows binary - pyRXP.pyd</outlineAdd>

<para>
ReportLab’s FTP server has a win32-dlls directory, which is
sub-divided into Python versions. Each of these has the version of the
pyd file suitable for use with that version of Python. So, the version
we use with Python 2.1 is at</para>
<pre style="programListing">
http://www.reportlab.com/ftp/win32-dlls/2.1/pyRXP.pyd 
</pre>
<para>
Download the pyRXP DLL from the ReportLab FTP site. Save the pyRXP.pyd
in the DLLs directory under your Python installation (eg this is the
<font face="Courier">C:\Python21\DLLs</font> directory for a standard
Windows installation of Python 2.1).
</para>

<h2>2.2 Source Code installation</h2>
<outlineAdd level="2">2.2 Source Code installation</outlineAdd>
<para>
The source code is open source under the GPL. This is available on
SourceForge.
</para>
<para>
The source for pyRXP and a slightly patched version of RXP is made
available by anonymous CVS at
</para>
<pre style="programListing">
:pserver:anonymous@cvs.reportlab.sourceforge.net:/cvsroot/reportlab
</pre>
<para>
To get the source use the commands 
</para>
<pre style="programListing">
cvs -d :pserver:anonymous@cvs.reportlab.sourceforge.net:/cvsroot/reportlab login 
cvs -d :pserver:anonymous@cvs.reportlab.sourceforge.net:/cvsroot/reportlab co rl_addons/pyRXP 
</pre>

<para>
enter a carriage return for the password.
</para>


<para>
If you have obtained the source code in the way described above, the
<font face="Courier">rl_addons/pyRXP</font> directory  should contain
a distutils script, <font face="Courier">setup.py</font> which should
be run with argument install or build. If successful a shared library
<font face="Courier">pyRXP.pyd</font> or <font face="Courier">pyRXP.so</font> should be built.
</para>

<h2>2.3 Examples</h2><outlineAdd level="2">2.3 Examples</outlineAdd>
<para>
We have made available a small directory of example stuff to play with.
This will be superceded by the release of the framework soon.  As such
there is no formal package location for it; unzip anywhere you want.
</para>
<pre style="programListing">
http://www.reportlab.com/ftp/pyRXP_examples.zip
</pre>

<para>
The examples directory includes a couple of substantial XML files with
DTDs, a wrapper module called <i>xmlutils</i> which provides easy access
to the tuple tree, and the beginnings of a benchmarking script.  The benchmark
script tries to find lots of XML parsers on your system.  Both are documented
in section 4 below.
</para>



<nextFrame/>

<h1>3.  Using pyRXP</h1>
<outlineAdd level="1">3. Using pyRXP</outlineAdd>

<h2>3.1.    Simple use without validation</h2>
<outlineAdd level="2">3.1. Simple use without validation</outlineAdd>

<para>
Firstly you have to import the pyRXP module (using Python’s <font face="Courier">import</font>
statement).  While we are here, pyRXP has a couple of attributes that
are worth knowing about: <font face="Courier">version</font> gives you a string with the
version number of the pyRXP module itself, and <font face="Courier">RXPVersion</font> gives
you string with the version information for the rxp library embedded
in the module.
</para>

<pre style="programListing">
C:\Python21>python
Python 2.1.1 (#20, Jul 20 2001, 01:19:29) [MSC 32 bit (Intel)] on win32
Type "copyright", "credits" or "license" for more information.
>>> import pyRXP 
>>> pyRXP.version
'0.7'
>>> pyRXP.RXPVersion
'RXP 1.2.5 Copyright Richard Tobin, LTG, HCRC, University of Edinburgh'
</pre>

<para>
Once you have imported pyRXP, you can instantiate a parser instance
using the <font face="Courier">Parser</font> class.
</para>

<pre style="programListing">
>>>p=pyRXP.Parser()
</pre>

<para>
This by itself isn’t very useful. But it does allow us to create a
single parser which we can reuse many times. It also allows us to type
a short variable name rather than ‘pyRXP.Parser’ every time we
need to use it. p is now an instance of Parser – Parser is a
constructer that creates an object with its own methods and
attributes. When you create a parser like this you can also set
multiple flags at the same time. This can save you from having to set
them separately, or having to set them all repeatedly each time you
need to do a parse.
</para>

<para>
To parse some XML, you use the <font face="Courier">parse</font>
method. The simplest way of doing this is to feed it a string. You
could create the string beforehand, or read it from disk (using
something like <font face="Courier">s=open('filename',
'r').read())</font>. PyRXP isn’t designed to allow you to read the
source directly from disk without an intermediate step like this.
</para>

<para>
As well as exposing this method, instances of Parser are
callable. This means that you can do this:
</para>

<pre style="programListing"><![CDATA[>>> p=pyRXP.Parser()
>>> p('<a>some text<a>')
]]></pre>

<para>
instead of this
</para>

<pre style="programListing"><![CDATA[>>> p=pyRXP.Parser()
>>> p.parse('<a>some text</a>')
]]></pre>

<para>
Both would give you exactly the same result (<font
face="Courier">('a', None, ['some text'], None))</font>)
</para>

<para>
We'll use the second style in this documentation, since it makes the
examples slightly clearer. Whether you do or not is up to you and your
programming style.
</para>

<para>
We’ll start with some very simple examples and leave validation for
later.
</para>

<pre style="programListing"><![CDATA[
>>> p.parse('<tag>content</tag>')
('tag', None, ['content'], None)
]]></pre>

<para>
This could also be expressed more long-windedly as 
<font face="Courier"><![CDATA[pyRXP.Parser().parse('<tag>content</tag>')]]></font>
</para>

<para>
The parser has taken this single line of XML and returned a
tuple-tree. This tuple is made up of the following elements:
</para>
<para style="bulletList" bulletText="l" spaceAfter="0">'tag':  the tag
name (aka element name).</para>
<para style="bulletList" bulletText="l" spaceAfter="0">None:  a
dictionary of the tag’s attributes (null here since it doesn’t
have any).</para>
<para style="bulletList" bulletText="l" spaceAfter="0">['content']:  a
list of included textual results. This is the contents of the
tag.</para>
<para style="bulletList" bulletText="l" spaceAfter="0">None:  the
fourth element is unused. It’s provided for debugging info, line
numbers or other possible future uses.</para>

<para>
This tree structure is equivalent to the input XML, at least in
information content. It is theoretically possible to recreate the
original XML from this tree since no information has been lost.
</para>

<para>
A tuple tree for more complex XML snippets will contain more of these
tuples, but they will all use the same structure as this one.
</para>

<pre style="programListing"><![CDATA[
>>> p.parse('<tag1><tag2>content</tag2></tag1>')
('tag1', None, [('tag2', None, ['content'], None)], None)
]]></pre>

<condPageBreak height="172"/>
<para>
This may be easier to understand if we lay it out differently:
</para>

<pre style="programListing"><![CDATA[
>>> p.parse('<tag1><tag2>content</tag2></tag1>')
('tag1',
 None,
     [('tag2',
       None,
       ['content'],
       None)
     ],
None)
]]></pre>

<para>
Tag1 is the name of the outer tag, which has no attributes. Its
contents is a list. This contents contains Tag2, which has its own
attribute dictionary (which is also empty since it has no attributes)
and its content, which is the string ‘content’. It has the closing
null element, then the list for Tag2 is closed, Tag1 has its own final
null element and it too is closed.
</para>

<para>
The XML that is passed to the parser must be balanced. Any opening and
closing tags must match. They wouldn’t be valid XML otherwise.
</para>

<para>
Look at the following three examples. The first one is a fairly
ordinary tag with contents. The second and third can both be
considered as empty tags – one is a tag with no content between its
opening and closing tag, and the other is the singleton form which by
definition has no content.
</para>

<pre style="programListing"><![CDATA[
>>> p.parse('<tag>my contents</tag>')
('tag', None, ['my contents'], None)

>>> p.parse('<tag></tag>')
('tag', None, [], None)

>>> p.parse('<tag/>')
('tag', None, None, None)
]]></pre>

<para>
Notice how the contents list is handled differently for the last two
examples. This is how we can tell the difference between an empty tag
and its singleton version. If the content list is empty then the tag
doesn’t have any content, but if the list is None, then it can not
have any content since it’s the singleton form which can’t
actually have any.
</para>

<para>
Another example:
</para>

<pre style="programListing"><![CDATA[
>>>p.parse('<outerTag><innerTag>bb</innerTag>aaa<singleTag/></outerTag>')
('outerTag', None, [('innerTag', None, ['bb'], None), 'aaa', ('singleTag', 
None, None, None)], None)
]]></pre>

<para>
Again, this is more understandable if we show it like this:
</para>

<pre style="programListing"><![CDATA[
('outerTag',
 None,
     [('innerTag',
       None,
       ['bb'],
       None),
          'aaa',
              ('singleTag',
               None,
               None,
               None)
      ],
 None)
]]></pre>

<para>
In this example, the tuple contains the outerTag (with no attribute
dictionary), whose list of contents are the innerTag, which contains
the string ‘bb’ as its contents, and the singleton singleTag whose
contents list is replaced by a null.
</para>

<condPageBreak height="140"/>
<para>
By default, PyRXP ignores comments whose contents are lost (this
behaviour can be changed – see the section of Flags later for
details).
</para>

<pre style="programListing"><![CDATA[
>>> p.parse('<tag/><!-- this is a comment about the tag -->')
('tag', None, None, None)

>>> p.parse('<!-- this is a comment -->')
Traceback (most recent call last):
  File "<interactive input>", line 1, in ?
Error: Error: Document ends too soon
 in unnamed entity at line 1 char 22 of [unknown]
]]></pre>

<para>
This causes an error, since the parser sees an empty string which
isn’t valid XML.
</para>

<para>
It also brings us on to the <font face="Courier">srcName</font>
attribute. The Parser has an attribute called srcName which is useful
when debugging. This is the name by which pyRXP refers to your code in
tracebacks. This can be useful – for example, if you have read the
XML in from a file, you can use the srcName attribute to show the
filename to the user. It doesn’t get used for anything other than
pyRXP Errors – SyntaxErrors  and IOErrors still won’t refer to
your XML by name.
</para>

<pre style="programListing"><![CDATA[
>>> p.parse('<a>aaa</a') # note the missing ‘>’
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
pyRXP.Error: Error: Expected > after name in end tag, but got <EOE>
 in unnamed entity at line 1 char 10 of yourString
]]></pre>

<para>
The XML that is passed to the parser must be balanced. Not only must
the opening and closing tags match (they wouldn’t be valid XML
otherwise), but there must also be one tag that encloses all the
others. If there are valid fragments that aren’t enclosed
by another valid tag, they are considered ‘multiple elements’ and
a pyRXP Error is raised.
</para>

<pre style="programListing"><![CDATA[
>>> p.parse('<a></a><b></b>')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
pyRXP.Error: Error: Document contains multiple elements
 in unnamed entity at line 1 char 9 of [unknown]

>>> p.parse('<outer><a></a><b></b></outer>')
('outer', None, [('a', None, [], None), ('b', None, [], None)], None)
]]></pre>

<h2>
3.2.    Validating against a DTD
</h2>
<outlineAdd level="2">3.2. Validating against a DTD</outlineAdd>

<para>
This section describes the default behaviours when validating against
a DTD. Most of these can be changed – see the section on flags later
in this document for details on how to do that.
</para>

<para>
For the following examples, we’re going to assume that you have a
single directory with the DTD and any test files in it.
</para>

<pre style="programListing"><![CDATA[
>>> import os
>>> os.getcwd()
'C:\\tmp\\pyRXP_tests'

>>> os.listdir('.')
['sample1.xml', 'sample2.xml', 'sample3.xml', 'sample4.xml', 'tinydtd.dtd']

>>> dtd = open('tinydtd.dtd', 'r').read()

>>> print dtd
<!-- A tiny sample DTD for use with the PyRXP documentation -->
<!-- $Header $-->

<!ELEMENT a (b)>
<!ELEMENT b (#PCDATA)*>
]]></pre>
                               
<para>
This is just to show you how trivial the DTD is for this example.
It’s about as simple as you can get – two tags, both mandatory.
Both a and b must appear in an xml file for it to conform to this DTD,
but you can have as many b’s as you want, and they can contain any
content.
</para>

<pre style="programListing"><![CDATA[
>>> fn=open('sample1.xml', 'r').read()

>>> print fn
<?xml version="1.0" encoding="iso-8859-1" standalone="no" ?>
<!DOCTYPE a SYSTEM "tinydtd.dtd">

<a>
<b>This is the contents</b>
</a>
]]></pre>

<para>                             
This is the simple example file.  The first line is the XML declaration,
and the <i>standalone="no"</i> part says that there should be an external DTD.  The
second line says where the DTD is, and gives the name of the root element – 
<i>a</i> in this case.  If you put this in your XML document, pyRXP will
attempt to validate it.
</para>

<pre style="programListing"><![CDATA[                              
>>p.parse(fn)
('a',
 None,
 ['\n', ('b', None, ['This tag is the contents'], None), '\n'],
 None)
>>>
]]></pre>

<para>This is a successful parse, and returns a tuple tree in the same way
as we have seen where the input was a string.</para>
                       
<para>
If you have a reference to a non-existant DTD file in a file (or one
that can’t be found over a network), then any attempt to parse it
will raise a pyRXP error.
</para>

<pre style="programListing"><![CDATA[                              
>>> fn=open('sample2.xml', 'r').read()

>>> print fn
<?xml version="1.0" encoding="iso-8859-1" standalone="no" ?>
<!DOCTYPE a SYSTEM "nonexistent.dtd">

<a>
<b>This is the contents</b>
</a>

>>> p.parse(fn)
C:\tmp\pyRXP_tests\nonexistent.dtd: No such file or directory
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
pyRXP.Error: Error: Couldn't open dtd entity file:///C:/tmp/pyRXP_tests/nonexistent.dtd
 in unnamed entity at line 2 char 38 of [unknown]
]]></pre>
                               

<para>
This is a different kind of error to one where no DTD is specified:
</para>

<pre style="programListing"><![CDATA[                              
>>> fn=open('sample4.xml', 'r').read()

>>> print fn
<?xml version="1.0" encoding="iso-8859-1" standalone="no" ?>
<a>
<b>This is the contents</b>
</a>

>>> p.parse(fn,NoNoDTDWarning=0)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
pyRXP.Error: Error: Document has no DTD, validating abandoned
 in unnamed entity at line 3 char 2 of [unknown]
]]></pre>


<condPageBreak height="252"/>
<para>
If you have errors in your XML and it does not validate against the
DTD, you will get a different kind of pyRXPError.
</para>

<pre style="programListing"><![CDATA[
>>> fn=open('sample3.xml', 'r').read()

>>> print fn
<?xml version="1.0" encoding="iso-8859-1" standalone="no" ?>
<!DOCTYPE a SYSTEM "tinydtd.dtd">

<x>
<b>This is the contents</b>
</x>

>>> p.parse(fn)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
pyRXP.Error: Error: Start tag for undeclared element x
 in unnamed entity at line 4 char 3 of [unknown]
>>>
]]></pre>

<para>     
Whether PyRXP validates against a DTD, together with a number of other
behaviours is decided by how the various flags are set.
</para>
<para>
By default, <font face="Courier">ErrorOnValidityErrors</font> is set
to 1, as is <font face="Courier">NoNoDTDWarning</font>. If you want
the XML you are parsing to actually validate against your DTD, you should
have both of these set to 1 (which is the default value), otherwise instead 
of raising a pyRXP error saying the XML that doesn’t conform to the DTD (which may or 
may not exist) this will be silently ignored. You should also have <font
face="Courier">Validate</font> set to 1, otherwise validation won’t
even be attempted.
</para>

<para>
Note that the first examples in this chapter - the ones without a DTD
- only worked because we had carefully chosen what seem like the
sensible defaults.  It is set to validate, but not to complain if
the DTD is missing.  So when you feed it something without a DTD
declaration, it notices the DTD is missing but continues in non-validating mode.
There are numerous flags set out below which affect the behaviour.
</para>


<h2>3.3    Interface Summary</h2>
<outlineAdd level="2">3.3 Interface Summary</outlineAdd>

<para>
The python module exports the following:
</para>

<blockTable
colWidths = "100, 351">
<tr><td><para><font face="Courier">Error</font></para></td><td>a python exception</td></tr>
<tr><td><para><font face="Courier">Version</font></para></td><td>the string version of the module</td></tr>
<tr><td><para><font face="Courier">RXPVersion</font></para></td><td>the version string of the rxp library embedded in the module</td></tr>
<tr><td><para><font face="Courier">parser_flags</font></para></td><td>a dictionary of parser flags - the values are the defaults for parsers</td></tr>
<tr><td><para><font face="Courier">Parser(*kw)</font></para></td><td>Create a parser</td></tr>
</blockTable>
							   
<h2>3.4    Parser Object Attributes and Methods</h2>
<outlineAdd level="2">3.4 Parser() Attributes and Methods</outlineAdd>

<para spaceAfter="0"><font face="Courier">parse(src)</font></para><para>
We have already seen that this is the main interface to the parser. It
returns ReportLab's standard tuple tree representation of the xml source. 
The string <i>src</i> contains the xml.
</para>

<para>The keyword arguments can modify the instance attributes for this call only. For example, we can do
</para>

<pre style="programListing"><![CDATA[>>>p.parse('<a>some text</a>', ReturnList=1, ReturnComments=1)
]]></pre>

<para>instead of</para>

<pre style="programListing"><![CDATA[>>>p.ReturnList=1
>>>p.ReturnComments=1
>>>p.parse('<a>some text</a>')
]]></pre>

<para>
Any other parses using p will be unaffacted by the values of
ReturnList and ReturnComments in the first example, whereas all
parses using p will have ReturnList and ReturnComments set to 1 after the second.
</para>
                               
<para spaceAfter="0"><font face="Courier">srcName</font></para><para>
A name used to refer to the source text in error and warning messages.
It is initially set as '&lt;unknown&gt;’.  If you know that the data
came from "spam.xml" and you want error messages to say so, you can
set this to the filename.
</para>

<para spaceAfter="0"><font face="Courier">warnCB  0,</font></para><para>
Warning callback.  Should either be None, 0, or a callable object (e.g. a function)
with a single argument which will receive warning messages. If None is used then warnings are
thrown away. If the default 0 value is used then warnings are written
to the internal error message buffer and will only be seen if an error
occurs.
</para>

<para spaceAfter="0"><font face="Courier">eoCB</font></para><para>
Entity-opening callback. The argument should be None or a callable method with a 
single argument. This method will be called when external entities are opened. The
method should return a (possibly modified) URI.  So, you could intercept a declaration
referring to <i>http://some.slow.box/somefile.dtd</i> and point at at the local
copy you know you have handy, or implement a DTD-caching scheme.
</para>

<nextFrame/>

<h2>
3.5    List of Flags
</h2>
<outlineAdd level="2">3.5 List of Flags</outlineAdd>

<para>
Flag attributes corresponding to the rxp flags; the values are the module standard defaults.
pyRXP.parser_flags returns these as a dictionary if you need to refer to these inline.
</para>

<blockTable style="plainTable"
colWidths="200, 90">
<tr><td>
Flag (1=on, 0=off)</td><td>Default</td></tr>
<tr><td>
AllowMultipleElements</td><td>0
</td></tr>
<tr><td>
AllowUndeclaredNSAttributes</td><td>0
</td></tr>
<tr><td>
CaseInsensitive</td><td>0
</td></tr>
<tr><td>
ErrorOnBadCharacterEntities</td><td>1
</td></tr>
<tr><td>
ErrorOnUndefinedAttributes</td><td>0
</td></tr>
<tr><td>
ErrorOnUndefinedElements</td><td>0
</td></tr>
<tr><td>
ErrorOnUndefinedEntities</td><td>1
</td></tr>
<tr><td>
ErrorOnUnquotedAttributeValues</td><td>1
</td></tr>
<tr><td>
ErrorOnValidityErrors</td><td>1
</td></tr>
<tr><td>
ExpandCharacterEntities</td><td>1
</td></tr>
<tr><td>
ExpandGeneralEntities</td><td>1
</td></tr>
<tr><td>
IgnoreEntities</td><td>0
</td></tr>
<tr><td>
IgnorePlacementErrors</td><td>0
</td></tr>
<tr><td>
MaintainElementStack</td><td>1
</td></tr>
<tr><td>
MergePCData</td><td>1
</td></tr>
<tr><td>
NoNoDTDWarning</td><td>1
</td></tr>
<tr><td>
NormaliseAttributeValues</td><td>1
</td></tr>
<tr><td>
ProcessDTD</td><td>0
</td></tr>
<tr><td>
RelaxedAny</td><td>0
</td></tr>
<tr><td>
ReturnComments</td><td>0
</td></tr>
<tr><td>
ReturnDefaultedAttributes</td><td>1
</td></tr>
<tr><td>
ReturnList</td><td>0
</td></tr>                              
<tr><td>
ReturnNamespaceAttributes</td><td>0
</td></tr>
<tr><td>
SimpleErrorFormat</td><td>0
</td></tr>
<tr><td>
TrustSDD</td><td>1
</td></tr>
<tr><td>
Validate</td><td>1
</td></tr>
<tr><td>
WarnOnRedefinitions</td><td>0
</td></tr>
<tr><td>
XMLExternalIDs</td><td>1
</td></tr>
<tr><td>
XMLLessThan</td><td>0
</td></tr>
<tr><td>
XMLMiscWFErrors</td><td>1
</td></tr>
<tr><td>
XMLNamespaces</td><td>0
</td></tr>
<tr><td>
XMLPredefinedEntities</td><td>1
</td></tr>
<tr><td>
XMLSpace</td><td>0
</td></tr>
<tr><td>
XMLStrictWFErrors</td><td>1
</td></tr>
<tr><td>
XMLSyntax</td><td>1
</td></tr>
</blockTable>

<nextFrame/>
<h2>
3.6    Flag explanations and examples
</h2>
<outlineAdd level="2">3.6 Flag explanations and examples</outlineAdd>

<para>
With so many flags, there is a lot of scope for interaction between
them. These interactions are not documented yet, but you should be
aware that they exist.
</para>

<para spaceAfter="6">
<b>AllowMultipleElements</b> 
</para>
<para spaceAfter="3">
Default: 0
</para>
<para spaceAfter="0">
Description:
</para>
<para spaceAfter="3">
A piece of XML that does not have a single root-tag enclosing all the
other tags is described as having multiple elements. By default, this
will raise a pyRXP error. Turning this flag on will ignore this and
not raise those errors.
</para>
<para spaceAfter="0">
Example:
</para>
<pre style="programListing"><![CDATA[
>>> p.AllowMultipleElements = 0
>>> p.parse('<a></a><b></b>')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
pyRXP.Error: Error: Document contains multiple elements
 in unnamed entity at line 1 char 9 of [unknown]

>>> p.AllowMultipleElements = 1
>>> p.parse('<a></a><b></b>')
('a', None, [], None)
>>>
]]></pre>

<para spaceAfter="3">
<b>AllowUndeclaredNSAttributes </b>
</para>
<para spaceAfter="3">
Default: 0
</para>
<para spaceAfter="0">
Description:
</para>
<para spaceAfter="3">
<i>[to be added]</i>
</para>
<para spaceAfter="0">
Example:
</para>
<para>
<i>[to be added]</i>
</para>

<para spaceAfter="3">
<b>CaseInsensitive</b>
</para>
<para spaceAfter="3">
Default: 0
</para>
<para spaceAfter="0">
Description:
</para>
<para spaceAfter="3">
This flag controls whether the parse is case sensitive or not.
</para>
<para spaceAfter="0">
Example:
</para>
<pre style="programListing"><![CDATA[
>>> p.CaseInsensitive=1
>>> p.parse('<a></A>')
('A', None, [], None)

>>> p.CaseInsensitive=0
>>> p.parse('<a></A>')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
pyRXP.Error: Error: Mismatched end tag: expected </a>, got </A>
 in unnamed entity at line 1 char 7 of [unknown]
>>>
]]></pre>

<para spaceAfter="3">
<b>ErrorOnBadCharacterEntities</b>
</para>
<para spaceAfter="3">
Default: 1
</para>
<para spaceAfter="0">
Description:
</para>
<para spaceAfter="3">
If this is set, character entities which expand to illegal values are
an error, otherwise they are ignored with a warning.
</para>
<para spaceAfter="0">
Example:
</para><pre style="programListing"><![CDATA[
>>> p.ErrorOnBadCharacterEntities=0
>>> p.parse('<a>&#999;</a>')
('a', None, [''], None)

>>> p.parse('<a>&#999;</a>')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
pyRXP.Error: Error: 0x3e7 is not a valid 8-bit XML character
 in unnamed entity at line 1 char 10 of [unknown]
]]></pre>

<para spaceAfter="3">
<b>ErrorOnUndefinedAttributes</b>
</para>
<para spaceAfter="3">
Default: 0
</para>
<para spaceAfter="0">
Description:
</para>
<para spaceAfter="3">
If this is set and there is a DTD, references to undeclared attributes
are an error.
</para>
<para>
See also: ErrorOnUndefinedElements
</para>

<para spaceAfter="3">
<b>ErrorOnUndefinedElements</b>
</para>
<para spaceAfter="3">
Default: 0
</para>
<para spaceAfter="0">
Description:
</para>
<para spaceAfter="3">
If this is set and there is a DTD, references to undeclared elements
are an error.
</para>
<para>
See also: ErrorOnUndefinedAttributes
</para>

<para spaceAfter="3">
<b>ErrorOnUndefinedEntities</b>
</para>
<para spaceAfter="3">
Default: 1
</para>
<para spaceAfter="0">
Description:
</para>
<para spaceAfter="3">
If this is set, undefined general entity references are an error,
otherwise a warning is given and a fake entity constructed whose value
looks the same as the entity reference.
</para>
<para spaceAfter="0">
Example:
</para>
<pre style="programListing"><![CDATA[
>>> p.ErrorOnUndefinedEntities=0
>>> p.parse('<a>&dud;</a>')
('a', None, ['&dud;'], None)

>>> p.ErrorOnUndefinedEntities=1
>>> p.parse('<a>&dud;</a>')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
pyRXP.Error: Error: Undefined entity dud
 in unnamed entity at line 1 char 9 of [unknown]
]]></pre>

<para spaceAfter="3">
<b>ErrorOnUnquotedAttributeValues</b>
</para>
<para spaceAfter="3">
Default: 1
</para>
<para spaceAfter="0">
Description:
</para>
<para>
<i>[to be added]</i>
</para>

<para spaceAfter="3">
<b>ErrorOnValidityErrors</b>
</para>
<para spaceAfter="3">
Default: 1
</para>
<para spaceAfter="0">
Description:
</para>
<para>
If this is on, validity errors will be reported as errors rather than
warnings.  This is useful if your program wants to rely on the
validity of its input.
</para>

<para spaceAfter="3">
<b>ExpandCharacterEntities</b>
</para>
<para spaceAfter="3">
Default: 1
</para>
<para spaceAfter="0">
Description:
</para>
<para spaceAfter="3">
If this is set, entity references are expanded.  If not, the
references are treated as text, in which case any text returned that
starts with an ampersand must be an entity reference (and provided
MergePCData is off, all entity references will be returned as separate
pieces).
</para>
<para spaceAfter="3">
See also: ExpandGeneralEntities, ErrorOnBadCharacterEntities
</para>
<para spaceAfter="0">
Example:
</para>
<pre style="programListing"><![CDATA[
>>> p.ExpandCharacterEntities=1
>>> p.parse('<a>&#109;</a>')
('a', None, ['m'], None)

>>> p.ExpandCharacterEntities=0
>>> p.parse('<a>&#109;</a>')
('a', None, ['&#109;'], None)
]]></pre>


<para spaceAfter="3">
<b>ExpandGeneralEntities</b>
</para>
<para spaceAfter="3">
Default: 1
</para>
<para spaceAfter="0">
Description:
</para>
<para spaceAfter="3">
If this is set, entity references are expanded.  If not, the
references are treated as text, in which case any text returned that
starts with an ampersand must be an entity reference (and provided
MergePCData is off, all entity references will be returned as separate
pieces).
</para>
<para spaceAfter="3">
See also: ExpandCharacterEntities
</para>
<para spaceAfter="0">
Example:
</para>
<pre style="programListing"><![CDATA[
>>> p.ExpandGeneralEntities=0
>>> p.parse('<a>&amp;</a>')
('a', None, ['&amp;'], None)

>>> p.ExpandGeneralEntities=1
>>> p.parse('<a>&amp;</a>')
('a', None, ['&'], None)
]]></pre>

<para spaceAfter="3">
<b>IgnoreEntities</b>
</para>
<para spaceAfter="3">
Default: 0
</para>
<para spaceAfter="0">
Description:
</para>
<para spaceAfter="3">
If this flag is on, normal entity substitution takes place. If it is
off, entities are passed through unaltered.
</para>
<para spaceAfter="0">
Example:
</para>
<pre style="programListing"><![CDATA[
>>> p.IgnoreEntities=0
>>> p.parse('<a>&amp;</a>')
('a', None, ['&'], None)

>>> p.IgnoreEntities=1
>>> p.parse('<a>&amp;</a>')
('a', None, ['&amp;'], None)
]]></pre>


<para spaceAfter="3">
<b>IgnorePlacementErrors</b>
</para>
<para spaceAfter="3">
Default: 0
</para>
<para spaceAfter="0">
Description:
</para>
<para>
<i>[to be added]</i>
</para>

<para spaceAfter="3">
<b>MaintainElementStack</b>
</para>
<para spaceAfter="3">
Default: 1
</para>
<para spaceAfter="0">
Description:
</para>
<para>
<i>[to be added]</i>
</para>


<para spaceAfter="3">
<b>MergePCData</b>
</para>
<para spaceAfter="3">
Default: 1
</para>
<para spaceAfter="0">
Description:
</para>
<para>
If this is set, text data will be merged across comments and entity
references.
</para>

<para spaceAfter="3">
<b>NoNoDTDWarning</b>
</para>
<para spaceAfter="3">
Default: 1
</para>
<para spaceAfter="0">
Description:
</para>
<para>
Usually, if <font face="Courier">Validate</font> is set, the parser
will produce a warning if the document has no DTD.  This flag
suppresses the warning (useful if you want to validate if possible,
but not complain if not).
</para>


<para spaceAfter="3">
<b>NormaliseAttributeValues</b>
</para>
<para spaceAfter="3">
Default: 1
</para>
<para spaceAfter="0">
Description:
</para>
<para>
If this is set, attributes are normalised according to the standard.
You might want to not normalise if you are writing something like an
editor.
</para>

<para spaceAfter="3">
<b>ProcessDTD</b>
</para>
<para spaceAfter="3">
Default: 0
</para>
<para spaceAfter="0">
Description:
</para>
<para spaceAfter="3">
If <font face="Courier">TrustSDD</font> is set and a <font
face="Courier">DOCTYPE</font> declaration is present, the internal
part is processed and if the document was not declared standalone or
if <font face="Courier">Validate</font> is set the external part is
processed.  Otherwise, whether the <font face="Courier">DOCTYPE</font>
is automatically processed depends on <font
face="Courier">ProcessDTD</font>; if <font
face="Courier">ProcessDTD</font> is not set the user must call <font
face="Courier">ParseDtd()</font> if desired.
</para>
<para>
See also:  TrustSDD
</para>


<para spaceAfter="3">
<b>RelaxedAny</b>
</para>
<para spaceAfter="3">
Default: 0
</para>
<para spaceAfter="0">
Description:
</para>
<para>
<i>[to be added]</i>
</para>


<para spaceAfter="3">
<b>ReturnComments</b>
</para>
<para spaceAfter="3">
Default: 0
</para>
<para spaceAfter="0">
Description:
</para>
<para spaceAfter="3">
If this is set, comments are returned, otherwise they are ignored.
</para>
<para spaceAfter="0">
Example:
</para>
<pre style="programListing"><![CDATA[
>>> p.ReturnComments = 1
>>> p.parse('<a><!-- this is a comment --></a>')
('a', None, ['<!-- this is a comment -->'], None)

>>> p.ReturnComments = 0
>>> p.parse('<a><!-- this is a comment --></a>')
('a', None, [], None)
]]></pre>
<para>
See also: ReturnList
</para>

<para spaceAfter="3">
<b>ReturnDefaultedAttributes</b>
</para>
<para spaceAfter="3">
Default: 1
</para>
<para spaceAfter="0">
Description:
</para>
<para>
If this is set, the returned attributes will include ones defaulted as
a result of ATTLIST declarations, otherwise missing attributes will
not be returned.
</para>


<para spaceAfter="3">
<b>ReturnList</b>
</para>
<para spaceAfter="3">
Default: 0
</para>
<para spaceAfter="0">
Description:
</para>
<para spaceAfter="3">
If both ReturnComments and ReturnList are both set to 1, the whole
list (including any comments) is returned from a parse. If ReturnList
is set to 0, only the first tuple in the list is returned (ie the
actual XML content rather than any comments before it).
</para>
<para spaceAfter="0">
Example:
</para>
<pre style="programListing"><![CDATA[
>>> p.ReturnComments=1
>>> p.ReturnList=1
>>> p.parse('<!-- comment --><a>Some Text</a><!-- another comment -->')
['<!-- comment -->', ('a', None, ['Some Text'], None), '<!-- another comment -->']
>>> p.ReturnList=0
>>> p.parse('<!-- comment --><a>Some Text</a><!-- another comment -->')
('a', None, ['Some Text'], None)
>>> 
]]></pre>
<para>
See also: ReturnComments
</para>


<para spaceAfter="3">
<b>ReturnNamespaceAttributes</b>
</para>
<para spaceAfter="3">
Default: 0
</para>
<para spaceAfter="0">
Description:
</para>
<para>
<i>[to be added]</i>
</para>


<para spaceAfter="3">
<b>SimpleErrorFormat</b>
</para>
<para spaceAfter="3">
Default: 0
</para>
<para spaceAfter="0">
Description:
</para>
<para spaceAfter="3">
This causes the output on errors to get shorter and more compact.
</para>
<para spaceAfter="0">
Example:
</para>
<pre style="programListing"><![CDATA[
>>> p.SimpleErrorFormat=0
>>> p.parse('<a>causes an error</b>')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
pyRXP.Error: Error: Mismatched end tag: expected </a>, got </b>
 in unnamed entity at line 1 char 22 of [unknown]

>>> p.SimpleErrorFormat=1
>>> p.parse('<a>causes an error</b>')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
pyRXP.Error: [unknown]:1:22: Mismatched end tag: expected </a>, got </b>
]]></pre>

<para spaceAfter="3">
<b>TrustSDD</b>
</para>
<para spaceAfter="3">
Default: 1
</para>
<para spaceAfter="0">
Description:
</para>
<para spaceAfter="3">
If <font face="Courier">TrustSDD</font> is set and a <font
face="Courier">DOCTYPE</font> declaration is present, the internal
part is processed and if the document was not declared standalone or
if <font face="Courier">Validate</font> it is set the external part is
processed.  Otherwise, whether the <font face="Courier">DOCTYPE</font>
is automatically processed depends on <font
face="Courier">ProcessDTD</font>; if <font
face="Courier">ProcessDTD</font> is not set the user must call <font
face="Courier">ParseDtd()</font> if desired.
</para>
<para>
See also: ProcessDTD
</para>

<para spaceAfter="3">
<b>Validate</b>
</para>
<para spaceAfter="3">
Default: 1
</para>
<para spaceAfter="0">
Description:
</para>
<para>
If this is on, the parser will validate the document. If it’s off,
it won’t. It is not usually a good idea to set this to 0.
</para>

<para spaceAfter="3">
<b>WarnOnRedefinitions</b>
</para>
<para spaceAfter="3">
Default: 0
</para>
<para spaceAfter="0">
Description:
</para>
<para>
If this is on, a warning is given for redeclared elements, attributes,
entities and notations.
</para>

<para spaceAfter="3">
<b>XMLExternalIDs</b>
</para>
<para spaceAfter="3">
Default: 1
</para>
<para spaceAfter="0">
Description:
</para>
<para>
<i>[to be added]</i>
</para>


<para spaceAfter="3">
<b>XMLLessThan</b> 
</para>
<para spaceAfter="3">
Default: 0
</para>
<para spaceAfter="0">
Description:
</para>
<para>
<i>[to be added]</i>
</para>

<para spaceAfter="3">
<b>XMLMiscWFErrors</b>
</para>
<para spaceAfter="3">
Default: 1
</para>
<para spaceAfter="0">
Description:
</para>
<para spaceAfter="3">
To do with  well-formedness errors.
</para>
<para>
See also: XMLStrictWFErrors
</para>

<para spaceAfter="3">
<b>XMLNamespaces</b>
</para>
<para spaceAfter="3">
Default: 0
</para>
<para spaceAfter="0">
Description:
</para>
<para spaceAfter="3">
If this is on, the parser processes namespace declarations (see
below).  Namespace declarations are <i>not</i> returned as part of the list
of attributes on an element.
</para>
<para>
See also: XMLSpace
</para>


<para spaceAfter="3">
<b>XMLPredefinedEntities</b>
</para>
<para spaceAfter="3">
Default: 1
</para>
<para spaceAfter="0">
Description:
</para>
<para spaceAfter="3">
If this is on, pyRXP recognises the standard preset XML entities <font
face="Courier"><![CDATA[&amp; &lt; &gt; &quot;]]></font> and <font
face="Courier"><![CDATA[&apos;]]></font>) . If this is off, all
entities including the standard ones must be declared in the DTD or an
error will be raised.
</para>
<condPageBreak height="70"/>
<para spaceAfter="0">
Example:
</para>
<pre style="programListing"><![CDATA[
>>> p.XMLPredefinedEntities=1
>>> p.parse('<a>&amp;</a>')
('a', None, ['&'], None)

>>> p.XMLPredefinedEntities=0
>>> p.parse('<a>&amp;</a>')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
pyRXP.Error: Error: Undefined entity amp
 in unnamed entity at line 1 char 9 of [unknown]
]]></pre>

<para spaceAfter="3">
<b>XMLSpace</b>
</para>
<para spaceAfter="3">
Default: 0
</para>
<para spaceAfter="0">
Description:
</para>
<para spaceAfter="3">
If this is on, the parser will keep track of xml:space attributes
</para>
<para>
See also: XMLNamespaces
</para>

<para spaceAfter="3">
<b>XMLStrictWFErrors</b>
</para>
<para spaceAfter="3">
Default: 1
</para>
<para spaceAfter="0">
Description:
</para>
<para>
If this is set, various well-formedness errors will be reported as
errors rather than warnings.
</para>


<para spaceAfter="3">
<b>XMLSyntax</b>
</para>
<para spaceAfter="3">
Default: 1
</para>
<para spaceAfter="0">
Description:
</para>
<para spaceAfter="0">
<i>[to be added]</i>
</para>

<nextFrame/>
<h1>4. The examples and utilities</h1>
<outlineAdd level="1">4. The examples and utilities</outlineAdd>

<para>The zip file of examples contains a couple of validatable documents in xml,
the samples used in this manual, and two utility modules: one for benchmarking
and one for navigating through tuple trees.</para>


<h2>4.1 Benchmarking</h2>
<outlineAdd level="2">4.1 Benchmarking</outlineAdd>
<para><i>benchmarks.py</i> is a script aiming to compare performance of various
parsers.  We include it to make our results reproducable.  It is not a work of
art and if you think you can make it fairer or better, tell us how!  Here's
an example run.</para>


<pre style="programListing">
<![CDATA[
C:\code\rlextra\radxml\samples>benchmarks.py

    Interactive benchmark suite for Python XML parsers.
    Parsers available:

opened sample XML file 444220 bytes long
        1.  pyRXP
        2.  rparsexml
        3.  minidom
        4.  msxml30
        5.  4dom
        6.  cdomlette

Shall we do memory tests?  i.e. you look at Task Manager? y/n  y
Test number (or x to exit)>1
testing pyRXP
Pre-parsing: please input python process memory in kb >2904
Post-parsing: please input python process memory in kb >7180
12618 tags, 8157 attributes
pyRXP: init 0.0315, parse 0.3579, traverse 0.1594, mem used 4276kb, mem factor 9.86
]]>
</pre>



<para>Instead of the traditional example (hamlet), we took as our example an
early version of the Report Markup Language user guide, which is about
half a megabyte.  Hamlet's XML has almost no attributes; ours contains lots
of attributes, many of which will need conversion to numbers one day, and so 
it provides a more rounded basis for benchmarks</para>
<para>We measure several factors.  First there is speed.  Obviously this 
depends on your PC.  The script exits after each test so you get a clean
process.  We measure (a) the time to load the parser and any code it
needs into memory (important if doing CGI); (b) time to produce the tree,
using whatever the parser natively produces; and (c) time to traverse
the tree counting the number of tags and attributes. Note, (c) might be important
with a 'very lazy' parser which searched the source text on every request.
Also, later on we will be able to look at the difference between traversing
a raw tuple tree and some objects with friendlier syntax.</para>

<para>Next is memory.  Actually you have to measure that!  If anyone can give
us the API calls on Windows and other platforms to find out the current
process size, we'd be grateful!  What we are interested in is how big the
structure is in memory.  The above shows that the memory allocated is 9.86
times as big as the original XML text.  That sounds a lot, but it's actually
much less than most DOM parsers.</para>


<para>By contrast, here's the result for the <i>minidom</i> parser included in the official
Python distro:</para>

<pre style="programListing">
minidom: init 0.3039, parse 12.6435, traverse 0.0000, mem used 29136kb, mem factor 67.16
</pre>

<para>Even though minidom uses pyexpat (which is in C) to parse the XML, it's
36 times slower and uses 7 times more memory.  And of course it does not validate.
</para>

<h2>4.2 xmlutils and the TagWrapper</h2>
<outlineAdd level="2">4.2 xmlutils and the TagWrapper</outlineAdd>
<para>Finally, we've included a 'tag wrapper' class which makes it easy to
navigate around the tuple tree.  This is randomly selected from many such
modules we have used in various projects; the next task for us is to
pick ONE and publish it!  Essentially, it uses lazy evaluation to try
and figure out which part of the XML you want.  If you ask for
'tag.spam', it will check if (a) there is an attribute called spam,
or (b) if there is a child tag whose tag name is 'spam'.  And you
can iterate over child nodes as a sequence.  And, the str() method of
a tag which just contains text is just the text.  The snippets below
should make it clear what we are doing.</para>

<pre style="programListing">
<![CDATA[
>>> tree = pyRXP.Parser().parse(srcText)
>>> srcText = open('rml_a.xml').read()
>>> tree = pyRXP.Parser().parse(srcText)
>>> import xmlutils
>>> tw = xmlutils.TagWrapper(tree)
>>> tw
TagWrapper<document>
>>> tw.filename
'RML_UserGuide_1_0.pdf'
>>> len(tw.story)  # how many tags in the story?
1566
>>> tw.template.pageSize
'(595, 842)'

>>> for elem in tw.story:
... 	if elem.tagName == 'h1':
... 		print elem
... 		
 RML User Guide

Part I - The Basics
Part II - Advanced Features
Part III - Tables
Appendix A - Colors recognized by RML
Appendix B - Glossary of terms and abbreviations
Appendix C - Letters used by the Greek tag
Appendix D - Command reference
Generic Flowables (Story Elements)
Graphical Drawing Operations
Graphical State Change Operations
Style Elements
Page Layout Tags
Special Tags
>>> 
]]>
</pre>


<para>We are NOT saying this is a particularly good or complete wrapper;
but we do intend to standardize on one such wrapper module in the near future,
because it makes access to XML information much more 'pythonic' and pleasant.
It could be used with tuple trees generated by any parser.  Please let
us know if you have any suggestions on how it should behave.</para>

<nextFrame/>
<h1>
5.  Future Directions
</h1>
<outlineAdd level="1">5. Future Directions</outlineAdd>

<h2>
5.1    Test Suite
</h2>
<outlineAdd level="2">5.1 Test Suite</outlineAdd>

<para>
We urgently need a unittest-based suite full of samples saying
‘parse this XML with these flags and assert fact X about the
output’.  If done right, this could be used to generate the
documentation on the parser flags as well.  It will be very
important when allowing pluggable parsers.
</para>

<para>
In the meantime, there are some simple tests. Look at the file <font
face="Courier">test\t.py</font>.
</para>

<h2>
5.2    Standardize the Wrapper
</h2>
<outlineAdd level="2">5.2 Standardize then Wrapper</outlineAdd>

<para>
A standard wrapper class to let you ‘drill down’ into the tuple
tree.  This should be as pythonic as possible.
</para>

<h2>
5.3    Other parsers
</h2>
<outlineAdd level="2">5.3 Other parsers</outlineAdd>

<para>
Include tuple tree constructors based on other parsers.  One could use
pyexpat (in fact a few lines could be added to pyexpat itself to
produce a tuple tree in some future version of Python).  This would be
useful for people who cannot install extensions but have Python 2.0 or
above.  We also have our own parser, Aaron Watters’ rparsexml, which
uses no C code and is thus useful in places where you cannot build
extensions.  The latter is not guaranteed to be 100% standards
compliant, but this means we can modify it to handle bad XML.
</para>

<h2>
5.4    Better Benchmark Suite
</h2>
<outlineAdd level="2">5.4 Better Benchmark Suite</outlineAdd>

<para>
Extend this so that it knows about more parsers and (if possible) can
detect the memory used by them without needing to pause and look in
Task Manager.  Ensure we are being fair to competitors and using their
parsers optimally.
</para>

<h2>
5.5    Type Conversion Utility
</h2>
<outlineAdd level="2">5.5 Type Conversion Utility</outlineAdd>

<para>
In the parsed output, everything is a string.  Yet XML is full of
attributes which "mean" numeric values.  In particular our own
Report Markup Language has numerous attributes like <i>x, y, width,
height</i>, as well as color attributes.  It would be really useful to 
generalize the conversion step. Let’s say you can provide a mapping like this
</para>
<pre style="Normal">
1.  (tag, attribute) -> reader function
2.   attribute -> reader function
</pre>
<para>
Many of the reader functions are just <i>int</i> or <i>float</i>; others
could be written in Python or C.  For example we have standard length
expressions like "3cm" or "8.5in" which we convert to float values in points.
This could say that (a) if this tag name and attribute name has a converter
function, use it in-place;  (b) if the attribute name has a converter, use
that;  and if (c) there is nothing specified, leave it as a string.
</para>



<para>
So the tree could be converted “in place” with a simple API call, at C-like 
speeds.  And we'd be able to remove a lot of code from our application
and replace it with a very simple mapping.  Expect this real soon now!
</para>

<para>
Note that this type-conversion is not an XML standard.  The one true way is 
probably to use XML Schema; but for now this is not possible as we don't 
have a schema-validating parser, and we are big fans of stuff 
that works now.
</para>

<h2>
5.6    Source File References
</h2>
<outlineAdd level="2">5.6 Source File References</outlineAdd>

<para>
Debug/trace info:  add an extra structure to show the position in the
original source file where the tag starts and finished.  This would be
a parse-time option, as you might not want to take the time and
memory.  This would let an application raise an error saying not just that the
color tag contained a bad color value, but also that it occurred at
line 2352 of the input.  Useful!  This is why we reserved the final
tuple element for future use.
</para>

<condPageBreak height="36"/>
<h2>
5.7    (longer term and debatable) Richer Tuple Tree Structure
</h2>
<outlineAdd level="2">5.7 Richer Tuple Tree Structure</outlineAdd>

<para>It has been suggested that we expand the structure in a couple of
ways.  Instead of tuples we could make a new C-based node object with
a richer model.
</para>

<para>
Each node should have some pointer back to its parent.  This makes
navigation a lot easier, but means a little more housekeeping.
</para>

<para>
We could then also let you distinguish things like CDATA and entity
nodes and make it a fully rewritable DOM implementation, running at
C-like speeds.  We could even go further and keep references to things
like comments, which are not part of the XML standard.
</para>

<para>
PyRXP meets our needs already and we won’t rush into this.  Still, it
might be an attractive enhancement for a future version of Python;
essentially one would make a lightweight XML node into a built-in
type.
</para>
 
</story>

</document>