Article&toc - Python

An XML repository, 4Suite Server (4SS), co-developed by the authors, will be used as the application base for this example. 4Suite Server is an XML repository with many facilities for XML data and metadata management that make it very suitable for rapid development of Web Services, whether in Python or not.

The example in this article is written using 4Suite Server 0.11 and requires Python 1.5.2 or greater and 4Suite 0.11. Links to download all of these applications are available in the Resources section at the end of this article.

Online Software Repository

This article is the second in "Python Web Services Developer" column, but the first in a three part series on building an on-line software repository. In this segment, we will build our infrastructure. In the subsequent columns, we will feature searching indexed content as well as agent-based addition or retrieval of content using various protocols such as Simple Object Access Protocol (SOAP), HTTP, and WWW Distributed Authoring and Versioning (WebDAV).

Our online software repository service has a schema loosely based around the RDF schema in RPMFind.net. RPMFind is a system for cataloging packages of UNIX and Linux software in the popular Red Hat Package Manager (RPM) format. It contains key metadata about the software package, including author, version, and description, in the RDF format (see Listing 1). For a brief definition of RDF, read the previous issue of this column or go to the Resources section for a link to a basic introduction of this simple format.

The actual format of the XML is irrelevant. In fact, it doesn't even need to describe software, as the techniques described will work with any type of XML content. You can use this technique to describe a book catalog, employee information, or even a wine list.

All of the code and data files used in this example are downloadable from links in the Resources section at the end of this article.

Document Definitions

In 4SS's XML repository, document definitions allow you to specify a mapping between XML content and RDF metadata. To do this, you need to define sets of three XPath expressions: a subject expression, a predicate expression, and an object expression. An XPath expression allows you to define a set of node relationships in the document and return subsets of content from the document based on these relationships. These XPath expressions are evaluated against each XML document as it is added, modified, and deleted from the repository.

The resulting statements, also known as triples, are automatically added and removed from the RDF database (known as the model). If the document is modified, the triples are modified to reflect the changes, and if the document is deleted, then the triples are removed from the RDF server. Document definitions can inherit information from other document definitions allowing you to define a complex mapping of XML content to RDF metadata information.

In our example application, we will extend one of the default document definitions. The default document definition describes a mapping between Dublin Core tags embedded in XML content to Dublin Core statements. The Dublin Core is a metadata initiative that defines a set of standard properties for common Web-based objects such as Creator, Title, and Date. Our derived document definition will add one more statement for each document.

A simple declaration, as shown in the following, sets this document's creator metadata to the result of evaluating a certain

XPath:RdfStatement(subject='$uri',
  predicate="http://purl.org/dc/elements/1.1#Title",
  object="/rdf:RDF/s:Software/dc:Creator")

(The code above is a single line statement, but had to be indented to fit this format.)

To add or update the system default data, you should run the script populate.py that comes with 4SS. This downloads useful data from, ftp://ftp.fourthought.com to update your server. The downloaded data include some commonly used items like the Dublin Core document definition and a Docbook stylesheet (Docbook is a popular XML format for technical documentation).

The populate script is automatically installed into your demo application when 4SS is installed. On Unix-based machines, this is typically stored in /usr/doc/4SuiteServer-0.11 or /usr/local/doc/4SuiteServer-0.11. On Windows machines, these directories are typically c:\Program Files\Python or c:\Python20. Listing 2 shows the installation procedure for populating your 4SS-based application.

[molson@penny example]$ python /usr/doc/4SuiteServer-0.11/demo/populate.py
Downloading XML Documents
Downloading Stylesheets
Downloading DocDefs
Adding XML document: 'null'
Adding stylesheet: 'docbook_html1.xslt'
Adding stylesheet: 'presentation_toc.xslt'
Adding stylesheet: 'presentation.xslt'
Adding stylesheet: 'docbook_text1.xslt'
Adding document definition: 'dublin_core'
Adding document definition: 'docbook1' 

[molson@penny example]$ 4ss deserialize docdef software.docdef

The Content

We will use 4ss create document to add new content to the system from the command line. In the example download, there are two software listings, the XML files called software1.rdf and software2.rdf. To add these files to the system, we execute 4ss create document, giving it the document definition to use, the name of the file to add, and a list of aliases to assign to the resource in the system.

[molson@penny example]$ 4ss create container /softrepo
[molson@penny example]$ 4ss set acl --write=uo --world-read /softrepo

Then we add our sample download files to the repository. The 4SS repository is highly optimized for storing XML data, although it can pretty much store data in any format. When we add our .tar files to the repository, we specify the --imt option to set the Internet Media Type (IMT) of the file (in our case application/x-gzip). This IMT can, among other things, be used by the HTTP server when retrieving the content over the Web. Note that IMT is sometimes called "MIME type". See Listing 3 for the commands to add the content. Note that in a more complex project, you might consider placing the binaries in a separate container.

Fetching the Content

[molson@penny example]$ 4ss create document BASE_XSLT software.xslt softrepo/software.xslt

BASE_XSLT is a special document definition which tells 4SS to optimize this document as an XSLT stylesheet.

With the document added, you can now connect to the 4SS HTTP server (both plain Python and Apache servers are supported) with your Web browser and go to the page http://localhost:8080/softrepo/pong.xml. This will fetch the pong software description document from the repository.

If you are using a browser that understands the IMT text/xml such as Internet Explorer or Mozilla, then you can view the XML that was added to the repository. To tell the HTTP Listener that you would like to render the page (by running through XSLT) before it is returned, specify the xslt URI query argument http://localhost/softrepo/pong.xml?xslt=software.xslt.

Notice that the link on the page for the download of the package points to localhost as well. This link will also go through the HTTP Listener and fetch the resource that we added for pong-0.0.2.tgz. When it is returned to the browser it will specify the IMT that we defined when we added the resource to the system.

Generating Index Pages

To generate an index page, we will use some extension functions of 4SS for XSLT to access the RDF model. There are other solutions to the problem of generating an index page in 4SS. One such solution is to write a custom handler in Python for HTTP GET messages. Perhaps this would query the RDF model when index.html is requested. Another solution would be to use the 4SS event system to update a index.html document whenever a new document is added or removed from the system.

[molson@penny example]$ 4ss create document index.doc BASE_XML softrepo/index.doc
[molson@penny example]$ 4ss set acl --world-read softrepo/index.doc

We will use the extension function rdf.complete in our stylesheet to gather information about all of the software in the system. The extension function calls the complete method on the RDF model. The complete method allows you to search the RDF model for statements that match a specified pattern.

It takes up to three parameters: a subject, predicate, and, optionally, an object. Any of these parameters can be an empty string. It will return a list of statements that match all of the specified values. For example, if you pass in a subject of foo, and an object of bar it will return a list of statements that have a subject of foo, any predicate, and an object of bar.

4SS automatically creates RDF statements linking documents to their document definition. The subject of these statements is the document's URI, the predicate is http://schemas.4suite.org/4ss#metaxml.docdef and the object is the document definiton name. Knowing this, we use a simple complete call specifying the predicate and object to get a list of documents in our system using the software document definition.

The stylesheet we use to generate the index is called index.xslt. The template that matches the root of the source document has the first occurrence of rdf.complete. This function call does a complete operation on the RDF model for all of the statements that have http://schemas.4suite.org/4ss#metaxml.docdef as the predicate and software as the object. The result of the rdf.complete function call is a node set of Statement elements.

Each Statement element has three children: Subject, Predicate, and Object. We use xsl:apply-templates on the results of the function, and display each software item in the template that matches on Statement, as shown in Listing 4.

To view the generated index page point your browser to http://localhost:8080/softrepo/index.doc?xslt=index.xslt.

This index is not designed for looks; however, it is easy to see how this simple page could be expanded to display the software titles in any fashion. How each item is displayed can be modified in the stylesheet, and the data available for the stylesheet can be adjusted by adding more mappings to the document definition.