Testing the HMT project’s technical underpinnings

In February, we noted the release of new draft specifications for the CTS URN notation that we use to cite texts, and the CTS protocol that we use to retrieve texts in the Homer Multitext project. Since the publication of the draft specifications, we have released updates of a suite of test data and of software using the test data to assess the compliance of a given CTS service with the current version of the protocol.

Together with version 1.6 of this software, the ctsvalidator servlet, we are today releasing version 0.9.0 of our implementation of the CTS protocol, sparqlcts. The new version of sparqlcts passes 100% of the ctsvalidator tests.

To recapitulate what we have released in 2014 in our work on CTS:

• Formal specifications for the Canonical Text Services protocol, and CTS URNs. The specifications include Relax NG schemas for a CTS Text Inventory (the catalogue of a CTS library), and Relax NG schemas for validating the responses to CTS requests.
• A test data set, documented in a valid CTS Text Inventory, and available in three formats:
• valid and well-formed XML
• tabular data in simple delimited text files
• RDF triples in .ttl format
• A set of 68 tests applying CTS requests to the test data set. The tests are defined in an XML file listing the request and parameters to be submitted to a running CTS installation. For each test, a corresponding XML file gives the expected responses to the request.
• The CTS Validator, a web-app that runs the tests against any online CTS service hosting the corpus of test-data.
• An implementation of the Canonical Text Services, sparqlcts, a Java web-app drawing its data from a SPARQL endpoint.  When the SPARQL end point is hosting the corpus of test data, sparqlcts passes 68 out of 68 of our defined tests.

This of course does not mean that sparqlcts is necessarily flawless (there may be problems that ctsvalidator does not test for), but it is an important milestone. One of the most profound implications of digital scholarship is that when we can automate the testing of digital work, we should invert the humanist’s traditional order of composition and assessment: specify the automated test first, then work until you pass the test. This applies to the software we use, too. When we next update our online services, we can be confident that our text service has successfully passed 100% of a challenging series of tests.

Links

• ctsvalidator (servlet, and tests): github repository
• sparqlcts: github repository

Christopher Blackwell and Neel Smith, project architects

Technically speaking ...

For over a decade, the Homer Multitext project has been exploring how to represent a multitext in digital form.  For some of our essential work, we have been able to adopt well understood practices (such as how to use XML markup to structure a diplomatic edition of a text).  In other aspects of our work, we are faced with issues that have not been explored in prior work on digital scholarship, and have had to define new standards.

We have devoted special attention to the fundamental question of how to cite texts in a form that is independent of any specific technology and sufficiently rigorously defined for computers to use.  We have defined the syntax and semantics for a notation for citing texts that is based on the Internet Engineering Taskforce's Uniform Resource Name (URN) notation.  We call this notation the Canonical Text Service URN, or CTS URN.

We have also defined a protocol for a networked service that understands the CTS URN notation, and can retrieve passages of texts.  Unsurprisingly, we call this the Canonical Text Service protocol, or CTS protocol.

We have worked hard to ensure that the technical design of our notation and service fully satisfies the needs of the Homer Multitext project, but is not limited to or in any way specific to the HMT project's corpus of texts.  Both of us have applied the CTS notation and CTS service protocol to a range of other projects, not limited to Greek or Latin texts.  As our work on these two technical projects has matured, we have found more and more interest in it from scholars working with canonically citable texts.

This week, we were able to complete revisions for a new version of the specification for both the CTS URN notation and CTS protocol.  It was especially gratifying that we were able to complete this work during a visit to Leiden University, where we were graciously hosted by Ineke Sluiter and her colleagues, a new group of collaborators on HMT who first participated in the summer 2013 seminar at the Center for Hellenic Studies.

The specifications:

• The CTS URN specification (version 2.0, release candidate 1): http://www.homermultitext.org/hmt-docs/specifications/ctsurn/
• The CTS protocol specification (version 5.0, release candidate 1): http://www.homermultitext.org/hmt-docs/specifications/cts/

Christopher Blackwell and Neel Smith, HMT project architects

The HMT Manuscript Browser · News and Updates

Summary: The old version of the Manuscript Browser at chs75.harvard.edu is going away. Its replacement will appear at amphoreus.hpcc.uh.edu. In the interim, a beta application is available at http://beta.hpcc.uh.edu/tomcat/hmtapps/. 

An archive of every digital image in the HMT library is always available for manual and automatic downloading.

Background

In 2007, shortly after we completed photography of manuscripts Marciana 822, Marciana 821, and Marciana 841, we wrote and posted online an application for browsing the high-resolution images. This application used the Google Maps API to allow users to manipulate the images; it allowed searching by the enumeration of manuscript folios and by poetic book and line.
Technological and scholarly considerations alike dictate that we take this original application offline. Using the Google Maps API is not ideal for our purposes. Google Maps assumes that it is displaying images of the surface of the earth, for one thing, and using it for serious work on manuscripts requires many workarounds, compromises, and hacks. More serious, though, is the problem of scholarly citation. In the five years since we wrote the original facsimile browser, our understanding of scholarly citation in a digital world has matured. By standardizing on citation with URNs, we can deliver effective applications today, using scholarly material that can remain valid in the future. The old application does not support citation using URNs; its replacements do.

What Will Happen

The application at chs75.harvard.edu will cease working at any moment. New “Facsimile Browser” applications are being developed on two hosts: beta.hpcc.uh.edu and amphoreus.hpcc.uh.edu. beta hosts experimental versions of HMT applications; when we consider versions of an application ready for regular public use, we will also host them on amphoreus.
In the meantime, users can use two resources for accessing digital images of the Homeric manuscripts:

• Our beta version of a Facsimile Browser is online at http://beta.hpcc.uh.edu/tomcat/hmtapps/.
• As always, users can download complete sets of images of any of our manuscripts from http://amphoreus.hpcc.uh.edu.

About the Beta Facsimile Browser

The application that lives at http://beta.hpcc.uh.edu/tomcat/hmtapps/ allows users to look up digital images of manuscripts, and associated data, by requesting citations of folios or by poetic book and line.
We wrote this application to explore a URN-driven graph of HMT data. Its state as of December 26, 2012 reflects its origin as a test-bed for this way of integrating our data.
The application will evolve over the next few weeks as we redesign the user interface with a wider audience in mind. Once that redesign is complete, we will put the application on the amphoreus.hpcc.uh.edu server and announce its public availability.

A Final Word

The Homer Multitext consists of data–images, texts, collections of regularly structured data. It is a reality of the twenty-first century that technology changes rapidly, and thus how we interact with data will change rapidly. From the outset, the editors of the HMT assumed that any particular end-user application would have a lifespan of only a few years. Our goal has been to ensure that the data remain accessible regardless of technological changes, and that the discoveries and insights generated through widespread and free access to that data is never trapped in forms dependent on any particular technology.

From graphs to applications

In 2012, the quantity of material collected by the HMT project has grown rapidly.  To cope with this, we have been developing an automated system to identify the relations among all citable objects in the HMT data archive (texts, images, artifacts like manuscript pages, to name a few).   In mathematical terms, these relations form a graph.

In the HMT graph, all nodes are identified by URN values (CTS URNs for texts, or CITE URNs for other kinds of objects).  This simple, consistent reference format made it easy for us to develop a network service  for working with the HMT graph:  supply the service with a URN value, and the service finds all links to that URN.

This is an important development for the long-term development of our digital multitext, and will certainly be the subject of future blog posts.  For today, I simply want to announce a test site with end-user applications built on the graph service.

Like our other services for retrieving HMT data, the graph service replies with a simple XML format;  as in our other service implementations, we can include XSLT stylesheets to format these graph descriptions as web pages for human readers.
We have written three sets of stylesheets that turn the graph data into three quite distinct applications:

• a facsimile browser, for reading diplomatic edition of texts alongside documentary images
• a multitext reader, for reading multiple versions of a single text
• a graph navigator, for exploring links in the HMT project graph

You'll find test versions of all three of these apps at our new HMT Apps test page:  http://beta.hpcc.uh.edu/tomcat/hmtapps/

If you're curious about how the graph service works, try viewing the XML source of one of the application's web pages.  If you just want to try out an app, feel free.  Expect that the test versions on this site will evolve rapidly over the next several months.  We'll post announcements on this blog when we install more static release versions elsewhere.

Announcing the Open Paleography project

The Homer Multitext blog is an appropriate forum to announce a new project growing directly out of experience with the Homer Multitext project, and developing technology that will contribute directly to future work on the HMT project.

For almost three years, the HMT project has been collecting in structured notebooks paleographic observations about the manuscripts we are editing.  With the announcement of the Open Paleography project, we aim to expand this work to a general crowd-sourced collection of paleographic observations.

The Open Paleography project differs from other projects with similar aims in its application of the CITE architecture.  Paleographic observations identify a physical artifact, a textual passage, and a region of interest on a documentary image using technology-independent, machine-actionable URNs.  In turn, each observation itself is identified with a CITE URN.  The openly licensed data set is exposed to the software and end-users in the following ways:

• because all the data sets are CITE Collections, they are available through the CITE Collections Service API
• because data are stored in Google Fusion Tables,  they are available both through Google's programmatic API and through the user interfaces to Google Fusion Tables

The Open Paleography project is currently testing and helping develop two generic applications that work with any CITE Collection. The first is a collaborative CITE Collection editor allowing authorized contributors to add to a CITE Collection from a Web browser.  The second is a general querying and viewing application for end users.  Both of these applications will find immediate application in the HMT project.

Home page of the Open Paleography project:  http://shot.holycross.edu/op

ICT · The Image Citation Tool

The Homer Multitext has developed an Image Citation Tool for use with digital images served by the CITE Image Service.

Humanist scholarship is the act of forging new connections between ideas, and placing those connections before the eyes of readers. Those readers must be free and empowered to judge the value of the new connections.

The heart of humanist scholarship, then in quotation. Robert Sokolowski calls quotation a ‘curious conjunction of begin able to name and to contain’;* V.A. Howard is more succinct: quotation is ‘replication-plus-reference’.** We would re-phrase this as “reproduction plus citation”.

Reproduction in a quotation allows us to talk about a particular artifact of human thought without the burden of reproducing its entire context. As a practical matter, it is easier to say “Μῆνιν ἄειδε θεὰ Πηληϊάδεω Ἀχιλῆος” than to reproduce the whole of the Iliad.

Citation in a quotation provides a link from the reproduced selection back to context of that selection. “Iliad 1.1” names our quotation, as Howard says, but also invites the reader to explore Iliad 1.2, all of Book 1 of the Iliad, or the whole poem.

This is easy stuff. We were all taught to do it early in our educations, and we take it for granted.
It is easy, that is, with texts. Images are a different matter. How do you “quote” from an image? Most scholars use images from time to time in their work; few of those uses meet the rigorous standards of “quotation” that we take for granted with texts. The general practice is to open a digital image in an image-editor, cut out the area of the image under discussion, and paste that image into a document or web-page. If the image is cited, the citation is often to a page in a book that has published a version of the image, a museum’s accession number identifying an original work of art, or a URL to a web-page on which a digital version of the image appears. The citation does not provide a path from the selection to its context.

Nor is this kind of “image quotation” actionable in the way that a textual quotation (reproduction+citation) is. Given “Iliad 1.1”, it is simple to answer the question “What comes next?”. It is simple to know that “Iliad 1.1–1.10” includes Iliad 1.5. Given a cut-copied-pasted snippet of a digital image, and perhaps a citation to a web-page, it is not possible to answer with any degree of precision “Where is this snippet in its larger context? What parts of the image are adjacent to this snippet?” Given two snippets, it would require considerable computation to determine whether Snippet A contains Snippet B.

Since the Homer Multitext is committed to image-based scholarly editing, and subsequent discussion and argument based closely on digital imagery of manuscripts and papyri, we have worked to developed a means of quoting images as rigorously, and usefully, as we can quote texts.

The CITE Image Service allows us to identify images with canonical citations in URN-notation. These URNs take the form: urn:cite:hmt:chsimg.VA094VN-0597. This points to a notional image, which might be delivered at any scale, or by services hosted on various machines with various addresses.
A CITE Image URN can take a suffix that identifies a rectangular region-of-interest: urn:cite:hmt:chsimg.VA094VN-0597:0.3833,0.2441,0.0783,0.0463. This URN+ROI can resolve to a “quotation”, that is the region-of-interest on its own, or to a view of the larger image with the region-of-interest highlighted. These image-ROIs canonically cited with URN notation are concise, precise, and machine-actionable mechanisms for image quotation in the best tradition of humanist scholarship.

To help ourselves, our collaborators, and anyone else interested in working with the openly licensed images in our Homer Multitext Image Collection, we have developed a web-based tool for defining regions-of-interest on our digital images and capturing canonical citations for them. This is the Image Citation Tool.

A lengthy introduction to the tool and its use are included in our Homer Multitext documentation pages.   Downloadable source-code for the tool—a relatively simple web-application in HTML, CSS, and Javascript—is available from the HMT’s code repository.

References

* Sokolowski, Robert. “Quotation.” The Review of Metaphysics 37.4 (1984) : 699-723. Print. 24 May 2011.
** Howard, V.A., “On Musical Quotation”, Monist 58 (1974) 310.

Announcing version 1.0 of CHS Image Services

A recent post described our reorganization of the Homer Multitext project's archival image data.
We have been experimenting for some time with a preliminary internet service for working with canonically citable images.

Today, we are releasing version 1.0 of our implementation of the CHS Image Service, an extension to the CITE architecture's Collections.  The CHS Image Service supports extended citation of images including regions of interest, and provides methods for gathering various kinds of information about a canonically citable image, including retrieving binary image data.  We plan to follow up on this release shortly with a formal specification of version 1.0 of the CHS extension to CITE Collections.  

In the mean time, if you are a developer interested in using canonically citable images, see this summary of CHS Image Services in our overview of the CITE architecture.

If you would like to run your own installation, see this guide to running a CHS Image Service.

If you are an end-user who currently uses HMT apps, you should see no changes at all (except perhaps that the web pages at our reference installation of CHS Image Services, amphoreus.hpcc.uh.edu/tomcat/chsimg/, may have a little nicer skin) — that's a feature of the design of chsimg 1.0.  What you should expect to see over the next year or two is more rapid development of applications drawing on chsimg to incorporate canonically citable images in new ways to visualize and explore the Homer Multitext project's increasingly rich archive.

HTML CTS Kit

Abstract

Announcing for download a package of html, javascript, and css that allows embedding into an html page passages of text served by a Canonical Text Services implementation, by inserting a CTS-URN into a <blockquote></blockquote> element, with a @class attribute “cts-text”. E.g.:

<blockquote class="cts-text" cite="urn:cts:greekLit:tlg0012.tlg001.msA:1.1>Iliad, 1.1</blockquote> 

Background

CTS stands for Canonical Text Services; it is the networked service developed for the Homer Multitext that allows discovery and retrieval of passages of texts using citations in URN format. In short, if an electronic edition or translation of a text is in a CTS service, a user or machine can request that passage using a documented protocol. All of the electronic texts edited for the Homer Multitext are exposed via a CTS service.

The Homer Multitext (HMT) has also developed an image service, which allows citation by URN to images and parts of images.

HTML CTS Kit

Anyone with experience in making web-pages in HTML knows how easy it is to include an image in a page:

<img src="http://url-to-image"/>

A web-browser will interpret that tag as a request to embed the identified image in the page, to show the image to the reader. In other words, the http://url-to-image will be resolved to the image itself.

This is how citation has always worked… an author includes a citation in a piece of writing, and the reader can resolve the citation to the quotation to which it points. In the digital age, we expect that resolution to happen automatically.

Web-browsers have always allowed urls to images to be resolved for readers, even when the images are on different servers from the server hosting the HTML page. It would be nice if text were as easy.
Canonical citation has been the foundation of Classical philology for centuries, and it is the heart and sole linking mechanism of the HMT. In the digital realm we have found this to be a rich, scaleable, and flexible method for building a complex and diverse digital library. The HTML CTS Kit is a package of files that allows authors working in HTML to cite texts concisely using canonical CTS URNs, and have those URNs resolve to the passages to which they point.

Here is a demonstration of a page that uses URNs to cite both CTS texts and a region-of-interest on an image. The page that the reader sees has rich content; the underlying source is very concise:

<h1>High Resolution Scholarship</h1>

<p>The first five lines of the <i>Iliad</i> on the Venetus A:</p>

<blockquote class="cts-text" cite="urn:cts:greekLit:tlg0012.tlg001.msA:7.1-7.5">

Iliad 7.1-7.5</blockquote>

<p>The Summary of Book 7 from the Venetus A, in Dactylic Hexameter:</p>

<img class="cite-img" 

   src="urn:cite:hmt:chsimg.VA091RN-0263:0.2412,0.0845,0.4013,0.0295"/>

<blockquote class="cts-text" cite="urn:cts:greekLit:tlg5026.chs01.msA:7">

Book 7 Summary</blockquote>

How it Works

An author can discover the CTS URN for a text by browing the Homer Multitext’s CTS service, or any other implementation of CTS, such as this one, from Furman, containing Biblical texts. The URN for “Homer, Iliad, Edition based on the Venetus A, Book 7, lines 1–15” is:

urn:cts:greekLit:tlg0012.tlg001.msA:7.1-7.15.

To cite this passage in an HTML page, an author can use the standard HTML5 blockquote element. This element is defined as allowing an attribute named cite; that attribute will hold the CTS URN. In order for the scripts in the HTML CTS Kit to recognize this blockquote as containing a CTS URN that should be resolved, the blockquote element should also have a class attribute, with a value of “cts-text”. blockquote elements should not be empty, so it is a good idea to put a human-readable citation inside the element; if the citation cannot resolve for any reason, that will be what the reader sees. The final citation will look like this:

<blockquote class="cts-text" cite="urn:cts:greekLit:tlg0012.tlg001.msA:1.1>Iliad, 1.1</blockquote> 

Assuming the correct scripts and stylesheets have been included in the HTML page (instructions are here), this is what will happen.

• When the page loads, the script will find all of these <blockquote>…</blockquote> elements and perform an AJAX request, sending a “GetPassagePlus” request for each URN.
• As the results of those requests come in, the scripts will process the XML returned by the CTS Services, using XSLT stylesheets to turn the XML into fragments of HTML.
• Those HTML fragments will be inserted into the page.
• CSS stylesheets will give some attractive presentation to the newly inserted quotations.

The XSLT and CSS is of course entirely customizeable by anyone who wants to change the structure or appearance of the resulting texts; what we provide is simply a default.

HTML CTS Kit uses the jQuery Javascript Library for most of its work, and the Sarissa library to process XSLT via Javascript.

License and Download

Like all code and data in the Homer Multitext, the HTML CTS Kit is available under an open-content license, and we hope people will find it useful. The official guide is here. Download links are here.

Idea for improvement

None of this works inside Blogger. If anyone can make it work with Blogger, we would love to hear about it!