Scalable, document-centric addressing of semantic stores using the XPointer Framework and the REST architectural style.

5.1.1 Anatomy of an HTTP request - GET foo

Let's work through a couple of examples that illustrate how HTTP works. Let's say that you click on
http://www.myorg.org/mydoc

The sample request below assumes that the client is issuing an HTTP/1.1 request, which is common nowadays, for an XML representation ("Accept: text/xml") of the resource identified by that URI. The resource is located on the Internet host "www.myorg.org" and is found at the absolute pathname "/mydoc" on that host. The protocol scheme ("http") is not represented in the request. Instead, the protocol scheme was used to select the HTTP protocol. The (implied) port number (port 80) is used when the client initiates the HTTP connection with that socket on the identified host and also appears in the Host request header (when a non-default port is specified or the port is explicitly specified).

So, the request is:

GET /mydoc HTTP/1.1 Host: www.myorg.org Accept: text/xml

And the response is:

HTTP/1.1 200 Ok Accept-Ranges: xpointer Content-Type: text/xml <foo/>

The 200 is the HTTP status code for the response ³. Any 2xx status code indicates success. The 200 status code indicates success, but also informs the client that there is a response entity ⁴. In this case the response entity is an XML document - the one that we requested. The "xpointer" value for the "Accept-Ranges" response header indicates that this resource supports the Server-Side XPointer proposal for subresource addressing - more about that later.

5.1.2 PUT foo (updating a resource)

In this example we are going to work through the use of the HTTP PUT method to update the state of a resource. The semantics for PUT here are such that the state of the resource is updated to reflect the state of the interchanged document.

Since we are sending a request entity (our XML document) this time, we need two additional pieces of information: (1) the "Content-Type" header, which tells the server the Internet MIME Type of the document being interchanged ⁵; and (2) the actual document content ⁶.

The request:

PUT /mydoc HTTP/1.1 Host: www.myorg.org Content-Type: text/xml <!DOCTYPE foo [ <!ELEMENT foo (bar*)> <!ELEMENT bar (#PCDATA)> <!ATTLIST bar id ID #IMPLIED> ]> <foo> <bar id="a12">Hello</bar> <bar id="a13">World</bar> </foo>

The response:

HTTP/1.1 204 No Content

The 2xx status code indicates a successful request, i.e., the state of the resource was updated and now corresponds, more or less, to the request entity. A 204 status code is used to indicate that there is no response entity - that is, the resource did not send back a document along with the response. This is typical for PUT, and for DELETE as well.

The net effect of this request is that we have changed the state of the resource so that it now has two <bar/> elements, each of which has an "id" attribute. The request entity also includes a DOCTYPE declaration which provides a means for an XML processor to understand that the "id" attribute has ID type semantics. Declaring ID type semantics is critical for many subresource addressing schemes [xml:id], [xmlIDsemantics-32]. This example illustrates one way in which this information can be declared.

7.3.1 RDF Query Language and XPointer Scheme

For these examples, we define a custom XPointer scheme, called "graph-serql", based on the CONSTRUCTive subset of SeRQL. We restrict our scheme to CONSTRUCT based queries so that the resource addressed is always a graph. We further restrict our scheme to CONSTRUCT * where the store can always return a subgraph identified by the query (hence, minimal expressivity and inference). In a real system with a more liberal query language and expressive backend, there would be many opportunities for interesting sorts of negotiation. For example, it could be useful to keep distinct the return MIME types so that if the server determined that the query could not return a graph, but the only acceptable type was "application/rdf+xml", it would fault.

7.3.2 Query

Since this is not a W3C specified XPointer scheme, the rdf-query() scheme will need to be in an explicit namespace, we use http://www.myorg.org/xpointer-scheme/rdf-query#. Therefore we use the xmlns() scheme to setup the namespace context so that our graph-serql() scheme is correctly recognized by the XPointer processor. Once the namespace context has been established, we just wrap up the query and set the whole XPointer expression as the value of the "Range" header. When the request is received by the server it will use an XPointer processor to evaluate that express on behalf of the client and ship back the results.

The request:

GET /myStore HTTP/1.1 Host: www.myorg.org Accept: application/rdf+xml Range: xmlns(q=http://www.myorg.org/xpointer-scheme/rdf-query#) q:graph-serql( CONSTRUCT * WHERE {x} foaf:name "John Smith"; {x} foaf:mbox {mbox} USING NAMESPACE foaf = <!http://xmlns.com/foaf/0.1/> )

The response:

HTTP/1.1 206 Partial Content Content-Type: application/rdf+xml <rdf:RDF xmlns:rdf="...">...</rdf:RDF>

What is interesting about this example is that multiple subresources are being returned using a simple (vs. multipart) MIME type. We can get away with this since (a) we are using logical (vs. syntactic) addressing scheme; (b) we are presuming that any subresources of an RDF graph are also an RDF graph (i.e., you can't address below the level of a triple); (c) any RDF graph can be interchanged as a single "application/rdf+xml" document.

7.3.3 Update

If the semantic web server performs no inference at all, then update of a store is essentially equivalent to editing an RDF/XML document and is quite straightfoward. Again, it will typically be much easier to update the graph model than the model of the XML serialization. However, if the server performs significant inference, then update becomes much trickier. For example, adding an assertion might cause the knowledge base to become inconsistent and thus unreliable (assuming the store's formalism is rich enough to express contradictions). We can mitigate this by checking for consistency after each assertion (or batch of assertions), but this is a very computationally expensive operation. Deletes are much worse. If the assertion we are trying to delete is implied by the rest of the knowledge base, effective deletion might require deleting a further set of assertions (those which entail our target). There may be many different candidate sets of support for our assertion, leading to a non-deterministic choice of what else to delete. The most popular existing protocol for remote access for stores supporting languages with the expressivity of OWL, the DIG [DL Implementors Group] interface [DIG][DIG Interface], only allows inserts to the store. Effectively supporting deletes is a hard open problem.

7.5.1 Query

Since this is not a W3C specified XPointer scheme for Topic Maps, the tm-query() scheme will need to be in an explicit namespace, we use http://www.myorg.org/xpointer-scheme/tm-query. Therefore we use the xmlns() scheme to setup the namespace context so that our tm-query() scheme is correctly recognized by the XPointer processor. Once the namespace context has been established, we just wrap up the query and set the whole XPointer expression as the value of the "Range" header. When the request is received by the server it will use an XPointer processor to evaluate that expression on behalf of the client and ship back the results.

The request:

GET /myStore HTTP/1.1 Host: www.myorg.org Accept: application/rdf+xml Range: xmlns(q=http://www.myorg.org/xpointer-scheme/tm-query) q:tm-query( WHERE exists [ $toc @ S : * ] RETURN )

The response:

HTTP/1.1 206 Partial Content Content-Type: application/xtm+xml <xtm:topicMap xmlns:xtm="...">...</xtm:topicMap>

In this example we are retrieving a subgraph of the topic map. In conjunction with an appropriate fragment algorithm, not given here, we are able to retrieve a valid topic map graph as an XTM instance.

7.5.2 SNAPI (Semantic Network API) Binding

The RDF Net API, and more generally SNAPI, provides a basic set of operations based on REST principles that provide for the update, delete and construction of an RDF model or Topic Map. The protocol is defined in an abstract fashion that does not prescribe any specific binding. The approach presented here provides one alternative way of binding into the abstract protocol.

The SNAPI protocol is a genericised version of the RDF Net API but the basic principles are the same. There is a small collection of operations, Query, GetKnowledgeStructure, InsertKnowledgeStructure, RemoveKnowledgeStructure, PutKnowlegeStructure and UpdateKnowledgeStructure. In the case of Topic Maps the knowledge structure in question is a Topic Map fragment and in RDF it is an RDF Model. The operations all work on a by value basis. This means that in delete for example the values to delete are passed up by value. In addition, this protocol has no modification semantic whereby updates are done in the form of delete followed by an add. The following sections discuss the core protocol and relate it to how it can be implemented in terms of this framework. We also discuss how it can extend and refine the semantics of the protocol.

The basic Query operation in SNAPI is directly analogous to the example provided above. In this case the new XPointer scheme is used to wrap the query for the given topic or topics. SNAPI does not express the scope of the result, i.e. how many topics should be returned. Thus any implementation that returns a valid XTM fragment is compliant.

The SNAPI "Update" operation is perhaps the most interesting operation with regard to this framework. Update replaces the knowledge identified with information provided in the body of the request. The identification of the subresources to be replaced matches the intended use of the extended XPointer scheme in this framework. What this does is provides greater clarity of the protocol. In other bindings the query to identify the subresource has been done in a non-standard fashion. This framework provides an semantic basis on which to base a binding to the abstract protocol. It should be noted that update is defined in terms of a DELETE and POST. However, it could be defined in terms of a PUT where the resources removed are subresources of the Topic Map.

The DeleteKnowledge operation can make use of the machinery provided to identify and delete subresources of the Topic Map, i.e. Topics and Associations. Again, this provides a clearer understanding and communication of the semantics of the operation when it has a concrete binding such as this.

8.3.1 The Content-Range response header.

While the use of the Content-Range response header is indicated by the HTTP/1.1 specification, there is certainly room to clear up what its semantics are when using multiple pointer parts, e.g., "#element(a12)xpointer(/foo/bar)", or when using a logical addressing scheme (vs. a syntactic one). For example, the server response could indicate which pointer part matched or it could provide an xpointer (or other range-unit expression) which it choose to consider as canonical.

8.3.2 Should we register a node-list MIME type?

This proposal suggests the use of the MIME "multipart/mixed" type for the interchange of ordered XML node sets, such as those produced by an XPath processor. It might be convenient to register a MIME type that specialized "multipart/mixed" to indicate explicitly that the parts were XML information items, e.g., "multipart/xml-node-set".

8.3.3 Content negotiation.

While it is not drawn out specifically in the text above, this proposal takes the position that content negotiation occurs before the server applies the XPointer Framework, and, as a consequence, that the negotiated content type is found on the individual body parts in the multipart/mixed response entity. Some people may consider this "broken", in which case the client would need to negotiated directly for "multipart/mixed", but that leaves the client without a means to negotiated the representation to which the XPointer expression is applied by the server and results in an unworkable approach for resources that what to support content negotiation.

For example, consider a weblog that supports content negotiation. Depending on whether the client prefers "application/rss+xml" or "application/atom+xml", they can negotiate either an RSS or an Atom view of the weblog. Further, if the client wants to use subresource addressing as outline above, it is critical that they first negotiate the Internet MIME type of the representation since: (a) the semantics of the fragment identifier are determined by the Internet MIME type of the representation; and (b) the syntax of the two representations will be different, so different addressing expressions will be required depending on which view is negotiated.

8.4.1 Caching

Caching facilitates a scalable content distribution network by decentralizing access to leased representations of resource state. The HTTP protocol provides a variety of mechanisms that a server may use to describe this lease, including that NO lease is provided, i.e., the representation is not cacheable. At the same time, HTTP provides the client with mechanisms that it can use to drill through caching proxies and obtain an authoritative representation of the state of a resource.

Much of the protocol complexity concerning the HTTP Range has to do with breaking transparency in caching proxies in order to achieve a more efficient (scalable) content distribution network. In particular, example, for a 206 (Partial Content) response when the range-unit is "bytes" and the Content-Type is "multipart/byteranges", some HTTP/1.1 caches break transparency and, where permitted by the "lease terms", combine cached representations using adjacent or overlapping byte ranges.

However, for the purposes of this recommendation, we would recommend that HTTP proxies NOT break transparency and NOT attempt to combine physically or logically adjacent representations within cache.

Instead, if a resource wants to provide a cacheable representation, we recommend that it redirect the client to a URI whose representation provides a cacheable response to the subresource query, for example by responding with a 303 (See Other) status code and including the URI of a secondary resource that is responsive to the client's query. See section 10.3.4 of RFC 2616 [RFC 2616].

The HTTP/1.1 protocol (§ 13.10) makes provisions to support the automatic invalidation of cached resource representations after updates or deletion operations using POST, PUT or DELETE. This automatic invalidation facilitates the use of these methods for the direct manipulation of resource state. For example, after using PUT to update the state of a resource, any caching proxy which observes that request will invalidate any cached representations for that resource. This has the desired effect that a subsequent GET will obtain a current representation of the resource, which will then be cached (if allowed) for other clients. However, there are limitations. For example, a cache that does not see an update or deletion request will not invalidate its representation.

The question of correct (or at least good) cache control behaviors for the direct manipulation of subresources is closely related to the question of cache control for a resource that exposes different representations using different URIs. In both cases, a caching proxy needs to be able to determine the (parameterized) dependency relationships between the resource update and the cached representations that should be invalidated as a result of that update or delete operation.

This is a much larger issue facing semantic web servers, especially those that hope to expose both RDF and XTM facing service aspects. Unless the service reveals the dependency among the subresources in the response, a caching proxy will be unable to detect when an update on one subresource (or in one view) causes another subresource to become stale. If the server does not expose the dependency relationships, then it must either accept that stale representations will be provided to clients at least some of the time or it must disable caching altogether. This is very much a quality of service issue in which the role of the protocol is to enable the service and the client to achieve the compromise that best reflects their requirements.

8.4.2 Security

HTTP provides an extensible architecture for resource level security, e.g., through the configuration of HTTP proxies and network firewalls. In an HTTP security scenario, a proxy will typically examine three things: (1) the principle (if authenticated); (2) the URI of the addressed resource; and (3) the requested method (GET, POST, PUT, DELETE, etc.). Based on those three items, and perhaps on whether or not the channel is secure, the security agent will make a decision on whether or not to reject the client's request, to redirect the client, or to challenge the client to authenticate themselves.

This approach facilitates security policies that control which agents may read or directly manipulate the state of which resources. For example, a workflow agent may be issued credentials such that it can update the state of various business resources while other users may have read-only access to the same resources and agents not belonging to the host organization may be denied any access to the resource.

It is problematic to apply security constraints at the network level at a finer grain than the resource since there are multiple ways to ask for a given subresource, which could result in security loopholes that are difficult to identify and to plug. The same problem exists when there is more than one URI for a given resource, but it is made many, many times worse when you consider securing an extensible subresource addressing mechanism.

While it would be possible to expand the set of properties that a security proxy considers for a request, the multiplicity of different ways in which a client would ask for the same data in an extensible subresource addressing scheme could make it difficult to properly permit or deny a given request. One approach worth exploring would be to redirect the client to a URI which provided only a view of the addressed subresource and establishing security constrained based on those secondary URI. Ideally, those secondary URIs would be generatively produced in terms of some canonical subresource identification scheme known internally to the server and coordinated with the security agent.

8.6.1 Why not use the query string?

It suprised me, but a search of the HTTP/1.1 specification (RFC 2616) shows that HTTP/1.1 says nearly nothing about the meaning of the URL query string. Given this lack of constraint, one could use the query string to pass in a logical subresource address. However, this is likely to confound pre-existing application semantics for GET using the URL Query String. Further, unlike the XPointer Framework, this approach does not provide for extensible orthogonal addressing schemes. Finally, the XPointer Framework itself is explicitly a proposal for the semantics of the URL fragment identifier. This proposal suggests that the client simply indicate the logical subresources to be addressed. Getting the XPointer expression into the URL query string would mean re-writing the URL, not just adding information to the HTTP request.

There was a day when the practical length limits of a URI was a real concern. Today most software can handle relatively large URIs (4kb or more) (see § 3.2.1 and § 10.4.15 of [RFC 2616]). However, it is worth noting that the proposed approach places the fragment identifier on the Range header, and it does not occur as part of the Request-URI at all. This means that using XPointer together with the HTTP Range header, not only do you keep from intruding on existing application semantics for the URL query string, but the length of the Request-URI does not grow with the size and complexity of the query being made.

8.6.2 Use the HTTP Link header.

Roy Fielding offered a critique of this recommendation and a counter-proposal based on the HTTP Link header [Re: HTTP Methods] which would allow an agent to discover an alternative, parameterized URI by which the resource would expose access to subresources.

The HTTP Link header was described in RFC 2068 [RFC 2068], which was an earlier draft of HTTP/1.1, however it is not included in RFC 2616 [RFC 2616], which is the current version of the HTTP/1.1 specification. The semantics of the Link header are the same as those of the HTML [HTML 4.01 Specification]<LINK> element. The HTTP LINK and UNLINK methods from RFC 2068 were also dropped in RFC 2616. Those methods provided the client with an opportunity to directly manage the Link headers associated with a resource.

In [Re: HTTP Methods], Fielding writes:

In general, the HTTP Link header, the HTML LINK element, and the XLink Recommendation each provide a means to express inter-document links with labeled arc roles. There is an overlap with the role of RDF and XML Topic Maps to specify such labeled linking relationships. It would be a good thing if a standard was established that showed how the same linking information could be represented using each of these approaches. For example, see [Expressing Dublin Core in HTMK/XHTML] for a recommendation on how to encode the Dublin Core Metadata using the HTML <LINK> and <META> elements and [Dublin Core in RDF/XML] for a recommendation on how to encode the Dublin Core Metadata in RDF/XML.

8.6.3 Avoid data structure dependency in addressing

In a follow-up message [Re: HTTP Methods], Fielding highlights the concern with a data dependency in addressing schemes:

I unpack this as being several inter-related design issues. The first is how linked the addressing scheme is to the syntax of a resource representation. For example, the Groves paradigm emphasizes property sets, which may be coupled to the syntax or may be related to more abstract characteristics - so at some level this is a design decision within the architecture. The more invariant the addressing scheme with respect to the possible representations, the greater the longevity and evolvability of the web.

The second is that the client should be able to address the same (sub-)resource using the same (fragment) identifier regardless of the representation which they negotiate with the server. However, I think format specific fragment identifiers can be quite valuable (e.g., the svgView()) - and I think that the use of redirection may provide a solution that maintains the assignable name characteristic of the architecture. So, in this approach the server could choose to transparently redirect the client to a URI that serves as an "assignable name" for the addressed subresource(s). In this manner, any fragment identifier scheme (whether syntactic, abstract, or simply an opaque name) can be converted into an assigned name chosen by the service.

However, this approach clearly raises the level of complexity of the resource realization, since it must now maintain state about the secondary resources that are exposed to clients through redirection, including what dependencies they have on the resource from which they were derived, how they are secured, and the terms of their caching contracts and other service provisioning concerns.

This does not look like an either/or choice. Good choices for subresource addressing schemes can minimize the dependency on the syntax while transparent redirection can make it possible to have uniform addressing over subresources through assigned names.