Sahana Eden

S3XML

S3XML is a data exchange format for Sahana Eden.

S3XML is a meta-format and does not specify any particular data elements. The interface is entirely introspective to the underlying data model, thus the specific constraints defined in the data model also apply for S3XML documents.

Conventions

Name Space

In the current implementation of S3XML, no name space identifier shall be used. Where a name space identifier for the native S3XML format is needed (e.g. when embedding S3XML in other XML), it shall be: 

xmlns:s3xml="http://eden.sahanafoundation.org/wiki/S3XML"

    Character Encoding

    Generally, XML documents can specify their character encoding in the XML header: 

     <?xml version="1.0" encoding="utf-8"?>

    Sources in non-XML formats (JSON, CSV) used with S3XML on-the-fly conversion/transformation are expected to be UTF-8 encoded.

    All exported data are always UTF-8 encoded.

    Import Sources

    There are 3 different ways to specify or submit data sources for import:

    Files on the Server

    A source file in the server file system can be specified using the filename URL variable: 

    PUT http://<server>/<controller>/<resource>.xml?filename=<path>

    Multiple files can be specified as list of comma-separated pathnames: 

    PUT http://<server>/<controller>/<resource>.xml?filename=<path>,<path>

    Remote Files

    A source file can be specified by its URL using the fetchurl URL variable: 

    PUT http://<server>/<controller>/<resource>.xml?fetchurl=<url>

    Multiple files can be specified as list of comma-separated pathnames: 

    PUT http://<server>/<controller>/<resource>.xml?fetchurl=<url>,<url>

    Supported protocols are http, ftp and file, where file is interpreted in the server file system context. URLs of different protocols can be mixed.

    The specified URLs must be accessible either without authentication, or (if you specify credentials in the URLs) they must support unsolicited HTTP basic authentication - HTTP 403 retries are not handled by the interface.

    The URLs must be properly quoted (see http://www.w3schools.com/tags/ref_urlencode.asp  for more details), and must not contain commas.

    Request Attachments

    Source files can also be attached to a multipart-request. In this case the file extension of the source file must match the request URL file extension. Multiple files can be attached.

    Multiple Sources

    Where multiple sources are specified or attached, they are first converted and transformed one-by-one and then combined into a single element tree before import.

    Duplicate Resolution

    The S3XML Importer does not handle duplicates within the same source. As the order of elements in the resulting element tree is not defined, and the last update time attribute is not mandatory in source elements, there is no predictable rule of precedence.

    Records in the source must not be fractionated, but submitted in one element. Fractions of records will not be merged by the Importer, and which of the fractions finally would be imported is not predictable

    Source elements using unique keys are automatically matched with existing records. Where the match is ambiguous (e.g. a set of keys matching multiple existing records), the import element will be rejected as invalid. For certain resources, the server may have additional duplicate finders and resolvers configured. How duplicates are handled by these resolvers, can differ from resource to resource.

    The duplicate resolution strategy in standard import mode is to update the existing record with the values from the source record. In synchronization mode the default strategy is to accept/keep the newest data (the last update time attribute is mandatory in this case).

    XML Format

    Document Types and Structure

    S3XML defines 3 types of documents:

    Document Type
    		Description
    Schema Documents
    		describe the data schema for a resource
    Field Option Documents
    		describe the currently acceptable options for fields in a record
    Data Documents
    		provide the current contents (data) of resources
     

    Schema Documents

    Schema documents describe the data schema for a resource. Clients can use these documents e.g. for automatic generation of forms.

    Schema documents can be retrieved from Sahana Eden by sending an empty GET request (i.e. without source) to the create.xml method of a resource, e.g.: 

    GET http://localhost:8000/eden/pr/person/create.xml

    Document Tree: 

    <s3xml>
  <resource>
    <field>
    ...
    <resource>
      <field>
      ...
    </resource>
  </resource>
</s3xml>

    or (if requested from the fields.xml method): 

    <fields resource="name">
  <field/>
  <field/>
  <field/>
  ...
</fields>

    Note:

    • These documents can only be requested (GET), but not submitted for import
    • Schema documents support on-the-fly transformation (see chapter Web Services)
    • the URL query parameter ?options=true adds a list of field options to those fields where options are defined, and combined with the parameter &reference=true, even options for foreign key references will be included
    • the URL query parameter ?meta=true will include the meta fields (as <meta> elements). In data documents, the meta fields appear as attributes of the <resource> element

    Field Options Documents

    Field options documents describe the currently acceptable options for fields in a record. Clients can use these documents e.g. for automatic generation and/or client-side validation of forms.

    Field options documents can be requested from Sahana Eden by sending a GET request to the options.xml method of a resource, e.g.: 

    GET http://localhost:8000/eden/pr/person/options.xml

    Document Tree: 

    <options>
  <select>
    <option>
    <option>
    <option>
    ...
  </select>
  <select>
    ...
  </select>
  ...
</options>

    Note:

    • the field URL variable can be used to specify a particular field in the resource, the enclosing <options> element would then be omitted (i.e. <select> becomes root element)
    • on-the-fly transformation of field options documents is not supported
    • Field option documents can only be requested (GET), but not submitted for import

    Data Documents

    Data documents provide the current contents (data) of resources.

    Data documents can be requested from Sahana Eden by sending a GET request to the URL of the resource, e.g.: 

    GET http://localhost:8000/eden/pr/person.xml

    Data documents can be submitted to Sahana Eden by sending PUT requests to the URL of the resource, e.g.: 

    PUT http://localhost:8000/eden/pr/person.xml

    Note that sending data with POST will enter an interactive review of the source data before importing them, thus POST cannot be used by merely non-interactive clients.

    Document Tree: 

    <s3xml>
  <resource> <!-- primary resource element -->
    <data> <!-- field data -->
    <data>
    ...
    <resource> <!-- component resource inside the primary resource -->
      <data>
      <data>
      <reference/> <!-- reference -->
      ...
    </resource>
    <reference/> <!-- reference -->
    <reference> <!-- reference with embedded resource element -->
       <resource>
         <data>
         ...
       </resource>
    </reference>
  </resource>
</s3xml>

    Components

    Components of resources are <resource> elements nested inside the master <resource> element. Component records will be automatically imported and the required key references be added (=no explicit reference-element required).

    Foreign key references of component records to their primary record will not be exported, and where they appear in import sources, they will be ignored.

    Components of components are not allowed (maximum depth 1), and where they appear in import sources, they will be ignored.

    References

    Foreign key references (except those linking components to their primary record) are represented by <reference> elements.

    Foreign keys can be importable UIDs (uuid-attribute, which will be both imported and used to find and/or link to existing records in the DB) or temporary UIDs (tuid-attribute, which will not be imported but only used to find records within the current tree), If a <resource> element with a matching UID key attribute is found in the same tree, it will be automatically imported.

    References inside referenced elements will be resolved (unlimited depth) and also be imported. Circular references will be detected and properly resolved.

    Multi-references (list:reference type in web2py) use a list of UID keys separated by vertical dashes like uuid=|uid1|uid2|uid3|. The leading and trailing vertical dashes must be present.

    If a <resource> element is nested inside the <reference>, either or both of the UID keys can be omitted. Where both keys are however used, they must match. Multiple embedded <resource> elements are allowed for multi-references.

    Element Descriptions

    <s3xml>

    This is the root element (in schema and data documents). 

    <s3xml success="true" results="2" domain="mycomputer" url="http://127.0.0.1:8000/eden" latmin="-90.0" latmax="90.0" lonmin="-180.0" lonmax="180.0">
   ...
</s3xml>
    Parent elements: none (root element)
    Child elements: <resource>
    Contents: empty

    Attributes:

    Name Type Description mandatory?
    domain string the domain name of the data repository no
    url string the URL of the data repository no
    success boolean true if the page contains any records, otherwise false no
    results integer the total number of records matching the request no
    start integer the index of the first record returned (in paginated requests) no
    limit integer the maximum number of records returned (in paginated requests) no
    latmin, latmax, lonmin, lonmax float geo-location boundary box of the results no

    <resource>

    This element represents a record (in data documents) or a database table (in schema documents). 

    <s3xml>
  <resource name="xxx_yyy">
     ...
  </resource>
</s3xml>
    Parent elements: <s3xml>, <resource>, <reference>
    Child elements: <resource>, <data>, <field>
    Contents: empty

    Attributes:

     Name  Type  Description mandatory?
     name  string  the name of the database table
    		 yes
     uuid  string  a unique identifier for the record  no*
     tuid  string  a temporary unique identifier for the record  no*
     created_on  datetime  date and time when the record was created  no**
     modified_on  datetime  date and time when the record was last updated  no, default: time of the request** ***
     created_by  string email-address of the user who created the record  no
     modified_by  string email-address of the user who last updated the record  no
     owned_by_user  string email-address of the user who owns the record*****  no
     owned_by_role  string  name of the user group who collectively own the record*****  no
     mci  integer  master-copy-index  no, default: 2*** ****

    • (*) Records will be identified within the input file by their uuid, or, if no uuid is specified, by their tuid.
    • (**) as YYYY-MM-DDTHH:mm:ssZ, always UTC
    • (***) the last update date/time and mci are required in synchronization
    • (****) the master copy index specifies how often a record has been copied across sites, see below
    • (*****) record ownership will be retained if the record owners can be matched against existing users/user groups

    The uuid will be stored in the database together with the record. If uuid is present and matches an existing record in the database, then this record will be updated. If there's no match or no uuid specified in the resource element, then the importer will create a new record in the database (and automatically generate a uuid if required).

    The mci - master-copy-index - indicates how often this record has been copied across sites:

    • when importing a new record the mci value is always *imported* as-is from the source
    • when updating a record, the mci of the database record remains unchanged
    • the mci of a record is *exported* as its current database value + 1.
    • the repository first creating a record sets mci=0 in the database record, which appears as mci=1 in the exported XML.
    • a copying site then imports mci=1 into its database, which appears as mci=2 in its export XML, and so forth...

    The mci can be used to filter records for whether they have been originated at a repository or not. If there's a fixed set of synchronization paths between a number of Sahana Eden instances, the mci can be used for conflict resolution. If the mci is not specified, it defaults to 2.

    MCI handling is optional for non-synchronizing peers.

    <data>

    This element represents the value of a single field in the record. 

    <s3xml>
  <resource>
    <data field="fieldname" value="value">...</data>
  </resource>
</s3xml>
    Parent elements: resource
    Child elements: none (leaf element)
    Contents: Text

    Attributes:

    Name Type Description
    		mandatory?
    field string the field name in the record yes
    value JSON
    		the native field value no
    url URL the URL to download the contents from* no
    filename filename the filename of the attached contents* no

    (*) If the field is for file upload, a url attribute should be provided to specify the location of the file. The importer will try to download and store the file (file transfer) from that URL (pull). It is also possible to send the file together with the HTTP request - in this case the filename must be specified instead of the url (push). The push variant for uploads is meant for peers which do not support pulling for some reason (e.g. mobile phones). Normal servers would always provide a URL for download in order to allow the consuming site to decide which files to download and when (saves bandwidth).

    The text node in the data element provides a human-readable representation of the field value.

    The value attribute contains a JSON representation of the field value, retaining the original data type (i.e. strings must be double-quoted) except for date, time and datetime values, which are to be represented as simple strings in the respective standard format (no double quotes). The standard format for datetime values is YYYY-MM-ddTHH:mm:ssZ (ISO format, UTC), date shall be represented as YYYY-MM-dd, and time as HH:mm:ss.

    data elements representing passwords can contain the clear text password in the value attribute, or the encrypted password in the text node. Where a clear text password is given as value attribute, it will be stored encrypted, otherwise the password will be stored as-is. Note that clear-text representation of passwords will be accepted by the interface, but never be exported.

    <reference>

    Represents a foreign key reference. 

    <s3xml>
    <resource name="xxx_yyy">
        <reference field="xy" resource="aaabbb" uuid="urn:uuid:e4bcb9fd-d890-4f2f-b221-1d75fff79e2d"/>
    </resource>
</s3xml>
    Parent elements: <resource>
    Child elements: <resource>
    Contents: Text

    Attributes:

     Name  Type  Description  mandatory?
     field  string  the field name in the record  yes
     resource  string  the name of the referenced database table
    		 yes
     uuid  string  the unique identifier of the referenced record (foreign key)*  (yes)**
     tuid  string  a temporary identifier for a referenced record (foreign key)*  (yes)**
    (*) Referenced records would always be exported in the same output file. If a referenced record is found in the same input file, then it will be automatically imported.

    (**) Records will be identified within the input file by their uuid, or, if no uuid is specified, by their tuid.

    If the referenced record is enclosed in the reference element, then uuid and tuid can be omitted: 

    <s3xml>
   <resource name="xxxyyy">
       <!-- content of the record goes here -->
       <reference field="xy" resource="aaabbb">
          <resource name="aaabbb">
            <!-- content of the referenced record goes here -->
          </resource>
       </reference>
   </resource>
</s3xml>