Merge pull request #48 from resync/develop

Release 2.0.1

.gitignore

@@ -2,18 +2,22 @@
 build
 dist
 MANIFEST
+.tox
+.eggs
+*.egg
+# Testing
+.coverage
+htmlcov
 # Files generated by client / local config
 .resync-client-status.cfg
 resync-client.log
-resync.library.cornell.edu_sim100
-resync.library.cornell.edu_sim1000
-resync.library.cornell.edu_sim10000
 localhost_8888
+tmp
 # Related to packaging
 resync.egg-info
 # Other...
 .DS_store
 *.pyc
 *~
-rc
-re
+.cache
+

.travis.yml

@@ -1,12 +1,16 @@
 language: python
 python:
-  - "2.6"
-  - "2.7"
-# command to install dependencies
-# need "python setup.py install" to install resync so that client tests work
+  - "3.5"
+  - "3.6"
+  - "3.7"
+  - "3.8"
 install:
-  - pip install requests
-  - pip install python-dateutil
+  - pip install coveralls pycodestyle pep257 testfixtures
   - python setup.py install
-# command to run tests
-script: py.test
+script:
+  - python setup.py test
+  - pycodestyle --ignore=E501,W503 resync tests resync-sync resync-build resync-explorer
+  - pep257 resync bin tests resync-sync resync-build resync-explorer
+  - coverage run --source=resync setup.py test
+after_success:
+  - coveralls

CHANGES.md

@@ -1,12 +1,65 @@
-resync change log
-=================
+# resync change log
 
-The first two components of the version tags are tied to the ResourceSync 
-core specification version. Versions 1.0.x implement the v1.0 
-ResourceSync specification which was standardized as ANSI/NISO Z39.99-2014
-<http://www.openarchives.org/rs/1.0/toc>.
+v2.0.1 2021-03-23
+  * Route all URI and file requests through `resync/url_or_file_open.py` so that settings such as authentication headers can be consistently applied
+  * Do not exclude any directories from sync by default, specify with --exclude
+  * Fix comparison of using possibly multiple checksums
+  * Improve test coverage
+  * Remove Python 2 cruft
 
-v1.0.0 2014-06-XX
+v2.0.0 2020-12-16
+  * Supports ResourceSync v1.1 ANSI/NISO Z39.99-2017 <http://www.openarchives.org/rs/1.1/toc> as default
+  * Support for the prior v1.0 ANSI/NISO Z39.99-2014 <http://www.openarchives.org/rs/1.0/toc> is retained with `spec-version='1.0'` option in scripts and `spec_version='1.0'` in various classes
+  * Split old `resync` script into `resync-sync` and `resync-build`
+  * Move scripts from `bin` dir to base dir for easier testing/development (now that there is no conflict with the module name)
+  * Switch from optparse to argparse, use exclusive argument group for commands
+  * Add --access_token option to pass bearer token with web requests
+  * Add --delay option to pause between successive web requests
+  * Drop Python 2.7, 3.3 & 3.4 from tests, add 3.7 & 3.8
+  * Fix various depracation warnings and errors for 3.7 & 3.8
+  * Switch from pep8 to pycodestyle in tests
+  * Change README from RDS to github flavored Markdown
+  * Move libraries to support tests into test/testlib
+
+v1.0.9 2018-10-23
+  * Add -t/--tries and -T/--timeout options (https://github.com/resync/resync/issues/34)
+  * Includes partial replacement of urllib.request with requests
+
+v1.0.8 2017-08-30
+  * Fix support for explicitly writing ResourceList objects as indexes (https://github.com/resync/resync/issues/31)
+  * Change argument names for local modes to have `--write-` prefix, e.g. --write-resourcelist
+  * Improve modes to detect capabilities
+  * Abandon support for Python 2.6 (https://github.com/resync/resync/issues/23)
+
+v1.0.7 2017-03-27
+  * Fixed silly error in `resync-explorer` introduced in 1.0.6
+
+v1.0.6 2017-03-20
+  * Fixed md5 hash format (https://github.com/resync/resync/issues/25)
+  * Added support for sha-1 and sha-256 hashes
+
+v1.0.5 2017-01-30
+  * Fixed to support non-ASCII URIs/filenames (https://github.com/resync/resync/issues/22)
+  * Added tests for Python 3.6
+  * Temporarily disabled tests for Python 2.6 due to problems with coverage (https://github.com/resync/resync/issues/23)
+  
+v1.0.4 2017-01-27
+  * pep8 formatting
+  * Added contributing note
+
+v1.0.2 2016-05-06
+  * Python 3 fixes in resync-explorer
+  * Improved test coverage
+  * pep257 formatting
+
+v1.0.1 2016-03-22
+  * Work with Python 2 (2.6/2.7) and Python 3 (3.3/3.4/3.5)
+  * Improved test coverage
+  * Use defusedxml for safer XML parsing
+  * Changed parameter str to str_data in ListBase.parse(), str still supported
+    but deprecated. Issue pointed out by @uweschmitt
+
+v1.0.0 2014-06-18
   * Update for v1.0, Z39.99-2014 specification (http://www.openarchives.org/rs/1.0/)
   * Partial implementation of dump output (no read yet)
 

CONTRIBUTING.md

@@ -0,0 +1,24 @@
+# Contributing to resync
+
+Issues and pull requests are appreciated. I apologize if it takes me some time to respond to issues.
+
+## Submitting issues
+
+In the case of bugs please describe in detail, preferably with a recipe to repeat and demonstrate the problem. Although you should include test data as necessary, please be careful not to include secrets or passwords!
+
+If you suggest a new feature, please give an example of expected behavior and use. For changes to modules and methods a tests case would be idea.
+
+## Coding style
+
+If submitting a pull request:
+
+   * Understand that this code and any merged contributions are covered by an Apache 2.0 license
+   * Please discuss in a issue before submitting a pull request for significant changes.
+   * Please submit pull requests against the `develop` branch (changes are collected there before making a release).
+   * Please follow [PEP8](https://www.python.org/dev/peps/pep-0008/) and [PEP257](https://www.python.org/dev/peps/pep-0257/) style rules.
+   * Please don't repeat code.
+   * Please cover the code with tests.
+
+## Local conventions
+
+   * Tests: Use `from tests.testcase_with_tmpdir import TestCase` in place of `unittest.TestCase` to get a test object with `self.tmpdir` as temporary location to write to.

MANIFEST.in

@@ -1,2 +0,0 @@
-include *.txt
-recursive-include docs *.txt

README

@@ -1,118 +0,0 @@
-======
-resync
-======
-
-.. image:: https://travis-ci.org/resync/resync.png?branch=master
-  :target: https://travis-ci.org/resync/resync
-  :alt: Build status
-
-**resync** is a ResourceSync client and library in python. `ResourceSync 
-<http://www.openarchives.org/rs/>`_ is a synchronization framework for 
-the web consisting of various capabilities that allow third party 
-systems to remain synchronized with a server's evolving resources. 
-The capabilities can be combined in a modular manner to meet local 
-or community requirements.
-
-Client usage
-------------
-
-Typical client usage to synchronize from a source at 
-``http://source.example.com/`` to a set of local files would be::
-
-    resync http://source.example.com/
-
-which will create or update a local directory ``./source.example.com``. 
-Alternatively, the destination directory may be specified explicitly::
-
-    resync http://source.example.com/ /tmp/my_copy
-
-Option details and a number of different modes are described with::
-
-    resync -h
-
-Library usage
--------------
-
-Typical library use in a source (create and output a Resource List)::
-
-    from resync import Resource,ResourceList
-
-    rl = ResourceList()
-    rl.add( Resource('http://example.com/res1', lastmod='2013-01-01') )
-    rl.add( Resource('http://example.com/res2', lastmod='2013-01-02') )
-    print rl.as_xml()
-
-Typical library use in a destination (get and examine a Capability List):: 
-
-    from resync import CapabilityList
-
-    # Read Capability List and show supported capabilities
-    cl = CapabilityList()
-    cl.read("https://raw.github.com/resync/resync/1.0/resync/test/testdata/examples_from_spec/resourcesync_ex_2_6.xml")
-    for resource in cl:
-        print "supports %s (at %s)" % (resource.capability,resource.uri)
-
-Installation
-------------
-
-The client and library are designed to work with Python 2.6 or 2.7.
-
-**Automatic installation**::
-
-    easy_install resync
-
-rsync is listed in `PyPI
-<http://pypi.python.org/pypi/resync>`_ and can be installed with 
-``pip`` or ``easy_install``. 
-
-**Manual installation from github**::
-
-    cd /tmp
-    git clone git://github.com/resync/resync.git
-    cd resync/
-    python setup.py build
-    sudo python setup.py install
-
-This will install the library code in the appropriate place within 
-your python setup, and the client ``resync`` in an appropriate system 
-path (perhaps ``/usr/local/bin`` or ``/usr/bin`` depending on your system).
-
-The source code is maintained on `Github
-<https://github.com/resync/resync>`_
-and there may be branches/versions available there that are not 
-yet packaged for `PyPI
-<http://pypi.python.org/pypi/resync>`_.
-
-See also
---------
-
-`ResourceSync simulator
-<http://github.com/resync/simulator>`_
-
-Contributors
-------------
-
-Thanks to:
-`Bernhard Haslhofer
-<https://github.com/behas>`_,
-`Robert Sanderson
-<https://github.com/azaroth42>`_
-
-Copyright and License
----------------------
-
-Copyright 2012--2014 Simeon Warner
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-   http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.
-   
-See LICENSE.txt

README.md

@@ -0,0 +1,124 @@
+# resync
+
+[![Build Status](https://travis-ci.org/resync/resync.svg?branch=main)](https://travis-ci.org/resync/resync)
+[![Test Coverage](https://coveralls.io/repos/github/resync/resync/badge.svg?branch=main)](https://coveralls.io/github/resync/resync)
+
+**resync** is a ResourceSync library with supporting client scipts,
+written in python.
+[ResourceSync](http://www.openarchives.org/rs/) is a synchronization
+framework for the web consisting of various capabilities that allow
+third party systems to remain synchronized with a server's evolving
+resources. The capabilities can be combined in a modular manner to
+meet local or community requirements.
+
+## Client usage
+
+Typical client usage to synchronize from a source at
+`http://source.example.com/` to a set of local files would be
+
+```
+resync-sync http://source.example.com/
+```
+
+which will create or update a local directory `./source.example.com`.
+Alternatively, the destination directory may be specified explicitly::
+
+```
+resync-sync http://source.example.com/ /tmp/my_copy
+```
+
+Option details and a number of different modes are described with::
+
+```
+resync-sync -h
+```
+
+## Python library usage
+
+Typical library use in a source (create and output a Resource List)::
+
+```
+from resync import Resource, ResourceList
+
+rl = ResourceList()
+rl.add( Resource('http://example.com/res1', lastmod='2013-01-01') )
+rl.add( Resource('http://example.com/res2', lastmod='2013-01-02') )
+print rl.as_xml()
+```
+
+Typical library use in a destination (get and examine a Capability List)::
+
+```
+from resync import CapabilityList
+
+# Read Capability List and show supported capabilities
+cl = CapabilityList()
+cl.read("https://raw.github.com/resync/resync/1.0/resync/test/testdata/examples_from_spec/resourcesync_ex_2_6.xml")
+for resource in cl:
+    print "supports %s (at %s)" % (resource.capability,resource.uri)
+```
+
+## Installation
+
+The client and library are designed to work with Python 3.5 and up. (The
+last version supporting Python 2.7 was
+[1.0.9, also on PyPI](https://pypi.org/project/resync/1.0.9/))
+
+### Automatic installation
+
+```
+pip install resync
+```
+
+rsync is listed in [PyPI](http://pypi.python.org/pypi/resync) and can be
+installed with `pip` or `easy_install`.
+
+### Manual installation from github
+
+```
+cd /tmp
+git clone git://github.com/resync/resync.git
+cd resync/
+python setup.py build
+python setup.py install
+```
+
+This will install the library code in the appropriate place within
+a user-space python setup, including the clients `resync-sync`,
+`resync-build` and `resync-explorer`. Use of `sudo` to install in
+system spaces is generally discouraged.
+
+The source code is maintained on [Github](https://github.com/resync/resync)
+and there may be branches/versions available there that are not
+yet packaged for PyPI.
+
+## See also
+
+  * [ResourceSync simulator](http://github.com/resync/simulator)
+
+## Contributors
+
+Thanks to: [Bernhard Haslhofer](https://github.com/behas),
+[Robert Sanderson](<https://github.com/azaroth42)
+and [other contributors](https://github.com/resync/resync/graphs/contributors).
+
+See [CONTRIBUTING.md](https://github.com/resync/resync/blob/main/CONTRIBUTING.md)
+for guidelines for contributing.
+
+## Copyright and License
+
+Copyright 2012--2020 Simeon Warner and contributors
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+See `LICENSE.txt`

README.rst

@@ -1 +0,0 @@
-README

TODO.md

@@ -3,12 +3,12 @@ To do list for resync client and library
 
 See also issues on github tracker. Other big threads that need attention are:
 
-- work through and test creation of Resource and Change Dump ZIPs
-- add use of a Resource and Change Dump ZIPs
-- add archives (ResoourceListArchive, ChangeListArchive, ResourceDumpArchive, ChangeDumpArchive)
-- consider breaking out handling of links that is somewhat duplicated in resource.py and resource_container.py
+  * work through and test creation of Resource and Change Dump ZIPs
+  * add use of a Resource and Change Dump ZIPs
+  * add archives (ResoourceListArchive, ChangeListArchive, ResourceDumpArchive, ChangeDumpArchive)
+  * consider breaking out handling of links that is somewhat duplicated in resource.py and resource_container.py
 
-Issues with specification
-=========================
+Planned changes and deprecation
+-------------------------------
 
-- v1.0 spec Figure 4 shows Capability List indexes but these are not allowed per section 9
+  * Remove support for ListBase.parse(str=...) which was deprecated in favor of ListBase.parse(str_data=...) 2016-03-22 in 1.0.1

bin/resync

@@ -1,280 +0,0 @@
-#!/usr/bin/env python
-"""
-resync: The ResourceSync command line client
-
-Copyright 2012,2013 Simeon Warner
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License
-"""
-
-import optparse
-import sys
-
-from resync import __version__
-from resync.client import Client, ClientFatalError
-from resync.client_utils import init_logging,count_true_args,parse_links,parse_capabilities,parse_capability_lists
-from resync.utils import UTCFormatter
-
-DEFAULT_LOGFILE = 'resync-client.log'
-
-def main():
-
-    if (sys.version_info < (2,6)):
-        sys.exit("This program requires python version 2.6 or later")
-    
-    # Options and arguments
-    p = optparse.OptionParser(description='ResourceSync command line client',
-                              usage='usage: %prog [options] uri_path local_path  (-h for help)',
-                              version='%prog '+__version__ )
-
-    # Modes
-    # a. modes using remote sitemap/resources
-    rem = p.add_option_group('REMOTE MODES',
-        'These modes use a remote source that is specified in a set of uri=path mappings '
-        'and potentially also using an explicit --sitemap location. The default mode is '
-        '--baseline')
-    rem.add_option('--baseline', '-b', action='store_true',
-                   help='baseline sync of resources from remote source (src) to local filesystem (dst)')
-    rem.add_option('--incremental', '--inc', '-i', action='store_true',
-                   help='incremental sync of resources from remote source (src) to local filesystem (dst). Uses either timestamp recorded from last baseline or incremental sync for this source, or explicit --from parameter, to determine the earlier update timestamp to act on.')
-    rem.add_option('--audit', '-a', action='store_true',
-                   help="audit sync state of destination wrt source")
-    rem.add_option('--parse', '-p', action='store_true',
-                   help="parse a remote sitemap/sitemapindex (from mapping or explicit --sitemap) and show summary information including document type and number of entries")
-    rem.add_option('--explore','-x', action='store_true',
-                   help="read discovery information and show capabilities. Will use either an explicit --sitemap or the first mapping URI specified to determine which server to look for .well-known discovery information or which URI to try")
-    # b. modes based solely on files on local disk
-    loc = p.add_option_group('LOCAL MODES',
-        'These modes act on files on the local disk')
-    loc.add_option('--resourcelist', '--resource-list', '-r', action='store_true',
-                   help="write a resource list based on files on disk using uri=path mappings "
-                        "in reverse to calculate URIs from the local paths. Scans local disk "
-                        "based either on explicit --paths setting, else starting from all local "
-                        "paths specified in the mappings. Writes to STDOUT by default, override "
-                        "with --outfile")
-    loc.add_option('--changelist', '--change-list', '-c', action='store_true',
-                   help="write a change list based on comparison of a reference sitemap "
-                        "(specify file with --reference) and either files on disk (using "
-                        "the mapping provided) or a second sitemap (specify file with "
-                        "--newreference). Otherwise follows --resourcelist options. Also accepts "
-                        "the --empty option (with no mapping) to write and empty changelist.")
-    loc.add_option('--capabilitylist', '--capability-list', type=str, action='store',
-                   help="write a capability list based on the set of capabilities and "
-                        "URIs supplied in cap_name=URI,cap_name=URI format. Otherwise "
-                        "follows --resourcelist options.")
-    loc.add_option('--sourcedescription', '--source-description', type=str, action='store',
-                   help="write a Source Description document based on the set of capability "
-                        "list URIs supplied as a comma separated list. Otherwise "
-                        "follows --resourcelist options.")
-    loc.add_option('--resourcedump', '--resource-dump', '-d', action='store_true',
-                   help="write a Resource Dump. Specify output file with --outfile and use other "
-                        "options as for --resourcelist")
-    loc.add_option('--changedump', '--change-dump', action='store_true',
-                   help="write a Resource Dump. Specify output file with --outfile and use other "
-                        "options as for --changelist")
-
-    # Specification of map between remote URI and local file paths, and remote sitemap
-    nam = p.add_option_group('FILE/URI NAMING OPTIONS')
-    nam.add_option('--outfile', type=str, action='store',
-                   help="write output to specified file rather than STDOUT or default")
-    nam.add_option('--paths', type=str, action='store',
-                   help="explicit set of paths for disk scan --resourceslist or --changelist "
-                        "generation")
-    nam.add_option('--sitemap', type=str, action='store',
-                   help="explicitly set sitemap name, overriding default sitemap.xml "
-                        "appended to first source URI specified in the mappings")
-    nam.add_option('--reference', type=str, action='store',
-                   help="reference sitemap name for --changelist calculation")
-    nam.add_option('--newreference', type=str, action='store',
-                   help="updated reference sitemap name for --changelist calculation")
-    nam.add_option('--changelist-uri','--change-list-uri', type=str, action='store',
-                   help="explicitly set the changelist URI that will be use in --inc mode, "
-                        "overrides process of getting this from the sitemap")
-
-    lks = p.add_option_group("LINK GENERATION")
-    lks.add_option('--link', type=str, action='append',
-                   help="add discovery links to the output sitemap, "
-                        "format: rel,href[,att1=val1,att2=val2] "
-                        "(repeat option for multiple links)")
-    lks.add_option('--describedby-link', type=str, action='store',
-                   help="add an <rs:md rel=\"describedby\" link to "
-                        "a description of the feed at the URI given") 
-    lks.add_option('--sourcedescription-link', '--source-description-link', 
-                   type=str, action='store',
-                   help="for a Capability List add a <rs:md rel=\"up\" link to the"
-                        "Source Description document at the URI given, else ignored") 
-    lks.add_option('--capabilitylist-link', '--capability-list-link',
-                   type=str, action='store',
-                   help="for all documents except a Capability List or a "
-                        "Source Description, add an <rs:md rel=\"up\" link "
-                        "to the Capability List at the URI given") 
-
-    # Options that apply to multiple modes
-    opt = p.add_option_group('MISCELANEOUS OPTIONS')
-    opt.add_option('--checksum', action='store_true',
-                   help="use checksum (md5) in addition to last modification time and size")
-    opt.add_option('--delete', action='store_true',
-                   help="allow files on destination to be deleted")
-    opt.add_option('--from', type=str, action='store', dest='from_datetime', metavar="DATETIME",
-                   help="explicit datetime value used to filter updates in change list for "
-                        "--incremental sync")
-    opt.add_option('--exclude', type=str, action='append',
-                   help="exclude resources with URI or filename matching pattern "
-                        "(repeat option for multiple excludes)")
-    opt.add_option('--empty', action='store_true',
-                   help="combine with --changelist to write and empty changelist, perhaps with links")
-    opt.add_option('--multifile', '-m', action='store_true',
-                   help="disable reading and output of sitemapindex for multifile sitemap")
-    opt.add_option('--noauth', action='store_true',
-                   help="disable all checking of URLs to ensure that the ResourceSync "
-                        "documents refer only to resources on the same server or sub-domains. "
-                        "Use with care.")
-    opt.add_option('--strictauth', action='store_true',
-                   help="use more strict checking of URLs to ensure that the ResourceSync "
-                        "documents refer only to resources on the same server or sub-domains, "
-                        "and on the same server to sub-paths. This is the authority model "
-                        "of Sitemaps but there are legitimate uses where these rules would "
-                        "not be followed.")
-    opt.add_option('--warc', action='store_true',
-                   help="write dumps in WARC format (instead of ZIP+Sitemap default)")
-    opt.add_option('--dryrun', '-n', action='store_true',
-                   help="don't update local resources, say what would be done")
-    opt.add_option('--ignore-failures', action='store_true',
-                   help="continue past download failures")
-    # These likely only useful for experimentation
-    opt.add_option('--max-sitemap-entries', type=int, action='store',
-                   help="override default size limits")
-    # Want these to show at the end
-    opt.add_option('--verbose', '-v', action='store_true',
-                   help="verbose, show additional informational messages")
-    opt.add_option('--logger', '-l', action='store_true',
-                   help="create detailed log of client actions (will write "
-                        "to %s unless specified with --logfile" %
-                        DEFAULT_LOGFILE)
-    opt.add_option('--logfile', type='str', action='store',
-                   help="create detailed log of client actions")
-    opt.add_option('--eval', '-e', action='store_true',
-                   help="output evaluation of source/client synchronization performance... "
-                        "be warned, this is very verbose")
-
-    (args, map) = p.parse_args()
-
-    # Implement exclusive arguments and default --baseline (support for exclusive
-    # groups in argparse is incomplete is python2.6)
-    if (not args.baseline and not args.incremental and not args.audit and 
-        not args.parse and not args.explore and
-        not args.resourcelist and not args.changelist and
-        not args.capabilitylist and not args.sourcedescription and
-        not args.resourcedump and not args.changedump):
-        if (len(map)==0):
-            # No args at all, show help
-            p.print_help()
-            return
-        else:
-            args.baseline=True
-    elif (count_true_args(args.baseline,args.incremental,args.audit,args.parse,args.explore,
-                          args.resourcelist,args.changelist,args.capabilitylist,
-                          args.sourcedescription,args.resourcedump,args.changedump)>1):
-        p.error("Only one of --baseline, --incremental, --audit, --parse, --explore, --resourcelist, --changelist, --capabilitylist, --sourcedescription, --resourcedump, --changedump modes allowed")
-
-    # Configure logging module and create logger instance
-    init_logging( to_file=args.logger, logfile=args.logfile, default_logfile=DEFAULT_LOGFILE, 
-                  verbose=args.verbose, eval_mode=args.eval )
-
-    c = Client( checksum=args.checksum,
-                verbose=args.verbose,
-                dryrun=args.dryrun )
-
-    try:
-        if (map):
-            # Mappings apply to (almost) everything
-            c.set_mappings(map)
-        if (args.sitemap):
-            c.sitemap_name=args.sitemap
-        if (args.warc):
-            c.dump_format='warc'
-        if (args.exclude):
-            c.exclude_patterns=args.exclude
-        if (args.multifile):
-            c.allow_multifile=not args.multifile
-        if (args.noauth):
-            c.noauth=args.noauth
-        if (args.strictauth):
-            c.strictauth=args.strictauth
-        if (args.max_sitemap_entries):
-            c.max_sitemap_entries=args.max_sitemap_entries
-        if (args.ignore_failures):
-            c.ignore_failures=args.ignore_failures
-
-        # Links apply to anything that writes sitemaps
-        links = parse_links(args.link)
-        # Add specific links is appropriate cases
-        if (args.capabilitylist_link and 
-            not args.capabilitylist and
-            not args.sourcedescription):
-            # rel="up" to Capability List in all but Capability List 
-            # and Source Description
-            links.insert(0,{'rel':'up','href':args.capabilitylist_link})
-        if (args.sourcedescription_link and args.capabilitylist):
-            # rel="up" to Source Description from Capability List
-            links.insert(0,{'rel':'up','href':args.sourcedescription_link})
-        if (args.describedby_link):
-            links.insert(0,{'rel':'describedby','href':args.describedby_link})
-
-
-        # Finally, do something...
-        if (args.baseline or args.audit):
-            c.baseline_or_audit(allow_deletion=args.delete,
-                                audit_only=args.audit)
-        elif (args.incremental):
-            c.incremental(allow_deletion=args.delete,
-                          change_list_uri=args.changelist_uri,
-                          from_datetime=args.from_datetime)
-        elif (args.parse):
-            c.parse_document()
-        elif (args.explore):
-            c.explore()
-        elif (args.resourcelist or args.resourcedump):
-            c.write_resource_list(paths=args.paths,
-                                  outfile=args.outfile,
-                                  links=links,
-                                  dump=args.resourcedump)
-        elif (args.changelist or args.changedump):
-            if (not args.reference and not args.empty):
-                p.error("Must supply --reference sitemap for --changelist, or --empty")
-            c.write_change_list(ref_sitemap=args.reference,
-                                newref_sitemap=( args.newreference if (args.newreference) else None ),
-                                empty=args.empty,
-                                paths=args.paths,
-                                outfile=args.outfile,
-                                links=links,
-                                dump=args.changedump)
-        elif (args.capabilitylist):
-            c.write_capability_list(
-                capabilities=parse_capabilities(args.capabilitylist),
-                outfile=args.outfile,
-                links=links)
-        elif (args.sourcedescription):
-            c.write_source_description(
-                capability_lists=parse_capability_lists(args.sourcedescription),
-                outfile=args.outfile,
-                links=links)
-        else:
-            p.error("Unknown mode requested")
-    # Any problem we expect will come as a ClientFatalError, anything else 
-    # is... an exception ;-)
-    except ClientFatalError as e:
-        sys.stderr.write("\nFatalError: " + str(e) + "\n")
-
-if __name__ == '__main__':
-    main()

bin/resync-explorer

@@ -1,120 +0,0 @@
-#!/usr/bin/env python
-"""
-resync-explorer: The ResourceSync explorer
-
-Copyright 2012,2013 Simeon Warner
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License
-"""
-
-import optparse
-import sys
-
-from resync import __version__
-from resync.client import ClientFatalError
-from resync.client_utils import init_logging
-from resync.explorer import Explorer
-
-DEFAULT_LOGFILE = 'resync-explorer.log'
-
-def main():
-
-    if (sys.version_info < (2,6)):
-        sys.exit("This program requires python version 2.6 or later")
-    
-    # Options and arguments
-    p = optparse.OptionParser(description='ResourceSync explorer',
-                              usage='usage: %prog [options] uri_path local_path  (-h for help)',
-                              version='%prog '+__version__ )
-
-    # Specification of map between remote URI and local file paths, and remote sitemap
-    nam = p.add_option_group('FILE/URI NAMING OPTIONS')
-    nam.add_option('--outfile', type=str, action='store',
-                   help="write sitemap to specified file rather than STDOUT")
-    nam.add_option('--paths', type=str, action='store',
-                   help="explicit set of paths for disk scan --resourceslist or --changelist "
-                        "generation")
-    nam.add_option('--sitemap', type=str, action='store',
-                   help="explicitly set sitemap name, overriding default sitemap.xml "
-                        "appended to first source URI specified in the mappings")
-    nam.add_option('--reference', type=str, action='store',
-                   help="reference sitemap name for --changelist calculation")
-    nam.add_option('--newreference', type=str, action='store',
-                   help="updated reference sitemap name for --changelist calculation")
-    nam.add_option('--dump', metavar='DUMPFILE', type=str, action='store',
-                   help="write dump to specified file for --resourcelist or --changelist")
-    nam.add_option('--changelist-uri','--change-list-uri', type=str, action='store',
-                   help="explicitly set the changelist URI that will be use in --inc mode, "
-                        "overrides process of getting this from the sitemap")
-
-    # Options that apply to multiple modes
-    opt = p.add_option_group('MISCELANEOUS OPTIONS')
-    opt.add_option('--checksum', action='store_true',
-                   help="use checksum (md5) in addition to last modification time and size")
-    opt.add_option('--from', type=str, action='store', dest='from_datetime', metavar="DATETIME",
-                   help="explicit datetime value used to filter updates in change list for "
-                        "--incremental sync")
-    opt.add_option('--exclude', type=str, action='append',
-                   help="exclude resources with URI or filename matching pattern "
-                        "(repeat option for multiple excludes)")
-    opt.add_option('--multifile', '-m', action='store_true',
-                   help="disable reading and output of sitemapindex for multifile sitemap")
-    opt.add_option('--noauth', action='store_true',
-                   help="disable checking of URL paths to ensure that the sitemaps refer "
-                        "only to resources on the same server/sub-path etc. Use with care.")
-    # These likely only useful for experimentation
-    opt.add_option('--max-sitemap-entries', type=int, action='store',
-                   help="override default size limits")
-    # Want these to show at the end
-    opt.add_option('--verbose', '-v', action='store_true',
-                   help="verbose")
-    opt.add_option('--logger', '-l', action='store_true',
-                   help="create detailed log of explorer actions (will write "
-                        "to %s unless specified with --logfile" %
-                        DEFAULT_LOGFILE)
-    opt.add_option('--logfile', type='str', action='store',
-                   help="create detailed log of client actions")
-
-    (args, map) = p.parse_args()
-
-    init_logging( to_file=args.logger, logfile=args.logfile, default_logfile=DEFAULT_LOGFILE,
-                  verbose=args.verbose )
-
-    print "----- ResourceSync Explorer -----"
-    c = Explorer( checksum=args.checksum,
-                  verbose=args.verbose )
-
-    try:
-        if (map):
-            # Mappings apply to (almost) everything
-            c.set_mappings(map)
-        if (args.sitemap):
-            c.sitemap_name=args.sitemap
-        if (args.exclude):
-            c.exclude_patterns=args.exclude
-        if (args.multifile):
-            c.allow_multifile=not args.multifile
-        if (args.noauth):
-            c.noauth=args.noauth
-        if (args.max_sitemap_entries):
-            c.max_sitemap_entries=args.max_sitemap_entries
-
-        c.explore()
-
-    # Any problem we expect will come as a ClientFatalError, anything else 
-    # is... an exception ;-)
-    except ClientFatalError as e:
-        sys.stderr.write("\nFatalError: " + str(e) + "\n")
-
-if __name__ == '__main__':
-    main()

bin/test/__init__.py

@@ -1 +0,0 @@
-# Tests for client code in bin/test

bin/test/test_client_link_options.py

@@ -1,59 +0,0 @@
-import sys
-import unittest
-import StringIO
-
-sys.path.insert(0, '.')
-
-from resync.resource import Resource
-from resync.resource_list import ResourceList
-from resync.capability_list import CapabilityList
-from resync.sitemap import Sitemap, SitemapIndexError, SitemapParseError
-
-import subprocess
-
-def run_resync(args):
-    args.insert(0,'bin/resync')
-    proc = subprocess.Popen(args, stdout=subprocess.PIPE)
-    (out, err) = proc.communicate()
-    return(out)
-
-class TestClientLinkOptions(unittest.TestCase):
-
-    def test01_no_links(self):
-        xml = run_resync(['--resourcelist',
-                          'http://example.org/t','resync/test/testdata/dir1'])
-        rl = ResourceList()
-        rl.parse(fh=StringIO.StringIO(xml))
-        self.assertEqual( len(rl), 2 )
-        self.assertEqual( rl.link('describedby'), None )
-
-    def test02_resource_list_links(self):
-        xml = run_resync(['--resourcelist',
-                          '--describedby-link=a',
-                          '--sourcedescription-link=b', #will be ignored
-                          '--capabilitylist-link=c',
-                          'http://example.org/t','resync/test/testdata/dir1'])
-        rl = ResourceList()
-        rl.parse(fh=StringIO.StringIO(xml))
-        self.assertEqual( len(rl), 2 )
-        self.assertNotEqual( rl.link('describedby'), None )
-        self.assertEqual( rl.link('describedby')['href'], 'a' )
-        self.assertNotEqual( rl.link('up'), None )
-        self.assertEqual( rl.link('up')['href'], 'c' )
-
-    def test03_capability_list_links(self):
-        xml = run_resync(['--capabilitylist=resourcelist=rl,changedump=cd',
-                          '--describedby-link=a',
-                          '--sourcedescription-link=b',
-                          '--capabilitylist-link=c' ]) #will be ignored
-        capl = CapabilityList()
-        capl.parse(fh=StringIO.StringIO(xml))
-        self.assertEqual( len(capl), 2 )
-        self.assertNotEqual( capl.link('describedby'), None )
-        self.assertEqual( capl.link('describedby')['href'], 'a' )
-        self.assertNotEqual( capl.link('up'), None )
-        self.assertEqual( capl.link('up')['href'], 'b' )
-
-if  __name__ == '__main__':
-    suite = unittest.TestLoader().loadTestsFromTestCase(TestClientLinkOptions)
-    unittest.TextTestRunner(verbosity=2).run(suite)

docs/examples/01_make_resourcelist.py

@@ -1,8 +0,0 @@
-from resync.resource_list import ResourceList
-from resync.resource import Resource
-from resync.sitemap import Sitemap
-
-rl = ResourceList()
-rl.add( Resource('http://example.com/res1', lastmod='2013-01-01') )
-rl.add( Resource('http://example.com/res2', lastmod='2013-01-02') )
-print rl.as_xml(pretty_xml=True)

docs/examples/99_make_spec_examples.pl

@@ -1,190 +0,0 @@
-from resync.resource_list import ResourceList
-from resync.change_list import ChangeList
-from resync.resource_dump import ResourceDump
-from resync.resource_dump_manifest import ResourceDumpManifest
-from resync.capability_list import CapabilityList
-from resync.resource import Resource
-from resync.sitemap import Sitemap
-
-def head(txt):
-    print "\n=== "+txt+" ===\n"
-
-print "\nExample script to build objects to write the examples in the spec\n"
-
-head("Example 2.1: A Resource List")
-rl = ResourceList()
-rl.modified='2013-01-03T09:00:00Z'
-rl.add( Resource('http://example.com/res1') )
-rl.add( Resource('http://example.com/res2') )
-print rl.as_xml(pretty_xml=True)
-
-head("Example 2.2: A Resource List with additional information")
-rl = ResourceList()
-rl.modified='2013-01-03T09:00:00Z'
-rl.add( Resource('http://example.com/res1',
-	         lastmod='2013-01-02T13:00:00Z', 
-		 md5='1584abdf8ebdc9802ac0c6a7402c03b') )
-rl.add( Resource('http://example.com/res2',
-                 lastmod='2013-01-02T14:00:00Z',
-		 md5='1e0d5cb8ef6ba40c99b14c0237be735e') )
-print rl.as_xml(pretty_xml=True)
-
-head("Example 2.3: A Change List")
-cl = ChangeList()
-cl.modified="2013-01-03T11:00:00Z"
-cl.add( Resource('http://example.com/res2.pdf',
-		 lastmod='2013-01-02T13:00:00Z',
-		 change='updated') )
-cl.add( Resource('http://example.com/res3.tiff',
-	 	 lastmod='2013-01-02T18:00:00Z',
-		 change='deleted') )
-print cl.as_xml(pretty_xml=True)
-
-head("Example 2.4: A Resource Dump")
-rd = ResourceDump()
-rd.modified="2013-01-03T09:00:00Z"
-rd.add( Resource('http://example.com/resourcedump.zip',
-		 lastmod='2013-01-03T09:00:00Z') )
-print rd.as_xml(pretty_xml=True)
-
-head("Example 2.5: A Resource Dump Manifest detailing the content of a ZIP file")
-rdm = ResourceDumpManifest()
-rdm.modified="2013-01-03T09:00:00Z"
-rdm.add( Resource('http://example.com/res1',
-	          lastmod='2013-01-02T13:00:00Z', 
-		  md5='1584abdf8ebdc9802ac0c6a7402c03b6',
-	          path='/resources/res1') )
-rdm.add( Resource('http://example.com/res2',
-                  lastmod='2013-01-02T14:00:00Z',
-	   	  md5='1e0d5cb8ef6ba40c99b14c0237be735e',
-	          path='/resources/res2') )
-print rdm.as_xml(pretty_xml=True)
-
-head("Example 2.6: A Capability List with the description of the ResourceSync capabilities of a Source")
-caplist=CapabilityList()
-caplist.modified="2013-01-02T14:00:00Z"
-caplist.ln.append({'href':"http://example.com/info-about-source.xml",
-                   'rel':"describedby",
-		   'type':"application/xml"})
-caplist.add( Resource('http://example.com/dataset1/resourcelist.xml',
-                      capability="resourcelist") )
-caplist.add( Resource('http://example.com/dataset1/resourcedump.xml',
-                      capability="resourcedump") )
-caplist.add( Resource('http://example.com/dataset1/changelist.xml',
-                      capability="changelist") )
-print caplist.as_xml(pretty_xml=True)
-
-head("Example 2.7: A Resource List Index expressed using the <sitemapindex> document format")
-rl = ResourceList()
-rl.modified="2013-01-03T09:00:00Z"
-rl.add( Resource("http://example.com/resourcelist-part2.xml",
-		 lastmod="2013-01-03T09:00:00Z") )
-rl.add( Resource("http://example.com/resourcelist-part1.xml",
-                 lastmod="2013-01-03T09:00:00Z") )
-print rl.index_as_xml(pretty_xml=True)
-
-head("Example 4.1: A Resource List")
-rl = ResourceList()
-rl.modified="2013-01-03T09:00:00Z"
-rl.ln.append({"rel":"resourcesync",
-              "href":"http://example.com/dataset1/capabilitylist.xml"})
-rl.add( Resource("http://example.com/res1",
-                 lastmod="2013-01-02T13:00:00Z",
-                 md5="1584abdf8ebdc9802ac0c6a7402c03b6",
-                 length="8876",
-                 type="text/html") )
-rl.add( Resource("http://example.com/res2",
-                 lastmod="2013-01-02T14:00:00Z",
-                 md5="1e0d5cb8ef6ba40c99b14c0237be735e",
-		 sha256="854f61290e2e197a11bc91063afce22e43f8ccc655237050ace766adc68dc784",
-                 length="14599",
-                 type="application/pdf") )
-print rl.as_xml(pretty_xml=True)
-
-head("Example 4.2: A Resource List Index")
-rl = ResourceList()
-rl.modified="2013-01-03T09:00:00Z"
-rl.ln.append({"rel":"resourcesync",
-              "href":"http://example.com/dataset1/capabilitylist.xml"})
-rl.add( Resource("http://example.com/resourcelist3.xml",
-                 lastmod="2013-01-03T09:00:00Z") )
-rl.add( Resource("http://example.com/resourcelist2.xml",
-                 lastmod="2013-01-03T09:00:00Z") )
-rl.add( Resource("http://example.com/resourcelist1.xml",
-                 lastmod="2013-01-03T09:00:00Z") )
-print rl.index_as_xml(pretty_xml=True)
-
-head("Example 4.3: A Resource List with a navigational link to its parent Resource List Index")
-rl = ResourceList()
-rl.modified="2013-01-03T09:00:00Z"
-rl.ln.append({"rel":"resourcesync",
-              "href":"http://example.com/dataset1/capabilitylist.xml"})
-rl.ln.append({"rel":"up",
-              "href":"http://example.com/dataset1/resourcelist-index.xml"})
-rl.add( Resource("http://example.com/res3",
-                 lastmod="2013-01-03T09:00:00Z",
-                 md5="1584abdf8ebdc9802ac0c6a7402c8753",
-                 length="4385",
-                 type="application/pdf") )
-rl.add( Resource("http://example.com/res2",
-                 lastmod="2013-01-03T09:00:00Z",
-                 md5="4556abdf8ebdc9802ac0c6a7402c9881",
-                 length="883",
-                 type="image/png") )
-print rl.as_xml(pretty_xml=True)
-
-head("Example 5.1: A Resource Dump document")
-rd = ResourceDump()
-rd.modified = "2013-01-03T09:00:00Z"
-rd.ln.append({"rel":"resourcesync",
-              "href":"http://example.com/dataset1/capabilitylist.xml"})
-rd.add( Resource("http://example.com/resourcedump-part3.zip",
-		 lastmod="2013-01-03T09:00:00Z",
-		 type="application/zip",
-		 length="4765") )
-rd.add( Resource("http://example.com/resourcedump-part2.zip",
-		 lastmod="2013-01-03T09:00:00Z",
-                 type="application/zip",
-		 length="9875") )
-rd.add( Resource("http://example.com/resourcedump-part1.zip",
-                 lastmod="2013-01-03T09:00:00Z",
-                 type="application/zip",
-                 length="2298") )
-print rd.as_xml(pretty_xml=True)
-
-head("Example 5.2: A Resource Dump Manifest")
-rdm = ResourceDumpManifest()
-rdm.modified="2013-01-03T09:00:00Z"
-rdm.ln.append({"rel":"resourcesync",
-               "href":"http://example.com/dataset1/capabilitylist.xml"})
-rdm.add( Resource("http://example.com/res1",
-		  lastmod="2013-01-02T13:00:00Z",
-		  md5="1584abdf8ebdc9802ac0c6a7402c03b6",
-		  length="8876",
-		  type="text/html",
-		  path="/resources/res1") )
-rdm.add( Resource("http://example.com/res2",
-		  lastmod="2013-01-02T14:00:00Z",
-                  md5="1e0d5cb8ef6ba40c99b14c0237be735e",
-                  sha256="854f61290e2e197a11bc91063afce22e43f8ccc655237050ace766adc68dc784",
-		  length="14599",
-		  type="application/pdf",
-		  path="/resources/res2") )
-print rdm.as_xml(pretty_xml=True)
-
-head("Example 5.3: A Resource Dump Manifest Index")
-rdm = ResourceDumpManifest()
-rdm.modified="2013-01-03T09:00:00Z"
-rdm.ln.append({"rel":"resourcesync",
-               "href":"http://example.com/dataset1/capabilitylist.xml"})
-rdm.add( Resource("/manifests/part3.xml",
-		  lastmod="2013-01-03T09:00:00Z",
-                  path="/manifests/part3.xml") )
-rdm.add( Resource("/manifests/part2.xml",
-		  lastmod="2013-01-03T09:00:00Z",
-		  path="/manifests/part2.xml") )
-rdm.add( Resource("/manifests/part1.xml",
-		  lastmod="2013-01-03T09:00:00Z",
-		  path="/manifests/part1.xml") )
-print rdm.index_as_xml(pretty_xml=True)
-

docs/pypi_upload.md

@@ -1,56 +1,79 @@
-=======================
-Updating resync on pypi
-=======================
+# Updating resync on pypi
 
-Notes to remind zimeon...
+Notes to remind @zimeon...
 
-resync is at https://pypi.python.org/pypi/resync
+resync is at <https://pypi.python.org/pypi/resync> on pypi
 
-Putting up a new version
-------------------------
+## Putting up a new version
 
-0. In dev branch: bump version number in resync/_version.py and check CHANGES.md is up to date
-1. Check all tests good (python setup.py test; py.test)
+0. In `develop` branch: bump version number in `resync/__init__.py` and check `CHANGES.md` is up to date
+1. Check all tests good with appropriate Python 3.x (`python setup.py test` and CI)
 2. Check code is up-to-date with github version
-3. Check out master and merge in dev
-4. Check all tests good (python setup.py test; py.test)
-5. Make sure master README has correct travis-ci icon link
-6. Check branches as expected (git branch -a)
-7. Check local build and version reported OK (python setup.py build; sudo python setup.py install; resync --version)
+3. Check out `main` and merge in `develop`
+4. Check all tests still good (`python setup.py test` and CI)
+5. Make sure main `README.md` has correct travis-ci icon link
+6. Check branches as expected (`git branch -a`)
+7. Check local build and version reported OK (`python setup.py build; python setup.py install; resync-sync -h`)
 8. Check client works with simulator:
 
    ```
-   simeon@RottenApple resync>resync --delete http://resync.library.cornell.edu/sim100
-   Status: NOT IN SYNC (same=92, to create=0, to update=1, to delete=0)
-   Will GET 1 resources, and delete 0 resources
-   Status:      SYNCED (same=92, created=0, updated=1, deleted=0)
-   simeon@RottenApple resync>resync -i --delete http://resync.library.cornell.edu/sim100
-   Status:  NO CHANGES (created=0, updated=0, deleted=0)
-   simeon@RottenApple resync>resync -i --delete http://resync.library.cornell.edu/sim100
-   Status:     CHANGES (created=1, updated=0, deleted=0)
-   simeon@RottenApple resync>resync -i --delete http://resync.library.cornell.edu/sim100
-   Status:  NO CHANGES (created=0, updated=0, deleted=0)
-   simeon@RottenApple resync>resync -a http://resync.library.cornell.edu/sim100
-   Status:     IN SYNC (same=94, to create=0, to update=0, to delete=0)
-   simeon@RottenApple resync>resync -a http://resync.library.cornell.edu/sim100
-   Status: NOT IN SYNC (same=94, to create=1, to update=0, to delete=0)
-   simeon@RottenApple resync>resync -i --delete http://resync.library.cornell.edu/sim100
-   Status:     CHANGES (created=1, updated=0, deleted=0)
-   simeon@RottenApple resync>resync -a http://resync.library.cornell.edu/sim100
-   Status:     IN SYNC (same=95, to create=0, to update=0, to delete=0)
-    ```
+   # Run simulator in one window
+   resync-simulator> ./resync-simulator
+
+   # Run client in another
+   resync> rm -rf localhost_8888
+
+   resync> ./resync-sync --baseline --delete http://localhost:8888/
+   Using URI mapping: http://localhost:8888/ -> localhost_8888
+   Status:     NOT IN SYNC (same=0, to create=1003, to update=0, to delete=0)
+   Will GET 1003 resources, and delete 0 resources
+   Status:          SYNCED (same=0, created=1003, updated=0, deleted=0)
+
+   resync> ./resync-sync --baseline --delete http://localhost:8888/
+   Using URI mapping: http://localhost:8888/ -> localhost_8888
+   Status:     NOT IN SYNC (same=1000, to create=2, to update=1, to delete=2)
+   Will GET 3 resources, and delete 2 resources
+   Status:          SYNCED (same=1000, created=2, updated=1, deleted=2)
+
+   resync> ./resync-sync --incremental --delete http://localhost:8888/
+   Using URI mapping: http://localhost:8888/ -> localhost_8888
+   Status: CHANGES APPLIED (created=4, updated=5, deleted=4)
+   Will apply 13 changes, and delete 4 resources
+   Status:      NO CHANGES (created=4, updated=5, deleted=4)
+
+   esync> ./resync-sync --incremental --delete http://localhost:8888/
+   Using URI mapping: http://localhost:8888/ -> localhost_8888
+   Status: CHANGES APPLIED (created=0, updated=3, deleted=0)
+   Will apply 3 changes, and delete 0 resources
+   Status:      NO CHANGES (created=0, updated=3, deleted=0)
+
+   resync> ./resync-sync --incremental --delete http://localhost:8888/; ./resync-sync --audit http://localhost:8888/
+   Using URI mapping: http://localhost:8888/ -> localhost_8888
+   Status: CHANGES APPLIED (created=1, updated=2, deleted=0)
+   Will apply 3 changes, and delete 0 resources
+   Status:      NO CHANGES (created=1, updated=2, deleted=0)
+   Using URI mapping: http://localhost:8888/ -> localhost_8888
+   Status:         IN SYNC (same=1001, to create=0, to update=0, to delete=0)
+   ```
 
 9. If all checks out OK, tag and push the new version to github:
 
-    ```
-    git tag -n1
-    #...current tags
-    git tag -a -m "ResourceSync v0.9 specification, add --paths" v0.9.3
-    git push --tags
+```
+git tag -n1
+#...current tags
+git tag -a -m "ResourceSync library and client" v2.0.0
+git push --tags
+```
 
-    python setup.py sdist upload
-    ```
+10. Upload to PyPI
 
-10. Then check on PyPI at https://pypi.python.org/pypi/resync
-11. Finally, back on dev branch start new version number by editing resync/_version.py and CHANGES.md
 
+```
+rm -r dist
+python setup.py sdist bdist_wheel; ls dist
+# Should be source and wheel files for just this version
+twine upload dist/*
+```
+
+10. Check on PyPI at <https://pypi.python.org/pypi/resync>
+11. Finally, back on `develop` branch start new version number by editing `resync/__init__.py` and `CHANGES.md`

docs/tutorial.md

@@ -1,45 +1,47 @@
-============================================
-ResourceSync tutorial exercises using resync
-============================================
 
+# ResourceSync tutorial exercises using resync
 
-Writing a Resource Dump
------------------------
+## Writing a Resource Dump
 
 Relative to the base directory of the github distribution there is a directory
-`resync/test/testdata/dir1` with two files, `file_a` and `file_b`. We can create 
+`tests/testdata/dir1` with two files, `file_a` and `file_b`. We can create
 a Resource Dump of this directory, as if it were mapped to `http://example.com/dir1`,
 with the following:
 
 ```
-simeon@RottenApple resync>resync --resourcedump --outfile /tmp/rd.zip http://example.com/dir1 resync/test/testdata/dir1
+resync> ./resync-build -v --write-resourcedump --outfile /tmp/rd http://example.com/dir1 tests/testdata/dir1
+Scanning disk from tests/testdata/dir1
+Writing resource dump to True...
 Total size of files to include in dump 65 bytes
-Wrote ZIP file dump /tmp/rd.zip with size 595 bytes
-
-simeon@RottenApple resync>zipinfo /tmp/rd.zip 
-Archive:  /tmp/rd.zip
-Zip file size: 595 bytes, number of entries: 3
-?rw-------  2.0 unx      409 b- defN 13-May-28 18:40 manifest.xml
+Writing sitemap /tmp/rd00000.xml...
+Wrote sitemap /tmp/rd00000.xml
+Wrote ZIP file dump /tmp/rd00000.zip with size 623 bytes
+Wrote 1 dump files
+resync> zipinfo /tmp/rd00000.zip
+Archive:  /tmp/rd00000.zip
+Zip file size: 623 bytes, number of entries: 3
+?rw-------  2.0 unx      453 b- defN 20-Dec-16 10:58 manifest.xml
 -rw-r--r--  2.0 unx       20 b- defN 12-Jul-25 13:13 file_a
 -rw-r--r--  2.0 unx       45 b- defN 01-Sep-08 21:46 file_b
-3 files, 474 bytes uncompressed, 297 bytes compressed:  37.3%
+3 files, 518 bytes uncompressed, 325 bytes compressed:  37.3%
 ```
 
 This can then be unpacked and the files checked:
 
 ```
-simeon@RottenApple resync>mkdir /tmp/rd
-simeon@RottenApple resync>cd /tmp/rd
-simeon@RottenApple rd>unzip ../rd.zip 
-Archive:  ../rd.zip
+resync> mkdir /tmp/rd
+mkdir: /tmp/rd: File exists
+resync> cd /tmp/rd
+rd> unzip ../rd00000.zip
+Archive:  ../rd00000.zip
   inflating: manifest.xml            
   inflating: file_a                  
   inflating: file_b                  
-simeon@RottenApple rd>more manifest.xml 
-<?xml version=`1.0` encoding=`UTF-8`?>
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"><url><loc>http://example.com/dir1/file_a</loc><lastmod>2012-07-25T17:13:46Z</lastmod><rs:md length="20" path="file_a" /></url><url><loc>http://example.com/dir1/file_b</loc><lastmod>2001-09-09T01:46:40Z</lastmod><rs:md length="45" path="file_b" /></url></urlset>
-simeon@RottenApple rd>more file_a 
+rd> more manifest.xml
+<?xml version='1.0' encoding='UTF-8'?>
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"><rs:md capability="resourcedump-manifest" /><url><loc>http://example.com/dir1/file_a</loc><lastmod>2012-07-25T17:13:46Z</lastmod><rs:md length="20" path="file_a" /></url><url><loc>http://example.com/dir1/file_b</loc><lastmod>2001-09-09T01:46:40Z</lastmod><rs:md length="45" path="file_b" /></url></urlset>
+rd> more file_a
 I am file a in dir1
-simeon@RottenApple rd>more file_b 
+rd> more file_b
 I am file b in dir1, I am bigger than file_a
 ```

examples/README_client_library_in_destination.py

@@ -1,11 +1,8 @@
 #!/usr/bin/env python
-if (True): #keep indentation of README
-
-    from resync import CapabilityList
-
-    # Read Capability List and show supported capabilities
-    cl = CapabilityList()
-    cl.read("https://raw.github.com/resync/resync/0.6/resync/test/testdata/examples_from_spec/resourcesync_ex_2_6.xml")
-    for resource in cl:
-        print "supports %s (at %s)" % (resource.capability,resource.uri)
+from resync import CapabilityList
 
+# Read Capability List and show supported capabilities
+cl = CapabilityList()
+cl.read("https://raw.github.com/resync/resync/0.6/resync/test/testdata/examples_from_spec/resourcesync_ex_2_6.xml")
+for resource in cl:
+    print "supports %s (at %s)" % (resource.capability,resource.uri)

examples/README_client_library_in_source.py

@@ -1,9 +1,7 @@
 #!/usr/bin/env python
-if (True): #keep indentation of README
+from resync import Resource, ResourceList
 
-    from resync import Resource,ResourceList
-
-    rl = ResourceList()
-    rl.add( Resource('http://example.com/res1', lastmod='2013-01-01') )
-    rl.add( Resource('http://example.com/res2', lastmod='2013-01-02') )
-    print rl.as_xml()
+rl = ResourceList()
+rl.add( Resource('http://example.com/res1', lastmod='2013-01-01') )
+rl.add( Resource('http://example.com/res2', lastmod='2013-01-02') )
+print rl.as_xml()

resync-build

@@ -0,0 +1,201 @@
+#!/usr/bin/env python
+"""resync-build: The ResourceSync command line list builder.
+
+Copyright 2012-2020 Simeon Warner
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License
+"""
+
+import argparse
+import sys
+
+from resync import __version__
+from resync.client import Client, ClientFatalError
+from resync.client_utils import init_logging, count_true_args, parse_links, parse_capabilities, parse_capability_lists, add_shared_misc_options, process_shared_misc_options
+
+DEFAULT_LOGFILE = 'resync-client.log'
+
+
+def main():
+    """Main function to implement command line script."""
+    if (sys.version_info < (3, 5)):
+        sys.exit("This program requires python version 3.5 or later")
+
+    # Options and arguments
+    parser = argparse.ArgumentParser(
+        description="ResourceSync build script (v" + __version__ + ")",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+
+    parser._optionals = parser.add_argument_group(
+        'MODES OF OPERATION (must specify one only). This script operates only '
+        'to create ResourceSync descriptions on the local filesystem based on '
+        'local content')
+    rem = parser.add_mutually_exclusive_group(required=True)
+    rem.add_argument('--write-resourcelist', '--write-resource-list', action='store_true',
+                     help="write a resource list based on files on disk using uri=path mappings "
+                          "in reverse to calculate URIs from the local paths. Scans local disk "
+                          "based either on explicit --paths setting, else starting from all local "
+                          "paths specified in the mappings. Writes to STDOUT by default, override "
+                          "with --outfile")
+    rem.add_argument('--write-changelist', '--write-change-list', action='store_true',
+                     help="write a change list based on comparison of a reference sitemap "
+                          "(specify file with --reference) and either files on disk (using "
+                          "the mapping provided) or a second sitemap (specify file with "
+                          "--newreference). Otherwise follows --write-resourcelist options. Also accepts "
+                          "the --empty option (with no mapping) to write and empty changelist.")
+    rem.add_argument('--write-capabilitylist', '--write-capability-list', type=str, action='store',
+                     help="write a capability list based on the set of capabilities and "
+                          "URIs supplied in cap_name=URI,cap_name=URI format. Otherwise "
+                          "follows --write-resourcelist options.")
+    rem.add_argument('--write-sourcedescription', '--write-source-description', type=str, action='store',
+                     help="write a Source Description document based on the set of capability "
+                          "list URIs supplied as a comma separated list. Otherwise "
+                          "follows --write-resourcelist options.")
+    rem.add_argument('--write-resourcedump', '--write-resource-dump', '-d', action='store_true',
+                     help="write a Resource Dump. Specify output file with --outfile and use other "
+                          "options as for --write-resourcelist")
+    rem.add_argument('--write-changedump', '--write-change-dump', action='store_true',
+                     help="write a Resource Dump. Specify output file with --outfile and use other "
+                          "options as for --write-changelist")
+
+    # Positional arguments
+    map = parser.add_argument_group('URI MAPPING TO FILESYSTEM for REMOTE modes')
+    map.add_argument(metavar='uri=path | uri path', dest='map', type=str, nargs='*',
+                     help="remote URI of source for remote synchronization operations (may "
+                          "also combine uri=local path)")
+
+    nam = parser.add_argument_group('FILE/URI NAMING OPTIONS')
+    nam.add_argument('--outfile', type=str, action='store',
+                     help="write output to specified file rather than STDOUT or default")
+    nam.add_argument('--paths', type=str, action='store',
+                     help="explicit set of paths for disk scan --resourceslist or --changelist "
+                          "generation")
+    nam.add_argument('--reference', type=str, action='store',
+                     help="reference sitemap name for --write-changelist calculation")
+    nam.add_argument('--newreference', type=str, action='store',
+                     help="updated reference sitemap name for --write-changelist calculation")
+
+    lks = parser.add_argument_group("LINK GENERATION")
+    lks.add_argument('--link', type=str, action='append',
+                     help="add discovery links to the output sitemap, "
+                          "format: rel,href[,att1=val1,att2=val2] "
+                          "(repeat option for multiple links)")
+    lks.add_argument('--describedby-link', type=str, action='store',
+                     help="add an <rs:md rel=\"describedby\" link to "
+                          "a description of the feed at the URI given")
+    lks.add_argument('--sourcedescription-link', '--source-description-link',
+                     type=str, action='store',
+                     help="for a Capability List add a <rs:md rel=\"up\" link to the"
+                          "Source Description document at the URI given, else ignored")
+    lks.add_argument('--capabilitylist-link', '--capability-list-link',
+                     type=str, action='store',
+                     help="for all documents except a Capability List or a "
+                          "Source Description, add an <rs:md rel=\"up\" link "
+                          "to the Capability List at the URI given")
+
+    # Options that apply to multiple modes
+    opt = parser.add_argument_group('MISCELANEOUS OPTIONS')
+    add_shared_misc_options(opt, default_logfile=DEFAULT_LOGFILE)
+    opt.add_argument('--empty', action='store_true',
+                     help="combine with --changelist to write and empty changelist, perhaps with links")
+    opt.add_argument('--warc', action='store_true',
+                     help="write dumps in WARC format (instead of ZIP+Sitemap default)")
+    opt.add_argument('--dryrun', '-n', action='store_true',
+                     help="don't update local resources, say what would be done")
+    # These likely only useful for experimentation
+    opt.add_argument('--max-sitemap-entries', type=int, action='store',
+                     help="override default size limits")
+    opt.add_argument('--eval', '-e', action='store_true',
+                     help="output evaluation of source/client synchronization performance... "
+                          "be warned, this is very verbose")
+
+    args = parser.parse_args()
+
+    # Configure logging module and create logger instance
+    init_logging(to_file=args.logger, logfile=args.logfile, default_logfile=DEFAULT_LOGFILE,
+                 verbose=args.verbose, eval_mode=args.eval)
+
+    process_shared_misc_options(args)
+
+    c = Client(spec_version=args.spec_version,
+               hashes=args.hash,
+               verbose=args.verbose,
+               dryrun=args.dryrun)
+
+    try:
+        if (args.map):
+            # Mappings apply to (almost) everything
+            c.set_mappings(args.map)
+        if (args.warc):
+            c.dump_format = 'warc'
+        if (args.exclude):
+            c.exclude_patterns = args.exclude
+        if (args.multifile):
+            c.allow_multifile = not args.multifile
+        if (args.max_sitemap_entries):
+            c.max_sitemap_entries = args.max_sitemap_entries
+
+        # Links apply to anything that writes sitemaps
+        links = parse_links(args.link)
+        # Add specific links is appropriate cases
+        if (args.capabilitylist_link
+                and not args.write_capabilitylist
+                and not args.write_sourcedescription):
+            # rel="up" to Capability List in all but Capability List
+            # and Source Description
+            links.insert(0, {'rel': 'up', 'href': args.capabilitylist_link})
+        if (args.sourcedescription_link and args.write_capabilitylist):
+            # rel="up" to Source Description from Capability List
+            links.insert(0, {'rel': 'up', 'href': args.sourcedescription_link})
+        if (args.describedby_link):
+            links.insert(0, {'rel': 'describedby',
+                             'href': args.describedby_link})
+
+        # Finally, do something...
+        if (args.write_resourcelist or args.write_resourcedump):
+            c.write_resource_list(paths=args.paths,
+                                  outfile=args.outfile,
+                                  links=links,
+                                  dump=args.write_resourcedump)
+        elif (args.write_changelist or args.write_changedump):
+            if (not args.reference and not args.empty):
+                parser.error("Must supply --reference sitemap for --changelist, or --empty")
+            c.write_change_list(ref_sitemap=args.reference,
+                                newref_sitemap=(args.newreference if (
+                                    args.newreference) else None),
+                                empty=args.empty,
+                                paths=args.paths,
+                                outfile=args.outfile,
+                                links=links,
+                                dump=args.write_changedump)
+        elif (args.write_capabilitylist):
+            c.write_capability_list(
+                capabilities=parse_capabilities(args.write_capabilitylist),
+                outfile=args.outfile,
+                links=links)
+        elif (args.write_sourcedescription):
+            c.write_source_description(
+                capability_lists=parse_capability_lists(
+                    args.write_sourcedescription),
+                outfile=args.outfile,
+                links=links)
+        else:
+            parser.error("Unknown mode requested")
+    # Any problem we expect will come as a ClientFatalError, anything else
+    # is... an exception ;-)
+    except ClientFatalError as e:
+        sys.stderr.write("\nFatalError: " + str(e) + "\n")
+
+
+if __name__ == '__main__':
+    main()

resync-explorer

@@ -0,0 +1,99 @@
+#!/usr/bin/env python
+"""resync-explorer: The ResourceSync explorer.
+
+Copyright 2012-2020 Simeon Warner
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License
+"""
+
+import argparse
+import sys
+
+from resync import __version__
+from resync.client import ClientFatalError
+from resync.client_utils import init_logging, add_shared_misc_options, process_shared_misc_options
+from resync.explorer import Explorer
+
+DEFAULT_LOGFILE = 'resync-explorer.log'
+
+
+def main():
+    """Main function to implement command line script."""
+    if (sys.version_info < (3, 5)):
+        sys.exit("This program requires python version 3.5 or later")
+
+    # Options and arguments
+    parser = argparse.ArgumentParser(
+        description='ResourceSync explorer (v' + __version__ + ')',
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+
+    # Specification of map between remote URI and local file paths, and remote
+    # sitemap
+    nam = parser.add_argument_group('FILE/URI NAMING OPTIONS')
+    nam.add_argument('--outfile', type=str, action='store',
+                     help="write sitemap to specified file rather than STDOUT")
+    nam.add_argument('--paths', type=str, action='store',
+                     help="explicit set of paths for disk scan --resourceslist or --changelist "
+                          "generation")
+    nam.add_argument('--sitemap', type=str, action='store',
+                     help="explicitly set sitemap name, overriding default sitemap.xml "
+                          "appended to first source URI specified in the mappings")
+    nam.add_argument('--reference', type=str, action='store',
+                     help="reference sitemap name for --changelist calculation")
+    nam.add_argument('--newreference', type=str, action='store',
+                     help="updated reference sitemap name for --changelist calculation")
+    nam.add_argument('--dump', metavar='DUMPFILE', type=str, action='store',
+                     help="write dump to specified file for --resourcelist or --changelist")
+    nam.add_argument('--changelist-uri', '--change-list-uri', type=str, action='store',
+                     help="explicitly set the changelist URI that will be use in --inc mode, "
+                          "overrides process of getting this from the sitemap")
+
+    # Options that apply to multiple modes
+    opt = parser.add_argument_group('MISCELANEOUS OPTIONS')
+    opt.add_argument('--max-sitemap-entries', type=int, action='store',
+                     help="override default size limits")
+    add_shared_misc_options(opt, default_logfile=DEFAULT_LOGFILE, include_remote=True)
+
+    args = parser.parse_args()
+
+    init_logging(to_file=args.logger, logfile=args.logfile, default_logfile=DEFAULT_LOGFILE,
+                 verbose=args.verbose)
+
+    print("----- ResourceSync Explorer -----")
+    process_shared_misc_options(args, include_remote=True)
+
+    c = Explorer(hashes=args.hash,
+                 verbose=args.verbose)
+
+    try:
+        if (args.sitemap):
+            c.sitemap_name = args.sitemap
+        if (args.exclude):
+            c.exclude_patterns = args.exclude
+        if (args.multifile):
+            c.allow_multifile = not args.multifile
+        if (args.noauth):
+            c.noauth = args.noauth
+        if (args.max_sitemap_entries):
+            c.max_sitemap_entries = args.max_sitemap_entries
+
+        c.explore()
+
+    # Any problem we expect will come as a ClientFatalError, anything else
+    # is... an exception ;-)
+    except ClientFatalError as e:
+        sys.stderr.write("\nFatalError: " + str(e) + "\n")
+
+
+if __name__ == '__main__':
+    main()

resync-sync

@@ -0,0 +1,163 @@
+#!/usr/bin/env python
+"""resync-sync: The ResourceSync command line synchronization client.
+
+Copyright 2012-2020 Simeon Warner
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License
+"""
+
+import argparse
+import sys
+
+from resync import __version__
+from resync.client import Client, ClientFatalError
+from resync.client_utils import init_logging, count_true_args, parse_links, parse_capabilities, parse_capability_lists, add_shared_misc_options, process_shared_misc_options
+
+DEFAULT_LOGFILE = 'resync-client.log'
+
+
+def main():
+    """Main function to implement command line script."""
+    if (sys.version_info < (3, 5)):
+        sys.exit("This program requires python version 3.5 or later")
+
+    # Options and arguments
+    parser = argparse.ArgumentParser(
+        description="ResourceSync command line client (v" + __version__ + ")",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+
+    parser._optionals = parser.add_argument_group(
+        'MODES OF OPERATION (must specify one only). A source that is specified '
+        'either in a set of uri=path mappings or else using an explicit --sitemap '
+        'location')
+    rem = parser.add_mutually_exclusive_group(required=True)
+    rem.add_argument('--baseline', '-b', action='store_true',
+                     help='baseline sync of resources from remote source (src) to local filesystem (dst)')
+    rem.add_argument('--incremental', '--inc', '-i', action='store_true',
+                     help='incremental sync of resources from remote source (src) to local filesystem (dst). Uses either timestamp recorded from last baseline or incremental sync for this source, or explicit --from parameter, to determine the earlier update timestamp to act on.')
+    rem.add_argument('--audit', '-a', action='store_true',
+                     help="audit sync state of destination wrt source")
+    rem.add_argument('--parse', '-p', action='store_true',
+                     help="parse a remote sitemap/sitemapindex (from mapping or explicit --sitemap) and show summary information including document type and number of entries")
+
+    # Positional arguments
+    map = parser.add_argument_group('URI MAPPING TO FILESYSTEM for REMOTE modes')
+    map.add_argument(metavar='uri=path | uri path', dest='map', type=str, nargs='*',
+                     help="remote URI of source for remote synchronization operations (may "
+                          "also combine uri=local path)")
+
+    # Specification of map between remote URI and local file paths, and remote
+    # sitemap
+    nam = parser.add_argument_group('FILE/URI NAMING OPTIONS')
+    nam.add_argument('--sitemap', type=str, action='store',
+                     help="explicitly set sitemap name, overriding default sitemap.xml "
+                          "appended to first source URI specified in the mappings")
+    nam.add_argument('--capabilitylist', '--capability-list', type=str, action='store',
+                     help="explicitly set capability list URI to search for instead of "
+                          "looking for the source description")
+    nam.add_argument('--reference', type=str, action='store',
+                     help="reference sitemap name for --write-changelist calculation")
+    nam.add_argument('--newreference', type=str, action='store',
+                     help="updated reference sitemap name for --write-changelist calculation")
+    nam.add_argument('--changelist-uri', '--change-list-uri', type=str, action='store',
+                     help="explicitly set the changelist URI that will be use in --inc mode, "
+                          "overrides process of getting this from the sitemap")
+
+    # Options that apply to multiple modes
+    opt = parser.add_argument_group('MISCELANEOUS OPTIONS')
+    add_shared_misc_options(opt, default_logfile=DEFAULT_LOGFILE, include_remote=True)
+    opt.add_argument('--delete', action='store_true',
+                     help="allow files on destination to be deleted")
+    opt.add_argument('--empty', action='store_true',
+                     help="combine with --changelist to write and empty changelist, perhaps with links")
+    opt.add_argument('--strictauth', action='store_true',
+                     help="use more strict checking of URLs to ensure that the ResourceSync "
+                          "documents refer only to resources on the same server or sub-domains, "
+                          "and on the same server to sub-paths. This is the authority model "
+                          "of Sitemaps but there are legitimate uses where these rules would "
+                          "not be followed.")
+    opt.add_argument('--dryrun', '-n', action='store_true',
+                     help="don't update local resources, say what would be done")
+    opt.add_argument('--ignore-failures', action='store_true',
+                     help="continue past download failures")
+    # These likely only useful for experimentation
+    opt.add_argument('--max-sitemap-entries', type=int, action='store',
+                     help="override default size limits")
+    opt.add_argument('--eval', '-e', action='store_true',
+                     help="output evaluation of source/client synchronization performance... "
+                          "be warned, this is very verbose")
+    opt.add_argument('--tries', '-t', type=int, action='store', metavar='TRIES',
+                     help="set number of tries to TRIES. The default is to retry 20 times, "
+                          "with the exception of fatal errors like \"connection refused\" "
+                          "or \"not found\" (404), which are not retried.")
+    opt.add_argument('--timeout', '-T', type=int, action='store', metavar='SECONDS',
+                     help="set the request timeout for resource downloads to SECONDS seconds")
+
+    args = parser.parse_args()
+
+    # Configure logging module and create logger instance
+    init_logging(to_file=args.logger, logfile=args.logfile, default_logfile=DEFAULT_LOGFILE,
+                 verbose=args.verbose, eval_mode=args.eval)
+
+    process_shared_misc_options(args, include_remote=True)
+
+    c = Client(spec_version=args.spec_version,
+               hashes=args.hash,
+               verbose=args.verbose,
+               dryrun=args.dryrun)
+
+    try:
+        if (args.map):
+            # Mappings apply to (almost) everything
+            c.set_mappings(args.map)
+        if (args.sitemap):
+            c.sitemap_name = args.sitemap
+        if (args.capabilitylist):
+            c.capability_list_uri = args.capabilitylist
+        if (args.exclude):
+            c.exclude_patterns = args.exclude
+        if (args.multifile):
+            c.allow_multifile = not args.multifile
+        if (args.noauth):
+            c.noauth = args.noauth
+        if (args.strictauth):
+            c.strictauth = args.strictauth
+        if (args.max_sitemap_entries):
+            c.max_sitemap_entries = args.max_sitemap_entries
+        if (args.ignore_failures):
+            c.ignore_failures = args.ignore_failures
+        if (args.tries):
+            c.tries = args.tries
+        if (args.timeout):
+            c.timeout = args.timeout
+
+        # Finally, do something...
+        if (args.baseline or args.audit):
+            c.baseline_or_audit(allow_deletion=args.delete,
+                                audit_only=args.audit)
+        elif (args.incremental):
+            c.incremental(allow_deletion=args.delete,
+                          change_list_uri=args.changelist_uri,
+                          from_datetime=args.from_datetime)
+        elif (args.parse):
+            c.parse_document()
+        else:
+            parser.error("Unknown mode requested")
+    # Any problem we expect will come as a ClientFatalError, anything else
+    # is... an exception ;-)
+    except ClientFatalError as e:
+        sys.stderr.write("\nFatalError: " + str(e) + "\n")
+
+
+if __name__ == '__main__':
+    main()

resync/__init__.py

@@ -1,16 +1,17 @@
-from _version import __version__
+"""Module config for resync.
 
-"""Enable easy import for core classes, e.g.
-from resync import Resource
+This is the one place the version number for resync is stored.
 """
-from source_description import SourceDescription
-from capability_list import CapabilityList
-from resource_list import ResourceList
-from change_list import ChangeList
-from resource_dump import ResourceDump
-from resource_dump_manifest import ResourceDumpManifest
-from change_dump import ChangeDump
-from resync.change_dump import ChangeDump
-from archives import ResourceListArchive,ResourceDumpArchive,ChangeListArchive,ChangeDumpArchive
-from resource import Resource
+__version__ = '2.0.1'
 
+# Enable easy import for core classes, e.g.
+# from resync import Resource
+from .source_description import SourceDescription
+from .capability_list import CapabilityList
+from .resource_list import ResourceList
+from .change_list import ChangeList
+from .resource_dump import ResourceDump
+from .resource_dump_manifest import ResourceDumpManifest
+from .change_dump import ChangeDump
+from .archives import ResourceListArchive, ResourceDumpArchive, ChangeListArchive, ChangeDumpArchive
+from .resource import Resource

resync/_version.py

@@ -1,6 +0,0 @@
-# This is the one place the version number for resync is stored
-#
-# Format: x.y.z where
-# x.y is spec version, see http://www.openarchives.org/rs/x.y/
-# z is incremented for revisions within that version, 1...
-__version__ = '1.0.0'

resync/archives.py

@@ -1,53 +1,72 @@
-"""ResourceSync Archive objects
+"""ResourceSync Archive objects.
 
 The ResourceSync Archives specification
 http://www.openarchives.org/rs/archives specifies capabilities
-that provide archives of the 4 core capabilities. While some 
+that provide archives of the 4 core capabilities. While some
 optional attributes are different the basic structure is the
-same for the Resource List Archive, Change List Archive, 
-Resource Dump Archive, and Change Dump Archive. 
+same for the Resource List Archive, Change List Archive,
+Resource Dump Archive, and Change Dump Archive.
 """
 
 import collections
-from urllib import URLopener
 
-from list_base_with_index import ListBaseWithIndex
-from resource import Resource
-from sitemap import Sitemap
+from .list_base_with_index import ListBaseWithIndex
+from .resource import Resource
+from .sitemap import Sitemap
+
 
 class ResourceListArchive(ListBaseWithIndex):
-    """Class representing an Change List Archive"""
+    """Class representing an Resource List Archive."""
 
     def __init__(self, resources=None, md=None, ln=None, uri=None,
+                 spec_version='1.1', add_lastmod=False,
                  resources_class=None):
+        """Initialize ResourceListArchive."""
         self.resources_class = list if resources_class is None else resources_class
         if (resources is None):
             resources = self.resources_class()
-        super(ResourceListArchive, self).__init__(resources=resources, md=md, ln=ln, uri=uri,
-                                                  capability_name='resourcelist-archive')
+        super(ResourceListArchive, self).__init__(
+            resources=resources, md=md, ln=ln, uri=uri,
+            capability_name='resourcelist-archive',
+            spec_version=spec_version, add_lastmod=add_lastmod)
+
 
 class ChangeListArchive(ListBaseWithIndex):
-    """Class representing an Change List Archive"""
+    """Class representing an Change List Archive."""
 
     def __init__(self, resources=None, md=None, ln=None, uri=None,
+                 spec_version='1.1', add_lastmod=False,
                  resources_class=None):
-        super(ChangeListArchive, self).__init__(resources=resources, md=md, ln=ln, uri=uri,
-                                                capability_name='changelist-archive')
+        """Initialize ChangeListArchive."""
+        super(ChangeListArchive, self).__init__(
+            resources=resources, md=md, ln=ln, uri=uri,
+            capability_name='changelist-archive',
+            spec_version=spec_version, add_lastmod=add_lastmod)
+
 
 class ResourceDumpArchive(ListBaseWithIndex):
-    """Class representing an Resource Dump Archive"""
+    """Class representing an Resource Dump Archive."""
 
     def __init__(self, resources=None, md=None, ln=None, uri=None,
+                 spec_version='1.1', add_lastmod=False,
                  resources_class=None):
-        super(ResourceDumpArchive, self).__init__(resources=resources, md=md, ln=ln, uri=uri,
-                                                  capability_name='resourcedump-archive',
-                                                  resources_class=resources_class)
+        """Initialize ResourceDumpArchive."""
+        super(ResourceDumpArchive, self).__init__(
+            resources=resources, md=md, ln=ln, uri=uri,
+            capability_name='resourcedump-archive',
+            spec_version=spec_version, add_lastmod=add_lastmod,
+            resources_class=resources_class)
+
 
 class ChangeDumpArchive(ListBaseWithIndex):
-    """Class representing an Change Dump Archive"""
+    """Class representing an Change Dump Archive."""
 
     def __init__(self, resources=None, md=None, ln=None, uri=None,
+                 spec_version='1.1', add_lastmod=False,
                  resources_class=None):
-        super(ChangeDumpArchive, self).__init__(resources=resources, md=md, ln=ln, uri=uri,
-                                                capability_name='changedump-archive',
-                                                resources_class=resources_class)
+        """Initialize ChangeDumpArchive."""
+        super(ChangeDumpArchive, self).__init__(
+            resources=resources, md=md, ln=ln, uri=uri,
+            capability_name='changedump-archive',
+            spec_version=spec_version, add_lastmod=add_lastmod,
+            resources_class=resources_class)

resync/capability_list.py

@@ -1,36 +1,38 @@
-"""ResourceSync Capability List object
+"""ResourceSync Capability List object.
 
-An Capability List is a set of capabilitys with some metadata for 
-each capability. The Capability List object may also contain metadata 
+An Capability List is a set of capabilitys with some metadata for
+each capability. The Capability List object may also contain metadata
 and links like other lists.
 """
 
-import collections
+import collections.abc
+
+from .resource import Resource
+from .resource_set import ResourceSet
+from .list_base import ListBase
+from .sitemap import Sitemap
 
-from resource import Resource
-from resource_set import ResourceSet
-from list_base import ListBase
-from sitemap import Sitemap
 
 class CapabilitySet(ResourceSet):
-    """Class for storage of resources in a Capability List
+    """Class for storage of resources in a Capability List.
 
-    Extends the ResourceSet to add checks to ensure that there are 
-    never two entries for the same resource, and that values are 
+    Extends the ResourceSet to add checks to ensure that there are
+    never two entries for the same resource, and that values are
     returned in the canonical order.
     """
 
     def __init__(self):
-        self.order = [ 'resourcelist', 'resourcedump',
-                       'changelist', 'changedump',
-                       'resourcelist-archive', 'resourcedump-archive',
-                       'changelist-archive', 'changedump-archive' ]
+        """Initialize CpabilitySet."""
+        self.order = ['resourcelist', 'resourcedump',
+                      'changelist', 'changedump',
+                      'resourcelist-archive', 'resourcedump-archive',
+                      'changelist-archive', 'changedump-archive']
 
     def __iter__(self):
-        """Iterator over all the resources in capability order
+        """Iterator over all the resources in capability order.
 
         Deals with the case of unknown capabilities or duplicate entries
-        by using uri order for duplicates and adding any unknown ones 
+        by using uri order for duplicates and adding any unknown ones
         at the end
         """
         self._iter_next_list = []
@@ -39,7 +41,7 @@ class CapabilitySet(ResourceSet):
         for uri in self.keys():
             cap = self[uri].capability
             if (cap not in uris):
-                uris[cap]=[]
+                uris[cap] = []
             uris[cap].append(uri)
         # build list or uris in defined order for iterator
         for cap in self.order:
@@ -47,7 +49,8 @@ class CapabilitySet(ResourceSet):
                 for uri in sorted(uris[cap]):
                     self._iter_next_list.append(uri)
                 del uris[cap]
-        # add any left over capabilities we don't know about in alphabetical order
+        # add any left over capabilities we don't know about in alphabetical
+        # order
         for cap in uris:
             for uri in sorted(uris[cap]):
                 self._iter_next_list.append(uri)
@@ -55,44 +58,51 @@ class CapabilitySet(ResourceSet):
         return(iter(self._iter_next, None))
 
     def _iter_next(self):
-        if (len(self._iter_next_list)>0):
+        if (len(self._iter_next_list) > 0):
             return(self[self._iter_next_list.pop()])
         else:
             return(None)
 
 
 class CapabilityList(ListBase):
-    """Class representing a Capability List
+    """Class representing a Capability List.
 
-    An Capability List will admit only one resource with any given 
+    An Capability List will admit only one resource with any given
     URI. The iterator over resources is expected to return them in
     canonical order of capability names as defined in main specification
     section 7 and archives specification section 6.
     """
 
-    def __init__(self, resources=None, md=None, ln=None, uri=None):
+    def __init__(self, resources=None, md=None, ln=None, uri=None,
+                 spec_version='1.1', add_lastmod=False):
+        """Initialize CapabilityList."""
         if (resources is None):
             resources = CapabilitySet()
-        super(CapabilityList, self).__init__(resources=resources, md=md, ln=ln, uri=uri,
-                                             capability_name='capabilitylist')
+        super(CapabilityList, self).__init__(
+            resources=resources, md=md, ln=ln, uri=uri,
+            capability_name='capabilitylist',
+            spec_version=spec_version,
+            add_lastmod=add_lastmod)
 
     def add(self, resource, replace=False):
-        """Add a resource or an iterable collection of resources
+        """Add a resource or an iterable collection of resources.
 
         Will throw a ValueError if the resource (ie. same uri) already
         exists in the capability_list, unless replace=True.
-        """
-        if isinstance(resource, collections.Iterable):
-            for r in resource:
-                self.resources.add(r,replace)
-        else:
-            self.resources.add(resource,replace)
 
-    def add_capability(self,capability=None,uri=None,name=None):
-        """Specific add function for capabilities
-        
+        See add_capability() for normal method of adding capabilities.
+        """
+        if isinstance(resource, collections.abc.Iterable):
+            for r in resource:
+                self.resources.add(r, replace)
+        else:
+            self.resources.add(resource, replace)
+
+    def add_capability(self, capability=None, uri=None, name=None):
+        """Specific add function for capabilities.
+
         Takes either:
-        - a capability object (derived from ListBase) as the first argument 
+        - a capability object (derived from ListBase) as the first argument
           from which the capability name is extracted, and the URI if given
         - or a plain name string
         and
@@ -101,17 +111,17 @@ class CapabilityList(ListBase):
         if (capability is not None):
             name = capability.capability_name
             if (capability.uri is not None):
-                uri=capability.uri
-        self.add( Resource(uri=uri,capability=name) )
+                uri = capability.uri
+        self.add(Resource(uri=uri, capability=name))
 
-    def has_capability(self,name=None):
-        """True if the Capability List includes the named capability"""
-        return( self.capability_info(name) is not None )
+    def has_capability(self, name=None):
+        """True if the Capability List includes the named capability."""
+        return(self.capability_info(name) is not None)
 
-    def capability_info(self,name=None):
-        """Return information about the requested capability from this list
+    def capability_info(self, name=None):
+        """Return information about the requested capability from this list.
 
-        Will return None if there is no information about the requested capability
+        Will return None if there is no information about the requested capability.
         """
         for r in self.resources:
             if (r.capability == name):

resync/change_dump.py

@@ -1,7 +1,7 @@
-"""ResourceSync Change Dump object
+"""ResourceSync Change Dump object.
 
 A Change Dump is a set of content dump package resources
-with some metadata for each resource. 
+with some metadata for each resource.
 
 The Change Dump may also contain metadata and links like
 other ResourceSync documents.
@@ -10,17 +10,26 @@ Described in specification at:
 http://www.openarchives.org/rs/resourcesync#ChangeDump
 """
 
-from resource_list import ResourceList
+from .resource_list import ResourceList
+
 
 class ChangeDump(ResourceList):
-    """Class representing an Change Dump
+    """Class representing an Change Dump.
 
     A ChangeDump comprises a set of content dump packages that
     are the resources listed. Properties similar to a ResourceList
     and implemented as a sub-class of ResourceList.
     """
 
-    def __init__(self, resources=None, md=None, ln=None, uri=None, allow_multifile=None, mapper=None):
-        super(ChangeDump, self).__init__(resources=resources, md=md, ln=ln, uri=uri,
-                                         mapper=mapper)
-        self.capability_name='changedump' 
+    def __init__(self, resources=None, md=None, ln=None,
+                 uri=None, allow_multifile=None, mapper=None,
+                 spec_version='1.1', add_lastmod=False):
+        """Initialize ChangeDump.
+
+        Simply sets capability_name to 'changedump' when
+        subclassing ResourceList.
+        """
+        super(ChangeDump, self).__init__(
+            resources=resources, md=md, ln=ln, uri=uri, mapper=mapper,
+            spec_version=spec_version, add_lastmod=add_lastmod)
+        self.capability_name = 'changedump'

resync/change_dump_manifest.py

@@ -1,27 +1,36 @@
-"""ResourceSync ChangeDumpManifest object
+"""ResourceSync ChangeDumpManifest object.
 
-A ChangeDumpManifest lists the set of files/resources included 
-within a content package that is included in a ChangeDump. 
+A ChangeDumpManifest lists the set of files/resources included
+within a content package that is included in a ChangeDump.
 
 The ChangeeDumpManifest object will include the change type,
-a path for each item (except in the case of "deleted"), and may 
+a path for each item (except in the case of "deleted"), and may
 also contain metadata and links.
 
 Described in specification at:
 http://www.openarchives.org/rs/resourcesync#ChangeDumpManifest
 """
 
-from change_list import ChangeList
+from .change_list import ChangeList
+
 
 class ChangeDumpManifest(ChangeList):
-    """Class representing a Change Dump Manifest
+    """Class representing a Change Dump Manifest.
 
     A ChangeDumpManifest comprises a set of files/resources
     in a content package. Properties much Like a ChangeList
     and implemented as a sub-class of ChangeeList.
     """
 
-    def __init__(self, resources=None, md=None, ln=None, uri=None, allow_multifile=None, mapper=None):
-        super(ChangeDumpManifest, self).__init__(resources=resources, md=md, ln=ln, uri=uri, mapper=mapper)
-        self.capability_name = 'changedump-manifest'
+    def __init__(self, resources=None, md=None, ln=None,
+                 uri=None, allow_multifile=None, mapper=None,
+                 spec_version='1.1', add_lastmod=False):
+        """Initialize ChangeDumpManifest.
 
+        Simply sets capability_name to 'changedump-manifest' when
+        subclassing ChangeList.
+        """
+        super(ChangeDumpManifest, self).__init__(
+            resources=resources, md=md, ln=ln, uri=uri, mapper=mapper,
+            spec_version=spec_version, add_lastmod=add_lastmod)
+        self.capability_name = 'changedump-manifest'

resync/change_list.py

@@ -1,4 +1,4 @@
-"""ResourceSync ChangeList object
+"""ResourceSync ChangeList object.
 
 A ChangeList is a list of resource descriptions which includes
 both metadata associated with the resource at some point in
@@ -8,55 +8,79 @@ are Resource objects.
 
 Different from an resource_list, a change_list may include multiple
 descriptions for the same resource. The change_list is ordered
-from first entry to last entry. 
+from first entry to last entry.
 
-Different from an resource_list, dereference by a URI yields a 
-ChangeList containing descriptions pertaining to that 
+Different from an resource_list, dereference by a URI yields a
+ChangeList containing descriptions pertaining to that
 particular resource.
 """
 
-import collections
-from urllib import URLopener
+import collections.abc
+
+from .list_base_with_index import ListBaseWithIndex
+from .resource import Resource, ChangeTypeError
+from .sitemap import Sitemap
 
-from list_base_with_index import ListBaseWithIndex
-from resource import Resource,ChangeTypeError
-from sitemap import Sitemap
 
 class ChangeList(ListBaseWithIndex):
-    """Class representing an Change List"""
+    """Class representing an Change List."""
 
     def __init__(self, resources=None, md=None, ln=None, uri=None,
-                 mapper=None, resources_class=list):
-        super(ChangeList, self).__init__(resources=resources, md=md, ln=ln, uri=uri,
-                                         capability_name='changelist', mapper=mapper,
-                                         resources_class=resources_class)
+                 mapper=None, spec_version='1.1', add_lastmod=False,
+                 resources_class=list):
+        """Initialize ChangeList."""
+        super(ChangeList, self).__init__(
+            resources=resources, md=md, ln=ln, uri=uri,
+            capability_name='changelist', mapper=mapper,
+            spec_version=spec_version, add_lastmod=add_lastmod,
+            resources_class=resources_class)
 
     def add_if_changed(self, resource):
-        """Add resource if change is not None else ChangeTypeError"""
+        """Add resource if change is not None else ChangeTypeError."""
         if (resource.change is not None):
-            self.resources.append(resource)    
+            self.resources.append(resource)
         else:
             raise ChangeTypeError(resource.change)
 
     def add(self, resource):
-        """Add a resource change or an iterable collection of them to 
-        this ChangeList
-      
-        Allows multiple resource_change objects for the same 
+        """Add a resource change or an iterable collection of them.
+
+        Allows multiple resource_change objects for the same
         resource (ie. URI) and preserves the order of addition.
         """
-        if isinstance(resource, collections.Iterable):
+        if isinstance(resource, collections.abc.Iterable):
             for r in resource:
                 self.add_if_changed(r)
         else:
             self.add_if_changed(resource)
 
     def add_changed_resources(self, resources, change=None):
-        """Add items from a ResourceContainer resources to this ChangeList
+        """Add items from a ResourceContainer resources.
 
-        If change is specified then the attribute is set in the Resource 
+        If change is specified then the attribute is set in the Resource
         objects created.
         """
         for resource in resources:
-            rc = Resource( resource=resource, change=change )
+            rc = Resource(resource=resource, change=change)
             self.add(rc)
+
+    def prune_updates_before(self, timestamp, spec_version='1.1'):
+        """Remove all resource updates earlier than the given timestamp.
+
+        Returns the number of entries removed. Will raise an excpetion
+        if there are any entries without a datetime (1.1) or
+        timestamp (1.0).
+        """
+        n = 0
+        pruned = []
+        use_timestamp = (spec_version == '1.0')  # Else use datetime
+        for r in self.resources:
+            ts = r.timestamp if use_timestamp else r.ts_datetime
+            if (ts is None):
+                raise Exception("Entry %s has no update datetime/timestamp" % (r.uri))
+            elif (ts >= timestamp):
+                pruned.append(r)
+            else:
+                n += 1
+        self.resources = pruned
+        return(n)

resync/client_state.py

@@ -1,4 +1,4 @@
-"""ResourceSync client state class
+"""ResourceSync client state class.
 
 The client requires memory of state to support incremental
 synchronization. At minimum it must store the source timestamp
@@ -6,55 +6,70 @@ of the last change seen.
 """
 
 import sys
-import urllib
-import urlparse
 import os.path
 import datetime
-import distutils.dir_util 
+import distutils.dir_util
 import re
 import time
 import logging
-import ConfigParser
+try:  # python3
+    from configparser import ConfigParser, NoSectionError, NoOptionError
+except ImportError:  # pragma: no cover python2
+    from ConfigParser import SafeConfigParser as ConfigParser, NoSectionError, NoOptionError  # pragma: no cover
 
 
 class ClientState(object):
-    """
-    """
+    """Read and store client state on disk."""
 
     def __init__(self):
+        """Initialize ClientState object with default status file name."""
         self.status_file = '.resync-client-status.cfg'
 
-    def set_state(self,site,timestamp=None):
-        """Write status dict to client status file
-        
+    def set_state(self, site, timestamp=None):
+        """Write status dict to client status file.
+
         FIXME - should have some file lock to avoid race
         """
-        parser = ConfigParser.SafeConfigParser()
+        parser = ConfigParser()
         parser.read(self.status_file)
         status_section = 'incremental'
         if (not parser.has_section(status_section)):
             parser.add_section(status_section)
         if (timestamp is None):
-            parser.remove_option(status_section, self.config_site_to_name(site))
+            parser.remove_option(
+                status_section,
+                self.config_site_to_name(site))
         else:
-            parser.set(status_section, self.config_site_to_name(site), str(timestamp))
-        with open(self.status_file, 'wb') as configfile:
+            parser.set(
+                status_section,
+                self.config_site_to_name(site),
+                str(timestamp))
+        with open(self.status_file, 'w') as configfile:
             parser.write(configfile)
             configfile.close()
 
-    def get_state(self,site):
-        """Read client status file and return dict"""
-        parser = ConfigParser.SafeConfigParser()
+    def get_state(self, site):
+        """Read client status file and return dict."""
+        parser = ConfigParser()
         status_section = 'incremental'
         parser.read(self.status_file)
         timestamp = None
         try:
-            timestamp = float(parser.get(status_section,self.config_site_to_name(site)))
-        except ConfigParser.NoSectionError as e:
+            timestamp = float(
+                parser.get(
+                    status_section,
+                    self.config_site_to_name(site)))
+        except NoSectionError as e:
             pass
-        except ConfigParser.NoOptionError as e:
+        except NoOptionError as e:
             pass
         return(timestamp)
 
     def config_site_to_name(self, name):
-        return( re.sub(r"[^\w]",'_',name) )
+        """Convert site name to safe string for config.
+
+        Simply replaces any non-word chars with underscore. This could
+        lead to multiple site names mapping to one config name but
+        probably not too likely.
+        """
+        return(re.sub(r"[^\w]", '_', name))

resync/client_utils.py

@@ -1,142 +1,222 @@
-"""Client Utilities
+"""ResourceSync Client Utilities.
 
-Factor out code shared by both the resync and resync-explorer
-clients.
-
-Copyright 2012,2013 Simeon Warner
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License
+Code shared by client scripts.
 """
 
+import argparse
+from datetime import datetime
 import logging
 import logging.config
-import optparse
-import sys
 
-from resync.client import ClientFatalError
-from resync.explorer import Explorer
-from resync.utils import UTCFormatter
+from .url_or_file_open import set_url_or_file_open_config
+
+
+class ClientFatalError(Exception):
+    """Non-recoverable error in client, should include message to user."""
+
+    pass
+
+
+class ClientError(Exception):
+    """Possibly recoverable error in client, should include message to user."""
+
+    pass
+
+
+class UTCFormatter(logging.Formatter):
+    """Format datetime values as ISO8601 UTC Z form.
+
+    Based on http://bit.ly/T2n3Xk
+    """
+
+    def formatTime(self, record, datefmt=None):
+        """Format datetime of record.created as ISO8601 UTC Z form."""
+        timestamp = record.created
+        return datetime.utcfromtimestamp(timestamp).isoformat() + 'Z'
+
 
 def init_logging(to_file=False, logfile=None, default_logfile='/tmp/resync.log',
-                 human=True, verbose=False, eval_mode=False, 
+                 human=True, verbose=False, eval_mode=False,
                  default_logger='client', extra_loggers=None):
-    """Initialize logging
+    """Initialize logging.
 
     Use of log levels:
     DEBUG - very verbose, for evaluation of output (-e)
     INFO - verbose, only seen by users if they ask for it (-v)
     WARNING - messages output messages to console
 
-    Logging to a file: If to_file is True then output will be written to 
+    Logging to a file: If to_file is True then output will be written to
     a file. This will be logfile if set, else default_logfile (which may
     also be overridden).
     """
-    
     fmt = '%(asctime)s | %(name)s | %(levelname)s | %(message)s'
     formatter = UTCFormatter(fmt)
-    
+
     if human:
         # Create a special handler designed just for human readable output
         hh = logging.StreamHandler()
-        hh.setLevel( logging.INFO if (verbose) else logging.WARNING )
+        hh.setLevel(logging.INFO if (verbose) else logging.WARNING)
         hh.setFormatter(logging.Formatter(fmt='%(message)s'))
     if to_file:
         if (logfile is None):
             logfile = default_logfile
         fh = logging.FileHandler(filename=logfile, mode='a')
         fh.setFormatter(formatter)
-        fh.setLevel( logging.DEBUG if (eval_mode) else logging.INFO )
+        fh.setLevel(logging.DEBUG if (eval_mode) else logging.INFO)
 
-    loggers = [default_logger,'resync']
+    loggers = [default_logger, 'resync']
     if (extra_loggers is not None):
         for logger in extra_loggers:
             loggers.append(logger)
     for logger in loggers:
         log = logging.getLogger(logger)
-        log.setLevel(logging.DEBUG) #control at handler instead
+        log.setLevel(logging.DEBUG)  # control at handler instead
         if human:
             log.addHandler(hh)
         if to_file:
             log.addHandler(fh)
 
-    log=logging.getLogger(default_logger)
+    log = logging.getLogger(default_logger)
     if (to_file):
         log.info("Writing detailed log to %s" % (logfile))
 
 
 def count_true_args(*args):
-    """Count number of list of arguments that evaluate True"""
-    count=0
+    """Count number of list of arguments that evaluate True."""
+    count = 0
     for arg in args:
         if (arg):
-            count+=1
+            count += 1
     return(count)
 
+
 def parse_links(args_link):
-    links=[]
+    """Parse --link options.
+
+    Uses parse_link() to parse each option.
+    """
+    links = []
     if (args_link is not None):
         for link_str in args_link:
             try:
                 links.append(parse_link(link_str))
-            except ValueError as e:
-                raise ClientFatalError("Bad --link option '%s' (%s)"%(link_str,str(e)))
+            except ClientFatalError as e:
+                raise ClientFatalError(
+                    "Bad --link option '%s' (%s)" %
+                    (link_str, str(e)))
     return(links)
 
+
 def parse_link(link_str):
-    """Parse --link option to add to <rs:ln> links
+    """Parse one --link option to add to <rs:ln> links.
 
     Input string of the form: rel,href,att1=val1,att2=val2
     """
-    atts={}
-    help_str = "--link option '%s' (format rel,href,att1=val1...)"%(link_str)
+    atts = {}
+    help_str = "--link option '%s' (format rel,href,att1=val1...)" % (link_str)
     try:
         segs = link_str.split(',')
         # First segments are relation and subject
         atts['rel'] = segs.pop(0)
         atts['href'] = segs.pop(0)
-        if (atts['href']==''):
+        if (atts['href'] == ''):
             raise ClientFatalError("Missing uri in " + help_str)
         # Remaining segments are attributes
         for term in segs:
-            (k,v)=term.split('=')
-            if (k=='' or v==''):
-                raise ClientFatalError("Bad attribute (%s) in " % (term) + help_str)
-            atts[k]=v
+            (k, v) = term.split('=')
+            if (k == '' or v == ''):
+                raise ClientFatalError(
+                    "Bad attribute (%s) in " %
+                    (term) + help_str)
+            atts[k] = v
     except ValueError as e:
         raise ClientFatalError("Bad component of " + help_str)
     except IndexError as e:
         raise ClientFatalError("Incomplete component of " + help_str)
-    
+
     return(atts)
 
+
 def parse_capabilities(caps_str):
-    """Parse list of capabilities in --capabilitylist option
+    """Parse list of capabilities in --capabilitylist option.
 
     Input string of the form: cap_name=uri,cap_name=uri
     """
-    capabilities={}
+    capabilities = {}
     try:
         segs = caps_str.split(',')
         for term in segs:
-            (k,v)=term.split('=')
-            capabilities[k]=v
+            (k, v) = term.split('=')
+            capabilities[k] = v
     except ValueError as e:
-        raise ClientFatalError("Bad component of --capabilitylist option '%s' (%s)"%(caps_str,str(e)))
+        raise ClientFatalError(
+            "Bad component of --capabilitylist option '%s' (%s)" %
+            (caps_str, str(e)))
     return(capabilities)
 
+
 def parse_capability_lists(cls_str):
-    """Parse list of capability lists in --capabilitylistindex option
+    """Parse list of capability lists in --capabilitylistindex option.
 
     Input string of the form: uri,uri
     """
     return(cls_str.split(','))
+
+
+def add_shared_misc_options(opt, default_logfile, include_remote=False):
+    """Add shared miscellaneous options to the argument_group opt.
+
+    Options that the resync-sync, resync-build and resync-explorer scripts use.
+    """
+    opt.add_argument('--hash', type=str, action='append',
+                     help="use specified hash types in addition to last modification time "
+                          "and size (repeatable, may include `md5`, `sha-1` and `sha-256`)")
+    opt.add_argument('--checksum', action='store_true',
+                     help="use md5 checksum in addition to last modification time and size "
+                          "(same as --hash=md5)")
+    opt.add_argument('--from', type=str, action='store', dest='from_datetime', metavar="DATETIME",
+                     help="explicit datetime value used to filter updates in change list for "
+                          "--incremental sync")
+    opt.add_argument('--exclude', type=str, action='append',
+                     help="exclude resources with URI or filename matching the python regex "
+                          "supplied (see: <https://docs.python.org/3/howto/regex.html> for regex "
+                          "information, repeat option for multiple excludes)")
+    opt.add_argument('--multifile', '-m', action='store_true',
+                     help="disable reading and output of sitemapindex for multifile sitemap")
+    if include_remote:
+        opt.add_argument('--noauth', action='store_true',
+                         help="disable checking of URL paths to ensure that the sitemaps refer "
+                              "only to resources on the same server/sub-path etc. Use with care.")
+        opt.add_argument('--access-token', type=str, default=None,
+                         help="include this access token (a bearer token) in web requests")
+        opt.add_argument('--delay', type=float, default=None,
+                         help="add a delay between web requests (default is None)")
+    # Want these to show at the end
+    opt.add_argument('--logger', '-l', action='store_true',
+                     help="create detailed log of client actions (will write "
+                          "to %s unless specified with --logfile" % (default_logfile))
+    opt.add_argument('--logfile', type=str, action='store',
+                     help="create detailed log of client actions")
+    opt.add_argument('--spec-version', default='1.1', choices=('1.0', '1.1'),
+                     help="follow given ResourceSync specification version. The key difference is that v1.0 "
+                          "used lastmod for the time of a change (often also the resource Last-Modification "
+                          "time but not always). In v1.1 the rs:md datetime attribute in a ChangeList "
+                          "indicates the time of the change, and use of lastmod is entirely optional")
+    opt.add_argument('--verbose', '-v', action='store_true',
+                     help="verbose, show additional informational messages")
+
+
+def process_shared_misc_options(args, include_remote=False):
+    """Process shared miscellaneous options in args.
+
+    Parse options that the resync-sync, resync-build and resync-explorer scripts use.
+    """
+    if args.checksum:
+        args.hash.append('md5')
+    if include_remote:
+        if args.access_token:
+            set_url_or_file_open_config('bearer_token', args.access_token)
+        if args.delay:
+            if args.delay < 0.0:
+                raise argparse.ArgumentTypeError("--delay must be non-negative!")
+            set_url_or_file_open_config('delay', args.delay)

resync/dump.py

@@ -1,15 +1,13 @@
-"""Dump handler for ResourceSync"""
+"""Dump handler for ResourceSync."""
 
 import logging
 import os.path
 from zipfile import ZipFile, ZIP_STORED, ZIP_DEFLATED
 from resync.resource_dump_manifest import ResourceDumpManifest
 
-class DumpError(Exception):
-    pass
 
 class Dump(object):
-    """Dump of content for a Resource Dump or Change Dump
+    """Dump of content for a Resource Dump or Change Dump.
 
     The resources itearable must be comprised of Resource objects
     which have the path attributes set to indicate the local
@@ -18,52 +16,59 @@ class Dump(object):
        rl = ResourceList()
        # ... add items by whatever means, may have >50k items and/or
        # >100MB total size of files ...
-       d = Dump()
-       d.write(resources=rl,basename="/tmp/rd_")
-       # will create dump files /tmp/rd_00001.zip etc.
+       d = Dump(resources=rl)
+       d.write(basename="/tmp/rd_")
+       # will create dump files /tmp/rd_00000.zip etc.
     """
 
     def __init__(self, resources=None, format=None, compress=True):
+        """Initialize Dump object and set defaults."""
         self.resources = resources
         self.format = ('zip' if (format is None) else format)
         self.compress = compress
-        self.manifest_class = ResourceDumpManifest #FIXME 
-        self.max_size = 100*1024*1024 #100MB
+        self.manifest_class = ResourceDumpManifest  # FIXME
+        self.max_size = 100 * 1024 * 1024  # 100MB
         self.max_files = 50000
         self.path_prefix = None
         self.logger = logging.getLogger('resync.dump')
-        
+
     def write(self, basename=None, write_separate_manifests=True):
-        """Write one or more dump files to complete this dump
+        """Write one or more dump files to complete this dump.
 
         Returns the number of dump/archive files written.
         """
         self.check_files()
-        n=0
+        n = 0
         for manifest in self.partition_dumps():
-            dumpbase="%s%05d" % (basename,n)
-            dumpfile="%s.%s" % (dumpbase,self.format)
+            dumpbase = "%s%05d" % (basename, n)
+            dumpfile = "%s.%s" % (dumpbase, self.format)
             if (write_separate_manifests):
-                manifest.write(basename=dumpbase+'.xml')
+                manifest.write(basename=dumpbase + '.xml')
             if (self.format == 'zip'):
-                self.write_zip(manifest.resources,dumpfile)
+                self.write_zip(manifest.resources, dumpfile)
             elif (self.format == 'warc'):
-                self.write_warc(manifest.resources,dumpfile)
+                self.write_warc(manifest.resources, dumpfile)
             else:
-                raise DumpError("Unknown dump format requested (%s)" % (self.format))
-            n+=1
+                raise DumpError(
+                    "Unknown dump format requested (%s)" %
+                    (self.format))
+            n += 1
         self.logger.info("Wrote %d dump files" % (n))
         return(n)
 
     def write_zip(self, resources=None, dumpfile=None):
-        """Write a ZIP format dump file
+        """Write a ZIP format dump file.
 
         Writes a ZIP file containing the resources in the iterable resources along with
         a manifest file manifest.xml (written first). No checks on the size of files
         or total size are performed, this is expected to have been done beforehand.
         """
-        compression = ( ZIP_DEFLATED if self.compress else ZIP_STORED )
-        zf = ZipFile(dumpfile, mode="w", compression=compression, allowZip64=True)
+        compression = (ZIP_DEFLATED if self.compress else ZIP_STORED)
+        zf = ZipFile(
+            dumpfile,
+            mode="w",
+            compression=compression,
+            allowZip64=True)
         # Write resources first
         rdm = ResourceDumpManifest(resources=resources)
         real_path = {}
@@ -71,25 +76,27 @@ class Dump(object):
             archive_path = self.archive_path(resource.path)
             real_path[archive_path] = resource.path
             resource.path = archive_path
-        zf.writestr('manifest.xml',rdm.as_xml())
+        zf.writestr('manifest.xml', rdm.as_xml())
         # Add all files in the resources
         for resource in resources:
             zf.write(real_path[resource.path], arcname=resource.path)
         zf.close()
         zipsize = os.path.getsize(dumpfile)
-        self.logger.info("Wrote ZIP file dump %s with size %d bytes" % (dumpfile,zipsize))
-        
+        self.logger.info(
+            "Wrote ZIP file dump %s with size %d bytes" %
+            (dumpfile, zipsize))
+
     def write_warc(self, resources=None, dumpfile=None):
-        """Write a WARC dump file
+        """Write a WARC dump file.
 
         WARC support is not part of ResourceSync v1.0 (Z39.99 2014) but is left
         in this library for experimentation.
         """
-        # Load library late as we want to be able to run rest of code 
+        # Load library late as we want to be able to run rest of code
         # without this installed
         try:
-            from warc import WARCFile,WARCHeader,WARCRecord
-        except:
+            from warc import WARCFile, WARCHeader, WARCRecord
+        except ImportError:
             raise DumpError("Failed to load WARC library")
         wf = WARCFile(dumpfile, mode="w", compress=self.compress)
         # Add all files in the resources
@@ -102,73 +109,83 @@ class Dump(object):
             wh.result_code = 200
             wh.checksum = 'aabbcc'
             wh.location = self.archive_path(resource.path)
-            wf.write_record( WARCRecord( header=wh, payload=resource.path ) )
+            wf.write_record(WARCRecord(header=wh, payload=resource.path))
         wf.close()
         warcsize = os.path.getsize(dumpfile)
-        self.logging.info("Wrote WARC file dump %s with size %d bytes" % (dumpfile,warcsize))
-        
-    def check_files(self,set_length=True,check_length=True):
-        """Go though and check all files in self.resources, add up size, and find
-        longest common path that can be used when writing the dump file. Saved in 
+        self.logging.info(
+            "Wrote WARC file dump %s with size %d bytes" %
+            (dumpfile, warcsize))
+
+    def check_files(self, set_length=True, check_length=True):
+        """Check all files in self.resources, find longest common prefix.
+
+        Go though and check all files in self.resources, add up size, and find
+        longest common path that can be used when writing the dump file. Saved in
         self.path_prefix.
 
         Parameters set_length and check_length control control whether then set_length
-        attribute should be set from the file size if not specified, and whether any 
+        attribute should be set from the file size if not specified, and whether any
         length specified should be checked. By default both are True. In any event, the
         total size calculated is the size of files on disk.
         """
-        total_size = 0 #total size of all files in bytes
+        total_size = 0  # total size of all files in bytes
         path_prefix = None
         for resource in self.resources:
             if (resource.path is None):
-                #explicit test because exception raised by getsize otherwise confusing
-                raise DumpError("No file path defined for resource %s" % resource.uri)
+                # explicit test because exception raised by getsize otherwise
+                # confusing
+                raise DumpError(
+                    "No file path defined for resource %s" %
+                    resource.uri)
             if (path_prefix is None):
                 path_prefix = os.path.dirname(resource.path)
             else:
-                path_prefix = os.path.commonprefix( [ path_prefix, os.path.dirname(resource.path) ])
+                path_prefix = os.path.commonprefix(
+                    [path_prefix, os.path.dirname(resource.path)])
             size = os.path.getsize(resource.path)
             if (resource.length is not None):
-                if (check_length and resource.length!=size):
+                if (check_length and resource.length != size):
                     raise DumpError("Size of resource %s is %d on disk, not %d as specified" %
-                                    (resource.uri, size, resource.length) )
+                                    (resource.uri, size, resource.length))
             elif (set_length):
                 resource.length = size
             if (size > self.max_size):
-                raise DumpError("Size of file (%s, %d) exceeds maximum (%d) dump size" % (resource.path,size,self.max_size))
+                raise DumpError(
+                    "Size of file (%s, %d) exceeds maximum (%d) dump size" %
+                    (resource.path, size, self.max_size))
             total_size += size
         self.path_prefix = path_prefix
         self.total_size = total_size
-        self.logger.info("Total size of files to include in dump %d bytes" % (total_size))
+        self.logger.info(
+            "Total size of files to include in dump %d bytes" %
+            (total_size))
         return True
 
     def partition_dumps(self):
-        """Yeild a set of manifest object that parition the dumps
+        """Yeild a set of manifest object that parition the dumps.
 
         Simply adds resources/files to a manifest until their are either the
         the correct number of files or the size limit is exceeded, then yields
-        that manifest. 
+        that manifest.
         """
-        manifest=self.manifest_class()
-        manifest_size=0
-        manifest_files=0
+        manifest = self.manifest_class()
+        manifest_size = 0
+        manifest_files = 0
         for resource in self.resources:
             manifest.add(resource)
-            manifest_size+=resource.length
-            manifest_files+=1
-            if (manifest_size>=self.max_size or
-                manifest_files>=self.max_files):
+            manifest_size += resource.length
+            manifest_files += 1
+            if (manifest_size >= self.max_size or manifest_files >= self.max_files):
                 yield(manifest)
                 # Need to start a new manifest
-                manifest=self.manifest_class()
-                manifest_size=0
-                manifest_files=0
-        if (manifest_files>0):
+                manifest = self.manifest_class()
+                manifest_size = 0
+                manifest_files = 0
+        if (manifest_files > 0):
             yield(manifest)
 
-
-    def archive_path(self,real_path):
-        """Return the archive path for file with real_path
+    def archive_path(self, real_path):
+        """Return the archive path for file with real_path.
 
         Mapping is based on removal of self.path_prefix which is determined
         by self.check_files().
@@ -176,4 +193,10 @@ class Dump(object):
         if (not self.path_prefix):
             return(real_path)
         else:
-            return( os.path.relpath(real_path,self.path_prefix) )
+            return(os.path.relpath(real_path, self.path_prefix))
+
+
+class DumpError(Exception):
+    """Error class used by Dump() objects."""
+
+    pass

resync/explorer.py

@@ -1,66 +1,44 @@
-"""ResourceSync explorer
+"""ResourceSync explorer.
 
 This is the guts of a client designed to 'explore' the ResourceSync
-facilities offered by a source. Will use standard practices to 
+facilities offered by a source. Will use standard practices to
 look for and interpret capabilities.
 """
 
 import sys
-import urllib
-import urlparse
+try:  # python3
+    from urllib.parse import urlparse, urlunparse, urljoin
+except ImportError:  # python2
+    from urlparse import urlparse, urlunparse, urljoin
+    input = raw_input
 import os.path
 import datetime
-import distutils.dir_util 
+import distutils.dir_util
 import re
 import time
 import logging
-import requests
 
-from resync.mapper import Mapper
-from resync.sitemap import Sitemap
-from resync.client import Client,ClientFatalError
-from resync.client_state import ClientState
-from resync.resource import Resource
-from w3c_datetime import str_to_datetime,datetime_to_str
+from .mapper import Mapper
+from .sitemap import Sitemap
+from .client import Client, ClientFatalError
+from .client_state import ClientState
+from .client_utils import ClientFatalError
+from .resource import Resource
+from .url_or_file_open import url_or_file_open
+from .w3c_datetime import str_to_datetime, datetime_to_str
 
-class XResource(object):
-    """Information about a resource for the explorer
-
-    Must have a uri but may also store:
-    
-    acceptable_capabilities - None for any acceptable, 'resource' if a 
-        resource rather than a capability document is expected, else 
-        a list of capability names
-    checks - a set of information to check when then XResource is inspected
-    base_uri - on creation interpret any relative URI specified in the 
-        context of base_uri, store the resulting full URI
-    """
-    def __init__(self, uri, acceptable_capabilities=None, checks=None, context=None):
-        self.uri=urlparse.urljoin(context,uri)
-        self.acceptable_capabilities=acceptable_capabilities
-        self.checks=checks
-
-class HeadResponse(object):
-    """Object to mock up requests.head(...) response"""
-    def __init__(self):
-        self.status_code=None
-        self.headers={}
-
-class ExplorerQuit(Exception):
-    """Exception raised when user quits normally, no error"""
-    pass
 
 class Explorer(Client):
-    """Extension of the client code to explore a ResourceSync source
+    """Extension of the client code to explore a ResourceSync source.
 
     Designed to support a text-based command-line client that starts from
-    a given URI, server, or local file, and then allows interactive 
+    a given URI, server, or local file, and then allows interactive
     exploration of ResourceSync capabilities offered by the source.
     """
 
     def explore(self):
-        """INTERACTIVE exploration source capabilities
-        
+        """INTERACTIVE exploration source capabilities.
+
         Will use sitemap URI taken either from explicit self.sitemap_name
         or derived from the mappings supplied.
         """
@@ -69,129 +47,144 @@ class Explorer(Client):
         # capabilities
         starts = []
         if (self.sitemap_name is not None):
-            print "Starting from explicit --sitemap %s" % (uri)
-            starts.append( XResource(uri) )
-        elif (len(self.mapper)>0):
+            print("Starting from explicit --sitemap %s" % (self.sitemap_name))
+            starts.append(XResource(self.sitemap_name))
+        elif (len(self.mapper) > 0):
             uri = self.mapper.default_src_uri()
-            (scheme, netloc, path, params, query, fragment) = urlparse.urlparse(uri)
+            (scheme, netloc, path, params, query, fragment) = urlparse(uri)
             if (not scheme and not netloc):
                 if (os.path.isdir(path)):
                     # have a dir, look for 'likely' file names
-                    print "Looking for capability documents in local directory %s" % (path)
-                    for name in ['resourcesync','capabilities.xml',
-                                 'resourcelist.xml','changelist.xml']:
-                        file = os.path.join(path,name)
+                    print(
+                        "Looking for capability documents in local directory %s" %
+                        (path))
+                    for name in ['resourcesync', 'capabilities.xml',
+                                 'resourcelist.xml', 'changelist.xml']:
+                        file = os.path.join(path, name)
                         if (os.path.isfile(file)):
-                            starts.append( XResource(file) )
-                    if (len(starts)==0):
-                        raise ClientFatalError( "No likely capability files found in local directory %s" %
-                                                (path) )
+                            starts.append(XResource(file))
+                    if (len(starts) == 0):
+                        raise ClientFatalError("No likely capability files found in local directory %s" %
+                                               (path))
                 else:
                     # local file, might be anything (or not exist)
-                    print "Starting from local file %s" % (path)
-                    starts.append( XResource(path) )
+                    print("Starting from local file %s" % (path))
+                    starts.append(XResource(path))
             else:
-                # remote, can't tell whether we have a sitemap or a server name or something 
+                # remote, can't tell whether we have a sitemap or a server name or something
                 # else, build list of options depending on whether there is a path and whether
                 # there is an extension/name
-                well_known = urlparse.urlunparse( [ scheme,netloc,'/.well-known/resourcesync','','','' ] )
+                well_known = urlunparse(
+                    [scheme, netloc, '/.well-known/resourcesync', '', '', ''])
                 if (not path):
                     # root, just look for .well-known
-                    starts.append( XResource(well_known, ['capabilitylist','capabilitylistindex']) )
+                    starts.append(
+                        XResource(
+                            well_known, [
+                                'capabilitylist', 'capabilitylistindex']))
                 else:
-                    starts.append( XResource(uri) )
-                    starts.append( XResource(well_known, ['capabilitylist','capabilitylistindex']) )
-                print "Looking for discovery information based on mappings"
+                    starts.append(XResource(uri))
+                    starts.append(
+                        XResource(
+                            well_known, [
+                                'capabilitylist', 'capabilitylistindex']))
+                print("Looking for discovery information based on mappings")
         else:
-            raise ClientFatalError("No source information (server base uri or capability uri) specified, use -h for help")
-        # 
+            raise ClientFatalError(
+                "No source information (server base uri or capability uri) specified, use -h for help")
+        #
         # Have list of one or more possible starting point, try them in turn
         try:
             for start in starts:
                 # For each starting point we create a fresh history
-                history = [ start ]
+                history = [start]
                 input = None
-                while (len(history)>0):
-                    print
+                while (len(history) > 0):
+                    print()
                     xr = history.pop()
-                    new_xr = self.explore_uri(xr,len(history)>0)
+                    new_xr = self.explore_uri(xr, len(history) > 0)
                     if (new_xr):
                         # Add current and new to history
-                        history.append( xr )
-                        history.append( new_xr )
+                        history.append(xr)
+                        history.append(new_xr)
         except ExplorerQuit:
-            pass # expected way to exit
-        print "\nresync-explorer done, bye...\n"
+            pass  # expected way to exit
+        print("\nresync-explorer done, bye...\n")
 
     def explore_uri(self, explorer_resource, show_back=True):
-        """INTERACTIVE exploration of capabilities document(s) starting at a given URI
+        """INTERACTIVE exploration of capabilities document(s) starting at a given URI.
 
-        Will flag warnings if the document is not of type listed in caps
+        Will flag warnings if the document is not of type listed in caps.
         """
         uri = explorer_resource.uri
         caps = explorer_resource.acceptable_capabilities
         checks = explorer_resource.checks
-        print "Reading %s" % (uri)
-        options={}
-        capability=None
+        print("Reading %s" % (uri))
+        options = {}
+        capability = None
         try:
-            if (caps=='resource'):
+            if (caps == 'resource'):
                 # Not expecting a capability document
-                self.explore_show_head(uri,check_headers=checks)
-            else: 
-                s=Sitemap()
-                list = s.parse_xml(urllib.urlopen(uri))
-                (options,capability)=self.explore_show_summary(list,s.parsed_index,caps,context=uri)
+                self.explore_show_head(uri, check_headers=checks)
+            else:
+                s = Sitemap()
+                list = s.parse_xml(url_or_file_open(uri))
+                (options, capability) = self.explore_show_summary(
+                    list, s.parsed_index, caps, context=uri)
         except IOError as e:
-            print "Cannot read %s (%s)" % (uri,str(e))
+            print("Cannot read %s (%s)" % (uri, str(e)))
         except Exception as e:
-            print "Cannot parse %s (%s)" % (uri,str(e))
+            print("Cannot parse %s (%s)" % (uri, str(e)))
         #
-        # Loop until we have some valide input
+        # Loop until we have some valid input
         #
         while (True):
             # don't offer number option for no resources/capabilities
-            num_prompt = '' if (len(options)==0) else 'number, '
+            num_prompt = '' if (len(options) == 0) else 'number, '
             up_prompt = 'b(ack), ' if (show_back) else ''
-            input = raw_input( "Follow [%s%sq(uit)]?" % (num_prompt,up_prompt) )
-            if (input in options.keys()):
+            if (self.fake_input):
+                inp = self.fake_input
+            else:
+                inp = input("Follow [%s%sq(uit)]?" % (num_prompt, up_prompt))
+            if (inp in options.keys()):
                 break
-            if (input == 'q'):
+            if (inp == 'q'):
                 raise ExplorerQuit()
-            if (input == 'b'):
+            if (inp == 'b'):
                 return(None)
         #
         # Got input that is one of the options
         #
         checks = {}
-        r = options[input]
-        if ( r.capability is None ):
-            if (capability in ['resourcelist','changelist',
-                               'resourcedump','changedump']):
+        r = options[inp]
+        if (r.capability is None):
+            if (capability in ['resourcelist', 'changelist',
+                               'resourcedump', 'changedump']):
                 caps = 'resource'
             else:
                 caps = self.allowed_entries(capability)
-        elif (r.capability is 'resource'):
+        elif (r.capability == 'resource'):
             caps = r.capability
         else:
             caps = [r.capability]
         # Record anything we know about the resource to check
         if (r.length is not None):
-            checks['content-length']=r.length
+            checks['content-length'] = r.length
         if (r.lastmod is not None):
-            checks['last-modified']=r.lastmod
+            checks['last-modified'] = r.lastmod
         if (r.mime_type is not None):
-            checks['content-type']=r.mime_type
+            checks['content-type'] = r.mime_type
         # FIXME - could add fixity checks here too
-        return( XResource(options[input].uri, caps, checks) )
+        return(XResource(r.uri, caps, checks))
 
-    def explore_show_summary(self, list, index=False, expected=None, context=None):
-        """Show summary of one capability document
+    def explore_show_summary(self, list, index=False,
+                             expected=None, context=None):
+        """Show summary of one capability document.
 
-        Given a capability document or index (in list, index True if it is an 
-        index), write out a simply textual summary of the document with all 
-        related documents shown as numbered options (of the form 
-        [#] ...description...) which will then form a menu for the next 
+        Given a capability document or index (in list, index True if it is an
+        index), write out a simply textual summary of the document with all
+        related documents shown as numbered options (of the form
+        [#] ...description...) which will then form a menu for the next
         exploration.
 
         If expected is not None then it should be a list of expected document
@@ -206,127 +199,129 @@ class Explorer(Client):
             capability = list.md['capability']
         if (index):
             capability += 'index'
-        print "Parsed %s document with %d entries:" % (capability,num_entries)
+        print(
+            "Parsed %s document with %d entries:" %
+            (capability, num_entries))
         if (expected is not None and capability not in expected):
-            print "WARNING - expected a %s document" % (','.join(expected))
-        if (capability not in ['description','descriptionoindex','capabilitylist',
-                               'resourcelist','resourcelistindex','changelist','changelistindex',
-                               'resourcedump','resourcedumpindex','changedump','changedumpindex',
+            print("WARNING - expected a %s document" % (','.join(expected)))
+        if (capability not in ['description', 'descriptionoindex', 'capabilitylist',
+                               'resourcelist', 'resourcelistindex', 'changelist', 'changelistindex',
+                               'resourcedump', 'resourcedumpindex', 'changedump', 'changedumpindex',
                                'resourcelist-archive', 'resourcedump-archive',
                                'changelist-archive', 'changedump-archive']):
-            print "WARNING - unknown %s document type" % (capability)
+            print("WARNING - unknown %s document type" % (capability))
         to_show = num_entries
-        if (num_entries>21):
+        if (num_entries > 21):
             to_show = 20
-        # What capability entries are allowed/expected? 
-        entry_caps = self.allowed_entries(capability);
+        # What capability entries are allowed/expected?
+        entry_caps = self.allowed_entries(capability)
         options = {}
-        n=0
+        n = 0
         # Look for <rs:ln> elements in this document
         ln_describedby = list.link('describedby')
         if (ln_describedby):
             if ('href' in ln_describedby):
                 uri = ln_describedby['href']
-                print "[%s] rel='describedby' link to %s" % ('d',uri)
-                uri = self.expand_relative_uri(context,uri)
-                options['d']=Resource(uri,capability='resource')
+                print("[%s] rel='describedby' link to %s" % ('d', uri))
+                uri = self.expand_relative_uri(context, uri)
+                options['d'] = Resource(uri, capability='resource')
             else:
-                print "WARNING - describedby link with no href, ignoring"
+                print("WARNING - describedby link with no href, ignoring")
         ln_up = list.link('up')
         if (ln_up):
             if ('href' in ln_up):
                 uri = ln_up['href']
-                print "[%s] rel='up' link to %s" % ('u',uri)
-                uri = self.expand_relative_uri(context,uri)
-                options['u']=Resource(uri)
+                print("[%s] rel='up' link to %s" % ('u', uri))
+                uri = self.expand_relative_uri(context, uri)
+                options['u'] = Resource(uri)
             else:
-                print "WARNING - up link with no href, ignoring"
+                print("WARNING - up link with no href, ignoring")
         ln_index = list.link('index')
         if (ln_index):
             if ('href' in ln_index):
                 uri = ln_index['href']
-                print "[%s] rel='index' link to %s" % ('i',uri)
-                uri = self.expand_relative_uri(context,uri)
-                options['i']=Resource(uri)
+                print("[%s] rel='index' link to %s" % ('i', uri))
+                uri = self.expand_relative_uri(context, uri)
+                options['i'] = Resource(uri)
             else:
-                print "WARNING - index link with no href, ignoring"
+                print("WARNING - index link with no href, ignoring")
         # Show listed resources as numbered options
         for r in list.resources:
-            if (n>=to_show):
-                print "(not showing remaining %d entries)" % (num_entries-n)
+            if (n >= to_show):
+                print("(not showing remaining %d entries)" % (num_entries - n))
                 break
-            n+=1
-            options[str(n)]=r
-            print "[%d] %s" % (n,r.uri)
+            n += 1
+            options[str(n)] = r
+            print("[%d] %s" % (n, r.uri))
             if (self.verbose):
-                print "  " + str(r)
-            r.uri = self.expand_relative_uri(context,r.uri)
+                print("  " + str(r))
+            r.uri = self.expand_relative_uri(context, r.uri)
             if (r.capability is not None):
                 warning = ''
                 if (r.capability not in entry_caps):
                     warning = " (EXPECTED %s)" % (' or '.join(entry_caps))
-                print "  %s%s" % (r.capability,warning)
-            elif (len(entry_caps)==1):
-                r.capability=entry_caps[0]
-                print "  capability not specified, should be %s" % (r.capability)
-        return(options,capability)
+                print("  %s%s" % (r.capability, warning))
+            elif (len(entry_caps) == 1):
+                r.capability = entry_caps[0]
+                print(
+                    "  capability not specified, should be %s" %
+                    (r.capability))
+        return(options, capability)
 
-    def explore_show_head(self,uri,check_headers=None):
-        """Do HEAD on uri and show infomation
+    def explore_show_head(self, uri, check_headers=None):
+        """Do HEAD on uri and show infomation.
 
-        Will also check headers against any values specified in 
+        Will also check headers against any values specified in
         check_headers.
         """
-        print "HEAD %s" % (uri)
+        print("HEAD %s" % (uri))
         if (re.match(r'^\w+:', uri)):
             # Looks like a URI
-            response = requests.head(uri)
+            with url_or_file_open(uri, method='HEAD') as response:
+                status_code = response.code()
+                headers = response.headers()
         else:
             # Mock up response if we have a local file
-            response = self.head_on_file(uri)
-        print "  status: %s" % (response.status_code)
-        if (response.status_code=='200'):
-            # generate normalized lastmod
-#        if ('last-modified' in response.headers):
-#            response.headers.add['lastmod'] = datetime_to_str(str_to_datetime(response.headers['last-modified']))
-        # print some of the headers
-            for header in ['content-length','last-modified','lastmod','content-type','etag']:
-                if header in response.headers:
-                    check_str=''
-                    if (check_headers is not None and
-                        header in check_headers):
-                        if (response.headers[header] == check_headers[header]):
-                            check_str=' MATCHES EXPECTED VALUE'
+            (status_code, headers) = self.head_on_file(uri)
+        print("  status: %s" % (status_code))
+        if (status_code == '200'):
+            # print some of the headers
+            for header in ['content-length', 'last-modified',
+                           'lastmod', 'content-type', 'etag']:
+                if header in headers:
+                    check_str = ''
+                    if (check_headers is not None and header in check_headers):
+                        if (headers[header] == check_headers[header]):
+                            check_str = ' MATCHES EXPECTED VALUE'
                         else:
-                            check_str=' EXPECTED %s' % (check_headers[header])
-                    print "  %s: %s%s" % (header, response.headers[header], check_str)
+                            check_str = ' EXPECTED %s' % (
+                                check_headers[header])
+                    print("  %s: %s%s" % (header, headers[header], check_str))
 
-    def head_on_file(self,file):
-        """Mock up requests.head(..) response on local file
-        """
-        response = HeadResponse()
-        if (not os.path.isfile(file)):
-            response.status_code='404'
-        else:
-            response.status_code='200'
-            response.headers['last-modified']=datetime_to_str(os.path.getmtime(file))
-            response.headers['content-length']=os.path.getsize(file)
-        return(response)
+    def head_on_file(self, file):
+        """Get fake status code and headers from local file."""
+        status_code = '404'
+        headers = {}
+        if os.path.isfile(file):
+            status_code = '200'
+            headers['last-modified'] = datetime_to_str(os.path.getmtime(file))
+            headers['content-length'] = os.path.getsize(file)
+        return(status_code, headers)
 
-    def allowed_entries(self,capability):
-        """Given a current capability document, return list of allowed entries
+    def allowed_entries(self, capability):
+        """Return list of allowed entries for given capability document.
 
-        Includes handling of capability = *index where the only acceptable 
+        Includes handling of capability = *index where the only acceptable
         entries are *.
         """
-        index = re.match(r'(.+)index$',capability)
-        archive = re.match('r(.+)\-archive$',capability)
+        index = re.match(r'(.+)index$', capability)
+        archive = re.match(r'(.+)\-archive$', capability)
         if (capability == 'capabilitylistindex'):
-            return([]) #not allowed so no valid references
+            return([])  # not allowed so no valid references
         elif (index):
-            return(index.group(0)) #name without index ending
+            return([index.group(1)])  # name without index ending
         elif (archive):
-            return(archive.group(0)) #name without -archive ending
+            return([archive.group(1)])  # name without -archive ending
         elif (capability == 'description'):
             return(['capabilitylist'])
         elif (capability == 'capabilitylist'):
@@ -336,13 +331,40 @@ class Explorer(Client):
                     'changelist-archive', 'changedump-archive'])
         return([])
 
-    def expand_relative_uri(self,context,uri):
-        """If uri is relative then expand in context
-        
-        Prints warning if expansion happens
+    def expand_relative_uri(self, context, uri):
+        """If uri is relative then expand in context.
+
+        Prints warning if expansion happens.
         """
-        full_uri = urlparse.urljoin(context,uri)
+        full_uri = urljoin(context, uri)
         if (full_uri != uri):
-            print "  WARNING - expanded relative URI to %s" % (full_uri)
+            print("  WARNING - expanded relative URI to %s" % (full_uri))
             uri = full_uri
         return(uri)
+
+
+class XResource(object):
+    """Information about a resource for the explorer.
+
+    Must have a uri but may also store:
+
+    acceptable_capabilities - None for any acceptable, 'resource' if a
+        resource rather than a capability document is expected, else
+        a list of capability names
+    checks - a set of information to check when then XResource is inspected
+    base_uri - on creation interpret any relative URI specified in the
+        context of base_uri, store the resulting full URI
+    """
+
+    def __init__(self, uri, acceptable_capabilities=None,
+                 checks=None, context=None):
+        """Initialize XResource object."""
+        self.uri = urljoin(context, uri)
+        self.acceptable_capabilities = acceptable_capabilities
+        self.checks = checks
+
+
+class ExplorerQuit(Exception):
+    """Exception raised when user quits normally, no error."""
+
+    pass

resync/hashes.py

@@ -0,0 +1,116 @@
+"""util.py: A collection of utility functions for source and/or client."""
+import base64
+import hashlib
+
+
+class Hashes(object):
+    """Compute hash digests for ResourceSync.
+
+    These are all base64 encoded according to the rules of
+    http://www.ietf.org/rfc/rfc4648.txt
+
+    MD5
+
+    ResourceSync defined to be the same as for Content-MD5 in HTTP,
+    http://www.ietf.org/rfc/rfc2616.txt which, in turn, defined the
+    digest string as the "base64 of 128 bit MD5 digest as per RFC 1864"
+    http://www.ietf.org/rfc/rfc1864.txt
+
+    Unfortunately, RFC1864 is rather vague and contains only and example
+    which doesn't use encoding characters for 62 or 63. It points to
+    RFC1521 to describe base64 which is explicit that the encoding alphabet
+    is [A-Za-z0-9+/] with = to pad.
+
+    The above corresponds with the alphabet of "3. Base 64 Encoding" in RFC3548
+    http://www.ietf.org/rfc/rfc3548.txt
+    and not the url safe version, "Base 64 Encoding with URL and Filename Safe
+    Alphabet" which replaces + and / with - and _ respectively.
+
+    This is the same as the alphabet of "4. Base 64 Encoding" in RFC4648
+    http://www.ietf.org/rfc/rfc4648.txt.
+
+    This algorithm is implemented by base64.standard_b64encode() or
+    base64.b64encode() with no altchars specified. Available in python2.4 and
+    up [http://docs.python.org/library/base64.html]
+    """
+
+    NAME_TO_ATTRIBUTE = {'md5': 'md5', 'sha-1': 'sha1', 'sha-256': 'sha256'}
+
+    def __init__(self, hashes=None, file=None):
+        """Initialize Hashes object with types of hash to caluclate.
+
+        If file is supplied then compute for that file.
+        """
+        self.hashes = set()
+        for hash in hashes:
+            if (hash not in self.NAME_TO_ATTRIBUTE.keys()):
+                raise Exception("Hash type %s not supported" % (hash))
+            self.hashes.add(hash)
+        #
+        self.md5_calc = None
+        self.sha1_calc = None
+        self.sha256_calc = None
+        #
+        if (file is not None):
+            self.compute_for_file(file)
+
+    def initialize_hashes(self):
+        """Create new hashlib objects for each hash we are going to calculate."""
+        if ('md5' in self.hashes):
+            self.md5_calc = hashlib.md5()
+        if ('sha-1' in self.hashes):
+            self.sha1_calc = hashlib.sha1()
+        if ('sha-256' in self.hashes):
+            self.sha256_calc = hashlib.sha256()
+
+    def compute_for_file(self, file, block_size=2**14):
+        """Compute hash digests for a file.
+
+        Calculate the hashes based on one read through the file.
+        Optional block_size parameter controls memory used to do
+        calculations. This should be a multiple of 128 bytes.
+        """
+        self.initialize_hashes()
+        f = open(file, 'rb')
+        while True:
+            data = f.read(block_size)
+            if not data:
+                break
+            if self.md5_calc is not None:
+                self.md5_calc.update(data)
+            if self.sha1_calc is not None:
+                self.sha1_calc.update(data)
+            if self.sha256_calc is not None:
+                self.sha256_calc.update(data)
+        f.close()
+
+    def set(self, resource):
+        """Set hash values for resource from current file.
+
+        Assumes that resource has appropriate attributes or setters
+        with names md5, sha1, etc. and that hashes have been calculated.
+        """
+        for hash in self.hashes:
+            att = self.NAME_TO_ATTRIBUTE[hash]
+            setattr(resource, att, getattr(self, att))
+
+    @property
+    def md5(self):
+        """Return MD5 hash calculated."""
+        if (self.md5_calc is None):
+            return None
+        return self.md5_calc.hexdigest()
+
+    @property
+    def sha1(self):
+        """Return SHA-1 hash calculated."""
+        if (self.sha1_calc is None):
+            return None
+        return self.sha1_calc.hexdigest()
+
+    @property
+    def sha256(self):
+        """Return SHA-256 hash calculated."""
+        if (self.sha256_calc is None):
+            return None
+        return self.sha256_calc.hexdigest()

resync/list_base.py

@@ -1,43 +1,55 @@
-"""Base class for ResourceSync capabilities with list of resources
+"""Base class for ResourceSync capabilities with list of resources.
 
 Adds Sitemap based IO to the ResourceContainer class and is
 intended as the base class for ResourceList, ChangeList,
 CapabilityList etc.. Adds common read() and write() methods.
 """
 
-import collections
-import os
 from datetime import datetime
+import io
+import os
 import re
 import sys
-import StringIO
 import logging
-from urllib import URLopener
 
-from resource_container import ResourceContainer
-from sitemap import Sitemap
+from .resource_container import ResourceContainer
+from .sitemap import Sitemap
+from .url_or_file_open import url_or_file_open
+
 
 class ListBase(ResourceContainer):
-    """Class that adds Sitemap based IO to ResourceContainer
+    """Class that adds Sitemap based IO to ResourceContainer.
 
     resources - an iterable of resources
 
-    count - add optional explicit setting of the number of items in 
+    count - add optional explicit setting of the number of items in
         resources which is useful when this is an iterator/generator.
         Is used instead of trying len(resources)
 
     md - metadata information for the list (<rs:md>)
-     
+
     ln - link information for the list (<rs:ln>)
 
-    sitemapindex - defaults to False, set True if this is an index object
+    uri - the URL of this list
+
+    capability_name -
+
+    spec_version - default to None for latest version supported, else explicit
+        version such as '1.0'
+
+    Internal variables:
+      sitemapindex - defaults to False, set True if this is an index object
+      pretty_xml - defaults to False, set True for more human readable output
     """
 
-    def __init__(self, resources=None, count=None, md=None, ln=None, uri=None, 
-                 capability_name='unknown'):
+    def __init__(self, resources=None, count=None, md=None, ln=None, uri=None,
+                 capability_name='unknown', spec_version='1.1', add_lastmod=False):
+        """Initialize ListBase."""
         super(ListBase, self).__init__(resources=resources, md=md, ln=ln, uri=uri,
                                        capability_name=capability_name)
         self.count = count
+        self.spec_version = spec_version
+        self.add_lastmod = add_lastmod  # Optional in v1.1
         self.sitemapindex = False
         self.pretty_xml = False
         #
@@ -46,11 +58,11 @@ class ListBase(ResourceContainer):
         self.parsed_index = None
 
     def __iter__(self):
-        """Default to iterator provided by resources object"""
+        """Default to iterator provided by resources object."""
         return(iter(self.resources))
 
     def __len__(self):
-        """Number of entries in this list
+        """Number of entries in this list.
 
         To handle the case where self.resources is a generator or an iterator and
         thus cannot provide len(...) we first check to see whether an explicit
@@ -64,62 +76,79 @@ class ListBase(ResourceContainer):
             return(self.count)
         return(len(self.resources))
 
-    ##### INPUT #####
+    # INPUT
 
-    def read(self,uri=None):
-        """Default case is just to parse document at this URI
+    def read(self, uri=None):
+        """Default case is just to parse document at this URI.
 
         Intention is that the read() method may be overridden to support reading
         of compound documents in more then one sitemapindex/sitemap.
         """
         self.parse(uri=uri)
 
-    def parse(self,uri=None,fh=None,str=None):
-        """Parse a single XML document for this list
+    def parse(self, uri=None, fh=None, str_data=None, **kwargs):
+        """Parse a single XML document for this list.
 
-        Accepts either a uri (uri or default if parameter not specified), 
-        or a filehandle (fh) or a string (str).
+        Accepts either a uri (uri or default if parameter not specified),
+        or a filehandle (fh) or a string (str_data). Note that this method
+        does not handle the case of a sitemapindex+sitemaps.
 
-        Does not handle the case of sitemapindex+sitemaps
+        LEGACY SUPPORT - the parameter str may be used in place of str_data
+        but is deprecated and will be removed in a later version.
         """
         if (uri is not None):
             try:
-                fh = URLopener().open(uri)
+                fh = url_or_file_open(uri)
             except IOError as e:
-                raise Exception("Failed to load sitemap/sitemapindex from %s (%s)" % (uri,str(e)))
-        elif (str is not None):
-            fh=StringIO.StringIO(str)
+                raise Exception(
+                    "Failed to load sitemap/sitemapindex from %s (%s)" %
+                    (uri, str(e)))
+        elif (str_data is not None):
+            fh = io.StringIO(str_data)
+        elif ('str' in kwargs):
+            # Legacy support for str argument, see
+            # https://github.com/resync/resync/pull/21
+            # One test for this in tests/test_list_base.py
+            self.logger.warning(
+                "Legacy parse(str=...), use parse(str_data=...) instead")
+            fh = io.StringIO(kwargs['str'])
         if (fh is None):
             raise Exception("Nothing to parse")
         s = self.new_sitemap()
-        s.parse_xml(fh=fh,resources=self,capability=self.capability_name,sitemapindex=False)
+        s.parse_xml(
+            fh=fh,
+            resources=self,
+            capability=self.capability_name,
+            sitemapindex=False)
         self.parsed_index = s.parsed_index
-        
-    ##### OUTPUT #####
+
+    # OUTPUT
 
     def as_xml(self):
-        """Return XML serialization of this list
+        """Return XML serialization of this list.
 
-        This code does not support the case where the list is too big for 
+        This code does not support the case where the list is too big for
         a single XML document.
         """
         self.default_capability()
         s = self.new_sitemap()
-        return s.resources_as_xml(self,sitemapindex=self.sitemapindex)
+        return s.resources_as_xml(self, sitemapindex=self.sitemapindex)
 
-    def write(self,basename="/tmp/resynclist.xml"):
-        """Write a single sitemap or sitemapindex XML document
+    def write(self, basename="/tmp/resynclist.xml"):
+        """Write a single sitemap or sitemapindex XML document.
 
         Must be overridden to support multi-file lists.
         """
         self.default_capability()
-        fh = open(basename,'w')
+        fh = open(basename, 'w')
         s = self.new_sitemap()
-        s.resources_as_xml(self,fh=fh,sitemapindex=self.sitemapindex)
+        s.resources_as_xml(self, fh=fh, sitemapindex=self.sitemapindex)
         fh.close()
 
-    ##### UTILITY #####
+    # UTILITY
 
     def new_sitemap(self):
-        """Create new Sitemap object with default settings"""
-        return Sitemap(pretty_xml=self.pretty_xml)
+        """Create new Sitemap object with default settings."""
+        return Sitemap(pretty_xml=self.pretty_xml,
+                       spec_version=self.spec_version,
+                       add_lastmod=self.add_lastmod)

resync/list_base_with_index.py

@@ -1,31 +1,28 @@
-"""Base class for ResourceSync capabilities with lists of resources including 
-support for both sitemaps and sitemapindexes.
+"""Base class for ResourceSync capabilities with large lists of resources.
 
-Extends ListBase to add support for sitemapindexes.
+Handles possibly large lists by including support for both sitemaps and
+sitemapindexes. Extends ListBase to add the support for sitemapindexes.
 """
 
 import collections
-import math 
+import math
 import os
 from datetime import datetime
 import re
 import sys
 import itertools
-from urllib import URLopener
 
-from list_base import ListBase
-from resource import Resource
-from sitemap import Sitemap
-from mapper import Mapper, MapperError
-from url_authority import UrlAuthority
-from utils import compute_md5_for_file
+from .hashes import Hashes
+from .list_base import ListBase
+from .mapper import Mapper, MapperError
+from .resource import Resource
+from .sitemap import Sitemap
+from .url_authority import UrlAuthority
+from .url_or_file_open import url_or_file_open
 
-class ListBaseIndexError(Exception):
-    """Exception for problems with sitemapindexes"""
-    pass
 
 class ListBaseWithIndex(ListBase):
-    """Class that add handling of sitemapindexes to ListBase
+    """Class that add handling of sitemapindexes to ListBase.
 
     Splitting of a list into multiple sitemaps with a sitemapindex is currently
     handled based solely on the number of entries in the list. The configurable
@@ -50,135 +47,165 @@ class ListBaseWithIndex(ListBase):
         to those that the component sitemap files will be exposed as
     """
 
-    def __init__(self, resources=None, count=None, md=None, ln=None, uri=None, 
+    def __init__(self, resources=None, count=None, md=None, ln=None, uri=None,
                  capability_name='unknown', allow_multifile=None, mapper=None,
-                 resources_class=None):
+                 spec_version='1.1', add_lastmod=False, resources_class=None):
+        """Initialize ListBaseWithIndex."""
         self.resources_class = list if resources_class is None else resources_class
         if (resources is None):
             resources = self.resources_class()
-        super(ListBaseWithIndex, self).__init__(resources=resources, count=count, md=md, ln=ln, 
-                                                uri=uri, capability_name=capability_name)
+        super(ListBaseWithIndex, self).__init__(
+            resources=resources,
+            count=count,
+            md=md,
+            ln=ln,
+            uri=uri,
+            capability_name=capability_name,
+            spec_version=spec_version,
+            add_lastmod=add_lastmod)
         # specific to lists with indexes
-        self.max_sitemap_entries=50000
+        self.max_sitemap_entries = 50000
         self.mapper = mapper
-        self.allow_multifile = (True if (allow_multifile is None) else allow_multifile)
+        self.allow_multifile = (
+            True if (
+                allow_multifile is None) else allow_multifile)
         self.check_url_authority = False
         self.content_length = 0
         self.num_files = 0            # Number of files read
         self.bytes_read = 0           # Aggregate of content_length values
 
-    ##### INPUT #####
+    # INPUT
 
     def read(self, uri=None, resources=None, index_only=False):
-        """Read sitemap from a URI including handling sitemapindexes
+        """Read sitemap from a URI including handling sitemapindexes.
 
         If index_only is True then individual sitemaps references in a sitemapindex
         will not be read. This will result in no resources being returned and is
         useful only to read the metadata and links listed in the sitemapindex.
 
-        Includes the subtlety that if the input URI is a local file and is a 
+        Includes the subtlety that if the input URI is a local file and is a
         sitemapindex which contains URIs for the individual sitemaps, then these
         are mapped to the filesystem also.
         """
         try:
-            fh = URLopener().open(uri)
+            fh = url_or_file_open(uri)
             self.num_files += 1
         except IOError as e:
-            raise IOError("Failed to load sitemap/sitemapindex from %s (%s)" % (uri,str(e)))
+            raise IOError(
+                "Failed to load sitemap/sitemapindex from %s (%s)" %
+                (uri, str(e)))
         # Get the Content-Length if we can (works fine for local files)
         try:
             self.content_length = int(fh.info()['Content-Length'])
             self.bytes_read += self.content_length
-            self.logger.debug( "Read %d bytes from %s" % (self.content_length,uri) )
-        except KeyError:
+            self.logger.debug(
+                "Read %d bytes from %s" %
+                (self.content_length, uri))
+        except (KeyError, TypeError):
             # If we don't get a length then c'est la vie
-            self.logger.debug( "Read ????? bytes from %s" % (uri) )
+            self.logger.debug("Read ????? bytes from %s" % (uri))
             pass
-        self.logger.info( "Read sitemap/sitemapindex from %s" % (uri) )
+        self.logger.info("Read sitemap/sitemapindex from %s" % (uri))
         s = self.new_sitemap()
-        s.parse_xml(fh=fh,resources=self,capability=self.capability_name)
+        s.parse_xml(fh=fh, resources=self, capability=self.capability_name)
         # what did we read? sitemap or sitemapindex?
         if (s.parsed_index):
             # sitemapindex
             if (not self.allow_multifile):
-                raise ListBaseIndexError("Got sitemapindex from %s but support for sitemapindex disabled" % (uri))
-            self.logger.info( "Parsed as sitemapindex, %d sitemaps" % (len(self.resources)) )
+                raise ListBaseIndexError(
+                    "Got sitemapindex from %s but support for sitemapindex disabled" %
+                    (uri))
+            self.logger.info(
+                "Parsed as sitemapindex, %d sitemaps" %
+                (len(
+                    self.resources)))
             sitemapindex_is_file = self.is_file_uri(uri)
             if (index_only):
                 # don't read the component sitemaps
                 self.sitemapindex = True
                 return
-            # now loop over all entries to read each sitemap and add to resources
+            # now loop over all entries to read each sitemap and add to
+            # resources
             sitemaps = self.resources
             self.resources = self.resources_class()
-            self.logger.info( "Now reading %d sitemaps" % len(sitemaps.uris()) )
+            self.logger.info("Now reading %d sitemaps" % len(sitemaps.uris()))
             for sitemap_uri in sorted(sitemaps.uris()):
-                self.read_component_sitemap(uri,sitemap_uri,s,sitemapindex_is_file)
+                self.read_component_sitemap(
+                    uri, sitemap_uri, s, sitemapindex_is_file)
         else:
             # sitemap
-            self.logger.info( "Parsed as sitemap, %d resources" % (len(self.resources)) )
+            self.logger.info("Parsed as sitemap, %d resources" %
+                             (len(self.resources)))
 
+    def read_component_sitemap(
+            self, sitemapindex_uri, sitemap_uri, sitemap, sitemapindex_is_file):
+        """Read a component sitemap of a Resource List with index.
 
-    def read_component_sitemap(self, sitemapindex_uri, sitemap_uri, sitemap, sitemapindex_is_file):
-        """Read a component sitemap of a Resource List with index
-
-        Each component must be a sitemap with the 
+        Each component must be a sitemap with the
         """
         if (sitemapindex_is_file):
             if (not self.is_file_uri(sitemap_uri)):
                 # Attempt to map URI to local file
                 remote_uri = sitemap_uri
                 sitemap_uri = self.mapper.src_to_dst(remote_uri)
-                self.logger.info("Mapped %s to local file %s" % (remote_uri, sitemap_uri))
+                self.logger.info(
+                    "Mapped %s to local file %s" %
+                    (remote_uri, sitemap_uri))
             else:
                 # The individual sitemaps should be at a URL (scheme/server/path)
                 # that the sitemapindex URL can speak authoritatively about
-                if (self.check_url_authority and
-                    not UrlAuthority(sitemapindex_uri).has_authority_over(sitemap_uri)):
-                    raise ListBaseIndexError("The sitemapindex (%s) refers to sitemap at a location it does not have authority over (%s)" % (sitemapindex_uri,sitemap_uri))
+                if (self.check_url_authority
+                        and not UrlAuthority(sitemapindex_uri).has_authority_over(sitemap_uri)):
+                    raise ListBaseIndexError(
+                        "The sitemapindex (%s) refers to sitemap at a location it does not have authority over (%s)" %
+                        (sitemapindex_uri, sitemap_uri))
         try:
-            fh = URLopener().open(sitemap_uri)
+            fh = url_or_file_open(sitemap_uri)
             self.num_files += 1
         except IOError as e:
-            raise ListBaseIndexError("Failed to load sitemap from %s listed in sitemap index %s (%s)" % (sitemap_uri,sitemapindex_uri,str(e)))
+            raise ListBaseIndexError(
+                "Failed to load sitemap from %s listed in sitemap index %s (%s)" %
+                (sitemap_uri, sitemapindex_uri, str(e)))
         # Get the Content-Length if we can (works fine for local files)
         try:
             self.content_length = int(fh.info()['Content-Length'])
             self.bytes_read += self.content_length
-        except KeyError:
+        except (KeyError, TypeError):
             # If we don't get a length then c'est la vie
             pass
-        self.logger.info( "Reading sitemap from %s (%d bytes)" % (sitemap_uri,self.content_length) )
-        component = sitemap.parse_xml( fh=fh, sitemapindex=False )
+        self.logger.info(
+            "Reading sitemap from %s (%d bytes)" %
+            (sitemap_uri, self.content_length))
+        component = sitemap.parse_xml(fh=fh, sitemapindex=False)
         # Copy resources into self, check any metadata
         for r in component:
             self.resources.add(r)
         # FIXME - if rel="up" check it goes to correct place
         # FIXME - check capability
 
-    ##### OUTPUT #####
+    # OUTPUT
 
     def requires_multifile(self):
-        """Returns False or the number of component sitemaps required
-        
+        """Return False or the number of component sitemaps required.
+
         In the case that no len() is available for self.resources then
         then self.count must be set beforehand to avoid an exception.
         """
-        if (self.max_sitemap_entries is None or
-            len(self)<=self.max_sitemap_entries):
+        if (self.max_sitemap_entries is None
+                or len(self) <= self.max_sitemap_entries):
             return(False)
-        return( int( math.ceil( len(self) / float(self.max_sitemap_entries) ) ) )
+        return(int(math.ceil(len(self) / float(self.max_sitemap_entries))))
 
     def as_xml(self, allow_multifile=False, basename="/tmp/sitemap.xml"):
-        """Return XML serialization of this list
+        """Return XML serialization of this list.
 
-        If this list can be serialized as a single sitemap then the 
+        If this list can be serialized as a single sitemap then the
         superclass method is used.
 
-        There is no single XML serailization sense in the case that the 
-        number of list resources is more than is allowed in a single sitemap
-        so will raise an exception unless allow_multifile is set True.
-        If allow_multifile is set true then will return the sitemapindex
+        If there is no single XML serialization (in the case that the
+        number of list resources is more than is allowed in a single sitemap)
+        then raise an exception unless allow_multifile is set True.
+        If allow_multifile is set True then will return the sitemapindex
         for the set of component sitemaps.
         """
         if (not self.requires_multifile()):
@@ -186,97 +213,114 @@ class ListBaseWithIndex(ListBase):
         elif (allow_multifile):
             return self.as_xml_index(basename)
         else:
-            raise ListBaseIndexError("Attempt to write single XML string for list with %d entries when max_sitemap_entries is set to %d" % (len(self),self.max_sitemap_entries))
+            raise ListBaseIndexError(
+                "Attempt to write single XML string for list with %d entries when max_sitemap_entries is set to %d" %
+                (len(self), self.max_sitemap_entries))
 
     def as_xml_index(self, basename="/tmp/sitemap.xml"):
-        """Return a string of the index for a large list that is split
-        
+        """Return a string of the index for a large list that is split.
+
         All we need to do is determine the number of component sitemaps will
         be is and generate their URIs based on a pattern.
 
         Q - should there be a flag to select generation of each component sitemap
         in order to calculate the md5sum?
-        
+
         Q - what timestamp should be used?
         """
         num_parts = self.requires_multifile()
         if (not num_parts):
-            raise ListBaseIndexError("Request for sitemapindex for list with only %d entries when max_sitemap_entries is set to %s" % (len(self),str(self.max_sitemap_entries)))
-        index=ListBase()
-        index.sitemapindex=True
+            raise ListBaseIndexError(
+                "Request for sitemapindex for list with only %d entries when max_sitemap_entries is set to %s" %
+                (len(self), str(
+                    self.max_sitemap_entries)))
+        index = ListBase()
+        index.sitemapindex = True
         index.capability_name = self.capability_name
         index.default_capability()
         for n in range(num_parts):
-            r = Resource( uri = self.part_name(basename,n) )
+            r = Resource(uri=self.part_name(basename, n))
             index.add(r)
-        return( index.as_xml() )
+        return(index.as_xml())
 
     def as_xml_part(self, basename="/tmp/sitemap.xml", part_number=0):
-        """Return a string of component sitemap part_number for a large list that is split
-        
+        """Return a string of component sitemap number part_number.
+
+        Used in the case of a large list that is split into component
+        sitemaps.
+
         basename is used to create "index" links to the sitemapindex
-        
+
         Q - what timestamp should be used?
         """
         if (not self.requires_multifile()):
-            raise ListBaseIndexError("Request for component sitemap for list with only %d entries when max_sitemap_entries is set to %s" % (len(self),str(self.max_sitemap_entries)))
+            raise ListBaseIndexError(
+                "Request for component sitemap for list with only %d entries when max_sitemap_entries is set to %s" %
+                (len(self), str(
+                    self.max_sitemap_entries)))
         start = part_number * self.max_sitemap_entries
-        if (start>len(self)):
-            raise ListBaseIndexError("Request for component sitemap with part_number too high, would start at entry %d yet the list has only %d entries" % (start,len(self)))
+        if (start > len(self)):
+            raise ListBaseIndexError(
+                "Request for component sitemap with part_number too high, would start at entry %d yet the list has only %d entries" %
+                (start, len(self)))
         stop = start + self.max_sitemap_entries
-        if (stop>len(self)):
-            stop=len(self)
-        part = ListBase( itertools.islice(self.resources,start,stop) )
+        if (stop > len(self)):
+            stop = len(self)
+        part = ListBase(itertools.islice(self.resources, start, stop))
         part.capability_name = self.capability_name
         part.default_capability()
         part.index = basename
         s = self.new_sitemap()
-        return( s.resources_as_xml(part) )
+        return(s.resources_as_xml(part))
 
     def write(self, basename='/tmp/sitemap.xml'):
-        """Write one or a set of sitemap files to disk
+        """Write one or a set of sitemap files to disk.
 
         resources is a ResourceContainer that may be an ResourceList or
         a ChangeList. This may be a generator so data is read as needed
         and length is determined at the end.
 
-        basename is used as the name of the single sitemap file or the 
+        basename is used as the name of the single sitemap file or the
         sitemapindex for a set of sitemap files.
 
-        Uses self.max_sitemap_entries to determine whether the resource_list can 
-        be written as one sitemap. If there are more entries and 
-        self.allow_multifile is set true then a set of sitemap files, 
+        Uses self.max_sitemap_entries to determine whether the resource_list can
+        be written as one sitemap. If there are more entries and
+        self.allow_multifile is set True then a set of sitemap files,
         with an sitemapindex, will be written.
         """
         # Access resources through iterator only
         resources_iter = iter(self.resources)
-        ( chunk, next ) = self.get_resources_chunk(resources_iter)
+        (chunk, nxt) = self.get_resources_chunk(resources_iter)
         s = self.new_sitemap()
-        if (next is not None):
+        if (nxt is not None):
             # Have more than self.max_sitemap_entries => sitemapindex
             if (not self.allow_multifile):
-                raise ListBaseIndexError("Too many entries for a single sitemap but multifile disabled")
+                raise ListBaseIndexError(
+                    "Too many entries for a single sitemap but multifile disabled")
             # Work out URI of sitemapindex so that we can link up to
             # it from the individual sitemap files
             try:
                 index_uri = self.mapper.dst_to_src(basename)
             except MapperError as e:
-                raise ListBaseIndexError("Cannot map sitemapindex filename to URI (%s)" % str(e))
+                raise ListBaseIndexError(
+                    "Cannot map sitemapindex filename to URI (%s)" %
+                    str(e))
             # Use iterator over all resources and count off sets of
             # max_sitemap_entries to go into each sitemap, store the
             # names of the sitemaps as we go. Copy md from self into
             # the index and use this for all chunks also
-            index=ListBase(md=self.md.copy(), ln=list(self.ln))
+            index = ListBase(md=self.md.copy(), ln=list(self.ln))
             index.capability_name = self.capability_name
             index.default_capability()
-            while (len(chunk)>0):
-                file = self.part_name(basename,len(index))
+            while (len(chunk) > 0):
+                file = self.part_name(basename, len(index))
                 # Check that we can map the filename of this sitemap into
                 # URI space for the sitemapindex
                 try:
                     uri = self.mapper.dst_to_src(file)
                 except MapperError as e:
-                    raise ListBaseIndexError("Cannot map sitemap filename to URI (%s)" % str(e))
+                    raise ListBaseIndexError(
+                        "Cannot map sitemap filename to URI (%s)" % str(e))
                 self.logger.info("Writing sitemap %s..." % (file))
                 f = open(file, 'w')
                 chunk.index = index_uri
@@ -284,16 +328,22 @@ class ListBaseWithIndex(ListBase):
                 s.resources_as_xml(chunk, fh=f)
                 f.close()
                 # Record information about this sitemap for index
-                r = Resource( uri = uri,
-                              timestamp = os.stat(file).st_mtime,
-                              md5 = compute_md5_for_file(file) )
+                r = Resource(uri=uri,
+                             timestamp=os.stat(file).st_mtime,
+                             md5=Hashes(['md5'], file).md5)
                 index.add(r)
                 # Get next chunk
-                ( chunk, next ) = self.get_resources_chunk(resources_iter,next)
+                (chunk, nxt) = self.get_resources_chunk(resources_iter, nxt)
             self.logger.info("Wrote %d sitemaps" % (len(index)))
             f = open(basename, 'w')
             self.logger.info("Writing sitemapindex %s..." % (basename))
-            s.resources_as_xml(index,sitemapindex=True,fh=f)
+            s.resources_as_xml(index, sitemapindex=True, fh=f)
+            f.close()
+            self.logger.info("Wrote sitemapindex %s" % (basename))
+        elif self.sitemapindex:
+            f = open(basename, 'w')
+            self.logger.info("Writing sitemapindex %s..." % (basename))
+            s.resources_as_xml(chunk, sitemapindex=True, fh=f)
             f.close()
             self.logger.info("Wrote sitemapindex %s" % (basename))
         else:
@@ -304,64 +354,69 @@ class ListBaseWithIndex(ListBase):
             self.logger.info("Wrote sitemap %s" % (basename))
 
     def index_as_xml(self):
-        """Return XML serialization of this list taken to be sitemapindex entries
-
-        """
+        """XML serialization of this list taken to be sitemapindex entries."""
         self.default_capability()
         s = self.new_sitemap()
-        return s.resources_as_xml(self,sitemapindex=True)
+        return s.resources_as_xml(self, sitemapindex=True)
 
-    ##### Utility #####
+    # Utility
 
     def get_resources_chunk(self, resource_iter, first=None):
-        """Return next chunk of resources from resource_iter, and next item
-        
+        """Return next chunk of resources from resource_iter, and next item.
+
         If first parameter is specified then this will be prepended to
         the list.
 
-        The chunk will contain self.max_sitemap_entries if the iterator 
+        The chunk will contain self.max_sitemap_entries if the iterator
         returns that many. next will have the value of the next value from
-        the iterator, providing indication of whether more is available. 
+        the iterator, providing indication of whether more is available.
         Use this as first when asking for the following chunk.
         """
-        chunk = ListBase( md=self.md.copy(), ln=list(self.ln) )
+        chunk = ListBase(md=self.md.copy(), ln=list(self.ln))
         chunk.capability_name = self.capability_name
         chunk.default_capability()
         if (first is not None):
             chunk.add(first)
         for r in resource_iter:
             chunk.add(r)
-            if (len(chunk)>=self.max_sitemap_entries):
+            if (len(chunk) >= self.max_sitemap_entries):
                 break
         # Get next to see whether there are more resources
         try:
-            next = resource_iter.next()
+            nxt = next(resource_iter)
         except StopIteration:
-            next = None
-        return(chunk,next)
+            nxt = None
+        return(chunk, nxt)
 
     def part_name(self, basename='/tmp/sitemap.xml', part_number=0):
-        """Name (file or URI) for one component sitemap
-        
+        """Name (file or URI) for one component sitemap.
+
         Works for both filenames and URIs because manipulates only the end
         of the string.
-        
+
         Abstracting this into a function that starts from the basename to get
         prefix and suffix each time seems a bit wasteful but perhaps not worth
-        worrying about. Allows same code to be used for the write() and 
+        worrying about. Allows same code to be used for the write() and
         as_xml_index() cases.
         """
-        # Work out how to name the sitemaps, attempt to add %05d before ".xml$", else append
+        # Work out how to name the sitemaps, attempt to add %05d before
+        # ".xml$", else append
         sitemap_prefix = basename
         sitemap_suffix = '.xml'
         if (basename[-4:] == '.xml'):
             sitemap_prefix = basename[:-4]
-        return( sitemap_prefix + ( "%05d" % (part_number) ) + sitemap_suffix )
- 
+        return(sitemap_prefix + ("%05d" % (part_number)) + sitemap_suffix)
+
     def is_file_uri(self, uri):
-        """Return true if uri looks like a local file URI, false otherwise
-        
+        """Return true if uri looks like a local file URI, false otherwise.
+
         Test is to see whether have either an explicit file: URI or whether
         there is no scheme name.
         """
-        return(re.match('file:',uri) or not re.match('\w{3,4}:',uri))
+        return(re.match(r'file:', uri) or not re.match(r'\w{3,4}:', uri))
+
+
+class ListBaseIndexError(Exception):
+    """Exception for problems with sitemapindexes in ListBaseIndex."""
+
+    pass

resync/mapper.py

@@ -1,78 +1,95 @@
-"""Map between source URIs and destination paths"""
+"""Map between source URIs and destination paths."""
+
 import os
 import os.path
 import re
-import urlparse
+try:  # python3
+    from urllib.parse import urlparse
+except ImportError:  # pragma: no cover  python2
+    from urlparse import urlparse  # pragma: no cover
 import logging
 
-class MapperError(Exception):
-    pass
 
 class Mapper():
-    
+    """Mapper object to map between source URIs and destination paths.
+
+    Implemented as a list of Map objects.
+    """
+
     def __init__(self, mappings=None, use_default_path=False):
+        """Initialize Mapper."""
         self.logger = logging.getLogger('resync.mapper')
-        self.mappings=[]
+        self.mappings = []
         if (mappings):
             self.parse(mappings, use_default_path)
 
     def __len__(self):
-        """Length is number of mappings"""
+        """Length is number of mappings."""
         return(len(self.mappings))
 
     def parse(self, mappings, use_default_path=False):
-        """Parse a list of map strings (mappings)
+        """Parse a list of map strings (mappings).
 
         Accepts two distinct formats:
-        1. If there are exactly two entries then these may be the source base URI
-        and the destination base path. Neither may contain an equals (=) sign.
-        2. For any number of mapping stings interpret each as a mapping URI=path. 
-        These are in the order they will be tried.
-        
+        1. If there are exactly two entries then these may be the source
+        base URI and the destination base path. Neither may contain an
+        equals (=) sign.
+        2. For any number of mapping stings interpret each as a mapping
+        URI=path. These are in the order they will be tried.
+
         And if use_default_path is True then a third:
-        3. If there is exactly one entry and it does not contain an equals (=)
-        sign then a safe local path is created based on the source base URI. In
-        the case that the source base URI is a local path already then an indentity 
-        mapping is used.
+        3. If there is exactly one entry and it does not contain an
+        equals (=) sign then a safe local path is created based on the
+        source base URI. In the case that the source base URI is a local
+        path already then an indentity mapping is used.
         """
-        if (use_default_path and
-            len(mappings)==1 and
-            re.search(r"=",mappings[0])==None):
+        if (use_default_path
+                and len(mappings) == 1
+                and re.search(r"=", mappings[0]) is None):
             path = self.path_from_uri(mappings[0])
-            self.logger.warning("Using URI mapping: %s -> %s" % (mappings[0],path))
-            self.mappings.append(Map(mappings[0],path))
-        elif (len(mappings)==2 and 
-            re.search(r"=",mappings[0])==None and 
-            re.search(r"=",mappings[1])==None):
-            self.mappings.append(Map(mappings[0],mappings[1]))
+            self.logger.warning("Using URI mapping: %s -> %s" %
+                                (mappings[0], path))
+            self.mappings.append(Map(mappings[0], path))
+        elif (len(mappings) == 2
+                and re.search(r"=", mappings[0]) is None
+                and re.search(r"=", mappings[1]) is None):
+            self.mappings.append(Map(mappings[0], mappings[1]))
         else:
             for mapping in mappings:
-                l=mapping.split('=')
-                if (len(l)!=2):
-                    raise MapperError("Bad mapping argument (%s), got %s"%(mapping,str(l)))
-                (src_uri,dst_path)=l
+                entry = mapping.split('=')
+                if (len(entry) != 2):
+                    raise MapperError(
+                        "Bad mapping argument (%s), got %s" %
+                        (mapping, str(entry)))
+                (src_uri, dst_path) = entry
                 # Check for dupes
                 for map in self.mappings:
                     if (src_uri == map.src_uri):
-                        raise MapperError("Attempt to set duplicate mapping for source URI %s (with destination path %s)" % (src_uri,dst_path))
+                        raise MapperError(
+                            "Attempt to set duplicate mapping for "
+                            "source URI %s (with destination path %s)" %
+                            (src_uri, dst_path))
                     if (dst_path == map.dst_path):
-                        raise MapperError("Attempt to set duplicate mapping for destination path %s (with source URI %s)" % (dst_path,src_uri))
+                        raise MapperError(
+                            "Attempt to set duplicate mapping for "
+                            "destination path %s (with source URI %s)" %
+                            (dst_path, src_uri))
                 self.mappings.append(Map(src_uri, dst_path))
 
     def default_src_uri(self):
-        """Default src_uri from mapping
+        """Default src_uri from mapping.
 
         This is take just to be the src_uri of the first entry
         """
-        if (len(self.mappings)>0):
+        if (len(self.mappings) > 0):
             return(self.mappings[0].src_uri)
         raise MapperError("Can't get default source URI from empty mapping")
 
     def unsafe(self):
-        """True is one or more mapping is unsafe
+        """True if one or more mapping is unsafe.
 
-        See Map.unsafe() for logic. Provide this as a test rather than 
-        building into object creation/parse because it is useful to 
+        See Map.unsafe() for logic. Provide this as a test rather than
+        building into object creation/parse because it is useful to
         allow unsafe mappings in situations where it doesn't matter.
         """
         for map in self.mappings:
@@ -80,41 +97,46 @@ class Mapper():
                 return(True)
         return(False)
 
-    def dst_to_src(self,dst_file):
-        """Map destination path to source URI"""
+    def dst_to_src(self, dst_file):
+        """Map destination path to source URI."""
         for map in self.mappings:
             src_uri = map.dst_to_src(dst_file)
             if (src_uri is not None):
                 return(src_uri)
         # Must have failed if loop exited
-        raise MapperError("Unable to translate destination path (%s) into a source URI." % (dst_file))
+        raise MapperError(
+            "Unable to translate destination path (%s) "
+            "into a source URI." % (dst_file))
 
-    def src_to_dst(self,src_uri):
-        """Map source URI to destination path"""
+    def src_to_dst(self, src_uri):
+        """Map source URI to destination path."""
         for map in self.mappings:
             dst_path = map.src_to_dst(src_uri)
             if (dst_path is not None):
                 return(dst_path)
         # Must have failed if loop exited
-        raise MapperError("Unable to translate source URI (%s) into a destination path." % (src_uri))
+        raise MapperError(
+            "Unable to translate source URI (%s) into "
+            "a destination path." % (src_uri))
 
-    def path_from_uri(self,uri):
-        """Make a safe path name from uri
+    def path_from_uri(self, uri):
+        """Make a safe path name from uri.
 
-        In the case that uri is already a local path then the same path
-        is returned.
+        In the case that uri is already a local path then the
+        same path is returned.
         """
-        (scheme, netloc, path, params, query, fragment) = urlparse.urlparse( uri )
+        (scheme, netloc, path, params, query, frag) = urlparse(uri)
         if (netloc == ''):
             return(uri)
-        path = '/'.join([netloc,path])
-        path = re.sub('[^\w\-\.]', '_', path)
-        path = re.sub('__+', '_', path)
-        path = re.sub('[_\.]+$', '', path)
-        path = re.sub('^[_\.]+', '', path)
+        path = '/'.join([netloc, path])
+        path = re.sub(r'[^\w\-\.]', '_', path)
+        path = re.sub(r'__+', '_', path)
+        path = re.sub(r'[_\.]+$', '', path)
+        path = re.sub(r'^[_\.]+', '', path)
         return(path)
 
     def __repr__(self):
+        """Human readable dump of all Maps in the Mapper."""
         s = 'Mapper with %d maps:\n' % (len(self.mappings))
         for map in self.mappings:
             s += str(map) + '\n'
@@ -122,62 +144,85 @@ class Mapper():
 
 
 class Map:
-    """A single map from source URI to destination path
+    """A single map from source URI to destination path.
 
     Both URI and destination paths are assumed to use / as the path
     separator. No account is take for other path separators used
-    for paths on non-unix systems. This translation must be done 
+    for paths on non-unix systems. This translation must be done
     elsewhere by consideration of os.sep.
     """
 
-    def __init__(self,src_uri=None,dst_path=None):
-        self.src_uri  = self.strip_trailing_slashes(src_uri)
+    def __init__(self, src_uri=None, dst_path=None):
+        """Initialize Map object.
+
+        Will strip trailing slashes from src_uri and dst_path.
+        """
+        self.src_uri = self.strip_trailing_slashes(src_uri)
         self.dst_path = self.strip_trailing_slashes(dst_path)
 
-    def strip_trailing_slashes(self,path):
-        """Return input path minus any trailing slashes"""
-        m=re.match(r"(.*)/+$",path)
+    def strip_trailing_slashes(self, path):
+        """Return input path minus any trailing slashes."""
+        m = re.match(r"(.*)/+$", path)
         if (m is None):
             return(path)
         return(m.group(1))
 
-    def dst_to_src(self,dst_file):
-        """Return the src URI from the dst filepath
+    def dst_to_src(self, dst_file):
+        """Return the src URI from the dst filepath.
 
-        This does not rely on the destination filepath actually existing on the local
-        filesystem, just on pattern matching. Return source URI on success, None on 
-        failure.
+        This does not rely on the destination filepath actually
+        existing on the local filesystem, just on pattern matching.
+        Return source URI on success, None on failure.
+
+        Relies upon self.dst_path and self.src_path not including trailing
+        slashes. However, a match of just self.dst_path withouth a trailing
+        slash will return self.src_path with a trailing slash.
         """
-        m=re.match(self.dst_path+"/(.*)$",dst_file)
+        m = re.match(self.dst_path + "(/.*)?$", dst_file)
         if (m is None):
             return(None)
-        rel_path=m.group(1)
-        return(self.src_uri+'/'+rel_path)
-        
-    def src_to_dst(self,src_uri):
-        """Return the dst filepath from the src URI
-        
-        Returns None on failure, destination path on success.
+        rel_path = m.group(1)
+        if rel_path is None:
+            rel_path = '/'
+        return self.src_uri + rel_path
+
+    def src_to_dst(self, src_uri):
+        """Return the dst filepath from the src URI.
+
+        Returns None on failure, local destination path on success.
+
+        Relies upon self.dst_path and self.src_path not including trailing
+        slashes. However, a match of just self.src_path withouth a trailing
+        slash will return self.dst_path with a trailing slash.
         """
-        m=re.match(self.src_uri+"/(.*)$",src_uri)
+        m = re.match(self.src_uri + "(/.*)?$", src_uri)
         if (m is None):
             return(None)
-        rel_path=m.group(1)
-        return(self.dst_path+'/'+rel_path)
+        rel_path = m.group(1)
+        if rel_path is None:
+            rel_path = '/'
+        return self.dst_path + rel_path
 
     def unsafe(self):
-        """True is the mapping is unsafe for an update
+        """True if the mapping is unsafe for an update.
 
-        Applies only to local source. Returns True if the paths for source and 
+        Applies only to local source. Returns True if the paths for source and
         destination are the same, or if one is a component of the other path.
         """
-        (scheme, netloc, path, params, query, fragment) = urlparse.urlparse( self.src_uri )
+        (scheme, netloc, path, params, query, frag) = urlparse(self.src_uri)
         if (scheme != ''):
             return(False)
         s = os.path.normpath(self.src_uri)
         d = os.path.normpath(self.dst_path)
-        lcp = os.path.commonprefix([s,d])
+        lcp = os.path.commonprefix([s, d])
         return(s == lcp or d == lcp)
 
     def __repr__(self):
+        """Human readable string for one map."""
         return("Map( %s -> %s )" % (self.src_uri, self.dst_path))
+
+
+class MapperError(Exception):
+    """Exception for errors in Mapper class."""
+
+    pass

resync/resource.py

@@ -1,92 +1,80 @@
-"""Information about a web resource
-
-Each web resource is identified by a URI and may optionally 
-have other metadata such as timestamp, length, md5, etc..
-
-The lastmod property provides ISO8601 format string access 
-to the timestamp. The timestamp is stored internally as a 
-unix timestamp value in UTC. This limits the range of 
-possible lastmod values but covers all web-era values for
-a good way into the future.
-
-This object is optimized for size in the case whether there 
-is not large a large amount of data in the attributes. This 
-is done using __slots__ for the core attributes so that there
-is no __dict__ defined for a Resource object. Core attributes
-are:
-
-    uri - Resource URI
-    timestamp - Last-Modification time, has lastmod accessor
-    length - size in bytes
-    mime_type - MIME type 
-    md5, sha1, sha256 - digests, have hash accessor
-    change - change type
-    path - path in dump
-
-If non-core attributes are needed then the '_extra' attribute 
-has a dict of values. The ones explicitly used here are:
-
-    capability - Capability name
-    ts_at - at time, has md_at accessor 
-    ts_completed - completed time, has md_completed accessor 
-    ts_from - from time, has md_from accessor 
-    ts_until - until time, has md_until accessor 
-
-The accessor names mime_type, md_from etc. are used to avoid conflict 
-with Python built-in functions and keywords type(), from etc..
-
-The 'ln' attribute is used when it is necessary to add links 
-or other information to the object. Use of non-core attributes 
-or links results in dicts being created which is convenient 
-but will significantly increase the size of each Resource 
-object that contains such information.
-"""
+"""ResourceSync Resources - information about a web resource and changes."""
 
 import re
-from urlparse import urlparse
 from posixpath import basename
-from w3c_datetime import str_to_datetime, datetime_to_str
+from urllib.parse import urlparse
 
-
-class ChangeTypeError(Exception):
-    """Exception class raised by Resource for bad change attribute
-
-    The change attribute of a Resource object may be either None
-    or one of the values in Resource.CHANGE_TYPES. Checking is
-    disabled by setting Resource.CHANGE_TYPES False.
-    """
-    
-    def __init__(self, val):
-        self.supplied = val
-
-    def __repr__(self):
-        return "<ChangeTypeError: got %s, expected one of %s>" % \
-               (self.supplied,str(Resource.CHANGE_TYPES))
-
-    def __str__(self):
-        return repr(self)
+from .w3c_datetime import str_to_datetime, datetime_to_str
 
 
 class Resource(object):
-    __slots__=('uri', 'timestamp', 'length', 'mime_type', 
-               'md5', 'sha1', 'sha256', 'change', 'path',
-               '_extra', 'ln' )
+    """ResourceSync Resource object.
+
+    Each web resource is identified by a URI and may optionally
+    have other metadata such as timestamp, length, md5, etc..
+
+    The lastmod property provides ISO8601 format string access
+    to the timestamp. The timestamp is stored internally as a
+    unix timestamp value in UTC. This limits the range of
+    possible lastmod values but covers all web-era values for
+    a good way into the future.
+
+    This object is optimized for size in the case whether there
+    is not large a large amount of data in the attributes. This
+    is done using __slots__ for the core attributes so that there
+    is no __dict__ defined for a Resource object. Core attributes
+    are:
+
+        uri - Resource URI
+        timestamp - Last-Modification time, has lastmod accessor
+        length - size in bytes
+        mime_type - MIME type
+        md5, sha1, sha256 - digests, have hash accessor
+        change - change type
+        ts_datetime - time of the change, has datetime accessor
+        path - path in dump
+
+    If non-core attributes are needed then the '_extra' attribute
+    has a dict of values. The ones explicitly used here are:
+
+        capability - Capability name
+        ts_at - at time, has md_at accessor
+        ts_completed - completed time, has md_completed accessor
+        ts_from - from time, has md_from accessor
+        ts_until - until time, has md_until accessor
+
+    The accessor names mime_type, md_from etc. are used to avoid conflict
+    with Python built-in functions and keywords type(), from etc...
+
+    The 'ln' attribute is used when it is necessary to add links
+    or other information to the object. Use of non-core attributes
+    or links results in dicts being created which is convenient
+    but will significantly increase the size of each Resource
+    object that contains such information.
+    """
+
+    __slots__ = ('uri', 'timestamp', 'length', 'mime_type',
+                 'md5', 'sha1', 'sha256', 'change', 'ts_datetime',
+                 'path', 'ln', '_extra')
 
     CHANGE_TYPES = ['created', 'updated', 'deleted']
-    
-    def __init__(self, uri = None, timestamp = None, length = None, 
-                 md5 = None, sha1 = None, sha256 = None, mime_type = None,
-                 change = None, path = None, lastmod = None, 
-                 capability = None,
-                 ts_at = None, md_at = None,
-                 ts_completed = None, md_completed = None,
-                 ts_from = None, md_from = None,
-                 ts_until = None, md_until = None,
-                 resource = None, ln = None):
-        """Initialize object either from parameters specified or
-        from an existing Resource object. If explicit parameters
-        are specified then they will override values copied from
-        a resource object supplied.
+
+    def __init__(self, uri=None, timestamp=None, length=None,
+                 mime_type=None, md5=None, sha1=None, sha256=None,
+                 change=None, ts_datetime=None, path=None, ln=None,
+                 # the following in _extra
+                 capability=None,
+                 ts_at=None, ts_completed=None, ts_from=None, ts_until=None,
+                 # and the following via setters
+                 lastmod=None, datetime=None,
+                 md_at=None, md_completed=None, md_from=None, md_until=None,
+                 # and finally another Resource to copy from
+                 resource=None):
+        """Initialize Resource object.
+
+        Initialize either from parameters specified or from an existing
+        Resource object. If explicit parameters are specified then they
+        will override values copied from a Resource object supplied.
         """
         # Initialize core attributes
         self.uri = None
@@ -97,16 +85,18 @@ class Resource(object):
         self.sha1 = None
         self.sha256 = None
         self.change = None
+        self.ts_datetime = None  # Added in ResourceSync v1.1
         self.path = None
-        self._extra = None
         self.ln = None
-        # Create from a Resource-like object? Copy any relevant attributes
+        self._extra = None
+        # Create from a Resource-like object? Copy any attributes, both ones
+        # that have slots and ones that live in _extra
         if (resource is not None):
-            for att in ['uri','timestamp','length','md5','sha1','sha256',
-                        'change','path', 'capability',
-                        'ts_at','md_at', 'ts_completed', 'md_completed',
-                        'ts_from','md_from','ts_until','md_until','ln']:
-                if hasattr(resource,att):
+            for att in ['uri', 'timestamp', 'length', 'mime_type',
+                        'md5', 'sha1', 'sha256', 'change', 'ts_datetime', 'path', 'ln',
+                        # the following in _extra
+                        'capability', 'ts_at', 'ts_completed', 'ts_from', 'ts_until']:
+                if hasattr(resource, att):
                     setattr(self, att, getattr(resource, att))
         # Any arguments will then override
         if (uri is not None):
@@ -125,6 +115,8 @@ class Resource(object):
             self.mime_type = mime_type
         if (change is not None):
             self.change = change
+        if (ts_datetime is not None):
+            self.ts_datetime = ts_datetime
         if (path is not None):
             self.path = path
         if (ts_at is not None):
@@ -141,23 +133,31 @@ class Resource(object):
             self.ln = ln
         # Timestamp setters
         if (lastmod is not None):
-            self.lastmod=lastmod
+            self.lastmod = lastmod
+        if (datetime is not None):
+            self.datetime = datetime
         if (md_at is not None):
-            self.md_at=md_at
+            self.md_at = md_at
         if (md_completed is not None):
-            self.md_completed=md_completed
+            self.md_completed = md_completed
         if (md_from is not None):
-            self.md_from=md_from
+            self.md_from = md_from
         if (md_until is not None):
-            self.md_until=md_until
+            self.md_until = md_until
         # Sanity check
         if (self.uri is None):
-            raise ValueError("Cannot create resource without a URI")
+            raise ValueError("Cannot create Resource without a URI")
 
     def __setattr__(self, prop, value):
+        """Attribute setter with check and support for extra attributes.
+
+        Because this class is optimized to use __slots__ it cannot
+        ordinarily be dynamically extended. We implement extension with
+        the idea of extra properties.
+        """
         # Add validity check for self.change
-        if (prop == 'change' and Resource.CHANGE_TYPES and
-            value is not None and not value in Resource.CHANGE_TYPES):
+        if (prop == 'change' and Resource.CHANGE_TYPES
+                and value is not None and value not in Resource.CHANGE_TYPES):
             raise ChangeTypeError(value)
         else:
             try:
@@ -167,12 +167,12 @@ class Resource(object):
                 self._set_extra(prop, value)
 
     def _set_extra(self, prop, value):
-        # Use self._extra dict to hold non-core attributes to 
+        # Use self._extra dict to hold non-core attributes to
         # save space
         if (self._extra is None):
             self._extra = dict()
         self._extra[prop] = value
-    
+
     def _get_extra(self, prop):
         # Returns None for a property not set
         if (self._extra is None):
@@ -181,85 +181,126 @@ class Resource(object):
 
     @property
     def lastmod(self):
-        """The Last-Modified data in W3C Datetime syntax, Z notation"""
+        """The Last-Modified date for the resource in W3C Datetime syntax, Z notation."""
         return datetime_to_str(self.timestamp)
 
     @lastmod.setter
     def lastmod(self, lastmod):
-        """Set timestamp from a W3C Datetime Last-Modified value"""
+        """Set last modified timestamp from a W3C Datetime Last-Modified value."""
         self.timestamp = str_to_datetime(lastmod, context='lastmod')
 
+    @property
+    def datetime(self):
+        """The datetime of the resource change in W3C Datetime syntax, Z notation."""
+        return datetime_to_str(self.ts_datetime)
+
+    @datetime.setter
+    def datetime(self, datetime):
+        """Set timestamp from a W3C Datetime Last-Modified value."""
+        self.ts_datetime = str_to_datetime(datetime, context='ts_datetime')
+
+    @property
+    def ts_at(self):
+        """ts_at is the timestamp for md_at."""
+        return self._get_extra('ts_at')
+
     @property
     def md_at(self):
-        """md_at values in W3C Datetime syntax, Z notation"""
+        """md_at values in W3C Datetime syntax, Z notation."""
         return datetime_to_str(self._get_extra('ts_at'))
 
     @md_at.setter
     def md_at(self, md_at):
-        """Set at value from a W3C Datetime value"""
-        self._set_extra( 'ts_at', str_to_datetime(md_at, context='md_at datetime') )
+        """Set at value from a W3C Datetime value."""
+        self._set_extra(
+            'ts_at',
+            str_to_datetime(md_at, context='md_at datetime'))
+
+    @property
+    def ts_completed(self):
+        """ts_completed is the timestamp for md_completed."""
+        return self._get_extra('ts_completed')
 
     @property
     def md_completed(self):
-        """md_completed value in W3C Datetime syntax, Z notation"""
+        """md_completed value in W3C Datetime syntax, Z notation."""
         return datetime_to_str(self._get_extra('ts_completed'))
 
     @md_completed.setter
     def md_completed(self, md_completed):
-        """Set md_completed value from a W3C Datetime value"""
-        self._set_extra( 'ts_completed', str_to_datetime(md_completed, context='md_completed datetime') )
+        """Set md_completed value from a W3C Datetime value."""
+        self._set_extra(
+            'ts_completed',
+            str_to_datetime(md_completed, context='md_completed datetime'))
+
+    @property
+    def ts_from(self):
+        """ts_from is the timestamp for md_from."""
+        return self._get_extra('ts_from')
 
     @property
     def md_from(self):
-        """md_from value in W3C Datetime syntax, Z notation"""
+        """md_from value in W3C Datetime syntax, Z notation."""
         return datetime_to_str(self._get_extra('ts_from'))
 
     @md_from.setter
     def md_from(self, md_from):
-        """Set md_from value from a W3C Datetime value"""
-        self._set_extra( 'ts_from', str_to_datetime(md_from, context='md_from datetime') )
+        """Set md_from value from a W3C Datetime value."""
+        self._set_extra(
+            'ts_from',
+            str_to_datetime(md_from, context='md_from datetime'))
+
+    @property
+    def ts_until(self):
+        """ts_until is the timestamp for md_until."""
+        return self._get_extra('ts_until')
 
     @property
     def md_until(self):
-        """md_until value in W3C Datetime syntax, Z notation"""
+        """md_until value in W3C Datetime syntax, Z notation."""
         return datetime_to_str(self._get_extra('ts_until'))
 
     @md_until.setter
     def md_until(self, md_until):
-        """Set md_until value from a W3C Datetime value"""
-        self._set_extra( 'ts_until', str_to_datetime(md_until, context='md_until datetime') )
+        """Set md_until value from a W3C Datetime value."""
+        self._set_extra(
+            'ts_until',
+            str_to_datetime(md_until, context='md_until datetime'))
 
     @property
     def capability(self):
-        """Get Capability name string"""
+        """Get Capability name string."""
         return self._get_extra('capability')
 
     @capability.setter
     def capability(self, capability):
-        """Set Capability name string"""
+        """Set Capability name string."""
         self._set_extra('capability', capability)
 
     @property
     def hash(self):
-        """Provide access to the complete hash string
+        """Provide access to the complete hash string.
 
         The hash string may have zero or more hash values with
-        appropriate prefixes
+        appropriate prefixes. All hash values are assumed to be
+        strings
         """
         hashvals = []
         if (self.md5 is not None):
-            hashvals.append('md5:'+self.md5)
+            hashvals.append('md5:' + self.md5)
         if (self.sha1 is not None):
-            hashvals.append('sha-1:'+self.sha1)
+            hashvals.append('sha-1:' + self.sha1)
         if (self.sha256 is not None):
-            hashvals.append('sha-256:'+self.sha256)
-        if (len(hashvals)>0):
+            hashvals.append('sha-256:' + self.sha256)
+        if (len(hashvals) > 0):
             return(' '.join(hashvals))
         return(None)
 
     @hash.setter
-    def hash(self, hash):
-        """Parse space separated set of values
+    def hash(self, hashes_str):
+        """Parse space separated set of values to set md5, sha1 and sha256.
+
+        Any existing values for md5, sha1 and sha256 will be removed first.
 
         See specification at:
         http://tools.ietf.org/html/draft-snell-atompub-link-extensions-09
@@ -268,188 +309,220 @@ class Resource(object):
         self.md5 = None
         self.sha1 = None
         self.sha256 = None
-        if (hash is None):
-            return
         hash_seen = set()
         errors = []
-        for entry in hash.split():
-            ( hash_type, value ) = entry.split(':',1)
-            if (hash_type in hash_seen):
-                errors.append("Ignored duplicate hash type %s" % (hash_type))
+        for entry in hashes_str.split():
+            (hash_type, value) = entry.split(':', 1)
+            if hash_type in hash_seen:
+                errors.append("duplicate hash type %s" % (hash_type))
             else:
                 hash_seen.add(hash_type)
-                if (hash_type == 'md5'):
-                    self.md5=value
-                elif (hash_type == 'sha-1'):
-                    self.sha1=value
-                elif (hash_type == 'sha-256'):
-                    self.sha256=value
+                if hash_type == 'md5':
+                    self.md5 = value
+                elif hash_type == 'sha-1':
+                    self.sha1 = value
+                elif hash_type == 'sha-256':
+                    self.sha256 = value
                 else:
-                    errors.append("Ignored unsupported hash type (%s)" % (hash_type))
-        if (len(errors)>0):
-            raise ValueError(". ".join(errors))
+                    errors.append("unsupported hash type %s" % (hash_type))
+        if len(errors) > 0:
+            raise ValueError("Ignored " + ", ".join(errors))
 
-    def link(self,rel):
-        """Look for link with specified rel, return else None
+    def link(self, rel):
+        """Look for link with specified rel, return else None.
 
         Searches through dicts in self.ln looking for one with the
-        specified rel value. If there are multiple links with the 
+        specified rel value. If there are multiple links with the
         same rel then just the first will be returned
         """
-        if (self.ln is None):
-            return(None)
-        for link in self.ln:
-            if ('rel' in link and
-                link['rel']==rel):
-                return(link)
-        return(None)
+        if self.ln is not None:
+            for link in self.ln:
+                if 'rel' in link and link['rel'] == rel:
+                    return link
+        return None
 
-    def link_href(self,rel):
-        """Look for link with specified rel, return href from it or None"""
+    def link_href(self, rel):
+        """Look for link with specified rel, return href from it or None."""
         link = self.link(rel)
-        if (link is not None):
-            link = link['href']
-        return(link)
+        return None if link is None else link['href']
 
-    def link_set(self,rel,href,allow_duplicates=False,**atts):
-        """Set/create link with specified rel, set href and any other attributes
+    def link_set(self, rel, href, allow_duplicates=False, **atts):
+        """Set/create link with specified rel, set href and any other attributes.
 
         Any link element must have both rel and href values, the specification
-        also defines the type attributes and others are permitted also. See 
+        also defines the type attributes and others are permitted also. See
         description of allowed formats in
 
         http://www.openarchives.org/rs/resourcesync.html#DocumentFormats
 
-        Be aware that adding links to a Resource object will 
+        Be aware that adding links to a Resource object will
         significantly increase the size of the object.
         """
-        if (self.ln is None):
+        if self.ln is None:
             # automagically create a self.ln list
             self.ln = []
             link = None
         else:
             link = self.link(rel)
-        if (link is not None and not allow_duplicates):
+        if link is not None and not allow_duplicates:
             # overwrite current value
             link['href'] = href
         else:
             # create new link
-            link = {'rel':rel,'href':href}
+            link = {'rel': rel, 'href': href}
             self.ln.append(link)
         for k in atts:
             link[k] = atts[k]
 
-    def link_add(self,rel,href,**atts):
-        """Create an link with specified rel even if one with that rel already exists"""
-        self.link_set(rel,href,allow_duplicates=True,**atts)
+    def link_add(self, rel, href, **atts):
+        """Create an link with specified rel.
+
+        Will add a link even if one with that rel already exists.
+        """
+        self.link_set(rel, href, allow_duplicates=True, **atts)
 
     @property
     def describedby(self):
-        """Convenient access to <rs:ln rel="describedby" href="uri">"""
+        """Convenient access to <rs:ln rel="describedby" href="uri">."""
         return(self.link_href('describedby'))
 
     @describedby.setter
-    def describedby(self,uri):
-        """Set ResourceSync Description link to given URI"""
-        self.link_set('describedby',uri)
+    def describedby(self, uri):
+        """Set ResourceSync Description link to given URI."""
+        self.link_set('describedby', uri)
 
     @property
     def up(self):
-        """Get the URI of any ResourceSync rel="up" link"""
+        """Get the URI of any ResourceSync rel="up" link."""
         return(self.link_href('up'))
 
     @up.setter
-    def up(self,uri):
-        """Set rel="up" link to given URI"""
-        self.link_set('up',uri)
+    def up(self, uri):
+        """Set rel="up" link to given URI."""
+        self.link_set('up', uri)
 
     @property
     def index(self):
-        """Get the URI of and ResourceSync rel="index" link"""
+        """Get the URI of and ResourceSync rel="index" link."""
         return(self.link_href('index'))
 
     @index.setter
-    def index(self,uri):
-        """Set rel="index" link to given URI"""
-        self.link_set('index',uri)
+    def index(self, uri):
+        """Set rel="index" link to given URI."""
+        self.link_set('index', uri)
 
     @property
     def contents(self):
-        """Get the URI of and ResourceSync rel="contents" link"""
-        return(self.link_href('index'))
+        """Get the URI of and ResourceSync rel="contents" link."""
+        return(self.link_href('contents'))
 
     @contents.setter
-    def contents(self,uri,type='application/xml'):
-        """Set rel="contents" link to given URI
+    def contents(self, uri, type='application/xml'):
+        """Set rel="contents" link to given URI.
 
         Will also set the type="application/xml" unless overridden
         """
-        self.link_set('contents',uri,type=type)
+        self.link_set('contents', uri, type=type)
 
     @property
     def basename(self):
-        """The resource basename (http://example.com/resource/1 -> 1)"""
-        parse_object = urlparse(self.uri)
-        return basename(parse_object.path)
+        """The resource basename.
 
-    def __eq__(self,other):
-        """Equality test for resources allowing <1s difference in timestamp
-        
-        See equal(...) for more details of equality test
+        For example from http://example.com/resource/1 returns 1
         """
-        return( self.equal(other,delta=1.0) )
+        parse_object = urlparse(self.uri)
+        return(basename(parse_object.path))
 
-    def equal(self,other,delta=0.0):
-        """Equality or near equality test for resources
-        
-        Equality means:
+    def __eq__(self, other):
+        """Equality test for resources allowing <1s difference in timestamp.
+
+        See equal(...) for more details of equality test.
+        """
+        return(self.equal(other, delta=1.0))
+
+    def equal(self, other, delta=0.0):
+        """Equality or near equality test for resources.
+
+        Equality means that any parameters that exist for both resources match,
+        essentially "no proof of inequality":
         1. same uri, AND
         2. same timestamp WITHIN delta if specified for either, AND
-        3. same md5 if specified for both, AND
+        3. same md5|sha1|sha256 if specified for both, AND
         4. same length if specified for both
         """
-        if (other is None): return False
-        
-        if (self.uri != other.uri):
-            return(False)
-        if ( self.timestamp is not None or other.timestamp is not None ):
+        if other is None:
+            return False
+        if self.uri != other.uri:
+            return False
+        if (self.timestamp is not None or other.timestamp is not None):
             # not equal if only one timestamp specified
-            if ( self.timestamp is None or 
-                 other.timestamp is None or
-                 abs(self.timestamp-other.timestamp)>=delta ):
-                return(False)
-        if ( ( self.md5 is not None and other.md5 is not None ) and
-             self.md5 != other.md5 ):
-            return(False)
-        if ( ( self.length is not None and other.length is not None ) and
-             self.length != other.length ):
-            return(False)
-        return(True)
-    
+            if (self.timestamp is None
+                    or other.timestamp is None
+                    or abs(self.timestamp - other.timestamp) >= delta):
+                return False
+        if ((self.md5 is not None and other.md5 is not None)
+                and self.md5 != other.md5):
+            return False
+        if ((self.sha1 is not None and other.sha1 is not None)
+                and self.sha1 != other.sha1):
+            return False
+        if ((self.sha256 is not None and other.sha256 is not None)
+                and self.sha256 != other.sha256):
+            return False
+        if ((self.length is not None and other.length is not None)
+                and self.length != other.length):
+            return False
+        return True
+
     def __str__(self):
-        """Return a human readable string for this resource
+        """Return a human readable string for this resource.
 
         Includes only the parts necessary for synchronizaion and
         designed to support logging.
         """
-        s = [ str(self.uri), str(self.lastmod), str(self.length),
-              str(self.md5 if self.md5 else self.sha1) ]
+        s = [str(self.uri), str(self.lastmod), str(self.length),
+             str(self.md5 if self.md5 else (self.sha1 if self.sha1 else self.sha256))]
         if (self.change is not None):
-            s.append(str(self.change))
+            ch = str(self.change)
+            if self.datetime is not None:
+                ch += " @ " + str(self.datetime)
+            s.append(ch)
         if (self.path is not None):
             s.append(str(self.path))
         return "[ " + " | ".join(s) + " ]"
-                                         
-    def __repr__(self):
-        """Return an unambigous representation of this resource
 
-        Uses format like Python's representation of a dict() for 
-        attributes. Includes only those attributes with values that 
+    def __repr__(self):
+        """Return an unambigous representation of this resource.
+
+        Uses format like Python's representation of a dict() for
+        attributes. Includes only those attributes with values that
         are not None. Order defined by __slots__.
         """
         s = []
         for attr in self.__slots__:
             val = getattr(self, attr, None)
             if (val is not None):
-                s.append( repr(attr) + ': ' + repr(val) )
+                s.append(repr(attr) + ': ' + repr(val))
         return "{" + ", ".join(s) + "}"
+
+
+class ChangeTypeError(Exception):
+    """Exception class raised by Resource for bad change attribute.
+
+    The change attribute of a Resource object may be either None
+    or one of the values in Resource.CHANGE_TYPES. Checking is
+    disabled by setting Resource.CHANGE_TYPES False.
+    """
+
+    def __init__(self, val):
+        """Initialize ChangeTypeError, store val."""
+        self.supplied = val
+
+    def __repr__(self):
+        """Helpful error message."""
+        return "<ChangeTypeError: got %s, expected one of %s>" % \
+               (self.supplied, str(Resource.CHANGE_TYPES))
+
+    def __str__(self):
+        """Helpful error message."""
+        return repr(self)

resync/resource_container.py

@@ -1,8 +1,8 @@
-"""ResourceSync Resource Container object
+"""ResourceSync Resource Container object.
 
 All documents in ResourceSync have the same types of information:
 they have some top-level metadata and links, and then a list of
-resources, each of which may also have metadata and links. This 
+resources, each of which may also have metadata and links. This
 class implements this model.
 
 This is a base class for the ListBase class which is in turn the
@@ -10,11 +10,28 @@ base class for ResourceList, ChangeList, etc. This class provides
 only the data storage and manipulation, the ListBase class
 adds IO.
 """
-import collections
-from w3c_datetime import datetime_to_str
+import collections.abc
+from .w3c_datetime import datetime_to_str
+
+
+def _str_datetime_now(x='now'):
+    """Return datetime string for use with time attributes.
+
+    Handling depends on input:
+      'now'   - returns datetime for now
+      other string - no change, return same value
+      otherwise - assume datetime value, generate string
+    """
+    if (x == 'now' or not isinstance(x, str)):
+        # Now, this is wht datetime_to_str() with no arg gives
+        return datetime_to_str(x)
+    else:
+        # Assume valid datetime string
+        return x
+
 
 class ResourceContainer(object):
-    """Class containing resource-like objects
+    """Class containing resource-like objects.
 
     Core functionality::
     - resources property that is the set/list of resources
@@ -25,203 +42,176 @@ class ResourceContainer(object):
     - uri is optional identifier of this container object
     - capability_name - name of this capability
 
+    The properties md_from, md_at, md_until and md_completed are datetime
+    strings. The setters acceopt either a string value (not checked), the value
+    'now' to generate the current datetime, or otherwise an integer timestamp
+    value that is converted to a string.
+
     Derived classes may add extra functionality such as len() etc..
     However, any code designed to work with any ResourceContainer
     should use only the core functionality.
     """
 
-    def __init__(self, resources=None, md=None, ln=None, uri=None, capability_name=None):
-        self.resources=(resources if (resources is not None) else [])
-        self.md=(md if (md is not None) else {})
-        self.ln=(ln if (ln is not None) else [])
-        self.uri=uri
-        self.capability_name=capability_name
+    def __init__(self, resources=None, md=None, ln=None,
+                 uri=None, capability_name=None):
+        """Initialize ResourceContainer."""
+        self.resources = (resources if (resources is not None) else [])
+        self.md = (md if (md is not None) else {})
+        self.ln = (ln if (ln is not None) else [])
+        self.uri = uri
+        self.capability_name = capability_name
 
     def __iter__(self):
-        """Iterator over all the resources in this resource list
+        """Iterator over all the resources in this resource list.
 
-        Baseline implementation use iterator given by resources property
+        Baseline implementation use iterator given by resources property.
         """
-        return(iter(self.resources))
+        return iter(self.resources)
 
-    def __getitem__(self,index):
-        """Feed through for __getitem__ of resources property"""
-        return(self.resources[index])
+    def __getitem__(self, index):
+        """Feed through for __getitem__ of resources property."""
+        return self.resources[index]
 
     @property
     def capability(self):
-        """Get/set the <rs:md capability="" .../> attribute"""
-        if ('capability' in self.md):
-            return(self.md['capability'])
-        else:
-            return(None)
+        """Get/set the <rs:md capability="" .../> attribute, None if not set."""
+        return self.md.get('capability')
 
     @capability.setter
-    def capability(self,capability):
-        self.md['capability']=capability
+    def capability(self, capability):
+        self.md['capability'] = capability
 
     @property
     def md_from(self):
-        """Get/set the <rs:md from="" .../> attribute"""
-        if ('md_from' in self.md):
-            return(self.md['md_from'])
-        else:
-            return(None)
+        """Get/set the <rs:md from="" .../> attribute, None if not set."""
+        return self.md.get('md_from')
 
     @md_from.setter
-    def md_from(self,md_from):
-        self.md['md_from']=self._str_datetime_now(md_from)
+    def md_from(self, md_from):
+        self.md['md_from'] = _str_datetime_now(md_from)
 
     @property
     def md_until(self):
-        """Get/set the <rs:md until="" .../> attribute"""
-        if ('md_until' in self.md):
-            return(self.md['md_until'])
-        else:
-            return(None)
+        """Get/set the <rs:md until="" .../> attribute, None if not set."""
+        return self.md.get('md_until')
 
     @md_until.setter
-    def md_until(self,md_until):
-        self.md['md_until']=self._str_datetime_now(md_until)
+    def md_until(self, md_until):
+        self.md['md_until'] = _str_datetime_now(md_until)
 
     @property
     def md_at(self):
-        """Get/set the <rs:md at="" attribute"""
-        if ('md_at' in self.md):
-            return(self.md['md_at'])
-        else:
-            return(None)
+        """Get/set the <rs:md at="" attribute."""
+        return self.md.get('md_at')
 
     @md_at.setter
-    def md_at(self,md_at):
-        self.md['md_at']=self._str_datetime_now(md_at)
+    def md_at(self, md_at):
+        self.md['md_at'] = _str_datetime_now(md_at)
 
     @property
     def md_completed(self):
-        """Get/set the <rs:md completed="" .../> attribute"""
-        if ('md_completed' in self.md):
-            return(self.md['md_completed'])
-        else:
-            return(None)
+        """Get/set the <rs:md completed="" .../> attribute, None if not set."""
+        return self.md.get('md_completed')
 
     @md_completed.setter
-    def md_completed(self,md_completed):
-        self.md['md_completed']=self._str_datetime_now(md_completed)
+    def md_completed(self, md_completed):
+        self.md['md_completed'] = _str_datetime_now(md_completed)
 
-    def link(self,rel):
-        """Look for link with specified rel, return else None"""
+    def link(self, rel):
+        """Look for link with specified rel, return else None."""
         for link in self.ln:
-            if ('rel' in link and
-                link['rel']==rel):
-                return(link)
-        return(None)
+            if ('rel' in link and link['rel'] == rel):
+                return link
+        return None
 
-    def link_href(self,rel):
-        """Look for link with specified rel, return href from it or None"""
+    def link_href(self, rel):
+        """Look for link with specified rel, return href from it or None."""
         link = self.link(rel)
         if (link is not None):
             link = link['href']
-        return(link)
+        return link
 
-    def link_set(self,rel,href,**atts):
-        """Set/create link with specified rel, set href and any other attributes
+    def link_set(self, rel, href, **atts):
+        """Set/create link with specified rel, set href and any other attributes.
 
         Any link element must have both rel and href values, the specification
-        also defines the type attributes and others are permitted also. See 
+        also defines the type attributes and others are permitted also. See
         description of allowed formats in
 
-        http://www.openarchives.org/rs/resourcesync.html#DocumentFormats
+        http://www.openarchives.org/rs/resourcesync#DocumentFormats
         """
         link = self.link(rel)
-        if (link is not None):
+        if link is None:
+            # create new link
+            link = {'rel': rel, 'href': href}
+            self.ln.append(link)
+        else:
             # overwrite current value
             link['href'] = href
-        else:
-            # create new link
-            link = {'rel':rel,'href':href}
-            self.ln.append(link)
         for k in atts:
             link[k] = atts[k]
 
     @property
     def describedby(self):
-        """Convenient access to <rs:ln rel="describedby" href="uri">"""
+        """Convenient access to <rs:ln rel="describedby" href="uri">."""
         return(self.link_href('describedby'))
 
     @describedby.setter
-    def describedby(self,uri):
-        """Set ResourceSync Description link to given URI"""
-        self.link_set('describedby',uri)
+    def describedby(self, uri):
+        """Set ResourceSync Description link to given URI."""
+        self.link_set('describedby', uri)
 
     @property
     def up(self):
-        """Get the URI of any ResourceSync rel="up" link"""
+        """Get the URI of any ResourceSync rel="up" link."""
         return(self.link_href('up'))
 
     @up.setter
-    def up(self,uri):
-        """Set ResourceSync rel="up" link to given URI"""
-        self.link_set('up',uri)
+    def up(self, uri):
+        """Set ResourceSync rel="up" link to given URI."""
+        self.link_set('up', uri)
 
     @property
     def index(self):
-        """Get the URI of and ResourceSync rel="index" link"""
+        """Get the URI of and ResourceSync rel="index" link."""
         return(self.link_href('index'))
 
     @index.setter
-    def index(self,uri):
-        """Set index link to given URI"""
-        self.link_set('index',uri)
+    def index(self, uri):
+        """Set index link to given URI."""
+        self.link_set('index', uri)
 
     def default_capability(self):
-        """Set capability name in md
+        """Set capability name in md.
 
         Every ResourceSync document should have the top-level
         capability attributes.
         """
         if ('capability' not in self.md and self.capability_name is not None):
-            self.md['capability']=self.capability_name
+            self.md['capability'] = self.capability_name
 
     def add(self, resource):
-        """Add a resource or an iterable collection of resources to this container
+        """Add a resource or an iterable collection of resources to this container.
 
-        Must be implemented in derived class
+        Must be implemented in derived class.
         """
-        if isinstance(resource, collections.Iterable):
+        if isinstance(resource, collections.abc.Iterable):
             for r in resource:
                 self.resources.append(r)
         else:
             self.resources.append(resource)
 
     def uris(self):
-        """Return list of all URIs, possibly including dupes"""
+        """Return list of all URIs, possibly including dupes."""
         uris = []
         for r in self.resources:
             uris.append(r.uri)
         return(uris)
 
-    def prune_before(self, timestamp):
-        """Remove all resources with timestamp earlier than that given
-        
-        Returns the number of entries removed. Will raise an excpetion
-        if there are any entries without a timestamp.
-        """
-        n = 0
-        pruned = []
-        for r in self.resources:
-            if (r.timestamp is None):
-                raise Exception("Entry %s has no timestamp" % (r.uri))
-            elif (r.timestamp >= timestamp):
-                pruned.append(r)
-            else:
-                n += 1
-        self.resources = pruned
-        return(n)
-
     def prune_dupes(self):
-        """Remove all but the last entry for a given resource URI
-        
-        Returns the number of entries removed. Also removes all entries for a 
+        """Remove all but the last entry for a given resource URI.
+
+        Returns the number of entries removed. Also removes all entries for a
         given URI where the first entry is a create and the last entry is a
         delete.
         """
@@ -233,12 +223,12 @@ class ResourceContainer(object):
             if (r.uri in seen):
                 n += 1
                 if (r.uri in deletes):
-                    deletes[r.uri]=r.change
+                    deletes[r.uri] = r.change
             else:
                 pruned1.append(r)
                 seen.add(r.uri)
                 if (r.change == 'deleted'):
-                    deletes[r.uri]=r.change
+                    deletes[r.uri] = r.change
         # go through all deletes and prune if first was create
         pruned2 = []
         for r in reversed(pruned1):
@@ -246,31 +236,5 @@ class ResourceContainer(object):
                 n += 1
             else:
                 pruned2.append(r)
-        self.resources=pruned2
+        self.resources = pruned2
         return(n)
-
-    def __str__(self):
-        """Return string of all resources in order given by interator"""
-        s = ''
-        for resource in self:
-            s += str(resource) + "\n"
-        return(s)
-
-    def _str_datetime_now(self, x=None):
-        """Return datetime string for use with time attributes
-        
-        Handling depends on input:
-          'now'   - returns datetime for now
-          number  - assume datetime values, generate string
-          other   - no change, return same value
-        """
-        if (x == 'now'):
-            # Now, this is wht datetime_to_str() with no arg gives
-            return( datetime_to_str() )
-        try:
-            # Test for number
-            junk = x + 0.0
-            return datetime_to_str(x)
-        except TypeError:
-            # Didn't look like a number, treat as string
-            return x

resync/resource_dump.py

@@ -1,7 +1,7 @@
-"""ResourceSync Resource Dump object
+"""ResourceSync Resource Dump object.
 
-A Resource Dump is a set of content package resources with 
-some metadata for each resource. 
+A Resource Dump is a descriptions of a set of content packages
+that contain copies of resources at a point in time.
 
 The Resource Dump object may also contain metadata and links.
 
@@ -9,10 +9,11 @@ Described in specification at:
 http://www.openarchives.org/rs/resourcesync#ResourceDump
 """
 
-from resource_list import ResourceList
+from .resource_list import ResourceList
+
 
 class ResourceDump(ResourceList):
-    """Class representing an Resource Dump
+    """Class representing an Resource Dump.
 
     A Resource Dump comprises a set of content packages that are
     the resources listed. Properties similar to a ResourceList
@@ -22,14 +23,20 @@ class ResourceDump(ResourceList):
     Dump from a ResourceList.
     """
 
-    def __init__(self, resources=None, md=None, ln=None, uri=None, allow_multifile=None, mapper=None):
-        super(ResourceDump, self).__init__(resources=resources, md=md, ln=ln, uri=uri, 
-                                           mapper=mapper)
-        self.capability_name='resourcedump'
+    def __init__(self, resources=None, md=None, ln=None, uri=None,
+                 allow_multifile=None, mapper=None,
+                 spec_version='1.1', add_lastmod=False):
+        """Initialize ResourceDump."""
+        super(ResourceDump, self).__init__(
+            resources=resources, md=md, ln=ln, uri=uri, mapper=mapper,
+            spec_version=spec_version, add_lastmod=add_lastmod)
+        self.capability_name = 'resourcedump'
 
-    def write(self, basename="/tmp/resource_dump.xml", dumpfile=None):
-    	"""Write out a Resource Dump document and optionally also the actual dump files
+    def write(self, basename="/tmp/resource_dump.xml"):
+        """Write out a ResourceDump document.
 
-    	Dump files will be written if the dumpfile argument is set to a base filename.
-    	"""
-    	super(ResourceDump,self).write(basename)
+        FIXME - should there be support for a tie to writing out
+        the Dump files themselves, complete with Resource Dump
+        Manifests?
+        """
+        super(ResourceDump, self).write(basename)

resync/resource_dump_manifest.py

@@ -1,6 +1,6 @@
-"""ResourceSync ResourceDumpManifest object
+"""ResourceSync ResourceDumpManifest object.
 
-A ResourceDumpManifest lists the set of files/resources included 
+A ResourceDumpManifest lists the set of files/resources included
 within a content package that is included in a ResourceDump.
 
 The ResourceDumpManifest object may also contain metadata and links.
@@ -9,16 +9,22 @@ Described in specification at:
 http://www.openarchives.org/rs/resourcesync#ResourceDumpManifest
 """
 
-from resource_list import ResourceList
+from .resource_list import ResourceList
+
 
 class ResourceDumpManifest(ResourceList):
-    """Class representing a Resource Dump Manifest
+    """Class representing a Resource Dump Manifest.
 
     A ResourceDumpManifest comprises a set of files/resources
     in a content package. Properties much Like a ResourceList
     and implemented as a sub-class of ResourceList.
     """
 
-    def __init__(self, resources=None, md=None, ln=None, uri=None, allow_multifile=None, mapper=None):
-        super(ResourceDumpManifest, self).__init__(resources=resources, md=md, ln=ln, uri=uri, mapper=mapper)
+    def __init__(self, resources=None, md=None, ln=None,
+                 uri=None, allow_multifile=None, mapper=None,
+                 spec_version='1.1', add_lastmod=False):
+        """Initialize ResourceDumpManifest."""
+        super(ResourceDumpManifest, self).__init__(
+            resources=resources, md=md, ln=ln, uri=uri, mapper=mapper,
+            spec_version=spec_version, add_lastmod=add_lastmod)
         self.capability_name = 'resourcedump-manifest'

resync/resource_list.py

@@ -1,207 +1,226 @@
-"""ResourceSync Resource List object
+"""ResourceSync Resource List object.
 
-An Resource List is a set of resources with some metadata for 
-each resource. Comparison of resource lists from a source and 
-a destination allows understanding of whether the two are in
-sync or whether some resources need to be updated at the
-destination.
+A Resource List is a set of resources with some metadata for
+each resource. Every resource must have a uri attribute.
+Comparison of resource lists from a source and a destination
+allows understanding of whether the two are in sync or whether
+some resources need to be updated at the destination.
 
 There may also be metadata about the Resource List, and links
-to other ResourceSync documents. Metadata include the 
-timestamp of the ResourceList (md_at) and, optionally, the 
+to other ResourceSync documents. Metadata include the
+timestamp of the Resource List (md_at) and, optionally, the
 timestamp when creation of the Resource List was completed
-(md_completed).at the top level. These include a creation timestamp (from) 
-and links to the Capability List.
+(md_completed) at the top level. These include a creation timestamp
+(from) and links to the Capability List.
 
 Described in specification at:
 http://www.openarchives.org/rs/resourcesync#DescResources
 """
 
-import collections
+import collections.abc
+from collections import OrderedDict
 import os
 from datetime import datetime
 import re
 import sys
-from urllib import URLopener
 
-from list_base_with_index import ListBaseWithIndex
-from sitemap import Sitemap
-from mapper import Mapper, MapperError
-from url_authority import UrlAuthority
+from .list_base_with_index import ListBaseWithIndex
+from .sitemap import Sitemap
+from .mapper import Mapper, MapperError
+from .url_authority import UrlAuthority
 
 
-class ResourceListDict(dict):
-    """Default implementation of class to store resources in ResourceList
+class SortedIterMixin(object):
+    """Mixin to provide sorted_iter() method."""
+
+    def sorted_iter(self):
+        """Iterator over all the resources in this dict by sorted key order."""
+        self._sorted_iter_next_list = sorted(self.keys(), reverse=True)
+        return iter(self._sorted_iter_next, None)
+
+    def _sorted_iter_next(self):
+        if (len(self._sorted_iter_next_list) > 0):
+            return self[self._sorted_iter_next_list.pop()]
+        else:
+            return None
+
+
+class ResourceListDict(dict, SortedIterMixin):
+    """Default implementation of class to store resources in ResourceList.
 
     Key properties of this class are:
     - has add(resource) method
-    - is iterable and results given in alphanumeric order by resource.uri
+    - is iterable and gives resources (not keys) in alphanumeric order by
+      resource.uri
     """
 
     def __iter__(self):
-        """Iterator over all the resources in this resource_list"""
-        self._iter_next_list = sorted(self.keys())
-        self._iter_next_list.reverse()
-        return(iter(self._iter_next, None))
-
-    def _iter_next(self):
-        if (len(self._iter_next_list)>0):
-            return(self[self._iter_next_list.pop()])
-        else:
-            return(None)
+        """Iterator over all the resources in this ResourceListDict."""
+        return self.sorted_iter()
 
     def uris(self):
-        return(sorted(self.keys()))
+        """Extract sorted list of URIs for resources in this ResourceListDict."""
+        return sorted(self.keys())
 
     def add(self, resource, replace=False):
-        """Add just a single resource"""
+        """Add just a single resource."""
         uri = resource.uri
-        if (uri in self and not replace):
-            raise ResourceListDupeError("Attempt to add resource already in resource_list") 
-        self[uri]=resource
+        if (uri in self.keys() and not replace):
+            raise ResourceListDupeError(
+                "Attempt to add resource already in resource_list")
+        self[uri] = resource
 
-class ResourceListOrdered(list):
-    """Alternative implementation of class to store resources in ResourceList
 
-    FIXME - This is a rather inefficient implementation which involves 
-    scanning all resources to check for duplicates. Designed just to enable
-    re-creation of examples in the spec. Something dictionary based would
-    likely be better. Might be best to use OrderedDict but that is available
-    natively only in python >= 2.7 and this library is designed for 2.6,2.7.
+class ResourceListOrdered(OrderedDict, SortedIterMixin):
+    """Alternative implementation of class to store resources in ResourceList.
 
     Key properties of this class are:
     - has add(resource) method
-    - is iterable and results given in order added (not the usual one!)
+    - is iterable and gives resources (not keys) in the order added
     """
 
+    def __iter__(self):
+        """Iterator over the resources in the list in the order added."""
+        return iter(self.values())
+
     def uris(self):
-        """Extract list of all resource URIs (in the order added)"""
-        uris = []
-        for r in self:
-            uris.append(r.uri)
-        return(uris)
+        """Extract list of all resource URIs in the order added."""
+        return self.keys()
 
     def add(self, resource, replace=False):
-        """Add a single resource, check for dupes"""
+        """Add just a single resource."""
         uri = resource.uri
-        for r in self:
-            if (uri == r.uri):
-                if (replace):
-                    r=resource
-                    return
-                else:
-                    raise ResourceListDupeError("Attempt to add resource already in resource_list") 
-        # didn't find it in list, add to end
-        self.append(resource)
+        if (uri in self and not replace):
+            raise ResourceListDupeError(
+                "Attempt to add resource already in resource_list")
+        self[uri] = resource
+
 
 class ResourceListDupeError(Exception):
+    """Exception in case of duplicate resource."""
+
     pass
 
+
 class ResourceList(ListBaseWithIndex):
-    """Class representing an resource_list of resources
+    """Class representing a ResourceList.
 
     This same class is used for both the source and the destination
     and is the central point of comparison the decide whether they
     are in sync or what needs to be copied to bring the destinaton
     into sync.
 
-    A Resource List will admit only one resource with any given URI.
+    A ResourceList will admit only one resource with any given URI.
 
-    Typical usage for a small Resource List is:
-    
-    rl = ResourceList()
-    rl.add( Resource(...) )
-    rl.add( Resource(...) )
-    print rl.as_xml()
+    Typical usage for a small ResourceList is:
 
-    The default storage is unordered but the iterator imposes a canonical 
-    order which is alphabetical by URI. If it is desired to have 
-    resources listed in the order they are added then the ResourceDictOrdered
+        rl = ResourceList()
+        rl.add( Resource(...) )
+        rl.add( Resource(...) )
+        print rl.as_xml()
+
+    The default storage is unordered but the iterator imposes a canonical
+    order which is alphabetical by URI. If it is desired to have
+    resources listed in the order they are added then the ResourceListOrdered
     class may be specified on creation:
 
-    rl = ResourceList( resources_class=ResourceDictOrdered )
-    
+        rl = ResourceList( resources_class=ResourceDictOrdered )
+
+    Use of ResourceListOrdered will be faster than the default.
+
     In normal use it is expected that any Resource List Index will be
     created automatically when writing out a large Resource List in
-    multiple sitemap files. However, should it be necessary to 
+    multiple sitemap files. However, should it be necessary to
     explicitly create an index then this may be specified with:
 
-    rli = ResourceList( resources_class=ResourceDictOrdered )
-    rli.sitemapindex=True
+        rli = ResourceList( resources_class=ResourceDictOrdered )
+        rli.sitemapindex=True
 
     See additional descriptions in ListBaseWithIndex and ListBase.
     """
 
-    def __init__(self, resources=None, count=None, md=None, ln=None, uri=None, 
-                 allow_multifile=None, mapper=None, resources_class=ResourceListDict):
-        super(ResourceList, self).__init__(resources=resources, count=count, md=md, ln=ln, uri=uri,
-                                           capability_name = 'resourcelist', 
-                                           allow_multifile=allow_multifile, mapper=mapper,
-                                           resources_class=resources_class)
+    def __init__(self, resources=None, count=None, md=None, ln=None, uri=None,
+                 allow_multifile=None, mapper=None,
+                 spec_version='1.1', add_lastmod=False,
+                 resources_class=ResourceListDict):
+        """Initialize ResourceList."""
+        super(ResourceList, self).__init__(
+            resources=resources,
+            count=count,
+            md=md,
+            ln=ln,
+            uri=uri,
+            capability_name='resourcelist',
+            allow_multifile=allow_multifile,
+            mapper=mapper, spec_version=spec_version,
+            add_lastmod=add_lastmod,
+            resources_class=resources_class)
 
     def add(self, resource, replace=False):
-        """Add a resource or an iterable collection of resources
+        """Add a resource or an iterable collection of resources.
 
         Will throw a ValueError if the resource (ie. same uri) already
-        exists in the resource_list, unless replace=True.
+        exists in the ResourceList, unless replace=True.
         """
-        if isinstance(resource, collections.Iterable):
+        if isinstance(resource, collections.abc.Iterable):
             for r in resource:
-                self.resources.add(r,replace)
+                self.resources.add(r, replace)
         else:
-            self.resources.add(resource,replace)
+            self.resources.add(resource, replace)
 
-    def compare(self,src):
-        """Compare the current resource_list object with the specified resource_list
+    def compare(self, src):
+        """Compare this ResourceList object with that specified as src.
 
-        The parameter src must also be an resource_list object, it is assumed
-        to be the source, and the current object is the destination. This 
+        The parameter src must also be a ResourceList object, it is assumed
+        to be the source, and the current object is the destination. This
         written to work for any objects in self and sc, provided that the
         == operator can be used to compare them.
 
-        The functioning of this method depends on the iterators for self and
-        src providing access to the resource objects in URI order.
+        The functioning of this method depends on the sorted_iter() iterators
+        for self and src providing access to the resource objects in URI order.
         """
-        dst_iter = iter(self.resources)
-        src_iter = iter(src.resources)
-        same=ResourceList()
-        updated=ResourceList()
-        deleted=ResourceList()
-        created=ResourceList()
-        dst_cur=next(dst_iter,None)
-        src_cur=next(src_iter,None)
+        dst_iter = self.resources.sorted_iter()
+        src_iter = src.resources.sorted_iter()
+        same = ResourceList()
+        updated = ResourceList()
+        deleted = ResourceList()
+        created = ResourceList()
+        dst_cur = next(dst_iter, None)
+        src_cur = next(src_iter, None)
         while ((dst_cur is not None) and (src_cur is not None)):
-            #print 'dst='+dst_cur+'  src='+src_cur
-            if (dst_cur.uri == src_cur.uri):
-                if (dst_cur==src_cur):
+            # print 'dst='+dst_cur+'  src='+src_cur
+            if dst_cur.uri == src_cur.uri:
+                if (dst_cur == src_cur):
                     same.add(dst_cur)
                 else:
                     updated.add(src_cur)
-                dst_cur=next(dst_iter,None)
-                src_cur=next(src_iter,None)
-            elif (not src_cur or dst_cur.uri < src_cur.uri):
+                dst_cur = next(dst_iter, None)
+                src_cur = next(src_iter, None)
+            elif dst_cur.uri < src_cur.uri:
                 deleted.add(dst_cur)
-                dst_cur=next(dst_iter,None)
-            elif (not dst_cur or dst_cur.uri > src_cur.uri):
+                dst_cur = next(dst_iter, None)
+            else:  # dst_cur.uri > src_cur.uri:
                 created.add(src_cur)
-                src_cur=next(src_iter,None)
-            else:
-                raise Exception("this should not be possible")
+                src_cur = next(src_iter, None)
         # what do we have leftover in src or dst lists?
         while (dst_cur is not None):
             deleted.add(dst_cur)
-            dst_cur=next(dst_iter,None)
+            dst_cur = next(dst_iter, None)
         while (src_cur is not None):
             created.add(src_cur)
-            src_cur=next(src_iter,None)
+            src_cur = next(src_iter, None)
         # have now gone through both lists
-        return(same,updated,deleted,created)
+        return(same, updated, deleted, created)
 
-    def has_md5(self):
-        """Return true if at least one contained resource-like object has md5 data"""
-        if (self.resources is None):
-            return(False)
-        for resource in self:
-            if (resource.md5 is not None):
-                return(True)
-        return(False)
+    def hashes(self):
+        """Return set of hashes uses in this resource_list."""
+        hashes = set()
+        if (self.resources is not None):
+            for resource in self:
+                if (resource.md5 is not None):
+                    hashes.add('md5')
+                if (resource.sha1 is not None):
+                    hashes.add('sha-1')
+                if (resource.sha256 is not None):
+                    hashes.add('sha-256')
+        return(hashes)

resync/resource_list_builder.py

@@ -1,92 +1,102 @@
-"""ResourceListBuilder to create ResourceList objects
-
-Currently implements build from files on disk only. 
-
-Attributes:
-- set_path set true to add path attribute for each resource
-- set_md5 set true to calculate MD5 sums for all files
-- set_length set true to include file length in resource_list (defaults true)
-- exclude_dirs is a list of directory names to exclude
-  (defaults to ['CVS','.git'))
-
-FIXME - should add options to set sha1 and sha256 in addition or as
-alternatives to md5.
-"""
+"""ResourceListBuilder to create ResourceList objects."""
 
 import os
 import os.path
 import re
 import time
 import logging
-from urllib import URLopener
-from xml.etree.ElementTree import parse
+from defusedxml.ElementTree import parse
+
+from .hashes import Hashes
+from .resource import Resource
+from .resource_list import ResourceList
+from .sitemap import Sitemap
+from .w3c_datetime import datetime_to_str
 
-from resource import Resource
-from resource_list import ResourceList
-from sitemap import Sitemap
-from utils import compute_md5_for_file
-from w3c_datetime import datetime_to_str
 
 class ResourceListBuilder():
+    """ResourceListBuilder to create ResourceList objects.
 
-    def __init__(self, mapper=None, set_md5=False, set_length=True, set_path=False):
-        """Create ResourceListBuilder object, optionally set options
+    Currently implements build from files on disk only.
+
+    Attributes:
+    - set_hashes to indicate which hashes should be calculated for each resource
+    - set_path set true to add path attribute for each resource
+    - set_length set true to include file length in resource_list (defaults true)
+    - exclude_patterns is a list of files and directory names patterns to exclude
+      using re.match(..). These patterns are left anchored so thus need to be
+      preceded with .* if there may be arbitrary leading characters (defaults to
+      empty)
+    """
+
+    def __init__(self, mapper=None, set_hashes=None,
+                 set_length=True, set_path=False):
+        """Create ResourceListBuilder object, optionally set options.
 
         The mapper attribute must be set before a call to from_disk() in order to
         map between local filenames and URIs.
 
-        The following attributes may be set to determine information added to 
+        The following attributes may be set to determine information added to
         each Resource object based on the disk scan:
-        - set_md5 - True to add md5 digests for each resource. This may add 
-        significant time to the scan process as each file has to be read to 
-        compute the sum
+        - set_hashes - Set of digests to compute for each resource. This may add
+        significant time to the scan process as each file has to be read to
+        compute the hash. Empty set or None means no hashes calculated, names
+        defined in Hashes object
         - set_length - False to not add length for each resources
         - set_path - True to add local path information for each file/resource
         """
         self.mapper = mapper
         self.set_path = set_path
-        self.set_md5 = set_md5
+        self.set_hashes = set_hashes if (set_hashes and len(set_hashes) > 0) else None
         self.set_length = set_length
-        self.exclude_files = ['sitemap\d{0,5}.xml']
-        self.exclude_dirs = ['CVS','.git']
+        self.exclude_patterns = []
         self.include_symlinks = False
+        self.log_count_increment = 50000  # Write log message after 50000 files
         # Used internally only:
         self.logger = logging.getLogger('resync.resource_list_builder')
-        self.compiled_exclude_files = []
+        self.compiled_exclude_patterns = []
 
-    def add_exclude_files(self, exclude_patterns):
-        """Add more patterns of files to exclude while building resource_list"""
+    def add_exclude_patterns(self, exclude_patterns):
+        """Add more patterns of files or directories to exclude while building resource_list.
+
+        After patterns are added the set of compiled patterns is updated.
+        """
         for pattern in exclude_patterns:
-            self.exclude_files.append(pattern)
+            self.exclude_patterns.append(pattern)
+        self._compile_excludes()
 
-    def compile_excludes(self):
-        """Compile a set of regexps for files to be exlcuded from scans"""
-        self.compiled_exclude_files = []
-        for pattern in self.exclude_files:
-            self.compiled_exclude_files.append(re.compile(pattern))
+    def _compile_excludes(self):
+        # Compile a set of regexps for files and directories to be exlcuded from scans
+        self.compiled_exclude_patterns = []
+        for pattern in self.exclude_patterns:
+            try:
+                self.compiled_exclude_patterns.append(re.compile(pattern))
+            except re.error as e:
+                raise ValueError(
+                    "Bad python regex in exclude pattern '%s': %s" % (pattern, str(e)))
 
-    def exclude_file(self, file):
-        """True if file should be exclude based on name pattern"""
-        for pattern in self.compiled_exclude_files:
-            if (pattern.match(file)):
-                return(True)
-        return(False)
+    def _exclude(self, file):
+        # True if file should be exclude based on name pattern.
+        for pattern in self.compiled_exclude_patterns:
+            if pattern.match(file):
+                return True
+        return False
 
     def from_disk(self, resource_list=None, paths=None):
-        """Create or extend resource_list with resources from disk scan
+        """Create or extend resource_list with resources from disk scan.
 
-        Assumes very simple disk path to URL mapping (in self.mapping): chop 
+        Assumes very simple disk path to URL mapping (in self.mapping): chop
         path and replace with url_path. Returns the new or extended ResourceList
         object.
 
         If a resource_list is specified then items are added to that rather
         than creating a new one.
 
-        If paths is specified then these are used instead of the set 
+        If paths is specified then these are used instead of the set
         of local paths in self.mapping.
 
         Example usage with mapping start paths:
-        
+
         mapper=Mapper('http://example.org/path','/path/to/files')
         rlb = ResourceListBuilder(mapper=mapper)
         m = rlb.from_disk()
@@ -97,18 +107,17 @@ class ResourceListBuilder():
         rlb = ResourceListBuilder(mapper=mapper)
         m = rlb.from_disk(paths=['/path/to/files/a','/path/to/files/b'])
         """
-        num=0
+        num = 0
         # Either use resource_list passed in or make a new one
         if (resource_list is None):
             resource_list = ResourceList()
-        # Compile exclude pattern matches
-        self.compile_excludes()
         # Work out start paths from map if not explicitly specified
         if (paths is None):
-            paths=[]
+            paths = []
             for map in self.mapper.mappings:
                 paths.append(map.dst_path)
-        # Set start time unless already set (perhaps because building in chunks)
+        # Set start time unless already set (perhaps because building in
+        # chunks)
         if (resource_list.md_at is None):
             resource_list.md_at = datetime_to_str()
         # Run for each map in the mappings
@@ -120,60 +129,60 @@ class ResourceListBuilder():
         return(resource_list)
 
     def from_disk_add_path(self, path=None, resource_list=None):
-        """Add to resource_list with resources from disk scan starting at path
-        """
+        """Add to resource_list with resources from disk scan starting at path."""
         # sanity
         if (path is None or resource_list is None or self.mapper is None):
             raise ValueError("Must specify path, resource_list and mapper")
-        # is path a directory or a file? for each file: create Resource object, 
+        # is path a directory or a file? for each file: create Resource object,
         # add, increment counter
         if os.path.isdir(path):
-            num_files=0
-            for dirpath, dirs, files in os.walk(path,topdown=True):
+            num_files = 0
+            for dirpath, dirs, files in os.walk(path, topdown=True):
                 for file_in_dirpath in files:
-                    num_files+=1
-                    if (num_files%50000 == 0):
-                        self.logger.info("ResourceListBuilder.from_disk_add_path: %d files..." % (num_files))
-                    self.add_file(resource_list=resource_list,dir=dirpath,file=file_in_dirpath)
-                    # prune list of dirs based on self.exclude_dirs
-                    for exclude in self.exclude_dirs:
-                        if exclude in dirs:
-                            self.logger.debug("Excluding dir %s" % (exclude))
-                            dirs.remove(exclude)
+                    num_files += 1
+                    if (num_files % self.log_count_increment == 0):
+                        self.logger.info(
+                            "ResourceListBuilder.from_disk_add_path: %d files..." % (num_files))
+                    self.add_file(resource_list=resource_list,
+                                  dir=dirpath, file=file_in_dirpath)
+                # prune list of dirs based on self.exclude_dirs
+                prune = []
+                for dir in dirs:
+                    if self._exclude(dir):
+                        self.logger.debug("Excluding dir '%s'" % (dir))
+                        prune.append(dir)
+                for dir in prune:
+                    dirs.remove(dir)
         else:
             # single file
-            self.add_file(resource_list=resource_list,file=path)
+            self.add_file(resource_list=resource_list, file=path)
 
     def add_file(self, resource_list=None, dir=None, file=None):
-        """Add a single file to resource_list
-        
-        Follows object settings of set_path, set_md5 and set_length.
+        """Add a single file to resource_list.
+
+        Follows object settings of set_path, set_hashes and set_length.
         """
-        try:
-            if self.exclude_file(file):
-                self.logger.debug("Excluding file %s" % (file))
-                return
-            # get abs filename and also URL
-            if (dir is not None):
-                file = os.path.join(dir,file)
-            if (not os.path.isfile(file) or not (self.include_symlinks or not os.path.islink(file))):
-                return
-            uri = self.mapper.dst_to_src(file)
-            if (uri is None):
-                raise Exception("Internal error, mapping failed")
-            file_stat=os.stat(file)
-        except OSError as e:
-            sys.stderr.write("Ignoring file %s (error: %s)" % (file,str(e)))
+        if self._exclude(file):
+            self.logger.debug("Excluding file '%s'" % (file))
             return
-        timestamp = file_stat.st_mtime #UTC
-        r = Resource(uri=uri,timestamp=timestamp)
-        if (self.set_path):
-            # add full local path
-            r.path=file
-        if (self.set_md5):
-            # add md5
-            r.md5=compute_md5_for_file(file)
-        if (self.set_length):
-            # add length
-            r.length=file_stat.st_size
+        # get abs filename and also URL
+        if (dir is not None):
+            file = os.path.join(dir, file)
+        if os.path.islink(file) and not self.include_symlinks:
+            self.logger.warning("Ignoring symlink '%s'" % (file))
+            return
+        try:
+            uri = self.mapper.dst_to_src(file)  # might throw MapperError
+            file_stat = os.stat(file)
+        except OSError as e:
+            self.logger.warning("Ignoring file '%s' (error: %s)" % (file, str(e)))
+            return
+        timestamp = file_stat.st_mtime  # UTC
+        r = Resource(uri=uri, timestamp=timestamp)
+        if self.set_path:  # add full local path
+            r.path = file
+        if self.set_hashes:  # add any hashes requested
+            Hashes(self.set_hashes, file).set(r)
+        if self.set_length:  # add length
+            r.length = file_stat.st_size
         resource_list.add(r)

resync/resource_set.py

@@ -1,13 +1,14 @@
-"""A set of Resource objects used for Capability List
-Indexes and ResourceSync Description documents.
+"""Base class representing a set of Resource objects."""
 
-Ordinging is currently alphanumeric (using sorted(..)) on the
-uri which is the key. 
-"""
 
 class ResourceSet(dict):
-    """Implementation of class to store resources in Capability List
-    Indexes and ResourceSync Description documents.
+    """Base class representing a set of Resource objects.
+
+    The ResourceSet class is used for Capability List Indexes and
+    ResourceSync Description documents.
+
+    Ordering of Resources is currently alphanumeric (using sorted(..))
+    on the uri which is the key.
 
     Key properties of this class are:
     - has add(resource) method
@@ -15,23 +16,31 @@ class ResourceSet(dict):
     """
 
     def __iter__(self):
-        """Iterator over all the resources in this resource_list"""
+        """Iterator over all the resources in this ResourceSet."""
         self._iter_next_list = sorted(self.keys())
         self._iter_next_list.reverse()
         return(iter(self._iter_next, None))
 
     def _iter_next(self):
-        if (len(self._iter_next_list)>0):
+        if (len(self._iter_next_list) > 0):
             return(self[self._iter_next_list.pop()])
         else:
             return(None)
 
     def add(self, resource, replace=False):
-        """Add just a single resource"""
+        """Add just a single resource."""
         uri = resource.uri
         if (uri in self and not replace):
-            raise ResourceSetDupeError("Attempt to add resource already in this set") 
-        self[uri]=resource
+            raise ResourceSetDupeError(
+                "Attempt to add resource already in this set")
+        self[uri] = resource
+
+    def first(self):
+        """First item obtained from iterator."""
+        return(next(iter(self)))
+
 
 class ResourceSetDupeError(Exception):
+    """Exception for case of attempt to add duplicate resource."""
+
     pass

resync/sitemap.py

@@ -1,14 +1,14 @@
-"""Read and write ResourceSync documents as sitemaps"""
+"""Read and write ResourceSync documents as sitemaps."""
 
-import re
+import io
+import logging
 import os
 import sys
-import logging
-from xml.etree.ElementTree import ElementTree, Element, parse, tostring
-import StringIO
+from defusedxml.ElementTree import parse
+from xml.etree.ElementTree import ElementTree, Element, tostring
 
-from resource import Resource
-from resource_container import ResourceContainer
+from .resource import Resource
+from .resource_container import ResourceContainer
 
 SITEMAP_NS = 'http://www.sitemaps.org/schemas/sitemap/0.9'
 RS_NS = 'http://www.openarchives.org/rs/terms/'
@@ -21,55 +21,92 @@ XML_ATT_NAME = {
     'md_until': 'until'
 }
 
+
 class SitemapIndexError(Exception):
-    """Exception on attempt to read a sitemapindex instead of sitemap 
-    or vice-versa
+    """Exception if sitemapindex read instead of sitemap or vice-versa.
 
     Provides both a message and a place to store the etree so that
-    the parse tree may be reused as a sitemapingex.
+    the parse tree may be reused as a sitemapindex.
     """
 
     def __init__(self, message=None, etree=None):
+        """Initialize SitemapIndexError with supplied parameters."""
         self.message = message
         self.etree = etree
 
-    def __repr__(self):
+    def __str__(self):
+        """Return just the message attribute."""
         return(self.message)
 
+
 class SitemapParseError(Exception):
+    """Exception for sitemap parsing structural error."""
+
     pass
 
+
 class SitemapDupeError(Exception):
+    """Exception for case of duplicate resources in sitemap."""
+
     pass
 
-class Sitemap(object):
-    """Read and write sitemaps
 
-    Implemented as a separate class that uses ResourceContainer 
-    (ResourceList or ChangeList) and Resource classes as data objects. 
+class Sitemap(object):
+    """Read and write sitemaps.
+
+    Implemented as a separate class that uses ResourceContainer
+    (ResourceList or ChangeList) and Resource classes as data objects.
     Reads and write sitemaps, including multiple file sitemaps.
 
     This class does not automatically handle the reading or writing
     of a sitemapindex and multiple sitemap documents. Instead it will
     take a default and throw and exception if the other case is found
     so that the calling code can handle it.
+
+    Default is to follow ResourceSync v1.1 which distinguishes the meanings
+    of the <lastmod> element (resource modified time as used for Last-Modified
+    header) and the newly introduced <rs:md datetime="..."> attribute (the
+    change time in ChangeList and similar resources). There are two parameters
+    to control these behaviors:
+
+        spec_version - set to either '1.0' or '1.1'. If set to '1.0' then no
+            datetime attribute will be written or parsed
+        add_lastmod - set True to add a lastmod with the same value of datetime
+            if no specific lastmod is specified. Applies only when spec_version
+            '1.1' is selected, provide compatibility with systems that do not
+            understand datetime but instead rely on lastmod
     """
 
-    def __init__(self, pretty_xml=False):
+    def __init__(self, pretty_xml=False, spec_version='1.1', add_lastmod=False):
+        """Initialize Sitemap object."""
         self.logger = logging.getLogger('resync.sitemap')
-        self.pretty_xml=pretty_xml
+        self.pretty_xml = pretty_xml
+        self.spec_1_0 = (spec_version == '1.0')  # v1.0 else assume v1.1
+        self.add_lastmod = add_lastmod  # Optional in v1.1
         # Classes used when parsing
-        self.resource_class=Resource
+        self.resource_class = Resource
         # Information recorded for logging
-        self.resources_created=0    # Set during parsing sitemap
-        self.parsed_index=None      # Set True for sitemapindex, False for sitemap
+        self.resources_created = 0    # Set during parsing sitemap
+        self.parsed_index = None      # Set True for sitemapindex, False for sitemap
+        # rs:md attributes
+        self.md_att_keys = ['md_at', 'capability', 'change', 'datetime',
+                            'md_completed', 'md_from', 'hash', 'length',
+                            'path', 'mime_type', 'md_until']
+        # capabilities
+        self.capabilities = ['resourcelist', 'changelist', 'resourcedump',
+                             'changedump', 'resourcedump-manifest',
+                             'changedump-manifest', 'capabilitylist',
+                             'description',
+                             'resourcelist-archive', 'resourcedump-archive',
+                             'changelist-archive', 'changedump-archive']
+        if self.spec_1_0:
+            self.md_att_keys.remove('datetime')
 
-
-    ##### Write the XML for a sitemap or sitemapindex #####
+    # Write the XML for a sitemap or sitemapindex
 
     def resources_as_xml(self, resources, sitemapindex=False, fh=None):
-        """Write or return XML for a set of resources in sitemap format
-        
+        """Write or return XML for a set of resources in sitemap format.
+
         Arguments:
         - resources - either an iterable or iterator of Resource objects;
                       if there an md attribute this will go to <rs:md>
@@ -78,48 +115,50 @@ class Sitemap(object):
         - fh - write to filehandle fh instead of returning string
         """
         # element names depending on sitemapindex or not
-        root_element = ( 'sitemapindex' if (sitemapindex) else 'urlset' )
-        item_element = ( 'sitemap' if (sitemapindex) else 'url')
+        root_element = ('sitemapindex' if (sitemapindex) else 'urlset')
+        item_element = ('sitemap' if (sitemapindex) else 'url')
         # namespaces and other settings
-        namespaces = { 'xmlns': SITEMAP_NS, 'xmlns:rs': RS_NS }
+        namespaces = {'xmlns': SITEMAP_NS, 'xmlns:rs': RS_NS}
         root = Element(root_element, namespaces)
         if (self.pretty_xml):
-            root.text="\n"
+            root.text = "\n"
         # <rs:ln>
-        if (hasattr(resources,'ln')):
+        if (hasattr(resources, 'ln')):
             for ln in resources.ln:
-                self.add_element_with_atts_to_etree(root,'rs:ln',ln)
+                self.add_element_with_atts_to_etree(root, 'rs:ln', ln)
         # <rs:md>
-        if (hasattr(resources,'md')):
-            self.add_element_with_atts_to_etree(root,'rs:md',resources.md)
+        if (hasattr(resources, 'md')):
+            self.add_element_with_atts_to_etree(root, 'rs:md', resources.md)
         # <url> entries from either an iterable or an iterator
         for r in resources:
-            e=self.resource_etree_element(r, element_name=item_element)
+            e = self.resource_etree_element(r, element_name=item_element)
             root.append(e)
         # have tree, now serialize
-        tree = ElementTree(root);
-        xml_buf=None
+        tree = ElementTree(root)
+        xml_buf = None
         if (fh is None):
-            xml_buf=StringIO.StringIO()
-            fh=xml_buf
-        if (sys.version_info < (2,7)):
-            tree.write(fh,encoding='UTF-8')
-        else:
-            tree.write(fh,encoding='UTF-8',xml_declaration=True,method='xml')
+            xml_buf = io.StringIO()
+            fh = xml_buf
+        tree.write(
+            fh,
+            encoding='unicode',
+            xml_declaration=True,
+            method='xml')
         if (xml_buf is not None):
             return(xml_buf.getvalue())
 
+    # Read/parse an XML sitemap or sitemapindex
 
-    ##### Read/parse an XML sitemap or sitemapindex #####
+    def parse_xml(self, fh=None, etree=None, resources=None,
+                  capability=None, sitemapindex=None):
+        """Parse XML Sitemap and add to resources object.
 
-    def parse_xml(self, fh=None, etree=None, resources=None, capability=None, sitemapindex=None):
-        """Parse XML Sitemap from fh or etree and add resources to a
-        resorces object (which must support the add method). Returns 
-        the resources object.
+        Reads from fh or etree and adds resources to a resorces object
+        (which must support the add method). Returns the resources object.
 
-        Also sets self.resources_created to be the number of resources created. 
-        We adopt a very lax approach here. The parsing is properly namespace 
-        aware but we search just for the elements wanted and leave everything 
+        Also sets self.resources_created to be the number of resources created.
+        We adopt a very lax approach here. The parsing is properly namespace
+        aware but we search just for the elements wanted and leave everything
         else alone.
 
         This method will read either sitemap or sitemapindex documents. Behavior
@@ -128,223 +167,230 @@ class Sitemap(object):
         - False - SitemapIndexError exception if sitemapindex detected
         - True - SitemapIndexError exception if sitemap detected
 
-        Will set self.parsed_index based on whether a sitemap or sitemapindex 
+        Will set self.parsed_index based on whether a sitemap or sitemapindex
         document was read:
         - False - sitemap
         - True - sitemapindex
         """
         if (resources is None):
-            resources=ResourceContainer()
+            resources = ResourceContainer()
         if (fh is not None):
-            etree=parse(fh)
+            etree = parse(fh)
         elif (etree is None):
             raise ValueError("Neither fh or etree set")
         # check root element: urlset (for sitemap), sitemapindex or bad
         root_tag = etree.getroot().tag
-        resource_tag = None # will be <url> or <sitemap> depending on type
+        resource_tag = None  # will be <url> or <sitemap> depending on type
         self.parsed_index = None
-        if (root_tag == '{'+SITEMAP_NS+"}urlset"):
+        if (root_tag == '{' + SITEMAP_NS + "}urlset"):
             self.parsed_index = False
             if (sitemapindex is not None and sitemapindex):
-                raise SitemapIndexError("Got sitemap when expecting sitemapindex",etree)
-            resource_tag = '{'+SITEMAP_NS+"}url"
-        elif (root_tag == '{'+SITEMAP_NS+"}sitemapindex"):
+                raise SitemapIndexError(
+                    "Got sitemap when expecting sitemapindex", etree)
+            resource_tag = '{' + SITEMAP_NS + "}url"
+        elif (root_tag == '{' + SITEMAP_NS + "}sitemapindex"):
             self.parsed_index = True
             if (sitemapindex is not None and not sitemapindex):
-                raise SitemapIndexError("Got sitemapindex when expecting sitemap",etree)
-            resource_tag = '{'+SITEMAP_NS+"}sitemap"
+                raise SitemapIndexError(
+                    "Got sitemapindex when expecting sitemap", etree)
+            resource_tag = '{' + SITEMAP_NS + "}sitemap"
         else:
-            raise SitemapParseError("XML is not sitemap or sitemapindex (root element is <%s>)" % root_tag)
-        
+            raise SitemapParseError(
+                "XML is not sitemap or sitemapindex (root element is <%s>)" %
+                root_tag)
+
         # have what we expect, read it
         in_preamble = True
         self.resources_created = 0
         seen_top_level_md = False
-        for e in etree.getroot().getchildren():
-            # look for <rs:md> and <rs:ln>, first <url> ends
-            # then look for resources in <url> blocks
+        for e in list(etree.getroot()):
+            # look for <rs:md> and <rs:ln>, first <url>/<sitemap> ends
+            # then look for resources in <url>/<sitemap> blocks.
+            # ignore any elements we don't recognize
             if (e.tag == resource_tag):
-                in_preamble = False #any later rs:md or rs:ln is error
+                in_preamble = False  # any later rs:md or rs:ln is error
                 r = self.resource_from_etree(e, self.resource_class)
-                try:
-                    resources.add( r )
-                except SitemapDupeError:
-                    self.logger.warning("dupe of: %s (lastmod=%s)" % (r.uri,r.lastmod))
-                self.resources_created+=1
-            elif (e.tag == "{"+RS_NS+"}md"):
+                resources.add(r)
+                self.resources_created += 1
+            elif (e.tag == "{" + RS_NS + "}md"):
                 if (in_preamble):
                     if (seen_top_level_md):
-                        raise SitemapParseError("Multiple <rs:md> at top level of sitemap")
+                        raise SitemapParseError(
+                            "Multiple <rs:md> at top level of sitemap")
                     else:
-                        resources.md = self.md_from_etree(e,'preamble')
+                        resources.md = self.md_from_etree(e, 'preamble')
                         seen_top_level_md = True
                 else:
-                    raise SitemapParseError("Found <rs:md> after first <url> in sitemap")
-            elif (e.tag == "{"+RS_NS+"}ln"):
+                    raise SitemapParseError(
+                        "Found <rs:md> after first <url> in sitemap")
+            elif (e.tag == "{" + RS_NS + "}ln"):
                 if (in_preamble):
-                    resources.ln.append(self.ln_from_etree(e,'preamble'))
+                    resources.ln.append(self.ln_from_etree(e, 'preamble'))
                 else:
-                    raise SitemapParseError("Found <rs:ln> after first <url> in sitemap")
-            else:
-                # element we don't recognize, ignore
-                pass
+                    raise SitemapParseError(
+                        "Found <rs:ln> after first <url> in sitemap")
         # check that we read to right capability document
         if (capability is not None):
             if ('capability' not in resources.md):
                 if (capability == 'resourcelist'):
-                    self.logger.warning('No capability specified in sitemap, assuming resourcelist')
+                    self.logger.warning(
+                        'No capability specified in sitemap, assuming resourcelist')
                     resources.md['capability'] = 'resourcelist'
                 else:
                     raise SitemapParseError("Expected to read a %s document, but no capability specified in sitemap" %
-                                 (capability))
+                                            (capability))
             if (resources.md['capability'] != capability):
                 raise SitemapParseError("Expected to read a %s document, got %s" %
-                                 (capability,resources.md['capability']))
+                                        (capability, resources.md['capability']))
         # return the resource container object
         return(resources)
 
-
-    ##### Resource methods #####
+    # Resource methods
 
     def resource_etree_element(self, resource, element_name='url'):
-        """Return xml.etree.ElementTree.Element representing the resource
+        """Return xml.etree.ElementTree.Element representing the resource.
 
-        Returns and element for the specified resource, of the form <url> 
+        Returns and element for the specified resource, of the form <url>
         with enclosed properties that are based on the sitemap with extensions
         for ResourceSync.
         """
         e = Element(element_name)
         sub = Element('loc')
-        sub.text=resource.uri
+        sub.text = resource.uri
         e.append(sub)
-        if (resource.timestamp is not None):
-            # Create appriate element for timestamp
-            sub = Element('lastmod')
-            sub.text = str(resource.lastmod) #W3C Datetime in UTC
-            e.append(sub)
+        lm = resource.lastmod  # W3C Datetime in UTC
+        if lm is not None or self.spec_1_0 or self.add_lastmod:
+            # In 1.0 we either use the lastmod specified or else use the
+            # datetime value because there should always be a lastmod
+            if lm is None and (self.spec_1_0 or self.add_lastmod):
+                lm = resource.datetime  # W3C Datetime in UTC
+            if lm is not None:
+                sub = Element('lastmod')
+                sub.text = lm
+                e.append(sub)
         md_atts = {}
-        for att in ('capability','change','hash','length','path','mime_type',
-                    'md_at','md_completed','md_from','md_until'):
-            val = getattr(resource,att,None)
+        for att in self.md_att_keys:
+            val = getattr(resource, att, None)
             if (val is not None):
-                md_atts[self._xml_att_name(att)] = str(val)
-        if (len(md_atts)>0):
-            md = Element('rs:md',md_atts)
-            e.append(md)
+                md_atts[att] = str(val)
+        if (len(md_atts) > 0):
+            self.add_element_with_atts_to_etree(e, 'rs:md', md_atts, add_return=False)
         # add any <rs:ln>
-        if (hasattr(resource,'ln') and
-            resource.ln is not None):
+        if (hasattr(resource, 'ln') and resource.ln is not None):
             for ln in resource.ln:
-                self.add_element_with_atts_to_etree(e,'rs:ln',ln)
+                self.add_element_with_atts_to_etree(e, 'rs:ln', ln)
         if (self.pretty_xml):
-            e.tail="\n"
+            e.tail = "\n"
         return(e)
 
-    def resource_as_xml(self,resource,indent=' '):
-        """Return string for the the resource as part of an XML sitemap
+    def resource_as_xml(self, resource):
+        """Return string for the resource as part of an XML sitemap.
 
+        Returns a string with the XML snippet representing the resource,
+        without any XML declaration. (So much simpler now only Python 3.x
+        supported, see earlier versions for 2.6, 2.7 etc.)
         """
         e = self.resource_etree_element(resource)
-        if (sys.version_info < (2,7)):
-            #must not specify method='xml' in python2.6
-            return(tostring(e, encoding='UTF-8'))
-        else:
-            return(tostring(e, encoding='UTF-8', method='xml'))
+        return(tostring(e, encoding='unicode', method='xml'))
 
     def resource_from_etree(self, etree, resource_class):
-        """Construct a Resource from an etree
+        """Construct a Resource from an etree.
 
         Parameters:
          etree - the etree to parse
          resource_class - class of Resource object to create
 
-        The parsing is properly namespace aware but we search just 
-        for the elements wanted and leave everything else alone. Will 
+        The parsing is properly namespace aware but we search just
+        for the elements wanted and leave everything else alone. Will
         raise an error if there are multiple <loc> or multiple <lastmod>
-        elements. Otherwise, provided there is a <loc> element then will 
+        elements. Otherwise, provided there is a <loc> element then will
         go ahead and extract as much as possible.
 
         All errors raised are SitemapParseError with messages intended
         to help debug problematic sitemap XML.
         """
-        loc_elements = etree.findall('{'+SITEMAP_NS+"}loc")
-        if (len(loc_elements)>1):
-            raise SitemapParseError("Multiple <loc> elements while parsing <url> in sitemap");
-        elif (len(loc_elements)==0):
-            raise SitemapParseError("Missing <loc> element while parsing <url> in sitemap")
+        loc_elements = etree.findall('{' + SITEMAP_NS + "}loc")
+        if (len(loc_elements) > 1):
+            raise SitemapParseError(
+                "Multiple <loc> elements while parsing <url> in sitemap")
+        elif (len(loc_elements) == 0):
+            raise SitemapParseError(
+                "Missing <loc> element while parsing <url> in sitemap")
         else:
             loc = loc_elements[0].text
-        if (loc is None or loc==''):
-            raise SitemapParseError("Bad <loc> element with no content while parsing <url> in sitemap")
+        if (loc is None or loc == ''):
+            raise SitemapParseError(
+                "Bad <loc> element with no content while parsing <url> in sitemap")
         # must at least have a URI, make this object
-        resource=resource_class(uri=loc)
+        resource = resource_class(uri=loc)
         # and hopefully a lastmod datetime (but none is OK)
-        lastmod_elements = etree.findall('{'+SITEMAP_NS+"}lastmod")
-        if (len(lastmod_elements)>1):
-            raise SitemapParseError("Multiple <lastmod> elements while parsing <url> in sitemap");
-        elif (len(lastmod_elements)==1):
-            resource.lastmod=lastmod_elements[0].text
+        lastmod_elements = etree.findall('{' + SITEMAP_NS + "}lastmod")
+        if (len(lastmod_elements) > 1):
+            raise SitemapParseError(
+                "Multiple <lastmod> elements while parsing <url> in sitemap")
+        elif (len(lastmod_elements) == 1):
+            resource.lastmod = lastmod_elements[0].text
         # proceed to look for other resource attributes in an rs:md element
-        md_elements = etree.findall('{'+RS_NS+"}md")
-        if (len(md_elements)>1):
-            raise SitemapParseError("Found multiple (%d) <rs:md> elements for %s", (len(md_elements),loc))
-        elif (len(md_elements)==1):
+        md_elements = etree.findall('{' + RS_NS + "}md")
+        if (len(md_elements) > 1):
+            raise SitemapParseError(
+                "Found multiple (%d) <rs:md> elements for %s", (len(md_elements), loc))
+        elif (len(md_elements) == 1):
             # have on element, look at attributes
-            md = self.md_from_etree(md_elements[0],context=loc)
+            md = self.md_from_etree(md_elements[0], context=loc)
             # simple attributes that map directly to Resource object attributes
-            for att in ('capability','change','length','path','mime_type'):
+            for att in ('capability', 'change', 'datetime', 'length', 'path', 'mime_type'):
                 if (att in md):
-                    setattr(resource,att,md[att])
+                    setattr(resource, att, md[att])
             # The ResourceSync beta spec lists md5, sha-1 and sha-256 fixity
             # digest types. Parse and warn of errors ignored.
             if ('hash' in md):
                 try:
                     resource.hash = md['hash']
                 except ValueError as e:
-                    self.logger.warning("%s in <rs:md> for %s" % (str(e),loc))
+                    self.logger.warning("%s in <rs:md> for %s" % (str(e), loc))
         # look for rs:ln elements (optional)
-        ln_elements = etree.findall('{'+RS_NS+"}ln")
-        if (len(ln_elements)>0):
+        ln_elements = etree.findall('{' + RS_NS + "}ln")
+        if (len(ln_elements) > 0):
             resource.ln = []
             for ln_element in ln_elements:
-                resource.ln.append(self.ln_from_etree(ln_element,loc))
+                resource.ln.append(self.ln_from_etree(ln_element, loc))
         return(resource)
 
     def md_from_etree(self, md_element, context=''):
-        """Parse rs:md attributes returning a dict of the data
+        """Parse rs:md attributes returning a dict of the data.
 
         Parameters:
             md_element  - etree element <rs:md>
             context     - context for error reporting
+
+        ResourceSync v1.1 adds the datetime attribute
         """
         md = {}
         # grab all understood attributes into md dict
-        for att in ('capability','change','hash','length','path','mime_type',
-                    'md_at','md_completed','md_from','md_until'):
+        for att in self.md_att_keys:
             xml_att = self._xml_att_name(att)
-            val = md_element.attrib.get(xml_att,None)
+            val = md_element.attrib.get(xml_att, None)
             if (val is not None):
                 md[att] = val
-        # capability. Allow this to be missing but do a very simple syntax
-        # check on plausible values if present
-        if ('capability' in md):
-            if (re.match(r"^[\w\-]+$", md['capability']) is None):
-                raise SitemapParseError("Bad capability name '%s' in %s" % (capability,context))
-        # change should be one of defined values
-        if ('change' in md):
-            if (md['change'] not in ['created','updated','deleted'] ):
-                self.logger.warning("Bad change attribute in <rs:md> for %s" % (context))
-        # length should be an integer
+        # capability. Allow this to be missing or a new value but warn if it
+        # isn't recognized
+        if ('capability' in md and md['capability'] not in self.capabilities):
+            self.logger.warning("Unknown capability name '%s' in %s" % (md['capability'], context))
+        # change must be one of defined values
+        if ('change' in md and md['change'] not in ['created', 'updated', 'deleted']):
+            raise SitemapParseError("Bad change attribute in <rs:md> for %s" % (context))
+        # length must be an integer
         if ('length' in md):
             try:
-                md['length']=int(md['length'])
+                md['length'] = int(md['length'])
             except ValueError as e:
-                raise SitemapParseError("Invalid length element in <rs:md> for %s" % (context))
+                raise SitemapParseError("Invalid length element in <rs:md> for %s" %
+                                        (context))
         return(md)
 
-
-    def ln_from_etree(self,ln_element,context=''):
-        """Parse rs:ln element from an etree, returning a dict of the data
+    def ln_from_etree(self, ln_element, context=''):
+        """Parse rs:ln element from an etree, returning a dict of the data.
 
         Parameters:
             md_element - etree element <rs:md>
@@ -352,39 +398,45 @@ class Sitemap(object):
         """
         ln = {}
         # grab all understood attributes into ln dict
-        for att in ('hash','href','length','modified','path','rel','pri','mime_type'):
+        for att in ('hash', 'href', 'length', 'modified',
+                    'path', 'rel', 'pri', 'mime_type'):
             xml_att = self._xml_att_name(att)
-            val = ln_element.attrib.get(xml_att,None)
+            val = ln_element.attrib.get(xml_att, None)
             if (val is not None):
                 ln[att] = val
         # now do some checks and conversions...
         # href (MANDATORY)
         if ('href' not in ln):
-            raise SitemapParseError("Missing href in <rs:ln> in %s" % (context))
+            raise SitemapParseError("Missing href in <rs:ln> in %s" %
+                                    (context))
         # rel (MANDATORY)
         if ('rel' not in ln):
             raise SitemapParseError("Missing rel in <rs:ln> in %s" % (context))
         # length in bytes
         if ('length' in ln):
             try:
-                ln['length']=int(ln['length'])
+                ln['length'] = int(ln['length'])
             except ValueError as e:
-                raise SitemapParseError("Invalid length attribute value in <rs:ln> for %s" % (context))
+                raise SitemapParseError("Invalid length attribute value in <rs:ln> for %s" %
+                                        (context))
         # pri - priority, must be a number between 1 and 999999
         if ('pri' in ln):
             try:
-                ln['pri']=int(ln['pri'])
+                ln['pri'] = int(ln['pri'])
             except ValueError as e:
-                raise SitemapParseError("Invalid pri attribute in <rs:ln> for %s" % (context))
-            if (ln['pri']<1 or ln['pri']>999999):
-                raise SitemapParseError("Bad pri attribute value in <rs:ln> for %s" % (context))
+                raise SitemapParseError(
+                    "Invalid pri attribute in <rs:ln> for %s" %
+                    (context))
+            if (ln['pri'] < 1 or ln['pri'] > 999999):
+                raise SitemapParseError(
+                    "Bad pri attribute value in <rs:ln> for %s" %
+                    (context))
         return(ln)
 
+    # Metadata and link elements
 
-    ##### Metadata and link elements #####
-
-    def add_element_with_atts_to_etree(self, etree, name, atts):
-        """Add element with name and atts to etree iff there are any atts
+    def add_element_with_atts_to_etree(self, etree, name, atts, add_return=True):
+        """Add empty element with name and atts to etree iff there are any atts.
 
         Parameters:
             etree - an etree object
@@ -396,18 +448,27 @@ class Sitemap(object):
             val = atts[att]
             if (val is not None):
                 xml_atts[self._xml_att_name(att)] = str(val)
-        if (len(xml_atts)>0):
-            e = Element(name,xml_atts)
-            if (self.pretty_xml):
-                e.tail="\n"
+        if (len(xml_atts) > 0):
+            e = Element(name, xml_atts)
+            if (add_return and self.pretty_xml):
+                e.tail = "\n"
+            # There is no real reason why the attribute keys should be sorted
+            # but ElementTree serialization up to Python 3.7 always wrote XML
+            # with attributes in sort order. This was changed in Python 3.8
+            # to be the order of addition. Adding sorting here keeps consistent
+            # behavior for all versions. See:
+            # https://docs.python.org/3/library/xml.etree.elementtree.html#element-objects
+            attribs = sorted(e.attrib.items())
+            e.attrib.clear()
+            e.attrib.update(attribs)
             etree.append(e)
 
-    def _xml_att_name(self,att):
-        """Get XML attribute name corresponding to supplied Resource object attribute
+    def _xml_att_name(self, att):
+        """Get XML attribute name corresponding to supplied Resource object attribute.
 
         We cannot use the XML attribute names in Python because 'from' and others
         conflict with Python reserved words. To be consistent all the extra timestamps
-        are prefixed with md_. Assumption is that attibute name is same unless 
+        are prefixed with md_. Assumption is that attibute name is same unless
         specified in XML_ATT_NAME.
         """
-        return XML_ATT_NAME.get(att,att)
+        return XML_ATT_NAME.get(att, att)

resync/source_description.py

@@ -1,17 +1,17 @@
-"""ResourceSync Source Description object
+"""ResourceSync Source Description object.
 
-A ResourceSync Source Description enumerates the Capability 
-Lists offered by a Source. Since a Source has one Capability 
-List per set of resources that it distinguishes, the 
-ResourceSync Source Description will enumerate as many 
+A ResourceSync Source Description enumerates the Capability
+Lists offered by a Source. Since a Source has one Capability
+List per set of resources that it distinguishes, the
+ResourceSync Source Description will enumerate as many
 Capability Lists as the Source has distinct sets of resources.
 
 The ResourceSync Source Description is based on the <urlset>
 format. The specification allows for a very large Source Descriptions
-to use and index based on a <sitemapindex>, though this is probably 
+to use and index based on a <sitemapindex>, though this is probably
 rarely necessary!
 
-There is no meaning in the order of description of sets of resources 
+There is no meaning in the order of description of sets of resources
 in a Source Description so the default is to store these descriptions
 as a set (ResourceSet).
 
@@ -21,14 +21,15 @@ documents.
 See: http://www.openarchives.org/rs/resourcesync#SourceDesc
 """
 
-import collections
+import collections.abc
+
+from resync.resource import Resource
+from resync.resource_set import ResourceSet
+from resync.list_base_with_index import ListBaseWithIndex
 
-from resource import Resource
-from resource_set import ResourceSet
-from list_base_with_index import ListBaseWithIndex
 
 class SourceDescription(ListBaseWithIndex):
-    """Class representing the set of Capability Lists supported
+    """Class representing the set of Capability Lists supported.
 
     Will admit only one resource with any given URI.
 
@@ -37,37 +38,42 @@ class SourceDescription(ListBaseWithIndex):
     to override.
     """
 
-    def __init__(self, resources=None, md=None, ln=None):
-        super(SourceDescription, self).__init__(resources=resources, md=md, ln=ln,
-                                                capability_name='description',
-                                                resources_class=ResourceSet)
-        self.md['from']=None #usually don't want a from date
+    def __init__(self, resources=None, md=None, ln=None,
+                 spec_version='1.1', add_lastmod=False):
+        """Initialize based on superclass ListBaseWithIndex."""
+        super(SourceDescription, self).__init__(
+            resources=resources, md=md, ln=ln,
+            capability_name='description',
+            resources_class=ResourceSet,
+            spec_version=spec_version,
+            add_lastmod=add_lastmod)
+        self.md['from'] = None  # usually don't want a from date
 
     def add(self, resource, replace=False):
-        """Add a resource or an iterable collection of resources
+        """Add a resource or an iterable collection of resources.
 
         Will throw a ValueError if the resource (ie. same uri) already
         exists in the capability_list, unless replace=True.
         """
-        if isinstance(resource, collections.Iterable):
+        if isinstance(resource, collections.abc.Iterable):
             for r in resource:
-                self.resources.add(r,replace)
+                self.resources.add(r, replace)
         else:
-            self.resources.add(resource,replace)
+            self.resources.add(resource, replace)
 
-    def add_capability_list(self,capability_list=None):
-        """Add a capability list
+    def add_capability_list(self, capability_list=None):
+        """Add a capability list.
 
         Adds either a CapabiltyList object specified in capability_list
         or else creates a Resource with the URI given in capability_list
         and adds that to the Source Description
         """
-        if (hasattr(capability_list,'uri')):
-            r = Resource( uri=capability_list.uri,
-                          capability=capability_list.capability_name )
+        if (hasattr(capability_list, 'uri')):
+            r = Resource(uri=capability_list.uri,
+                         capability=capability_list.capability_name)
             if (capability_list.describedby is not None):
-                r.link_set( rel='describedby', href=capability_list.describedby )
+                r.link_set(rel='describedby', href=capability_list.describedby)
         else:
-            r = Resource( uri=capability_list,
-                          capability='capabilitylist')
-        self.add( r )
+            r = Resource(uri=capability_list,
+                         capability='capabilitylist')
+        self.add(r)

resync/test/test_capability_list.py

@@ -1,41 +0,0 @@
-import unittest
-import re
-from resync.resource import Resource
-from resync.resource_list import ResourceList
-from resync.change_list import ChangeList
-from resync.capability_list import CapabilityList
-
-class TestCapabilityList(unittest.TestCase):
-
-    def test01_resourcelist(self):
-        rl = ResourceList()
-        caps = CapabilityList()
-        caps.add_capability( rl, "http://example.org/resourcelist.xml" )
-        caps.md['from'] = "2013-02-07T22:39:00"
-        self.assertEqual( len(caps), 1 )
-        self.assertEqual( caps.as_xml(), '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"><rs:md capability="capabilitylist" from="2013-02-07T22:39:00" /><url><loc>http://example.org/resourcelist.xml</loc><rs:md capability="resourcelist" /></url></urlset>' )
-
-    def test02_multiple(self):
-        caps = CapabilityList()
-        rl = ResourceList()
-        caps.add_capability( rl, "rl.xml" )
-        cl = ChangeList()
-        caps.add_capability( cl, "cl.xml" )
-        self.assertEqual( len(caps), 2 )
-        xml = caps.as_xml()
-        self.assertTrue( re.search( r'<loc>rl.xml</loc><rs:md capability="resourcelist" />', xml ) )
-        self.assertTrue( re.search( r'<loc>cl.xml</loc><rs:md capability="changelist" />', xml) )
-
-    def test03_parse(self):
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"><rs:md capability="capabilitylist" from="2013-02-07T22:39:00" /><url><loc>http://example.org/resourcelist.xml</loc><rs:md capability="resourcelist" /></url></urlset>'
-        cl=CapabilityList()
-        cl.parse(str=xml)
-        self.assertEqual( cl.capability, 'capabilitylist')
-        self.assertEqual( len(cl.resources), 1, 'got 1 resource')
-        [r] = cl.resources
-        self.assertEqual( r.uri, 'http://example.org/resourcelist.xml', 'resourcelist uri')
-        self.assertEqual( r.capability, 'resourcelist')
-
-if __name__ == '__main__':
-    suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestCapabilityList)
-    unittest.TextTestRunner().run(suite)

resync/test/test_change_dump.py

@@ -1,58 +0,0 @@
-import unittest
-import StringIO
-import re
-from resync.resource import Resource
-from resync.change_dump import ChangeDump
-from resync.sitemap import SitemapParseError
-
-class TestChangeDump(unittest.TestCase):
-
-    def test01_as_xml(self):
-        rd = ChangeDump()
-        rd.add( Resource('a.zip',timestamp=1) )
-        rd.add( Resource('b.zip',timestamp=2) )
-        xml = rd.as_xml()
-        print xml
-        self.assertTrue( re.search(r'<rs:md .*capability="changedump"', xml), 'XML has capability' )
-        self.assertTrue( re.search(r'<url><loc>a.zip</loc><lastmod>1970-01-01T00:00:01Z</lastmod></url>', xml), 'XML has resource a' ) 
-
-    def test10_parse(self):
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
-<rs:md capability="changedump" from="2013-01-01"/>\
-<url><loc>http://example.com/a.zip</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="12345" /></url>\
-<url><loc>http://example.com/b.zip</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="56789" /></url>\
-</urlset>'
-        rd=ChangeDump()
-        rd.parse(fh=StringIO.StringIO(xml))
-        self.assertEqual( len(rd.resources), 2, 'got 2 resource dumps')
-        self.assertEqual( rd.md['capability'], 'changedump', 'capability set' )
-        self.assertEqual( rd.md_from, '2013-01-01' )
-        self.assertTrue( 'http://example.com/a.zip' in rd.resources )
-        self.assertTrue( rd.resources['http://example.com/a.zip'].length, 12345 )
-        self.assertTrue( 'http://example.com/b.zip' in rd.resources )
-        self.assertTrue( rd.resources['http://example.com/b.zip'].length, 56789 )
-
-    def test11_parse_no_capability(self):
-        # For a resource dump this should be an error
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
-<rs:md at="2013-01-01"/>\
-<url><loc>http://example.com/a.zip</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="12" /></url>\
-</urlset>'
-        rd=ChangeDump()
-        self.assertRaises( SitemapParseError, rd.parse, fh=StringIO.StringIO(xml) )
-
-    def test12_parse_bad_capability(self):
-        # the <rs:md capability="bad_capability".. should give error
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
-<rs:md capability="bad_capability" from="2013-01-01"/>\
-<url><loc>http://example.com/bad_res_1</loc><lastmod>2012-03-14T18:37:36Z</lastmod></url>\
-</urlset>'
-        rd=ChangeDump()
-        self.assertRaises( SitemapParseError, rd.parse, fh=StringIO.StringIO(xml) )
-
-if __name__ == '__main__':
-    suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestChangeDump)
-    unittest.TextTestRunner().run(suite)

resync/test/test_change_list.py

@@ -1,132 +0,0 @@
-import unittest
-import StringIO
-import re
-from resync.resource import Resource
-from resync.change_list import ChangeList
-from resync.resource_list import ResourceList
-from resync.sitemap import SitemapParseError
-
-class TestChangeList(unittest.TestCase):
-
-    def test1_set_with_repeats(self):
-        src = ChangeList()
-        src.add( Resource('a',timestamp=1,change='updated') )
-        src.add( Resource('b',timestamp=1,change='created') )
-        src.add( Resource('c',timestamp=1,change='deleted') )
-        src.add( Resource('a',timestamp=2,change='deleted') )
-        src.add( Resource('b',timestamp=2,change='updated') )
-        self.assertEqual(len(src), 5, "5 changes in change_list")
-
-    def test2_with_repeats_again(self):
-        r1 = Resource(uri='a',length=1,change='created')
-        r2 = Resource(uri='b',length=2,change='created')
-        i = ChangeList()
-        i.add(r1)
-        i.add(r2)
-        self.assertEqual( len(i), 2 )
-        # Can add another Resource with same URI
-        r1d = Resource(uri='a',length=10,change='created')
-        i.add(r1d)
-        self.assertEqual( len(i), 3 )
-
-    def test3_change_list(self):
-        src = ChangeList()
-        src.add( Resource('a',timestamp=1,change='created') )
-        src.add( Resource('b',timestamp=2,change='created') )
-        src.add( Resource('c',timestamp=3,change='created') )
-        src.add( Resource('d',timestamp=4,change='created') ) 
-        src.add( Resource('e',timestamp=5,change='created') )
-        self.assertEqual(len(src), 5, "5 things in src")
-
-    def test4_iter(self):
-        i = ChangeList()
-        i.add( Resource('a',timestamp=1,change='created') )
-        i.add( Resource('b',timestamp=2,change='created') )
-        i.add( Resource('c',timestamp=3,change='created') )
-        i.add( Resource('d',timestamp=4,change='created') )
-        resources=[]
-        for r in i:
-            resources.append(r)
-        self.assertEqual(len(resources), 4)
-        self.assertEqual( resources[0].uri, 'a')
-        self.assertEqual( resources[3].uri, 'd')
-
-    def test5_add_changed_resources(self):
-        added = ResourceList()
-        added.add( Resource('a',timestamp=1,change='created') )
-        added.add( Resource('d',timestamp=4,change='created') )
-        self.assertEqual(len(added), 2, "2 things in added resource_list")
-        changes = ChangeList()
-        changes.add_changed_resources( added, change='created' )
-        self.assertEqual(len(changes), 2, "2 things added")
-        i = iter(changes)
-        first = i.next()
-        self.assertEqual(first.uri, 'a', "changes[0].uri=a")
-        self.assertEqual(first.timestamp, 1, "changes[0].timestamp=1")
-        self.assertEqual(first.change, 'created') #, "changes[0].change=createdd")
-        second = i.next()
-        self.assertEqual(second.timestamp, 4, "changes[1].timestamp=4")
-        self.assertEqual(second.change, 'created', "changes[1].change=createdd")
-        # Now add some with updated (one same, one diff)
-        updated = ResourceList()
-        updated.add( Resource('a',timestamp=5,change='created') )
-        updated.add( Resource('b',timestamp=6,change='created') )
-        self.assertEqual(len(updated), 2, "2 things in updated resource_list")
-        changes.add_changed_resources( updated, change='updated' )
-        self.assertEqual(len(changes), 4, "4 = 2 old + 2 things updated")
-        # Make new resource_list from the changes which should not have dupes
-        dst = ResourceList()
-        dst.add( changes, replace=True )
-        self.assertEqual(len(dst), 3, "3 unique resources")
-        self.assertEqual(dst.resources['a'].timestamp, 5 ) # 5 was later in last the 1
-        self.assertEqual(dst.resources['a'].change, 'updated')
-        self.assertEqual(dst.resources['b'].timestamp, 6)
-        self.assertEqual(dst.resources['b'].change, 'updated')
-        self.assertEqual(dst.resources['d'].timestamp, 4)
-        self.assertEqual(dst.resources['d'].change, 'created')
-
-    def test20_as_xml(self):
-        cl = ChangeList()
-        cl.md_from = '1970-01-01T00:00:00Z'
-        cl.add( Resource('a',timestamp=1,change='updated') )
-        cl.add( Resource('b',timestamp=2,change='updated') )
-        xml = cl.as_xml()
-        self.assertTrue( re.search(r'<rs:md .*capability="changelist"', xml), 'XML has capability' )
-        self.assertTrue( re.search(r'<rs:md .*from="\d\d\d\d\-\d\d\-\d\dT\d\d:\d\d:\d\dZ"', xml), 'XML has from to seconds precision (and not more)' )
-        self.assertTrue( re.search(r'<url><loc>a</loc><lastmod>1970-01-01T00:00:01Z</lastmod>', xml), 'XML has resource a' ) 
-
-    def test30_parse(self):
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
-<rs:md capability="changelist" from="2013-01-01"/>\
-<url><loc>/tmp/rs_test/src/file_a</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md change="updated" length="12" /></url>\
-<url><loc>/tmp/rs_test/src/file_b</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md change="deleted" length="32" /></url>\
-</urlset>'
-        cl=ChangeList()
-        cl.parse(fh=StringIO.StringIO(xml))
-        self.assertEqual( len(cl.resources), 2, 'got 2 resources')
-        self.assertEqual( cl.md['capability'], 'changelist', 'capability set' )
-        self.assertEqual( cl.md['md_from'], '2013-01-01' )
-
-    def test31_parse_no_capability(self):
-        # missing capability is an error for changelist
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
-<url><loc>http://example.com/res1</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md change="updated"/></url>\
-</urlset>'
-        cl=ChangeList()
-        self.assertRaises( SitemapParseError, cl.parse, fh=StringIO.StringIO(xml) )
-
-    def test32_parse_bad_capability(self):
-        # the <rs:md capability="bad_capability".. should give error
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
-<rs:md capability="bad_capability" from="2013-01-01"/>\
-<url><loc>http://example.com/bad_res_1</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md change="updated"/></url>\
-</urlset>'
-        cl=ChangeList()
-        self.assertRaises( SitemapParseError, cl.parse, fh=StringIO.StringIO(xml) )
-
-if __name__ == '__main__':
-    suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestChangeList)
-    unittest.TextTestRunner().run(suite)

resync/test/test_client.py

@@ -1,110 +0,0 @@
-import unittest
-import re
-import logging
-import sys, StringIO, contextlib
-
-from resync.client import Client, ClientFatalError
-
-# From http://stackoverflow.com/questions/2654834/capturing-stdout-within-the-same-process-in-python
-class Data(object):
-    pass
-
-@contextlib.contextmanager
-def capture_stdout():
-    old = sys.stdout
-    capturer = StringIO.StringIO()
-    sys.stdout = capturer
-    data = Data()
-    yield data
-    sys.stdout = old
-    data.result = capturer.getvalue()
-
-
-class TestClient(unittest.TestCase):
-
-    @classmethod
-    def setUpClass(cls):
-        logging.basicConfig(level=logging.INFO)
-
-    def test01_make_resource_list_empty(self):
-        c = Client()
-        # No mapping is an error
-        self.assertRaises( ClientFatalError, c.build_resource_list )
-
-    def test02_bad_source_uri(self):
-        c = Client()
-        self.assertRaises( ClientFatalError, c.baseline_or_audit )
-        c.set_mappings( ['http://example.org/bbb','/tmp/this_does_not_exist'] )
-        self.assertRaises( ClientFatalError, c.baseline_or_audit )
-
-    def test06_write_capability_list(self):
-        c = Client()
-        with capture_stdout() as capturer:
-            c.write_capability_list( { 'a':'uri_a', 'b':'uri_b' } )
-        self.assertTrue( re.search(r'<urlset ',capturer.result) )
-        self.assertTrue( re.search(r'<rs:md capability="capabilitylist" />',capturer.result) )
-        self.assertTrue( re.search(r'<url><loc>uri_a</loc><rs:md capability="a"',capturer.result) )
-        self.assertTrue( re.search(r'<url><loc>uri_b</loc><rs:md capability="b"',capturer.result) )
-
-    def test07_write_source_description(self):
-        c = Client()
-        with capture_stdout() as capturer:
-            c.write_source_description( [ 'a','b','c' ] )
-        #print capturer.result
-        self.assertTrue( re.search(r'<urlset ',capturer.result) )
-        self.assertTrue( re.search(r'<rs:md capability="description" />',capturer.result) )
-        self.assertTrue( re.search(r'<url><loc>a</loc><rs:md capability="capabilitylist" /></url>',capturer.result) )
-        self.assertTrue( re.search(r'<url><loc>b</loc><rs:md capability="capabilitylist" /></url>',capturer.result) )
-
-    def test20_parse_document(self):
-        # Key property of the parse_document() method is that it parses the
-        # document and identifies its type
-        c = Client()
-        with capture_stdout() as capturer:
-            c.sitemap_name='resync/test/testdata/examples_from_spec/resourcesync_ex_1.xml'
-            c.parse_document()
-        self.assertTrue( re.search(r'Parsed resourcelist document with 2 entries',capturer.result) )
-        with capture_stdout() as capturer:
-            c.sitemap_name='resync/test/testdata/examples_from_spec/resourcesync_ex_17.xml'
-            c.parse_document()
-        self.assertTrue( re.search(r'Parsed resourcedump document with 3 entries',capturer.result) )
-        with capture_stdout() as capturer:
-            c.sitemap_name='resync/test/testdata/examples_from_spec/resourcesync_ex_19.xml'
-            c.parse_document()
-        self.assertTrue( re.search(r'Parsed changelist document with 4 entries',capturer.result) )
-        with capture_stdout() as capturer:
-            c.sitemap_name='resync/test/testdata/examples_from_spec/resourcesync_ex_22.xml'
-            c.parse_document()
-        self.assertTrue( re.search(r'Parsed changedump document with 3 entries',capturer.result) )
-
-    def test40_write_resource_list_mappings(self):
-        c = Client()
-        c.set_mappings( ['http://example.org/','resync/test/testdata/'] )
-        # with no explicit paths seting the mapping will be used
-        with capture_stdout() as capturer:
-            c.write_resource_list()
-        #sys.stderr.write(capturer.result)
-        self.assertTrue( re.search(r'<rs:md at="\S+" capability="resourcelist"', capturer.result ) )
-        self.assertTrue( re.search(r'<url><loc>http://example.org/dir1/file_a</loc>', capturer.result ) )
-        self.assertTrue( re.search(r'<url><loc>http://example.org/dir1/file_b</loc>', capturer.result ) )
-        self.assertTrue( re.search(r'<url><loc>http://example.org/dir2/file_x</loc>', capturer.result ) )
-
-    def test41_write_resource_list_path(self):
-        c = Client()
-        c.set_mappings( ['http://example.org/','resync/test/testdata/'] )
-        # with an explicit paths setting only the specified paths will be included
-        with capture_stdout() as capturer:
-            c.write_resource_list(paths='resync/test/testdata/dir1')
-        sys.stderr.write(capturer.result)
-        self.assertTrue( re.search(r'<rs:md at="\S+" capability="resourcelist"', capturer.result ) )
-        self.assertTrue( re.search(r'<url><loc>http://example.org/dir1/file_a</loc>', capturer.result ) )
-        self.assertTrue( re.search(r'<url><loc>http://example.org/dir1/file_b</loc>', capturer.result ) )
-        self.assertFalse( re.search(r'<url><loc>http://example.org/dir2/file_x</loc>', capturer.result ) )
-        # Travis CI does not preserve timestamps from github so test here for the file
-        # size but not the datestamp
-        #self.assertTrue( re.search(r'<url><loc>http://example.org/dir1/file_a</loc><lastmod>[\w\-:]+</lastmod><rs:md length="20" /></url>', capturer.result ) )
-        #self.assertTrue( re.search(r'<url><loc>http://example.org/dir1/file_b</loc><lastmod>[\w\-:]+</lastmod><rs:md length="45" /></url>', capturer.result ) )
-
-if __name__ == '__main__':
-    suite = unittest.TestLoader().loadTestsFromTestCase(TestClient)
-    unittest.TextTestRunner(verbosity=2).run(suite)

resync/test/test_client_utils.py

@@ -1,48 +0,0 @@
-import unittest
-from resync.client_utils import count_true_args,parse_links,parse_link,parse_capabilities,parse_capability_lists
-from resync.client import ClientFatalError
-
-class TestClientUtils(unittest.TestCase):
-
-    def test01_count_true_args(self):
-        self.assertEqual( count_true_args(), 0 )
-        self.assertEqual( count_true_args(True), 1 )
-        self.assertEqual( count_true_args(False), 0 )
-        self.assertEqual( count_true_args(0,1,2,3), 3 )
-
-    def test02_parse_links(self):
-        self.assertEqual( parse_links( [] ), [] )
-        self.assertEqual( parse_links( ['u,h'] ), [{'href': 'h', 'rel': 'u'}] )
-        self.assertEqual( parse_links( ['u,h','v,i'] ), [{'href': 'h', 'rel': 'u'},{'href': 'i', 'rel': 'v'}] )
-        self.assertRaises( ClientFatalError, parse_links, 'xx' )
-        self.assertRaises( ClientFatalError, parse_links, ['u'] )
-        self.assertRaises( ClientFatalError, parse_links, ['u,h','u'] )
-
-    def test03_parse_link(self):
-        # Input string of the form: rel,href,att1=val1,att2=val2
-        self.assertEqual( parse_link('u,h'), {'href': 'h', 'rel': 'u'} )
-        self.assertEqual( parse_link('u,h,a=b'), {'a': 'b', 'href': 'h', 'rel': 'u'} )
-        self.assertEqual( parse_link('u,h,a=b,c=d'), {'a': 'b', 'c': 'd', 'href': 'h', 'rel': 'u'} )
-        self.assertEqual( parse_link('u,h,a=b,a=d'), {'a': 'd', 'href': 'h', 'rel': 'u'} ) # desired??
-        self.assertRaises( ClientFatalError, parse_link, '' )
-        self.assertRaises( ClientFatalError, parse_link, 'u' )
-        self.assertRaises( ClientFatalError, parse_link, 'u,' )
-        self.assertRaises( ClientFatalError, parse_link, 'u,h,,' )
-        self.assertRaises( ClientFatalError, parse_link, 'u,h,a' )
-        self.assertRaises( ClientFatalError, parse_link, 'u,h,a=' )
-        self.assertRaises( ClientFatalError, parse_link, 'u,h,a=b,=c' )
-
-    def test04_parse_capabilities(self):
-        # Input string of the form: cap_name=uri,cap_name=uri 
-        self.assertRaises( ClientFatalError, parse_capabilities, 'a' )
-        #self.assertRaises( ClientFatalError, parse_capabilities, 'a=' )
-        self.assertRaises( ClientFatalError, parse_capabilities, 'a=b,' )
-        #self.assertRaises( ClientFatalError, parse_capabilities, 'a=b,c=' )
-
-    def test05_parse_capability_lists(self):
-        # Input string of the form: uri,uri
-        self.assertEqual( parse_capability_lists('a,b'), ['a','b'] )
-
-if __name__ == '__main__':
-    suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestClientUtils)
-    unittest.TextTestRunner(verbosity=2).run(suite)

resync/test/test_dump.py

@@ -1,169 +0,0 @@
-import os.path
-import unittest
-import tempfile
-import shutil
-import sys
-import zipfile
-from resync.dump import Dump, DumpError
-from resync.resource_list import ResourceList
-from resync.change_list import ChangeList
-from resync.resource import Resource
-from resync.resource_dump_manifest import ResourceDumpManifest
-from resync.change_dump_manifest import ChangeDumpManifest
-
-class TestDump(unittest.TestCase):
-
-    _tmpdir=None
-
-    @classmethod
-    def setUpClass(cls):
-        # Create tmp dir to write to and check
-        cls._tmpdir=tempfile.mkdtemp()
-        if (not os.path.isdir(cls._tmpdir)):
-            raise Exception("Failed to create tempdir to use for dump tests")
-
-    @classmethod
-    def tearDownClass(cls):
-        # Cleanup
-        if (not os.path.isdir(cls._tmpdir)):
-            raise Exception("Ooops, no tempdir (%s) to clean up?" % (tmpdir))
-        shutil.rmtree(cls._tmpdir)
-
-    @property
-    def tmpdir(self):
-        # read-only access to _tmpdir, just in case... The rmtree scares me
-        return(self._tmpdir)
-
-    def test00_dump_zip_resource_list(self):
-        rl=ResourceDumpManifest()
-        rl.add( Resource('http://ex.org/a', length=7, path='resync/test/testdata/a') )
-        rl.add( Resource('http://ex.org/b', length=21, path='resync/test/testdata/b') )
-        d=Dump()
-        zipf=os.path.join(self.tmpdir,"test00_dump.zip")
-        d.write_zip(resources=rl,dumpfile=zipf) # named args
-        self.assertTrue( os.path.exists(zipf) )
-        self.assertTrue( zipfile.is_zipfile(zipf) )
-        zo=zipfile.ZipFile(zipf,'r')
-        self.assertEqual( len(zo.namelist()), 3 )
-        zo.close()
-        os.unlink(zipf)
-
-    def test01_dump_zip_change_list(self):
-        cl=ChangeDumpManifest()
-        cl.add( Resource('http://ex.org/a', length=7, path='resync/test/testdata/a', change="updated") )
-        cl.add( Resource('http://ex.org/b', length=21, path='resync/test/testdata/b', change="updated") )
-        d=Dump()
-        zipf=os.path.join(self.tmpdir,"test01_dump.zip")
-        d.write_zip(cl,zipf) # positional args
-        self.assertTrue( os.path.exists(zipf) )
-        self.assertTrue( zipfile.is_zipfile(zipf) )
-        zo=zipfile.ZipFile(zipf,'r')
-        self.assertEqual( len(zo.namelist()), 3 )
-        zo.close()
-        os.unlink(zipf)
-
-    def test02_dump_check_files(self):
-        cl=ChangeList()
-        cl.add( Resource('http://ex.org/a', length=7, path='resync/test/testdata/a', change="updated") )
-        cl.add( Resource('http://ex.org/b', length=21, path='resync/test/testdata/b', change="updated") )
-        d=Dump(resources=cl)
-        self.assertTrue(d.check_files())
-        self.assertEqual(d.total_size, 28)
-        
-    def test03_dump_multi_file_max_size(self):
-        rl=ResourceList()
-        for letter in map(chr,xrange(ord('a'),ord('l')+1)):
-            uri='http://ex.org/%s' % (letter)
-            fname='resync/test/testdata/a_to_z/%s' % (letter)
-            rl.add( Resource(uri, path=fname) )
-        self.assertEqual( len(rl), 12 )
-        #d=Dump(rl) 
-        #tmpdir=tempfile.mkdtemp()
-        #tmpbase=os.path.join(tmpdir,'base')
-        #d.max_size=2000 # start new zip after size exceeds 2000 bytes
-        #n=d.write(tmpbase)
-        #self.assertEqual( n, 2, 'expect to write 2 dump files' )
-        # 
-        # Now repeat with large size limit but small number of files limit
-        d2=Dump(rl) 
-        tmpbase=os.path.join(self.tmpdir,'test03_')
-        d2.max_files=4
-        n=d2.write(tmpbase)
-        self.assertEqual( n, 3, 'expect to write 3 dump files' )
-        self.assertTrue( os.path.isfile(tmpbase+'00000.zip') )
-        self.assertTrue( os.path.isfile(tmpbase+'00001.zip') )
-        self.assertTrue( os.path.isfile(tmpbase+'00002.zip') )
-        # Look at the first file in detail
-        zipf=tmpbase+'00000.zip'
-        zo=zipfile.ZipFile(zipf,'r')
-        self.assertEqual( zo.namelist(), ['manifest.xml','a','b','c','d'] )
-        #self.assertEqual( zo.getinfo('manifest.xml').file_size, 470 )
-        self.assertEqual( zo.getinfo('a').file_size, 9 )
-        self.assertEqual( zo.getinfo('b').file_size, 1116 )
-        self.assertEqual( zo.getinfo('c').file_size, 32 )
-        self.assertEqual( zo.getinfo('d').file_size, 13 )
-        zo.close()
-        os.unlink(zipf)
-        # Check second and third files have expected contents
-        zipf=tmpbase+'00001.zip'
-        zo=zipfile.ZipFile(zipf,'r')
-        self.assertEqual( zo.namelist(), ['manifest.xml','e','f','g','h'] )
-        zo.close()
-        os.unlink(zipf)
-        zipf=tmpbase+'00002.zip'
-        zo=zipfile.ZipFile(zipf,'r')
-        self.assertEqual( zo.namelist(), ['manifest.xml','i','j','k','l'] )
-        zo.close()
-        os.unlink(zipf)
-
-    def test04_dump_multi_file_max_size(self):
-        rl=ResourceList()
-        for letter in map(chr,xrange(ord('a'),ord('l')+1)):
-            uri='http://ex.org/%s' % (letter)
-            fname='resync/test/testdata/a_to_z/%s' % (letter)
-            rl.add( Resource(uri, path=fname) )
-        self.assertEqual( len(rl), 12 )
-        d2=Dump(rl) 
-        tmpbase=os.path.join(self.tmpdir,'test0f_')
-        d2.max_size=2000
-        n=d2.write(tmpbase)
-        self.assertEqual( n, 2, 'expect to write 2 dump files' )
-        self.assertTrue( os.path.isfile(tmpbase+'00000.zip') )
-        self.assertTrue( os.path.isfile(tmpbase+'00001.zip') )
-        # Look at the first file in detail
-        zipf=tmpbase+'00000.zip'
-        zo=zipfile.ZipFile(zipf,'r')
-        self.assertEqual( zo.namelist(), ['manifest.xml','a','b','c','d','e','f'] )
-        #self.assertEqual( zo.getinfo('manifest.xml').file_size, 470 )
-        self.assertEqual( zo.getinfo('a').file_size, 9 )
-        self.assertEqual( zo.getinfo('b').file_size, 1116 )
-        self.assertEqual( zo.getinfo('c').file_size, 32 )
-        self.assertEqual( zo.getinfo('d').file_size, 13 )
-        self.assertEqual( zo.getinfo('e').file_size, 20 )
-        self.assertEqual( zo.getinfo('f').file_size, 1625 )
-        zo.close()
-        os.unlink(zipf)
-        # Check second and third files have expected contents
-        zipf=tmpbase+'00001.zip'
-        zo=zipfile.ZipFile(zipf,'r')
-        self.assertEqual( zo.namelist(), ['manifest.xml','g','h','i','j','k','l'] )
-        zo.close()
-        os.unlink(zipf)
-
-    def test10_no_path(self):
-        rl=ResourceList()
-        rl.add( Resource('http://ex.org/a', length=7, path='resync/test/testdata/a') )
-        rl.add( Resource('http://ex.org/b', length=21 ) )
-        d=Dump(rl)
-        self.assertRaises( DumpError, d.check_files )
-
-    def test11_bad_size(self):
-        rl=ResourceList()
-        rl.add( Resource('http://ex.org/a', length=9999, path='resync/test/testdata/a') )
-        d=Dump(rl)
-        self.assertTrue( d.check_files(check_length=False) )
-        self.assertRaises( DumpError, d.check_files )
-
-if __name__ == '__main__':
-    suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestDump)
-    unittest.TextTestRunner(verbosity=2).run(suite)

resync/test/test_examples_from_spec.py

@@ -1,854 +0,0 @@
-"""Tests on the examples in the specification documents
-"""
-import re
-import sys
-import unittest
-from resync.resource import Resource
-from resync.source_description import SourceDescription
-from resync.capability_list import CapabilityList
-from resync.resource_list import ResourceList,ResourceListOrdered
-from resync.resource_dump import ResourceDump
-from resync.resource_dump_manifest import ResourceDumpManifest
-from resync.change_list import ChangeList
-from resync.change_dump import ChangeDump
-from resync.change_dump_manifest import ChangeDumpManifest
-from resync.archives import ResourceListArchive,ResourceDumpArchive,ChangeListArchive,ChangeDumpArchive
-from resync.sitemap import Sitemap
-
-class TestExamplesFromSpec(unittest.TestCase):
-
-    def test_all_simple_read(self):
-        """Just try to read each one"""
-        for ex in ('archives_ex_2_1','archives_ex_2_2',
-                   'archives_ex_3_1','archives_ex_3_2',
-                   'archives_ex_4_1',
-                   'archives_ex_5_1',
-                   'archives_ex_6_1',
-                   'resourcesync_ex_1','resourcesync_ex_2','resourcesync_ex_3',
-                   'resourcesync_ex_4','resourcesync_ex_5','resourcesync_ex_6',
-                   'resourcesync_ex_7','resourcesync_ex_8','resourcesync_ex_12',
-                   'resourcesync_ex_13','resourcesync_ex_14','resourcesync_ex_15',
-                   'resourcesync_ex_16','resourcesync_ex_17','resourcesync_ex_18',
-                   'resourcesync_ex_19','resourcesync_ex_20','resourcesync_ex_21',
-                   'resourcesync_ex_22','resourcesync_ex_23','resourcesync_ex_24',
-                   'resourcesync_ex_25','resourcesync_ex_26','resourcesync_ex_27',
-                   'resourcesync_ex_28','resourcesync_ex_29','resourcesync_ex_30',
-                   'resourcesync_ex_31','resourcesync_ex_32','resourcesync_ex_33'):
-            s=Sitemap()
-            fh = self._open_ex(ex)
-            si = s.parse_xml( fh=fh )
-
-    def test_ex_01(self):
-        """resourcesync_ex_1 is a simple resource_list with 2 resources, no metadata"""
-        rl=ResourceList()
-        rl.parse(uri='resync/test/testdata/examples_from_spec/resourcesync_ex_1.xml')
-        self.assertEqual( rl.capability, 'resourcelist' )
-        self.assertEqual( len(rl.resources), 2, '2 resources')
-        sms = sorted(rl.uris())
-        self.assertEqual( sms, ['http://example.com/res1','http://example.com/res2'] )
-        self.assertEqual( rl.resources['http://example.com/res1'].lastmod, None )
-
-    def test_ex_02(self):
-        """resourcesync_ex_2 is a simple resource_list with 2 resources, some metadata"""
-        rl=ResourceList()
-        rl.parse(uri='resync/test/testdata/examples_from_spec/resourcesync_ex_2.xml')
-        self.assertEqual( len(rl.resources), 2, '2 resources')
-        sms = sorted(rl.uris())
-        self.assertEqual( sms, ['http://example.com/res1','http://example.com/res2'] )
-        self.assertEqual( rl.resources['http://example.com/res1'].lastmod, '2013-01-02T13:00:00Z' )
-        self.assertEqual( rl.resources['http://example.com/res2'].lastmod, '2013-01-02T14:00:00Z' )
-        self.assertEqual( rl.resources['http://example.com/res1'].md5, '1584abdf8ebdc9802ac0c6a7402c03b6' )
-        self.assertEqual( rl.resources['http://example.com/res2'].md5, '1e0d5cb8ef6ba40c99b14c0237be735e' )
-
-    def test_ex_03(self):
-        """resourcesync_ex_3 is a simple change_list with 2 resources"""
-        cl=ChangeList()
-        cl.parse('resync/test/testdata/examples_from_spec/resourcesync_ex_3.xml')
-        self.assertEqual( len(cl.resources), 2, '2 resources')
-        sms = sorted(cl.uris())
-        self.assertEqual( sms, ['http://example.com/res2.pdf','http://example.com/res3.tiff'] )
-        self.assertEqual( cl.resources[0].lastmod, '2013-01-02T13:00:00Z' )
-        self.assertEqual( cl.resources[1].lastmod, '2013-01-02T18:00:00Z' )
-        self.assertEqual( cl.resources[0].change, 'updated' )
-        self.assertEqual( cl.resources[1].change, 'deleted' )
-
-    def test_ex_04(self):
-        """resourcesync_ex_4 is a simple resource dump with one ZIP listed"""
-        rd=ResourceDump()
-        rd.parse('resync/test/testdata/examples_from_spec/resourcesync_ex_4.xml')
-        self.assertEqual( len(rd.resources), 1, '1 resources')
-        self.assertTrue( 'http://example.com/resourcedump.zip' in rd.resources )
-        self.assertEqual( rd.resources['http://example.com/resourcedump.zip'].lastmod, '2013-01-03T09:00:00Z' )
-
-    def test_ex_05(self):
-        """resourcesync_ex_5 is a simple resource dump manifest with two files listed"""
-        rdm=ResourceDumpManifest()
-        rdm.parse('resync/test/testdata/examples_from_spec/resourcesync_ex_5.xml')
-        self.assertEqual( len(rdm.resources), 2, '2 resources')
-        sms = sorted(rdm.uris())
-        self.assertEqual( sms, ['http://example.com/res1','http://example.com/res2'] )
-        self.assertEqual( rdm.resources['http://example.com/res1'].lastmod, '2013-01-03T03:00:00Z' )
-        self.assertEqual( rdm.resources['http://example.com/res1'].md5, '1584abdf8ebdc9802ac0c6a7402c03b6' )
-        self.assertEqual( rdm.resources['http://example.com/res1'].path, '/resources/res1' )
-        self.assertEqual( rdm.resources['http://example.com/res2'].lastmod, '2013-01-03T04:00:00Z' )
-        self.assertEqual( rdm.resources['http://example.com/res2'].md5, '1e0d5cb8ef6ba40c99b14c0237be735e' )
-        self.assertEqual( rdm.resources['http://example.com/res2'].path, '/resources/res2' )
-
-    def test_ex_06(self):
-        """resourcesync_ex_6 is a simple capability list with three capabilities"""
-        capl=CapabilityList()
-        capl.parse('resync/test/testdata/examples_from_spec/resourcesync_ex_6.xml')
-        self.assertEqual( len(capl.resources), 3, '3 capabilities')
-        # What capabilities are present?
-        self.assertTrue( capl.has_capability('resourcelist') )
-        self.assertEqual( capl.capability_info('resourcelist').uri, 'http://example.com/dataset1/resourcelist.xml')
-        self.assertTrue( capl.has_capability('resourcedump') )
-        self.assertEqual( capl.capability_info('resourcedump').uri, 'http://example.com/dataset1/resourcedump.xml')
-        self.assertTrue( capl.has_capability('changelist') )
-        self.assertEqual( capl.capability_info('changelist').uri, 'http://example.com/dataset1/changelist.xml')
-        # Check some that aren't
-        self.assertFalse( capl.has_capability() )
-        self.assertFalse( capl.has_capability('bogus') )
-        self.assertFalse( capl.has_capability('capabilitylist') )
-
-    def text_ex_07(self):
-        """resourcesync_ex_7 is a source description that list a single Capability List"""
-        sd=SourceDescription()
-        sd.read(uri='resync/test/testdata/examples_from_spec/resourcesync_ex_7.xml')
-        self.assertEqual( len(sd.resources), 1, '1 capability list' )
-        cl=sd.resources[0]
-        self.assertEqual( cl.uri, 'http://example.com/dataset1/capabilitylist.xml' )
-        self.assertEqual( cl.capability, 'resourcelist' )
-        self.assertEqual( cl.describedby, 'http://example.com/info_about_set1_of_resources.xml' )
-
-    def test_ex_08(self):
-        """resourcesync_ex_8 is a simple Resource List Index with 2 Resource Lists"""
-        rl=ResourceList()
-        rl.read(uri='resync/test/testdata/examples_from_spec/resourcesync_ex_8.xml',index_only=True)
-        self.assertEqual( rl.capability, 'resourcelist' )
-        self.assertEqual( rl.md_at, '2013-01-03T09:00:00Z' )
-        self.assertEqual( len(rl.resources), 2, '2 resources')
-        sms = sorted(rl.uris())
-        self.assertEqual( sms, ['http://example.com/resourcelist-part1.xml',
-                                'http://example.com/resourcelist-part2.xml'] )
-
-    # Examples 9, 10, 11 in the spec are not XML documents
-
-    def test_ex_12(self):
-        """resourcesync_ex_12 is a Source Description that talks about 3 sets of resources"""
-        sd=SourceDescription()
-        sd.read(uri='resync/test/testdata/examples_from_spec/resourcesync_ex_12.xml')
-        self.assertEqual( len(sd), 3 )
-        self.assertEqual( sd.uris(), ['http://example.com/capabilitylist1.xml',
-                                      'http://example.com/capabilitylist2.xml',
-                                      'http://example.com/capabilitylist3.xml'] )
-        cl1=sd['http://example.com/capabilitylist1.xml']
-        self.assertEqual( cl1.capability, 'capabilitylist' )
-        self.assertEqual( cl1.describedby, 'http://example.com/info_about_set1_of_resources.xml')
-
-
-##### BUILD EXAMPLES #####
-
-    def test_build_ex_01(self):
-        """Simple Resource List document """
-        rl = ResourceList()
-        rl.md_at = '2013-01-03T09:00:00Z'
-        rl.add( Resource('http://example.com/res1') )
-        rl.add( Resource('http://example.com/res2') )
-        ex_xml = self._open_ex('resourcesync_ex_1').read()
-        self._assert_xml_equal( rl.as_xml(), ex_xml )
-
-    def test_build_ex_02(self):
-        """Slightly more complex Resource List document """
-        rl = ResourceList()
-        rl.md_at = '2013-01-03T09:00:00Z'
-        rl.add( Resource(uri='http://example.com/res1',
-                         lastmod='2013-01-02T13:00:00Z',
-                         md5='1584abdf8ebdc9802ac0c6a7402c03b6') )
-        r2 =  Resource(uri='http://example.com/res2',
-                       lastmod='2013-01-02T14:00:00Z',
-                       md5='1e0d5cb8ef6ba40c99b14c0237be735e')
-        r2.link_set(rel="duplicate",href="http://mirror.example.com/res2")
-        rl.add( r2 )
-        ex_xml = self._open_ex('resourcesync_ex_2').read()
-        self._assert_xml_equal( rl.as_xml(), ex_xml )
-
-    def test_build_ex_03(self):
-        """Simple Change List document """
-        cl = ChangeList()
-        cl.md_from = '2013-01-02T00:00:00Z'
-        cl.md_until= '2013-01-03T00:00:00Z'
-        cl.add( Resource(uri='http://example.com/res2.pdf',
-                         lastmod='2013-01-02T13:00:00Z',
-                         change="updated") )
-        cl.add( Resource(uri='http://example.com/res3.tiff',
-                         lastmod='2013-01-02T18:00:00Z',
-                         change='deleted') )
-        ex_xml = self._open_ex('resourcesync_ex_3').read()
-        self._assert_xml_equal( cl.as_xml(), ex_xml )
-
-    def test_build_ex_04(self):
-        """Simple Resource Dump document """
-        rd = ResourceDump()
-        rd.md_at = '2013-01-03T09:00:00Z'
-        rd.add( Resource(uri='http://example.com/resourcedump.zip',
-                         lastmod='2013-01-03T09:00:00Z') )
-        ex_xml = self._open_ex('resourcesync_ex_4').read()
-        self._assert_xml_equal( rd.as_xml(), ex_xml )
-
-    def test_build_ex_05(self):
-        """Simple Resource Dump Manifest document """
-        rdm = ResourceDumpManifest()
-        rdm.md_at = '2013-01-03T09:00:00Z'
-        rdm.add( Resource(uri='http://example.com/res1',
-                          lastmod='2013-01-03T03:00:00Z',
-                          md5='1584abdf8ebdc9802ac0c6a7402c03b6',
-                          path='/resources/res1') )
-        rdm.add( Resource(uri='http://example.com/res2',
-                          lastmod='2013-01-03T04:00:00Z',
-                          md5='1e0d5cb8ef6ba40c99b14c0237be735e',
-                          path='/resources/res2') )
-        ex_xml = self._open_ex('resourcesync_ex_5').read()
-        self._assert_xml_equal( rdm.as_xml(), ex_xml )
-
-    def test_build_ex_06(self):
-        """Simple Capability List document """
-        cl = CapabilityList()
-        cl.describedby = 'http://example.com/info_about_set1_of_resources.xml'
-        cl.up = 'http://example.com/resourcesync_description.xml'
-        cl.add_capability( uri='http://example.com/dataset1/resourcelist.xml', name='resourcelist' )
-        cl.add_capability( uri='http://example.com/dataset1/resourcedump.xml', name='resourcedump' )
-        cl.add_capability( uri='http://example.com/dataset1/changelist.xml', name='changelist' )
-        ex_xml = self._open_ex('resourcesync_ex_6').read()
-        self._assert_xml_equal( cl.as_xml(), ex_xml )
-
-    def test_build_ex_07(self):
-        """A Source Description document """
-        sd = SourceDescription()
-        sd.describedby = 'http://example.com/info-about-source.xml'
-        r = Resource( uri='http://example.com/dataset1/capabilitylist.xml',
-                      capability='capabilitylist' )
-        r.link_set( rel='describedby',
-                    href='http://example.com/info_about_set1_of_resources.xml' )
-        sd.add( r )
-        ex_xml = self._open_ex('resourcesync_ex_7').read()
-        self._assert_xml_equal( sd.as_xml(), ex_xml )
-
-    def test_build_ex_08(self):
-        """Simple Resource List Index document
-        
-        This is not something that would usually be created directly 
-        but instead would be created as part of the process of 
-        writing a large Resource List in multiple files. However,
-        it is possible to create manually.
-        """
-        rli = ResourceList()
-        rli.sitemapindex=True
-        rli.md_at = '2013-01-03T09:00:00Z'
-        rli.add( Resource(uri='http://example.com/resourcelist-part1.xml') )
-        rli.add( Resource(uri='http://example.com/resourcelist-part2.xml') )
-        ex_xml = self._open_ex('resourcesync_ex_8').read()
-        self._assert_xml_equal( rli.as_xml(), ex_xml )
-
-    # Examples 9, 10, 11 in the spec are not XML documents
-
-    def test_build_ex_12(self):
-        """Source Description document with describedby links"""
-        sd = SourceDescription()
-        sd.describedby = 'http://example.com/info_about_source.xml'
-        cl1 = CapabilityList( uri='http://example.com/capabilitylist1.xml' )
-        cl1.describedby = 'http://example.com/info_about_set1_of_resources.xml'
-        sd.add_capability_list( cl1 )
-        cl2 = CapabilityList( uri='http://example.com/capabilitylist2.xml' )
-        cl2.describedby = 'http://example.com/info_about_set2_of_resources.xml'
-        sd.add_capability_list( cl2 )
-        cl3 = CapabilityList( uri='http://example.com/capabilitylist3.xml' )
-        cl3.describedby = 'http://example.com/info_about_set3_of_resources.xml'
-        sd.add_capability_list( cl3 )
-        ex_xml = self._open_ex('resourcesync_ex_12').read()
-        self._assert_xml_equal( sd.as_xml(), ex_xml )
-
-    def test_build_ex_13(self):
-        """Capability List document with 4 entries"""
-        cl = CapabilityList()
-        cl.describedby = 'http://example.com/info_about_set1_of_resources.xml'
-        cl.up = 'http://example.com/resourcesync_description.xml'
-        cl.add_capability( capability=ResourceList( uri='http://example.com/dataset1/resourcelist.xml' ) )
-        cl.add_capability( capability=ResourceDump( uri='http://example.com/dataset1/resourcedump.xml' ) )
-        cl.add_capability( capability=ChangeList( uri='http://example.com/dataset1/changelist.xml' ) )
-        cl.add_capability( capability=ChangeDump( uri='http://example.com/dataset1/changedump.xml' ) )
-        ex_xml = self._open_ex('resourcesync_ex_13').read()
-        self._assert_xml_equal( cl.as_xml(), ex_xml )
-
-    def test_build_ex_14(self):
-        """Resource List with 2 entries and some metadata"""
-        rl = ResourceList()
-        rl.up='http://example.com/dataset1/capabilitylist.xml'
-        rl.md_at="2013-01-03T09:00:00Z"
-        rl.md_completed="2013-01-03T09:01:00Z"
-        rl.add( Resource( uri='http://example.com/res1',
-                          lastmod='2013-01-02T13:00:00Z',
-                          md5='1584abdf8ebdc9802ac0c6a7402c03b6',
-                          length=8876,
-                          mime_type="text/html" ))
-        rl.add( Resource( uri='http://example.com/res2',
-                          lastmod='2013-01-02T14:00:00Z',
-                          md5='1e0d5cb8ef6ba40c99b14c0237be735e',
-                          sha256='854f61290e2e197a11bc91063afce22e43f8ccc655237050ace766adc68dc784',
-                          length=14599,
-                          mime_type="application/pdf" ))
-        ex_xml = self._open_ex('resourcesync_ex_14').read()
-        self._assert_xml_equal( rl.as_xml(), ex_xml )
-
-    def test_build_ex_15(self):
-        """Resource List Index with metadata"""
-        rl = ResourceList(resources_class=ResourceListOrdered) #order in example is non-canonical
-        rl.sitemapindex=True
-        rl.up='http://example.com/dataset1/capabilitylist.xml'
-        rl.md_at="2013-01-03T09:00:00Z"
-        rl.md_completed="2013-01-03T09:10:00Z"
-        rl.add( Resource( uri='http://example.com/resourcelist1.xml',
-                          md_at='2013-01-03T09:00:00Z' ))
-        rl.add( Resource( uri='http://example.com/resourcelist2.xml',
-                          md_at='2013-01-03T09:03:00Z' ))
-        rl.add( Resource( uri='http://example.com/resourcelist3.xml',
-                          md_at='2013-01-03T09:07:00Z' ))
-        ex_xml = self._open_ex('resourcesync_ex_15').read()
-        self._assert_xml_equal( rl.as_xml(), ex_xml )
-
-    def test_build_ex_16(self):
-        rl = ResourceList()
-        rl.up = 'http://example.com/dataset1/capabilitylist.xml'
-        rl.index = 'http://example.com/dataset1/resourcelist-index.xml'
-        rl.md_at="2013-01-03T09:00:00Z"
-        rl.add( Resource( uri='http://example.com/res3',
-                          lastmod='2013-01-02T13:00:00Z',
-                          md5='1584abdf8ebdc9802ac0c6a7402c8753',
-                          length=4385,
-                          mime_type="application/pdf" ))
-        rl.add( Resource( uri='http://example.com/res4',
-                          lastmod='2013-01-02T14:00:00Z',
-                          md5='4556abdf8ebdc9802ac0c6a7402c9881',
-                          length=883,
-                          mime_type="image/png" ))
-        ex_xml = self._open_ex('resourcesync_ex_16').read()
-        self._assert_xml_equal( rl.as_xml(), ex_xml )
-
-    def test_build_ex_17(self):
-        """Resource Dump with 3 entries and some metadata"""
-        rd = ResourceDump()
-        rd.up='http://example.com/dataset1/capabilitylist.xml'
-        rd.md_at="2013-01-03T09:00:00Z"
-        rd.md_completed="2013-01-03T09:04:00Z"
-        z1 = Resource( uri='http://example.com/resourcedump-part1.zip',
-                       mime_type="application/zip",
-                       length=4765,
-                       md_at="2013-01-03T09:00:00Z",
-                       md_completed="2013-01-03T09:02:00Z" )
-        z1.link_set( rel="contents",
-                     href="http://example.com/resourcedump_manifest-part1.xml",
-                     mime_type="application/xml" )
-        rd.add( z1 )
-        z2 = Resource( uri='http://example.com/resourcedump-part2.zip',
-                       mime_type="application/zip",
-                       length=9875,
-                       md_at="2013-01-03T09:01:00Z",
-                       md_completed="2013-01-03T09:03:00Z" )
-        z2.link_set( rel="contents",
-                     href="http://example.com/resourcedump_manifest-part2.xml",
-                     mime_type="application/xml" )
-        rd.add( z2 )
-        z3 = Resource( uri='http://example.com/resourcedump-part3.zip',
-                       mime_type="application/zip",
-                       length=2298,
-                       md_at="2013-01-03T09:03:00Z",
-                       md_completed="2013-01-03T09:04:00Z" )
-        z3.link_set( rel="contents",
-                     href="http://example.com/resourcedump_manifest-part3.xml",
-                     mime_type="application/xml" )
-        rd.add( z3 )
-        ex_xml = self._open_ex('resourcesync_ex_17').read()
-        self._assert_xml_equal( rd.as_xml(), ex_xml )
-
-    def test_build_ex_18(self):
-        """Resource Dump Manifest with 2 entries and some metadata"""
-        rdm = ResourceDumpManifest()
-        rdm.up='http://example.com/dataset1/capabilitylist.xml'
-        rdm.md_at="2013-01-03T09:00:00Z"
-        rdm.md_completed="2013-01-03T09:02:00Z"
-        rdm.add( Resource( uri='http://example.com/res1',
-                           lastmod='2013-01-02T13:00:00Z',
-                           md5='1584abdf8ebdc9802ac0c6a7402c03b6',
-                           length=8876,
-                           mime_type='text/html',
-                           path='/resources/res1') )
-        rdm.add( Resource( uri='http://example.com/res2',
-                           lastmod='2013-01-02T14:00:00Z',
-                           md5='1e0d5cb8ef6ba40c99b14c0237be735e',
-                           sha256='854f61290e2e197a11bc91063afce22e43f8ccc655237050ace766adc68dc784',
-                           length=14599,
-                           mime_type='application/pdf',
-                           path='/resources/res2') )
-        ex_xml = self._open_ex('resourcesync_ex_18').read()
-        self._assert_xml_equal( rdm.as_xml(), ex_xml )
-
-    def test_build_ex_19(self):
-        """Change List with 4 changes, 'open' as no until"""
-        cl = ChangeList()
-        cl.up = 'http://example.com/dataset1/capabilitylist.xml'
-        cl.md_from="2013-01-03T00:00:00Z"
-        cl.add( Resource( uri='http://example.com/res1.html',
-                          lastmod='2013-01-03T11:00:00Z',
-                          change='created' ) )
-        cl.add( Resource( uri='http://example.com/res2.pdf',
-                          lastmod='2013-01-03T13:00:00Z',
-                          change='updated' ) )
-        cl.add( Resource( uri='http://example.com/res3.tiff',
-                          lastmod='2013-01-03T18:00:00Z',
-                          change='deleted' ) )
-        cl.add( Resource( uri='http://example.com/res2.pdf',
-                          lastmod='2013-01-03T21:00:00Z',
-                          change='updated' ) )
-        ex_xml = self._open_ex('resourcesync_ex_19').read()
-        self._assert_xml_equal( cl.as_xml(), ex_xml )
-
-    def test_build_ex_20(self):
-        """Change List Index listing 3 Change Lists, the last one 'open'"""
-        cl = ChangeListArchive(resources_class=ResourceListOrdered) #order in example is non-canonical   
-        cl.sitemapindex=True
-        cl.capability_name='changelist'
-        cl.up = 'http://example.com/dataset1/capabilitylist.xml'
-        cl.md_from="2013-01-01T00:00:00Z"
-        cl.add( Resource( uri='http://example.com/20130101-changelist.xml',
-                          md_from='2013-01-01T00:00:00Z',
-                          md_until='2013-01-02T00:00:00Z') )
-        cl.add( Resource( uri='http://example.com/20130102-changelist.xml',
-                          md_from='2013-01-02T00:00:00Z',
-                          md_until='2013-01-03T00:00:00Z') )
-        cl.add( Resource( uri='http://example.com/20130103-changelist.xml',
-                          md_from='2013-01-03T00:00:00Z') )
-        ex_xml = self._open_ex('resourcesync_ex_20').read()
-        self._assert_xml_equal( cl.as_xml(), ex_xml )
-
-    def test_build_ex_21(self):
-        """Change List which points back to index"""
-        cl = ChangeList()
-        cl.up = 'http://example.com/dataset1/capabilitylist.xml'
-        cl.index = 'http://example.com/dataset1/changelist.xml'
-        cl.md_from="2013-01-02T00:00:00Z"
-        cl.md_until="2013-01-03T00:00:00Z"
-        cl.add( Resource( uri='http://example.com/res7.html',
-                          lastmod='2013-01-02T12:00:00Z',
-                          change='created' ) )
-        cl.add( Resource( uri='http://example.com/res9.pdf',
-                          lastmod='2013-01-02T13:00:00Z',
-                          change='updated' ) )
-        cl.add( Resource( uri='http://example.com/res5.tiff',
-                          lastmod='2013-01-02T19:00:00Z',
-                          change='deleted' ) )
-        cl.add( Resource( uri='http://example.com/res7.html',
-                          lastmod='2013-01-02T20:00:00Z',
-                          change='updated' ) )
-        ex_xml = self._open_ex('resourcesync_ex_21').read()
-        self._assert_xml_equal( cl.as_xml(), ex_xml )
-
-    def test_build_ex_22(self):
-        """Change Dump with three dump files"""
-        cd = ChangeDump()
-        cd.up = 'http://example.com/dataset1/capabilitylist.xml'
-        cd.md_from="2013-01-01T00:00:00Z"
-        z1 = Resource( uri='http://example.com/20130101-changedump.zip',
-                       lastmod='2013-01-01T23:59:59Z',
-                       length=3109,
-                       md_from="2013-01-01T00:00:00Z",
-                       md_until="2013-01-02T00:00:00Z",
-                       mime_type="application/zip" )
-        z1.contents='http://example.com/20130101-changedump-manifest.xml'
-        z2 = Resource( uri='http://example.com/20130102-changedump.zip',
-                       lastmod='2013-01-02T23:59:59Z',
-                       length=6629,
-                       md_from="2013-01-02T00:00:00Z",
-                       md_until="2013-01-03T00:00:00Z",
-                       mime_type="application/zip" )
-        z2.contents='http://example.com/20130102-changedump-manifest.xml'
-        z3 = Resource( uri='http://example.com/20130103-changedump.zip',
-                       lastmod='2013-01-03T23:59:59Z',
-                       length=8124,
-                       md_from="2013-01-03T00:00:00Z",
-                       md_until="2013-01-04T00:00:00Z",
-                       mime_type="application/zip" )
-        z3.contents='http://example.com/20130103-changedump-manifest.xml'
-        cd.add( [z1, z2, z3] )
-        ex_xml = self._open_ex('resourcesync_ex_22').read()
-        self._assert_xml_equal( cd.as_xml(), ex_xml ) 
-
-    def test_build_ex_23(self):
-        cdm = ChangeDumpManifest()
-        cdm.up = "http://example.com/dataset1/capabilitylist.xml"
-        cdm.md_from = "2013-01-02T00:00:00Z"
-        cdm.md_until = "2013-01-03T00:00:00Z"
-        cdm.add( Resource(uri="http://example.com/res7.html",
-                          lastmod="2013-01-02T12:00:00Z",
-                          change="created",
-                          md5="1c1b0e264fa9b7e1e9aa6f9db8d6362b",
-                          length=4339,
-                          mime_type="text/html",
-                          path="/changes/res7.html") )
-        cdm.add( Resource(uri="http://example.com/res9.pdf",
-                          lastmod="2013-01-02T13:00:00Z",
-                          change="updated",
-                          md5="f906610c3d4aa745cb2b986f25b37c5a",
-                          length=38297,
-                          mime_type="application/pdf",
-                          path="/changes/res9.pdf") )
-        cdm.add( Resource(uri="http://example.com/res5.tiff",
-                          lastmod="2013-01-02T19:00:00Z",
-                          change="deleted") )
-        cdm.add( Resource(uri="http://example.com/res7.html",
-                          lastmod="2013-01-02T20:00:00Z",
-                          change="updated",
-                          md5="0988647082c8bc51778894a48ec3b576",
-                          length="5426", #should also take string
-                          mime_type="text/html",
-                          path="/changes/res7-v2.html") )
-        self._assert_xml_equal_ex( cdm.as_xml(), 'resourcesync_ex_23' )
-
-    def test_build_ex_24(self):
-        cl = ChangeList()
-        cl.up = "http://example.com/dataset1/capabilitylist.xml"
-        cl.md_from = "2013-01-03T00:00:00Z"
-        c1 = Resource(uri="http://example.com/res1",
-                      lastmod="2013-01-03T18:00:00Z",
-                      change="updated",
-                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
-                      length=8876,
-                      mime_type="text/html")
-        # Resource.link_set with add or change link depending on one with 
-        # the particular rel exists unless allow_duplicates=True. 
-        # Resource.link_add will always add. Test both here...
-        c1.link_set(rel="duplicate",
-                    href="http://mirror1.example.com/res1",
-                    pri="1",
-                    modified="2013-01-03T18:00:00Z")
-        c1.link_set(rel="duplicate",
-                    href="http://mirror2.example.com/res1",
-                    pri="2",
-                    modified="2013-01-03T18:00:00Z",
-                    allow_duplicates=True)
-        c1.link_add(rel="duplicate",
-                    href="gsiftp://gridftp.example.com/res1",
-                    pri="3",
-                    modified="2013-01-03T18:00:00Z")
-        cl.add( c1 )
-        self._assert_xml_equal_ex( cl.as_xml(), 'resourcesync_ex_24' )
-
-    def test_build_ex_25(self):
-        cl = ChangeList()
-        cl.up = "http://example.com/dataset1/capabilitylist.xml"
-        cl.md_from = "2013-01-03T11:00:00Z"
-        c1 = Resource(uri="http://example.com/res1",
-                      lastmod="2013-01-03T18:00:00Z",
-                      change="updated")
-        c1.link_add(rel="alternate",
-                    href="http://example.com/res1.html",
-                    modified="2013-01-03T18:00:00Z",
-                    type="text/html") #FIXME - inconsistent
-        c1.link_add(rel="alternate",
-                    href="http://example.com/res1.pdf",
-                    modified="2013-01-03T18:00:00Z",
-                    type="application/pdf")
-        cl.add( c1 )
-        self._assert_xml_equal_ex( cl.as_xml(), 'resourcesync_ex_25' )
-
-    def test_build_ex_26(self):
-        cl = ChangeList()
-        cl.up = "http://example.com/dataset1/capabilitylist.xml"
-        cl.md_from = "2013-01-03T00:00:00Z"
-        c1 = Resource(uri="http://example.com/res1.html",
-                      lastmod="2013-01-03T18:00:00Z",
-                      change="updated",
-                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
-                      length=8876 )
-        c1.link_add(rel="canonical",
-                    href="http://example.com/res1",
-                    modified="2013-01-03T18:00:00Z")
-        cl.add( c1 )
-        self._assert_xml_equal_ex( cl.as_xml(), 'resourcesync_ex_26' )
-
-    def test_build_ex_27(self):
-        cl = ChangeList()
-        cl.up = "http://example.com/dataset1/capabilitylist.xml"
-        cl.md_from = "2013-01-03T00:00:00Z"
-        c1 = Resource(uri="http://example.com/res4",
-                      lastmod="2013-01-03T17:00:00Z",
-                      change="updated",
-                      sha256="f4OxZX_x_DFGFDgghgdfb6rtSx-iosjf6735432nklj",
-                      length=56778,
-                      mime_type="application/json" )
-        c1.link_set(rel="http://www.openarchives.org/rs/terms/patch",
-                    href="http://example.com/res4-json-patch",
-                    modified="2013-01-03T17:00:00Z",
-                    hash="sha-256:y66dER_t_HWEIKpesdkeb7rtSc-ippjf9823742opld", #FIXME - inconsistent
-                    length=73,
-                    type="application/json-patch")
-        c2 = Resource(uri="http://example.com/res5-full.tiff",
-                      lastmod="2013-01-03T18:00:00Z",
-                      change="updated",
-                      sha256="f4OxZX_x_FO5LcGBSKHWXfwtSx-j1ncoSt3SABJtkGk",
-                      length="9788456778",
-                      mime_type="image/tiff")
-        c2.link_set(rel="http://www.openarchives.org/rs/terms/patch",
-                    href="http://example.com/res5-diff",
-                    modified="2013-01-03T18:00:00Z",
-                    hash="sha-256:h986gT_t_87HTkjHYE76G558hY-jdfgy76t55sadJUYT",
-                    length=4533,
-                    type="application/x-tiff-diff" )
-        cl.add( [c1,c2] )
-        self._assert_xml_equal_ex( cl.as_xml(), 'resourcesync_ex_27' )
-
-    def test_build_ex_28(self):
-        cl = ChangeList()
-        cl.up = "http://example.com/dataset1/capabilitylist.xml"
-        cl.md_from = "2013-01-03T00:00:00Z"
-        c1 = Resource(uri="http://example.com/res2.pdf",
-                      lastmod="2013-01-03T18:00:00Z",
-                      change="updated",
-                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
-                      length=8876,
-                      mime_type="application/pdf" )
-        c1.link_set(rel="describedby",
-                    href="http://example.com/res2_dublin-core_metadata.xml",
-                    modified="2013-01-01T12:00:00Z",
-                    type="application/xml")
-        c2 = Resource(uri="http://example.com/res2_dublin-core_metadata.xml",
-                      lastmod="2013-01-03T19:00:00Z",
-                      change="updated",
-                      mime_type="application/xml")
-        c2.link_set(rel="describes",
-                    href="http://example.com/res2.pdf",
-                    modified="2013-01-03T18:00:00Z",
-                    hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6",
-                    length="8876",
-                    type="application/pdf")
-        c2.link_set(rel="profile",
-                    href="http://purl.org/dc/elements/1.1/")
-        cl.add( [c1,c2] )
-        self._assert_xml_equal_ex( cl.as_xml(), 'resourcesync_ex_28' )
-
-    def test_build_ex_29(self):
-        cl = ChangeList()
-        cl.up = "http://example.com/dataset1/capabilitylist.xml"
-        cl.md_from = "2013-01-03T00:00:00Z"
-        c1 = Resource(uri="http://example.com/res1",
-                      lastmod="2013-01-03T18:00:00Z",
-                      change="updated",
-                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
-                      length=8876,
-                      mime_type="text/html" )
-        c1.link_add(rel="memento",
-                    href="http://example.com/20130103070000/res1",
-                    modified="2013-01-02T18:00:00Z",
-                    hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6",
-                    length="8876",
-                    type="text/html")
-        c1.link_add(rel="timegate",
-                    href="http://example.com/timegate/http://example.com/res1")
-        c1.link_add(rel="timemap",
-                    href="http://example.com/timemap/http://example.com/res1",
-                    type="application/link-format")
-        cl.add( c1 )
-        self._assert_xml_equal_ex( cl.as_xml(), 'resourcesync_ex_29' )
-
-    def test_build_ex_30(self):
-        cl = ChangeList()
-        cl.up = "http://example.com/dataset1/capabilitylist.xml"
-        cl.md_from = "2013-01-03T00:00:00Z"
-        c1 = Resource(uri="http://example.com/res1",
-                      lastmod="2013-01-03T07:00:00Z",
-                      change="updated",
-                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
-                      length=8876,
-                      mime_type="text/html" )
-        c1.link_add(rel="collection",
-                    href="http://example.com/aggregation/0601007")
-        cl.add( c1 )
-        self._assert_xml_equal_ex( cl.as_xml(), 'resourcesync_ex_30' )
-
-    def test_build_ex_31(self):
-        cl = ChangeList()
-        cl.up = "http://example.com/dataset1/capabilitylist.xml"
-        cl.md_from = "2013-01-03T00:00:00Z"
-        c1 = Resource(uri="http://original.example.com/res1.html",
-                      lastmod="2013-01-03T07:00:00Z",
-                      change="updated",
-                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
-                      length=8876,
-                      mime_type="text/html" )
-        cl.add( c1 )
-        self._assert_xml_equal_ex( cl.as_xml(), 'resourcesync_ex_31' )
-
-    def test_build_ex_32(self):
-        cl = ChangeList()
-        cl.up = "http://aggregator1.example.com/dataset1/capabilitylist.xml"
-        cl.md_from = "2013-01-03T11:00:00Z"
-        c1 = Resource(uri="http://aggregator1.example.com/res1.html",
-                      lastmod="2013-01-03T20:00:00Z",
-                      change="updated",
-                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
-                      length=8876,
-                      mime_type="text/html" )
-        c1.link_add(rel="via",
-                    href="http://original.example.com/res1.html",
-                    modified="2013-01-03T07:00:00Z",
-                    hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6",
-                    length="8876",
-                    type="text/html")
-        cl.add( c1 )
-        self._assert_xml_equal_ex( cl.as_xml(), 'resourcesync_ex_32' )
-
-    def test_build_ex_33(self):
-        cl = ChangeList()
-        cl.up = "http://aggregator2.example.com/dataset1/capabilitylist.xml"
-        cl.md_from = "2013-01-03T12:00:00Z"
-        c1 = Resource(uri="http://aggregator2.example.com/res1.html",
-                      lastmod="2013-01-04T09:00:00Z",
-                      change="updated",
-                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
-                      length=8876,
-                      mime_type="text/html" )
-        c1.link_add(rel="via",
-                    href="http://original.example.com/res1.html",
-                    modified="2013-01-03T07:00:00Z",
-                    hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6",
-                    length="8876",
-                    type="text/html")
-        cl.add( c1 )
-        self._assert_xml_equal_ex( cl.as_xml(), 'resourcesync_ex_33' )
-
-    ##### Archives tests
-
-    def test_build_archives_ex_3_1(self):
-        """Resource List Archive listing 3 Resource Lists"""
-        rla = ResourceListArchive()
-        rla.up = 'http://example.com/dataset1/capabilitylist.xml'
-        rla.add( Resource( uri='http://example.com/resourcelist1.xml',
-                           md_at='2012-11-03T09:00:00Z') )
-        rla.add( Resource( uri='http://example.com/resourcelist2.xml',
-                           md_at='2012-12-03T09:00:00Z') )
-        rla.add( Resource( uri='http://example.com/resourcelist3.xml',
-                           md_at='2013-01-03T09:00:00Z') )
-        ex_xml = self._open_ex('archives_ex_3_1').read()
-        self._assert_xml_equal( rla.as_xml(), ex_xml )
-
-    def test_build_archives_ex_3_2(self):
-        """Resource List Archive Index listing 2 component Resource List Archives"""
-        rlai = ResourceListArchive()
-        rlai.sitemapindex = True
-        rlai.up = 'http://example.com/dataset1/capabilitylist.xml'
-        rlai.add( Resource( uri='http://example.com/resourcelistarchive00001.xml' ))
-        rlai.add( Resource( uri='http://example.com/resourcelistarchive00002.xml' ))
-        ex_xml = self._open_ex('archives_ex_3_2').read()
-        self._assert_xml_equal( rlai.as_xml(), ex_xml )
-
-    def test_build_archives_ex_4_1(self):
-        """Resource Dump Archive listing 2 Resource Dumps"""
-        rda = ResourceDumpArchive()
-        rda.up = 'http://example.com/dataset1/capabilitylist.xml'
-        rda.add( Resource( uri='http://example.com/resourcedump1.xml',
-                           lastmod='2012-11-03T09:05:42Z',
-                           md_at="2012-11-03T09:00:00Z",
-                           md_completed="2012-11-03T09:05:01Z" ) )
-
-        rda.add( Resource( uri='http://example.com/resourcedump2.xml',
-                           lastmod='2012-12-03T09:06:12Z',
-                           md_at="2012-12-03T09:00:00Z",
-                           md_completed="2012-12-03T09:05:17Z" ) )
-        ex_xml = self._open_ex('archives_ex_4_1').read()
-        self._assert_xml_equal( rda.as_xml(), ex_xml )
-
-    def test_build_archives_ex_5_1(self):
-        """Change List Archive listing 3 Change Lists"""
-        cla = ChangeListArchive()
-        cla.up = 'http://example.com/dataset1/capabilitylist.xml'
-        cla.add( Resource( uri='http://example.com/changelist1.xml',
-                           md_from='2013-01-01T09:00:00Z',
-                           md_until='2013-01-02T09:00:00Z') )
-        cla.add( Resource( uri='http://example.com/changelist2.xml',
-                           md_from='2013-01-02T09:00:00Z',
-                           md_until='2013-01-03T09:00:00Z') )
-        cla.add( Resource( uri='http://example.com/changelist3.xml',
-                           md_from='2013-01-03T09:00:00Z',
-                           md_until='2013-01-04T09:00:00Z') )
-        ex_xml = self._open_ex('archives_ex_5_1').read()
-        self._assert_xml_equal( cla.as_xml(), ex_xml )
-
-    def test_build_archives_ex_6_1(self):
-        """Change Dump Archive listing 2 Change Dumps"""
-        cda = ChangeDumpArchive()
-        cda.up = 'http://example.com/dataset1/capabilitylist.xml'
-        cda.add( Resource( uri='http://example.com/changedump-w1.xml',
-                           lastmod='2012-12-20T09:02:43Z',
-                           md_from="2012-01-13T09:00:00Z",
-                           md_until="2013-01-20T09:00:00Z" ) )
-        cda.add( Resource( uri='http://example.com/changedump-w2.xml',
-                           lastmod='2012-12-27T09:01:57Z',
-                           md_from="2012-01-20T09:00:00Z",
-                           md_until="2013-01-27T09:00:00Z" ) )
-        ex_xml = self._open_ex('archives_ex_6_1').read()
-        self._assert_xml_equal( cda.as_xml(), ex_xml )
-
-##### UTILITIES FOR (APPROX) COMPARISON OF XML IN EXAMPLES AND OUTPUT
-
-    def _assert_xml_equal_ex(self,xml,ex):
-        """Compare XML supplied with XML from example file ex"""
-        ex_xml = self._open_ex(ex).read()
-        self._assert_xml_equal( xml, ex_xml )
-
-    def _assert_xml_equal(self,a,b):
-        context = "Element mismatch in\n%s\nvs\n%s\n" % (a,b);
-        aa = self._xml_massage_split(a)
-        bb = self._xml_massage_split(b)
-        ia=iter(aa)
-        ib=iter(bb)
-        try:
-            while (1):
-                self._assert_xml_elements_equal( self._xml_reorder_attributes(ia.next()), 
-                                                 self._xml_reorder_attributes(ib.next()), 
-                                                 context )
-        except StopIteration:
-            # all is good provided there were the same number of elements
-            pass
-        self.assertEqual( len(aa), len(bb), "Same length check\n%s" % (context) )
-
-    def _assert_xml_elements_equal(self,a,b,context):
-        context = "Elements %s != %s\n%s" % (a,b,context)
-        self.assertEqual( a, b, context )
-
-    def _xml_reorder_attributes(self,xml):
-        """Manipulate string for single element with atts in alpha order
-        
-        This is a bit of a fudge because of pattern matching. Should give
-        correct match for all matches, but might give matches in rare cases 
-        that should not.
-        """
-        return(' '.join( sorted( xml.split(' ') ) ) )
-
-    def _xml_massage_split(self,xml):
-        """Massage XML for comparison and split by elements (on >)"""
-        xml = re.sub( r'\s+$', '', xml)
-        xml = re.sub( r'^\s+', '', xml)
-        xml = re.sub( r'\s+', ' ', xml)
-        xml = re.sub( r'\s*/>', ' />', xml) #always one space before end of self-closing element
-        xml = re.sub( r'>\s+<', '><', xml) #remove space between elements
-        # FUDGES, need to check these are OK
-        xml = re.sub( r"version='1.0'", 'version="1.0"', xml )
-        xml = re.sub( r"encoding='UTF-8'", 'encoding="UTF-8"', xml )
-        # return self.assertEqual( x, 'xx' )
-        return( xml.split('>') )
-
-    def _open_ex(self,ex):
-        return open('resync/test/testdata/examples_from_spec/%s.xml'%(ex),'r')
-
-if  __name__ == '__main__':
-    suite = unittest.TestLoader().loadTestsFromTestCase(TestExamplesFromSpec)
-    unittest.TextTestRunner(verbosity=2).run(suite)

resync/test/test_explorer.py

@@ -1,51 +0,0 @@
-import unittest
-import re
-import logging
-import sys, StringIO, contextlib
-
-from resync.client import Client, ClientFatalError
-from resync.capability_list import CapabilityList
-from resync.explorer import Explorer
-from resync.resource import Resource
-
-# From http://stackoverflow.com/questions/2654834/capturing-stdout-within-the-same-process-in-python
-class Data(object):
-    pass
-
-@contextlib.contextmanager
-def capture_stdout():
-    old = sys.stdout
-    capturer = StringIO.StringIO()
-    sys.stdout = capturer
-    data = Data()
-    yield data
-    sys.stdout = old
-    data.result = capturer.getvalue()
-
-
-class TestExplorer(unittest.TestCase):
-
-    @classmethod
-    def setUpClass(cls):
-        logging.basicConfig(level=logging.INFO)
-
-    def test01_create(self):
-        # dumb test that we can create Exporer object
-        e = Explorer()
-        self.assertTrue( e )
-
-    def test02_explore_show_summary(self):
-        # Create dummy capabilities object and display
-        cl = CapabilityList()
-        cl.add( Resource('uri:resourcelist') )
-        cl.add( Resource('uri:changelist') )
-        e = Explorer()
-        with capture_stdout() as capturer:
-            e.explore_show_summary(cl,False,[])
-        self.assertTrue( re.search(r'Parsed \(unknown capability\) document with 2 entries:',capturer.result) )
-        self.assertTrue( re.search(r'\[1\] uri:changelist',capturer.result) )
-        self.assertTrue( re.search(r'\[2\] uri:resourcelist',capturer.result) )
-
-if __name__ == '__main__':
-    suite = unittest.TestLoader().loadTestsFromTestCase(TestExplorer)
-    unittest.TextTestRunner(verbosity=2).run(suite)

resync/test/test_list_base.py

@@ -1,50 +0,0 @@
-import sys
-import unittest
-import StringIO
-from resync.resource import Resource
-from resync.list_base import ListBase
-from resync.sitemap import Sitemap, SitemapIndexError
-
-# etree gives ParseError in 2.7, ExpatError in 2.6
-etree_error_class = None
-if (sys.version_info < (2,7)):
-    from xml.parsers.expat import ExpatError
-    etree_error_class = ExpatError
-else:
-    from xml.etree.ElementTree import ParseError
-    etree_error_class = ParseError
-
-class TestListBase(unittest.TestCase):
-
-    def test_05_len_count(self):
-        # count sets explicit number of resources, len(resources) not used
-        lb = ListBase()
-        self.assertEqual( len(lb), 0 )
-        lb.add(Resource('a'))
-        self.assertEqual( len(lb), 1 )
-        lb = ListBase( count=100 )
-        self.assertEqual( len(lb), 100 )
-
-    def test_08_print(self):
-        lb = ListBase()
-        lb.add( Resource(uri='a',lastmod='2001-01-01',length=1234) )
-        lb.add( Resource(uri='b',lastmod='2002-02-02',length=56789) )
-        lb.add( Resource(uri='c',lastmod='2003-03-03',length=0) )
-        lb.md['from']=None #avoid now being added
-        #print lb
-        self.assertEqual( lb.as_xml(), '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"><rs:md capability="unknown" /><url><loc>a</loc><lastmod>2001-01-01T00:00:00Z</lastmod><rs:md length="1234" /></url><url><loc>b</loc><lastmod>2002-02-02T00:00:00Z</lastmod><rs:md length="56789" /></url><url><loc>c</loc><lastmod>2003-03-03T00:00:00Z</lastmod><rs:md length="0" /></url></urlset>' )
-
-    def test_11_parse_2(self):
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
-<rs:md capability="unknown" from="2013-02-12T14:09:00Z" />\
-<url><loc>/tmp/rs_test/src/file_a</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="12" /></url>\
-<url><loc>/tmp/rs_test/src/file_b</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="32" /></url>\
-</urlset>'
-        lb=ListBase()
-        lb.parse(fh=StringIO.StringIO(xml))
-        self.assertEqual( len(lb.resources), 2, 'got 2 resources')
-
-if  __name__ == '__main__':
-    suite = unittest.TestLoader().loadTestsFromTestCase(TestListBase)
-    unittest.TextTestRunner(verbosity=2).run(suite)

resync/test/test_list_base_with_index.py

@@ -1,59 +0,0 @@
-import sys
-import unittest
-import StringIO
-from resync.resource import Resource
-from resync.list_base_with_index import ListBaseWithIndex, ListBaseIndexError
-from resync.sitemap import Sitemap, SitemapIndexError
-
-# etree gives ParseError in 2.7, ExpatError in 2.6
-etree_error_class = None
-if (sys.version_info < (2,7)):
-    from xml.parsers.expat import ExpatError
-    etree_error_class = ExpatError
-else:
-    from xml.etree.ElementTree import ParseError
-    etree_error_class = ParseError
-
-class TestListBaseWithIndex(unittest.TestCase):
-
-    def test_08_print(self):
-        lb = ListBaseWithIndex()
-        lb.add( Resource(uri='a',lastmod='2001-01-01',length=1234) )
-        lb.add( Resource(uri='b',lastmod='2002-02-02',length=56789) )
-        lb.add( Resource(uri='c',lastmod='2003-03-03',length=0) )
-        lb.md['from']=None #avoid now being added
-        #print lb
-        self.assertEqual( lb.as_xml(), '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"><rs:md capability="unknown" /><url><loc>a</loc><lastmod>2001-01-01T00:00:00Z</lastmod><rs:md length="1234" /></url><url><loc>b</loc><lastmod>2002-02-02T00:00:00Z</lastmod><rs:md length="56789" /></url><url><loc>c</loc><lastmod>2003-03-03T00:00:00Z</lastmod><rs:md length="0" /></url></urlset>' )
-
-    def test_08_print_iter(self):
-        r = [ Resource(uri='a',lastmod='2001-01-01',length=1234),
-              Resource(uri='b',lastmod='2002-02-02',length=56789),
-              Resource(uri='c',lastmod='2003-03-03',length=0) ]
-        # without setting count will barf on len() attempt
-        lb = ListBaseWithIndex( resources=iter(r) )
-        self.assertRaises( TypeError, lb.as_xml )
-        # set explicit count larger than max_sitemap_entiries and as_xml will throw exception
-        lb = ListBaseWithIndex( resources=iter(r), count=3 )
-        lb.max_sitemap_entries = 2
-        self.assertRaises( ListBaseIndexError, lb.as_xml )
-        # set explicit count and all will be OK
-        lb = ListBaseWithIndex( resources=iter(r), count=3 )
-        lb.md['from']=None #avoid now being added
-        #print lb
-        self.assertEqual( lb.as_xml(), '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"><rs:md capability="unknown" /><url><loc>a</loc><lastmod>2001-01-01T00:00:00Z</lastmod><rs:md length="1234" /></url><url><loc>b</loc><lastmod>2002-02-02T00:00:00Z</lastmod><rs:md length="56789" /></url><url><loc>c</loc><lastmod>2003-03-03T00:00:00Z</lastmod><rs:md length="0" /></url></urlset>' )
-
-
-    def test_11_parse_2(self):
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
-<rs:md capability="unknown" from="2013-02-12T14:09:00Z" />\
-<url><loc>/tmp/rs_test/src/file_a</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="12" /></url>\
-<url><loc>/tmp/rs_test/src/file_b</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="32" /></url>\
-</urlset>'
-        lb=ListBaseWithIndex()
-        lb.parse(fh=StringIO.StringIO(xml))
-        self.assertEqual( len(lb.resources), 2, 'got 2 resources')
-
-if  __name__ == '__main__':
-    suite = unittest.TestLoader().loadTestsFromTestCase(TestListBaseWithIndex)
-    unittest.TextTestRunner(verbosity=2).run(suite)

resync/test/test_mapper.py

@@ -1,96 +0,0 @@
-import unittest
-from resync.mapper import Mapper, MapperError, Map
-
-class TestMapper(unittest.TestCase):
-
-    def test00_mapper_creation(self):
-        m1=Mapper( ['http://e.org/p/','/tmp/q/'] )
-        self.assertEqual( len(m1), 1 )
-        m2=Mapper( mappings=['http://e.org/p','/tmp/q'] )
-        self.assertEqual( len(m2), 1 )
-        self.assertEqual( str(m1), str(m2) )
-        m3=Mapper( ['http://e.org/p/=/tmp/q/'] )
-        self.assertEqual( len(m3), 1 )
-        self.assertEqual( str(m1), str(m3) )
-        m4=Mapper( ['http://e.org/p/=/tmp/q/','http://e.org/r/=/tmp/s/'] )
-        m5=Mapper( ['http://e.org/r/=/tmp/s/','http://e.org/p/=/tmp/q/'] )
-        self.assertEqual( len(m4), 2 )
-        self.assertEqual( len(m5), 2 )
-        self.assertNotEqual( str(m4), str(m5) )
-        
-
-    def test01_mapper_src_to_dst(self):
-        m=Mapper( ['http://e.org/p/','/tmp/q/'] )
-        self.assertEqual( m.src_to_dst('http://e.org/p/'), '/tmp/q/')
-        self.assertEqual( m.src_to_dst('http://e.org/p/aa'), '/tmp/q/aa')
-        self.assertEqual( m.src_to_dst('http://e.org/p/aa/bb'), '/tmp/q/aa/bb')
-        self.assertEqual( m.src_to_dst('http://e.org/p/aa/bb/'), '/tmp/q/aa/bb/')
-        self.assertRaises( MapperError, m.src_to_dst, 'http://e.org/p' )
-        self.assertRaises( MapperError, m.src_to_dst, 'http://e.org/pa' )
-        self.assertRaises( MapperError, m.src_to_dst, 'nomatch' )
-
-    def test02_mapper_dst_to_src(self):
-        m=Mapper( ['http://e.org/p/','/tmp/q/'] )
-        self.assertEqual( m.dst_to_src('/tmp/q/'), 'http://e.org/p/')
-        self.assertEqual( m.dst_to_src('/tmp/q/bb'), 'http://e.org/p/bb')
-        self.assertEqual( m.dst_to_src('/tmp/q/bb/cc'), 'http://e.org/p/bb/cc')
-        self.assertRaises( MapperError, m.dst_to_src, '/tmp/q' )
-        self.assertRaises( MapperError, m.dst_to_src, '/tmp/qa')
-        self.assertRaises( MapperError, m.dst_to_src, 'nomatch' )
-
-    def test03_mapper2_src_to_dst(self):
-        m=Mapper( ['http://e.org/p=/tmp/q','http://e.org/r=/tmp/s'] )
-        self.assertEqual( m.src_to_dst('http://e.org/p/'), '/tmp/q/')
-        self.assertEqual( m.src_to_dst('http://e.org/p/aa'), '/tmp/q/aa')
-        self.assertEqual( m.src_to_dst('http://e.org/r/'), '/tmp/s/')
-        self.assertEqual( m.src_to_dst('http://e.org/r/aa'), '/tmp/s/aa')
-
-    def test04_mapper2_dst_to_src(self):
-        m=Mapper( ['http://e.org/p=/tmp/q','http://e.org/r=/tmp/s'] )
-        self.assertEqual( m.dst_to_src('/tmp/q/'), 'http://e.org/p/')
-        self.assertEqual( m.dst_to_src('/tmp/q/bb'), 'http://e.org/p/bb')
-        self.assertEqual( m.dst_to_src('/tmp/s/'), 'http://e.org/r/')
-        self.assertEqual( m.dst_to_src('/tmp/s/bb'), 'http://e.org/r/bb')
-
-    def test05_path_from_uri(self):
-        m=Mapper()
-        self.assertEqual( m.path_from_uri('a_file'), 'a_file' )
-        self.assertEqual( m.path_from_uri('some_path/a_file'), 'some_path/a_file' )
-        self.assertEqual( m.path_from_uri('http://localhost/p'), 'localhost_p' )
-        self.assertEqual( m.path_from_uri('http://localhost:8888/p'), 'localhost_8888_p' )
-        self.assertEqual( m.path_from_uri('https://localhost:8888/p'), 'localhost_8888_p' )
-        self.assertEqual( m.path_from_uri('http://example.com'), 'example.com' )
-        self.assertEqual( m.path_from_uri('http://example.com/'), 'example.com' )
-        self.assertEqual( m.path_from_uri('http://example.com/ex1'), 'example.com_ex1' )
-        self.assertEqual( m.path_from_uri('http://example.com/ex1/'), 'example.com_ex1' )
-
-    def test06_mapper_unsafe(self):
-        self.assertFalse( Mapper( ['http://example.com/=/tmp/a'] ).unsafe() )
-        self.assertFalse( Mapper( ['http://example.com/=http://example.com/'] ).unsafe() )
-        self.assertFalse( Mapper( ['http://example.com/'], use_default_path=True ).unsafe() )
-        # Following hits case of single local arg supplied
-        self.assertTrue( Mapper( ['/tmp/a'], use_default_path=True ).unsafe() )
-        # One good, one bad -> bad
-        self.assertTrue( Mapper( ['http://example.com/=/tmp/a','/tmp/a=/tmp'] ).unsafe() )
-
-    def test07_default_src_uri(self):
-        self.assertEqual( Mapper(['a=b']).default_src_uri(), 'a')
-        self.assertEqual( Mapper(['a=b','b=c']).default_src_uri(), 'a')
-        self.assertRaises( MapperError, Mapper().default_src_uri )
-        
-    ##### Map #####
-
-    def test10_map_unsafe(self):
-        self.assertFalse( Map('http://example.com/','path').unsafe() )
-        # Note the first is a URI and the second is a (silly) path in the following
-        self.assertFalse( Map('http://example.com/','http://example.com/').unsafe() )
-        self.assertFalse( Map('a','b').unsafe() )
-        self.assertFalse( Map('path/a','path/b').unsafe() )
-        # The following are unsafe
-        self.assertTrue( Map('path','path').unsafe() )
-        self.assertTrue( Map('path/a','path').unsafe() )
-        self.assertTrue( Map('path','path/b').unsafe() )
-
-if __name__ == '__main__':
-    suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestMapper)
-    unittest.TextTestRunner(verbosity=2).run(suite)

resync/test/test_resource.py

@@ -1,178 +0,0 @@
-import unittest
-import re
-from resync.resource import Resource, ChangeTypeError
-
-class TestResource(unittest.TestCase):
-
-    def test01a_same(self):
-        r1 = Resource('a')
-        r2 = Resource('a')
-        self.assertEqual( r1, r1 )
-        self.assertEqual( r1, r2 )
-
-    def test01b_same(self):
-        r1 = Resource(uri='a',timestamp=1234.0)
-        r2 = Resource(uri='a',timestamp=1234.0)
-        self.assertEqual( r1, r1 )
-        self.assertEqual( r1, r2 )
-
-    def test01c_same(self):
-        """Same with lastmod instead of direct timestamp"""
-        r1 = Resource('a')
-        r1lm = '2012-01-01T00:00:00Z'
-        r1.lastmod = r1lm
-        r2 = Resource('a')
-        for r2lm in ('2012',
-                     '2012-01',
-                     '2012-01-01',
-                     '2012-01-01T00:00Z',
-                     '2012-01-01T00:00:00Z',
-                     '2012-01-01T00:00:00.000000Z',
-                     '2012-01-01T00:00:00.000000000000Z',
-                     '2012-01-01T00:00:00.000000000001Z', #below resolution
-                     '2012-01-01T00:00:00.00+00:00',
-                     '2012-01-01T00:00:00.00-00:00',
-                     '2012-01-01T02:00:00.00-02:00',
-                     '2011-12-31T23:00:00.00+01:00'
-                     ):
-            r2.lastmod = r2lm
-            self.assertEqual( r1.timestamp, r2.timestamp, ('%s (%f) == %s (%f)' % (r1lm,r1.timestamp,r2lm,r2.timestamp)) )
-            self.assertEqual( r1, r2 )
-
-    def test01d_same(self):
-        """Same with slight timestamp diff"""
-        r1 = Resource('a')
-        r1.lastmod='2012-01-02T01:02:03Z'
-        r2 = Resource('a')
-        r2.lastmod='2012-01-02T01:02:03.99Z'
-        self.assertNotEqual( r1.timestamp, r2.timestamp )
-        self.assertEqual( r1, r2 )
-
-    def test02a_diff(self):
-        r1 = Resource('a')
-        r2 = Resource('b')
-        self.assertNotEqual(r1,r2)
-
-    def test02b_diff(self):
-        r1 = Resource('a',lastmod='2012-01-11')
-        r2 = Resource('a',lastmod='2012-01-22')
-        #print 'r1 == r2 : '+str(r1==r2)
-        self.assertNotEqual( r1, r2 )
-
-    def test04_bad_lastmod(self):
-        def setlastmod(r,v):
-            r.lastmod=v
-        r = Resource('4')
-        # Bad formats
-        self.assertRaises( ValueError, setlastmod, r, "bad_lastmod" )
-        self.assertRaises( ValueError, setlastmod, r, "" )
-        self.assertRaises( ValueError, setlastmod, r, "2012-13-01" )
-        self.assertRaises( ValueError, setlastmod, r, "2012-12-32" )
-        self.assertRaises( ValueError, setlastmod, r, "2012-11-01T10:10:60" )
-        self.assertRaises( ValueError, setlastmod, r, "2012-11-01T10:10:59.9x" )
-        # Valid ISO8601 but not alloed in W3C Datetime
-        self.assertRaises( ValueError, setlastmod, r, "2012-11-01T01:01:01" )
-        self.assertRaises( ValueError, setlastmod, r, "2012-11-01 01:01:01Z" )
-        self.assertRaises( ValueError, setlastmod, r, "2012-11-01T01:01:01+0000" )
-        self.assertRaises( ValueError, setlastmod, r, "2012-11-01T01:01:01-1000" )
-   
-    def test05_lastmod_roundtrips(self):
-        r = Resource('a')
-        r.lastmod='2012-03-14'
-        self.assertEqual( r.lastmod, '2012-03-14T00:00:00Z' )
-        r.lastmod='2012-03-14T00:00:00+00:00'
-        #print r.timestamp
-        self.assertEqual( r.lastmod, '2012-03-14T00:00:00Z' )
-        r.lastmod='2012-03-14T00:00:00-00:00'
-        #print r.timestamp
-        self.assertEqual( r.lastmod, '2012-03-14T00:00:00Z' )
-        r.lastmod='2012-03-14T18:37:36Z'
-        #print r.timestamp
-        self.assertEqual( r.lastmod, '2012-03-14T18:37:36Z' )
-
-    def test06_str(self):
-        r1 = Resource('abc',lastmod='2012-01-01')
-        self.assertTrue( re.match( r"\[ abc \| 2012-01-01T", str(r1) ) )
-
-    def test07_repr(self):
-        r1 = Resource('def',lastmod='2012-01-01')
-        self.assertEqual( repr(r1), "{'uri': 'def', 'timestamp': 1325376000}" )
-
-    def test08_multiple_hashes(self):
-        r1 = Resource('abcd')
-        r1.md5 = "some_md5"
-        r1.sha1 = "some_sha1"
-        r1.sha256 = "some_sha256"
-        self.assertEqual( r1.md5, "some_md5" )
-        self.assertEqual( r1.sha1, "some_sha1" )
-        self.assertEqual( r1.sha256, "some_sha256" )
-        self.assertEqual( r1.hash, "md5:some_md5 sha-1:some_sha1 sha-256:some_sha256" )
-        r2 = Resource('def')
-        r2.hash = "md5:ddd"
-        self.assertEqual( r2.md5, 'ddd' )
-        self.assertEqual( r2.sha1, None )
-        r2.hash = "sha-1:eee"
-        self.assertEqual( r2.md5, None )
-        self.assertEqual( r2.sha1, 'eee' )
-        r2.hash = "md5:fff sha-1:eee sha-256:ggg"
-        self.assertEqual( r2.md5, 'fff' )
-        self.assertEqual( r2.sha1, 'eee' )
-        self.assertEqual( r2.sha256, 'ggg' )
-
-    def test09_changetypeerror(self):
-        r1 = Resource('a')
-        self.assertEqual( r1.change, None )
-        r1.change = 'deleted'
-        self.assertEqual( r1.change, 'deleted' )
-        self.assertRaises( ChangeTypeError, Resource, 'a', change="bad" )
-        # disable checking
-        Resource.CHANGE_TYPES = False
-        r1 = Resource( 'a', change="bad" )
-        self.assertEqual( r1.change, 'bad' )
-
-    def test10_md_at_roundtrips(self):
-        r = Resource('a')
-        r.md_at='2013-03-14'
-        self.assertEqual( r.md_at, '2013-03-14T00:00:00Z' )
-        r.md_at='2013-03-14T00:00:00+00:00'
-        self.assertEqual( r.md_at, '2013-03-14T00:00:00Z' )
-        r.md_at='2013-03-14T00:00:00-00:00'
-        self.assertEqual( r.md_at, '2013-03-14T00:00:00Z' )
-        r.md_at='2013-03-14T18:37:36Z'
-        self.assertEqual( r.md_at, '2013-03-14T18:37:36Z' )
-
-    def test11_md_completed_roundtrips(self):
-        r = Resource('a')
-        r.md_completed='2013-04-14'
-        self.assertEqual( r.md_completed, '2013-04-14T00:00:00Z' )
-        r.md_completed='2013-04-14T00:00:00+00:00'
-        self.assertEqual( r.md_completed, '2013-04-14T00:00:00Z' )
-        r.md_completed='2013-04-14T00:00:00-00:00'
-        self.assertEqual( r.md_completed, '2013-04-14T00:00:00Z' )
-        r.md_completed='2013-04-14T18:37:36Z'
-        self.assertEqual( r.md_completed, '2013-04-14T18:37:36Z' )
-
-    def test12_timevalues(self):
-        r = Resource(uri='tv',
-                     lastmod="2000-01-01",
-                     md_at="2000-01-02",
-                     md_completed="2000-01-03",
-                     md_from="2000-01-04",
-                     md_until="2000-01-05")
-        self.assertEqual( r.lastmod,   '2000-01-01T00:00:00Z' )
-        self.assertEqual( r.md_at,        '2000-01-02T00:00:00Z' )
-        self.assertEqual( r.md_completed, '2000-01-03T00:00:00Z' )
-        self.assertEqual( r.md_from,      '2000-01-04T00:00:00Z' )
-        self.assertEqual( r.md_until,     '2000-01-05T00:00:00Z' )
-
-    def test13_mime_type(self):
-        r = Resource(uri='tv1', mime_type='text/plain')
-        self.assertEqual( r.mime_type, 'text/plain' )
-        r.mime_type = None
-        self.assertEqual( r.mime_type, None )
-        r = Resource(uri='tv2')
-        self.assertEqual( r.mime_type, None )
-
-if __name__ == '__main__':
-    suite = unittest.TestLoader().loadTestsFromTestCase(TestResource)
-    unittest.TextTestRunner(verbosity=2).run(suite)

resync/test/test_resource_container.py

@@ -1,119 +0,0 @@
-import unittest
-from resync.resource import Resource
-from resync.resource_container import ResourceContainer
-
-class TestResourceContainer(unittest.TestCase):
-
-    def test01_create_and_add(self):
-        rc = ResourceContainer( resources=[] )
-        self.assertEqual( len(rc.resources), 0, "empty" )
-        rc.resources.append( Resource('a',timestamp=1) )
-        rc.resources.append( Resource('b',timestamp=2) )
-        self.assertEqual( len(rc.resources), 2, "two resources" )
-
-    def test02_iter(self):
-        rc = ResourceContainer( resources=[] )
-        rc.resources.append( Resource('a',timestamp=1) )
-        rc.resources.append( Resource('b',timestamp=2) )
-        rc.resources.append( Resource('c',timestamp=3) )
-        rc.resources.append( Resource('d',timestamp=4) )
-        resources=[]
-        for r in rc:
-            resources.append(r)
-        self.assertEqual(len(resources), 4)
-        self.assertEqual( resources[0].uri, 'a')
-        self.assertEqual( resources[3].uri, 'd')
-
-    def test03_prune_before(self):
-        rc = ResourceContainer()
-        rc.resources.append( Resource('a',timestamp=1) )
-        rc.resources.append( Resource('b',timestamp=2) )
-        rc.resources.append( Resource('c',timestamp=3) )
-        rc.resources.append( Resource('d',timestamp=4) )
-        rc.prune_before(3)
-        self.assertEqual( len(rc.resources), 2 )
-        i = iter(rc)
-        self.assertEqual( i.next().uri, 'c' )
-        self.assertEqual( i.next().uri, 'd' )
-        # put some more back out of order
-        rc.resources.append( Resource('a',timestamp=1) )
-        rc.resources.append( Resource('b',timestamp=2) )
-        rc.resources.append( Resource('e',timestamp=1000) )
-        rc.prune_before(3.5)
-        self.assertEqual( len(rc.resources), 2 )
-        i = iter(rc)
-        self.assertEqual( i.next().uri, 'd' )
-        self.assertEqual( i.next().uri, 'e' )
-
-    def test04_prune_dupes(self):
-        rc = ResourceContainer()
-        rc.resources.append( Resource('a',timestamp=1, change='created') )
-        rc.resources.append( Resource('b',timestamp=2, change='created') )
-        rc.resources.append( Resource('c',timestamp=3, change='updated') )
-        rc.resources.append( Resource('d',timestamp=4, change='deleted') )
-        rc.resources.append( Resource('a',timestamp=4, change='updated') )
-        rc.resources.append( Resource('b',timestamp=4, change='deleted') )
-        rc.resources.append( Resource('b',timestamp=5, change='created') )
-        rc.resources.append( Resource('b',timestamp=6, change='deleted') )
-        rc.resources.append( Resource('c',timestamp=7, change='deleted') )
-        rc.resources.append( Resource('d',timestamp=8, change='created') )
-        rc.prune_dupes()
-        self.assertEqual( len(rc.resources), 3 )
-        i = iter(rc)
-        i1 = i.next()
-        self.assertEqual( i1.uri, 'a' )
-        self.assertEqual( i1.change, 'updated' )
-        i2 = i.next()
-        self.assertEqual( i2.uri, 'c' )
-        self.assertEqual( i2.change, 'deleted' )
-        i3 = i.next()
-        self.assertEqual( i3.uri, 'd' )
-        self.assertEqual( i3.change, 'created' )
-
-    def test05_from_until(self):
-        rc = ResourceContainer()
-        # via convenience methods
-        self.assertEqual( rc.md_from, None )
-        self.assertEqual( rc.md_until, None )
-        rc.md_from = "ftime"
-        rc.md_until = "utime"
-        self.assertEqual( rc.md_from, "ftime" )
-        self.assertEqual( rc.md_until, "utime" )
-        # via underlying dict
-        rc.md['md_from'] = "ftime2"
-        rc.md['md_until'] = "utime2too"
-        self.assertEqual( rc.md_from, "ftime2" )
-        self.assertEqual( rc.md_until, "utime2too" )
-
-    def test06_source_description(self):
-        # up links used to point to the source description
-        rc = ResourceContainer()
-        # via convenience methods                                  
-        self.assertEqual( rc.up, None )
-        rc.up = "well-known"
-        self.assertEqual( rc.up, "well-known" )
-        rc.up = "well-known2"
-        self.assertEqual( rc.up, "well-known2" )
-        # via link
-        link = rc.link("up")
-        self.assertEqual( link['href'], "well-known2" )
-        link['href'] = "wk3"
-        self.assertEqual( rc.up, "wk3" )
-
-    def test07_describedby(self):
-        rc = ResourceContainer()
-        # via convenience methods                                  
-        self.assertEqual( rc.describedby, None )
-        rc.describedby = "db_uri"
-        self.assertEqual( rc.describedby, "db_uri" )
-
-    def test08_up(self):
-        rc = ResourceContainer()
-        # via convenience methods                                  
-        self.assertEqual( rc.up, None )
-        rc.up = "up_uri"
-        self.assertEqual( rc.up, "up_uri" )
-
-if __name__ == '__main__':
-    suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestResourceContainer)
-    unittest.TextTestRunner().run(suite)

resync/test/test_resource_dump.py

@@ -1,58 +0,0 @@
-import unittest
-import StringIO
-import re
-from resync.resource import Resource
-from resync.resource_dump import ResourceDump
-from resync.sitemap import SitemapParseError
-
-class TestResourceDump(unittest.TestCase):
-
-    def test01_as_xml(self):
-        rd = ResourceDump()
-        rd.add( Resource('a.zip',timestamp=1) )
-        rd.add( Resource('b.zip',timestamp=2) )
-        xml = rd.as_xml()
-        print xml
-        self.assertTrue( re.search(r'<rs:md .*capability="resourcedump"', xml), 'XML has capability' )
-        self.assertTrue( re.search(r'<url><loc>a.zip</loc><lastmod>1970-01-01T00:00:01Z</lastmod></url>', xml), 'XML has resource a' ) 
-
-    def test10_parse(self):
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
-<rs:md at="2013-01-01" capability="resourcedump"/>\
-<url><loc>http://example.com/a.zip</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="12345" /></url>\
-<url><loc>http://example.com/b.zip</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="56789" /></url>\
-</urlset>'
-        rd=ResourceDump()
-        rd.parse(fh=StringIO.StringIO(xml))
-        self.assertEqual( len(rd.resources), 2, 'got 2 resource dumps')
-        self.assertEqual( rd.capability, 'resourcedump', 'capability set' )
-        self.assertEqual( rd.md_at, '2013-01-01' )
-        self.assertTrue( 'http://example.com/a.zip' in rd.resources )
-        self.assertTrue( rd.resources['http://example.com/a.zip'].length, 12345 )
-        self.assertTrue( 'http://example.com/b.zip' in rd.resources )
-        self.assertTrue( rd.resources['http://example.com/b.zip'].length, 56789 )
-
-    def test11_parse_no_capability(self):
-        # For a resource dump this should be an error
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
-<rs:md from="2013-01-01"/>\
-<url><loc>http://example.com/a.zip</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="12" /></url>\
-</urlset>'
-        rd=ResourceDump()
-        self.assertRaises( SitemapParseError, rd.parse, fh=StringIO.StringIO(xml) )
-
-    def test12_parse_bad_capability(self):
-        # the <rs:md capability="bad_capability".. should give error
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
-<rs:md capability="bad_capability" from="2013-01-01"/>\
-<url><loc>http://example.com/bad_res_1</loc><lastmod>2012-03-14T18:37:36Z</lastmod></url>\
-</urlset>'
-        rd=ResourceDump()
-        self.assertRaises( SitemapParseError, rd.parse, fh=StringIO.StringIO(xml) )
-
-if __name__ == '__main__':
-    suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestResourceDump)
-    unittest.TextTestRunner().run(suite)

resync/test/test_resource_list.py

@@ -1,175 +0,0 @@
-import unittest
-import StringIO
-import re
-from resync.resource import Resource
-from resync.resource_list import ResourceList, ResourceListDupeError
-from resync.sitemap import SitemapParseError
-
-class TestResourceList(unittest.TestCase):
-
-    def test01_same(self):
-        src = ResourceList()
-        src.add( Resource('a',timestamp=1) )
-        src.add( Resource('b',timestamp=2) )
-        dst = ResourceList()
-        dst.add( Resource('a',timestamp=1) )
-        dst.add( Resource('b',timestamp=2) )
-        ( same, changed, deleted, added ) = dst.compare(src)
-        self.assertEqual( len(same), 2, "2 things unchanged" )
-        i = iter(same)
-        self.assertEqual( i.next().uri, 'a', "first was a" )
-        self.assertEqual( i.next().uri, 'b', "second was b" )
-        self.assertEqual( len(changed), 0, "nothing changed" )
-        self.assertEqual( len(deleted), 0, "nothing deleted" )
-        self.assertEqual( len(added), 0, "nothing added" )
-
-    def test02_changed(self):
-        src = ResourceList()
-        src.add( Resource('a',timestamp=1) )
-        src.add( Resource('b',timestamp=2) )
-        dst = ResourceList()
-        dst.add( Resource('a',timestamp=3) )
-        dst.add( Resource('b',timestamp=4) )
-        ( same, changed, deleted, added ) = dst.compare(src)
-        self.assertEqual( len(same), 0, "0 things unchanged" )
-        self.assertEqual( len(changed), 2, "2 things changed" )
-        i = iter(changed)
-        self.assertEqual( i.next().uri, 'a', "first was a" )
-        self.assertEqual( i.next().uri, 'b', "second was b" )
-        self.assertEqual( len(deleted), 0, "nothing deleted" )
-        self.assertEqual( len(added), 0, "nothing added" )
-
-    def test03_deleted(self):
-        src = ResourceList()
-        src.add( Resource('a',timestamp=1) )
-        src.add( Resource('b',timestamp=2) )
-        dst = ResourceList()
-        dst.add( Resource('a',timestamp=1) )
-        dst.add( Resource('b',timestamp=2) )
-        dst.add( Resource('c',timestamp=3) )
-        dst.add( Resource('d',timestamp=4) )
-        ( same, changed, deleted, added ) = dst.compare(src)
-        self.assertEqual( len(same), 2, "2 things unchanged" )
-        self.assertEqual( len(changed), 0, "nothing changed" )
-        self.assertEqual( len(deleted), 2, "c and d deleted" )
-        i = iter(deleted)
-        self.assertEqual( i.next().uri, 'c', "first was c" )
-        self.assertEqual( i.next().uri, 'd', "second was d" )
-        self.assertEqual( len(added), 0, "nothing added" )
-
-    def test04_added(self):
-        src = ResourceList()
-        src.add( Resource('a',timestamp=1) )
-        src.add( Resource('b',timestamp=2) )
-        src.add( Resource('c',timestamp=3) )
-        src.add( Resource('d',timestamp=4) )
-        dst = ResourceList()
-        dst.add( Resource('a',timestamp=1) )
-        dst.add( Resource('c',timestamp=3) )
-        ( same, changed, deleted, added ) = dst.compare(src)
-        self.assertEqual( len(same), 2, "2 things unchanged" )
-        self.assertEqual( len(changed), 0, "nothing changed" )
-        self.assertEqual( len(deleted), 0, "nothing deleted" )
-        self.assertEqual( len(added), 2, "b and d added" )
-        i = iter(added)
-        self.assertEqual( i.next().uri, 'b', "first was b" )
-        self.assertEqual( i.next().uri, 'd', "second was d" )
-
-    def test05_add(self):
-        r1 = Resource(uri='a',length=1)
-        r2 = Resource(uri='b',length=2)
-        i = ResourceList()
-        i.add(r1)
-        self.assertRaises( ResourceListDupeError, i.add, r1)
-        i.add(r2)
-        self.assertRaises( ResourceListDupeError, i.add, r2)
-        # allow dupes
-        r1d = Resource(uri='a',length=10)
-        i.add(r1d,replace=True)
-        self.assertEqual( len(i), 2 )
-        self.assertEqual( i.resources['a'].length, 10 ) 
-
-    def test06_add_iterable(self):
-        r1 = Resource(uri='a',length=1)
-        r2 = Resource(uri='b',length=2)
-        i = ResourceList()
-        i.add( [r1,r2] )
-        self.assertRaises( ResourceListDupeError, i.add, r1)
-        self.assertRaises( ResourceListDupeError, i.add, r2)
-        # allow dupes
-        r1d = Resource(uri='a',length=10)
-        i.add( [r1d] ,replace=True)
-        self.assertEqual( len(i), 2 )
-        self.assertEqual( i.resources['a'].length, 10 ) 
-
-    def test07_has_md5(self):
-        r1 = Resource(uri='a')
-        r2 = Resource(uri='b')
-        i = ResourceList()
-        self.assertFalse( i.has_md5() )
-        i.add(r1)
-        i.add(r2)
-        self.assertFalse( i.has_md5() )
-        r1.md5="aabbcc"
-        self.assertTrue( i.has_md5() )
-
-    def test08_iter(self):
-        i = ResourceList()
-        i.add( Resource('a',timestamp=1) )
-        i.add( Resource('b',timestamp=2) )
-        i.add( Resource('c',timestamp=3) )
-        i.add( Resource('d',timestamp=4) )
-        resources=[]
-        for r in i:
-            resources.append(r)
-        self.assertEqual(len(resources), 4)
-        self.assertEqual( resources[0].uri, 'a')
-        self.assertEqual( resources[3].uri, 'd')
-
-    def test20_as_xml(self):
-        rl = ResourceList()
-        rl.add( Resource('a',timestamp=1) )
-        rl.add( Resource('b',timestamp=2) )
-        xml = rl.as_xml()
-        print xml
-        self.assertTrue( re.search(r'<rs:md .*capability="resourcelist"', xml), 'XML has capability' )
-        self.assertTrue( re.search(r'<url><loc>a</loc><lastmod>1970-01-01T00:00:01Z</lastmod></url>', xml), 'XML has resource a' ) 
-
-    def test30_parse(self):
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
-<rs:md at="2013-08-07" capability="resourcelist" completed="2013-08-08" />\
-<url><loc>/tmp/rs_test/src/file_a</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md change="updated" length="12" /></url>\
-<url><loc>/tmp/rs_test/src/file_b</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="32" /></url>\
-</urlset>'
-        rl=ResourceList()
-        rl.parse(fh=StringIO.StringIO(xml))
-        self.assertEqual( len(rl.resources), 2, 'got 2 resources')
-        self.assertEqual( rl.md['capability'], 'resourcelist', 'capability set' )
-        self.assertEqual( rl.md_at, '2013-08-07' )
-        self.assertEqual( rl.md_completed, '2013-08-08' )
-
-    def test31_parse_no_capability(self):
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">\
-<url><loc>http://example.com/res1</loc><lastmod>2012-03-14T18:37:36Z</lastmod></url>\
-</urlset>'
-        rl=ResourceList()
-        rl.parse(fh=StringIO.StringIO(xml))
-        self.assertEqual( len(rl.resources), 1, 'got 1 resource')
-        self.assertEqual( rl.md['capability'], 'resourcelist', 'capability set by reading routine' )
-        self.assertFalse( 'from' in rl.md )
-
-    def test32_parse_bad_capability(self):
-        # the <rs:md capability="bad_capability".. should give error
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
-<rs:md capability="bad_capability" from="2013-01-01"/>\
-<url><loc>http://example.com/bad_res_1</loc><lastmod>2012-03-14T18:37:36Z</lastmod></url>\
-</urlset>'
-        rl=ResourceList()
-        self.assertRaises( SitemapParseError, rl.parse, fh=StringIO.StringIO(xml) )
-
-if __name__ == '__main__':
-    suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestResourceList)
-    unittest.TextTestRunner().run(suite)

resync/test/test_resource_list_builder.py

@@ -1,122 +0,0 @@
-import unittest
-import re
-import os
-import time
-from resync.resource_list_builder import ResourceListBuilder
-from resync.mapper import Mapper
-
-class TestResourceListBuilder(unittest.TestCase):
-
-    def setUp(self):
-        # Set timestamps (mtime) for test data. Timestamps on disk are
-        # in UTC so no conversion issues.
-        # Test case file_a: 1343236426 = 2012-07-25T17:13:46Z
-        # Test case file_b: 1000000000 = 2001-09-09T01:46:40Z
-        os.utime( "resync/test/testdata/dir1/file_a", (0, 1343236426 ) )
-        os.utime( "resync/test/testdata/dir1/file_b", (0, 1000000000 ) )
-
-    def test01_simple_scan(self):
-        rlb = ResourceListBuilder()
-        rlb.mapper = Mapper(['http://example.org/t','resync/test/testdata/dir1'])
-        rl = rlb.from_disk()
-        self.assertEqual( len(rl), 2 )
-        rli = iter(rl)
-        r = rli.next()
-        self.assertEqual( r.uri, 'http://example.org/t/file_a' )
-        self.assertEqual( r.lastmod, '2012-07-25T17:13:46Z' )
-        self.assertEqual( r.md5, None )
-        self.assertEqual( r.length, 20 )
-        self.assertEqual( r.path, None )
-        r = rli.next()
-        self.assertEqual( r.uri, 'http://example.org/t/file_b' )
-        self.assertEqual( r.lastmod, '2001-09-09T01:46:40Z' )
-        self.assertEqual( r.md5, None )
-        self.assertEqual( r.length, 45 )
-        self.assertEqual( r.path, None )
-        # Make sure at and completed were set
-        self.assertTrue( rl.md_at is not None )
-        self.assertTrue( rl.md_completed is not None )
-
-    def test02_no_length(self):
-        rlb = ResourceListBuilder(set_length=False)
-        rlb.mapper = Mapper(['http://example.org/t','resync/test/testdata/dir1'])
-        rl = rlb.from_disk()
-        self.assertEqual( len(rl), 2 )
-        rli = iter(rl)
-        r = rli.next()
-        self.assertEqual( r.uri, 'http://example.org/t/file_a' )
-        self.assertEqual( r.lastmod, '2012-07-25T17:13:46Z' )
-        self.assertEqual( r.md5, None )
-        self.assertEqual( r.length, None )
-        self.assertEqual( r.path, None )
-        r = rli.next()
-        self.assertEqual( r.uri, 'http://example.org/t/file_b' )
-        self.assertEqual( r.lastmod, '2001-09-09T01:46:40Z' )
-        self.assertEqual( r.md5, None )
-        self.assertEqual( r.length, None )
-        self.assertEqual( r.path, None )
-
-    def test03_set_md5(self):
-        rlb = ResourceListBuilder(set_md5=True)
-        rlb.mapper = Mapper(['http://example.org/t','resync/test/testdata/dir1'])
-        rl = rlb.from_disk()
-        self.assertEqual( len(rl), 2 )
-        rli = iter(rl)
-        r = rli.next()
-        self.assertEqual( r.uri, 'http://example.org/t/file_a' )
-        self.assertEqual( r.lastmod, '2012-07-25T17:13:46Z' )
-        self.assertEqual( r.md5, 'a/Jv1mYBtSjS4LR+qoft/Q==' )
-        self.assertEqual( r.length, 20 )
-        self.assertEqual( r.path, None )
-        r = rli.next()
-        self.assertEqual( r.uri, 'http://example.org/t/file_b' )
-        self.assertEqual( r.lastmod, '2001-09-09T01:46:40Z' )
-        self.assertEqual( r.md5, 'RS5Uva4WJqxdbnvoGzneIQ==' )
-        self.assertEqual( r.length, 45 )
-        self.assertEqual( r.path, None )
-
-    def test04_data(self):
-        rlb = ResourceListBuilder(set_path=True,set_md5=True)
-        rlb.mapper = Mapper(['http://example.org/t','resync/test/testdata/dir1'])
-        rl = rlb.from_disk()
-        self.assertEqual( len(rl), 2)
-        r = rl.resources.get('http://example.org/t/file_a')
-        self.assertTrue( r is not None )
-        self.assertEqual( r.uri, 'http://example.org/t/file_a' )
-        self.assertEqual( r.lastmod, '2012-07-25T17:13:46Z' )
-        self.assertEqual( r.md5, 'a/Jv1mYBtSjS4LR+qoft/Q==' )
-        self.assertEqual( r.path, 'resync/test/testdata/dir1/file_a' ) 
-
-    def test05_from_disk_paths(self):
-        rlb = ResourceListBuilder()
-        rlb.mapper = Mapper(['http://example.org/t','resync/test/testdata/dir1'])
-        # no path, should get no resources
-        rl = rlb.from_disk(paths=[])
-        self.assertEqual( len(rl), 0)
-        # full path, 2 resources
-        rl = rlb.from_disk(paths=['resync/test/testdata/dir1'])
-        self.assertEqual( len(rl), 2)
-        # new object with mapper covering larger space of disk
-        rlb = ResourceListBuilder(set_path=True)
-        rlb.mapper = Mapper(['http://example.org/t','resync/test/testdata'])
-        # same path with 2 resources
-        rl = rlb.from_disk(paths=['resync/test/testdata/dir1'])
-        self.assertEqual( len(rl), 2)
-        # same path with 2 resources
-        rl = rlb.from_disk(paths=['resync/test/testdata/dir1','resync/test/testdata/dir2'])
-        self.assertEqual( len(rl), 3)
-        # path that is just a single file
-        rl = rlb.from_disk(paths=['resync/test/testdata/dir1/file_a'])
-        self.assertEqual( len(rl), 1)
-        rli = iter(rl)
-        r = rli.next()
-        self.assertTrue( r is not None )
-        self.assertEqual( r.uri, 'http://example.org/t/dir1/file_a' )
-        self.assertEqual( r.lastmod, '2012-07-25T17:13:46Z' )
-        self.assertEqual( r.md5, None )
-        self.assertEqual( r.length, 20 )
-        self.assertEqual( r.path, 'resync/test/testdata/dir1/file_a' ) 
-
-if __name__ == '__main__':
-    suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestResourceListBuilder)
-    unittest.TextTestRunner(verbosity=2).run(suite)

resync/test/test_resource_list_multifile.py

@@ -1,102 +0,0 @@
-import sys
-import unittest
-import StringIO
-import tempfile
-import os.path
-import shutil
-
-from resync.list_base_with_index import ListBaseIndexError
-from resync.resource import Resource
-from resync.resource_list import ResourceList
-from resync.mapper import Mapper
-
-# etree gives ParseError in 2.7, ExpatError in 2.6
-etree_error_class = None
-if (sys.version_info < (2,7)):
-    from xml.parsers.expat import ExpatError
-    etree_error_class = ExpatError
-else:
-    from xml.etree.ElementTree import ParseError
-    etree_error_class = ParseError
-
-class TestResourceListMultifile(unittest.TestCase):
-
-    def test_01_read_local_filenames(self):
-        rl = ResourceList()
-        rl.read('resync/test/testdata/sitemapindex2/sitemap.xml')
-        self.assertEqual( len(rl.resources), 17, '17 resources from 3 sitemaps listed')
-        sr = sorted(rl.uris())
-        self.assertEqual( sr[0], 'http://localhost:8888/resources/1' )
-        self.assertEqual( sr[1], 'http://localhost:8888/resources/10' )
-        self.assertEqual( sr[2], 'http://localhost:8888/resources/100' )
-        self.assertEqual( sr[3], 'http://localhost:8888/resources/1000' )
-        self.assertEqual( sr[16], 'http://localhost:8888/resources/826' )
-
-    def test_02_read_with_mapper(self):
-        rl = ResourceList()
-        rl.mapper = Mapper(['http://localhost/=resync/test/testdata/sitemapindex2/'])
-        rl.read('resync/test/testdata/sitemapindex2/sitemap_mapper.xml')
-        self.assertEqual( len(rl.resources), 17, '17 resources from 3 sitemaps listed')
-        sr = sorted(rl.uris())
-        self.assertEqual( sr[0], 'http://localhost:8888/resources/1' )
-        self.assertEqual( sr[1], 'http://localhost:8888/resources/10' )
-        self.assertEqual( sr[2], 'http://localhost:8888/resources/100' )
-        self.assertEqual( sr[3], 'http://localhost:8888/resources/1000' )
-        self.assertEqual( sr[16], 'http://localhost:8888/resources/826' )
-
-    def test_11_write_multifile(self):
-        tempdir = tempfile.mkdtemp(prefix='test_resource_list_multifile')
-        rl = ResourceList()
-        rl.mapper = Mapper(['http://localhost/=%s/' % (tempdir)])
-        rl.add( Resource( uri='http://localhost/a' ) )
-        rl.add( Resource( uri='http://localhost/b' ) )
-        rl.add( Resource( uri='http://localhost/c' ) )
-        rl.add( Resource( uri='http://localhost/d' ) )
-        rl.max_sitemap_entries = 2
-        # first try writing without mutlifile allowed
-        rl.allow_multifile = False
-        self.assertRaises( ListBaseIndexError, rl.write, basename=os.path.join(tempdir,'sitemap.xml') )
-        # second actually do it
-        rl.allow_multifile = True
-        rl.write( basename = os.path.join(tempdir,'sitemap.xml') )
-        # check the two component sitemaps
-        rl1 = ResourceList()
-        rl1.read( os.path.join(tempdir,'sitemap00000.xml') )
-        self.assertEquals( len(rl1), 2 )
-        self.assertEquals( rl1.capability, 'resourcelist' )
-        self.assertFalse( rl1.sitemapindex )
-        i = iter(rl1)
-        self.assertEquals( i.next().uri, 'http://localhost/a' )
-        self.assertEquals( i.next().uri, 'http://localhost/b' )
-        rl2 = ResourceList()
-        rl2.read( os.path.join(tempdir,'sitemap00001.xml') )
-        self.assertEquals( len(rl2), 2 )
-        i = iter(rl2)
-        self.assertEquals( i.next().uri, 'http://localhost/c' )
-        self.assertEquals( i.next().uri, 'http://localhost/d' )
-        # check the sitemapindex (read just as index)
-        rli = ResourceList()
-        rli.read( os.path.join(tempdir,'sitemap.xml'), index_only=True )
-        self.assertEquals( len(rli), 2 )
-        i = iter(rli)
-        self.assertEquals( rli.capability, 'resourcelist' )
-        self.assertTrue( rli.sitemapindex )
-        self.assertEquals( i.next().uri, 'http://localhost/sitemap00000.xml' )
-        self.assertEquals( i.next().uri, 'http://localhost/sitemap00001.xml' )
-        # check the sitemapindex and components
-        rli = ResourceList( mapper=rl.mapper )
-        rli.read( os.path.join(tempdir,'sitemap.xml') )
-        self.assertEquals( len(rli), 4 )
-        self.assertEquals( rli.capability, 'resourcelist' )
-        self.assertFalse( rli.sitemapindex )
-        i = iter(rli)
-        self.assertEquals( i.next().uri, 'http://localhost/a' )
-        self.assertEquals( i.next().uri, 'http://localhost/b' )
-        self.assertEquals( i.next().uri, 'http://localhost/c' )
-        self.assertEquals( i.next().uri, 'http://localhost/d' )
-        # cleanup tempdir
-        shutil.rmtree(tempdir)
-
-if  __name__ == '__main__':
-    suite = unittest.TestLoader().loadTestsFromTestCase(TestResourceListMultifile)
-    unittest.TextTestRunner(verbosity=2).run(suite)

resync/test/test_resource_set.py

@@ -1,39 +0,0 @@
-import unittest
-import re
-from resync.resource import Resource
-from resync.resource_set import ResourceSet,ResourceSetDupeError
-
-class TestResourceSet(unittest.TestCase):
-
-    def test01_add(self):
-        rs = ResourceSet()
-        self.assertEqual( len(rs), 0 )
-        rs.add( Resource('a') )
-        self.assertEqual( len(rs), 1 )
-        rs.add( Resource('b') )
-        self.assertEqual( len(rs), 2 )
-        rs.add( Resource('c') )
-        self.assertEqual( len(rs), 3 )
-
-    def test02_order(self):
-        rs = ResourceSet()
-        rs.add( Resource('a2') )
-        rs.add( Resource('a3') )
-        rs.add( Resource('a1') )
-        i = iter(rs)
-        self.assertEqual( i.next().uri, 'a1' )
-        self.assertEqual( i.next().uri, 'a2' )
-        self.assertEqual( i.next().uri, 'a3' )
-        self.assertRaises( StopIteration, i.next )
-
-    def test03_dupe(self):
-        rs = ResourceSet()
-        self.assertEqual( len(rs), 0 )
-        rs.add( Resource('a') )
-        self.assertEqual( len(rs), 1 )
-        self.assertRaises( ResourceSetDupeError, rs.add, Resource('a') )
-
-
-if __name__ == '__main__':
-    suite = unittest.TestLoader().loadTestsFromTestCase(TestResourceSet)
-    unittest.TextTestRunner(verbosity=2).run(suite)

resync/test/test_sitemap.py

@@ -1,274 +0,0 @@
-import sys
-import unittest
-import StringIO
-from resync.resource import Resource
-from resync.resource_list import ResourceList
-from resync.sitemap import Sitemap, SitemapIndexError, SitemapParseError
-
-# etree gives ParseError in 2.7, ExpatError in 2.6
-etree_error_class = None
-if (sys.version_info < (2,7)):
-    from xml.parsers.expat import ExpatError
-    etree_error_class = ExpatError
-else:
-    from xml.etree.ElementTree import ParseError
-    etree_error_class = ParseError
-
-class TestSitemap(unittest.TestCase):
-
-    def test_01_resource_str(self):
-        r1 = Resource('a3')
-        r1.lastmod='2012-01-11T01:02:03Z'
-        self.assertEqual( Sitemap().resource_as_xml(r1), "<?xml version='1.0' encoding='UTF-8'?>\n<url><loc>a3</loc><lastmod>2012-01-11T01:02:03Z</lastmod></url>" )
-
-    def test_02_resource_str(self):
-        r1 = Resource('3b',1234.1,9999,'ab54de')
-        self.assertEqual( Sitemap().resource_as_xml(r1), "<?xml version='1.0' encoding='UTF-8'?>\n<url><loc>3b</loc><lastmod>1970-01-01T00:20:34.100000Z</lastmod><rs:md hash=\"md5:ab54de\" length=\"9999\" /></url>" )
-
-    def test_03_resource_str_hashes(self):
-        r1 = Resource('03hashes',1234.1)
-        r1.md5 = 'aaa'
-        r1.sha1 = 'bbb'
-        self.assertEqual( Sitemap().resource_as_xml(r1), "<?xml version='1.0' encoding='UTF-8'?>\n<url><loc>03hashes</loc><lastmod>1970-01-01T00:20:34.100000Z</lastmod><rs:md hash=\"md5:aaa sha-1:bbb\" /></url>" )
-        r1.sha256 = 'ccc'
-        self.assertEqual( Sitemap().resource_as_xml(r1), "<?xml version='1.0' encoding='UTF-8'?>\n<url><loc>03hashes</loc><lastmod>1970-01-01T00:20:34.100000Z</lastmod><rs:md hash=\"md5:aaa sha-1:bbb sha-256:ccc\" /></url>" )
-        r1.sha1 = None
-        self.assertEqual( Sitemap().resource_as_xml(r1), "<?xml version='1.0' encoding='UTF-8'?>\n<url><loc>03hashes</loc><lastmod>1970-01-01T00:20:34.100000Z</lastmod><rs:md hash=\"md5:aaa sha-256:ccc\" /></url>" )
-
-    def test_04_resource_str(self):
-        r1 = Resource(uri='4a',lastmod="2013-01-02",length=9999,md5='ab54de')
-        r1.ln = [{ 'rel':'duplicate',
-                   'pri':'1',
-                   'href':'http://mirror1.example.com/res1',
-                   'modified':'2013-01-02T18:00:00Z' }]
-        self.assertEqual( Sitemap().resource_as_xml(r1), "<?xml version='1.0' encoding='UTF-8'?>\n<url><loc>4a</loc><lastmod>2013-01-02T00:00:00Z</lastmod><rs:md hash=\"md5:ab54de\" length=\"9999\" /><rs:ln href=\"http://mirror1.example.com/res1\" modified=\"2013-01-02T18:00:00Z\" pri=\"1\" rel=\"duplicate\" /></url>" )
-        # add another two rs:ln's
-        r1.ln.append( { 'rel':'num2' } )
-        r1.ln.append( { 'rel':'num3' } )
-        self.assertEqual( Sitemap().resource_as_xml(r1), "<?xml version='1.0' encoding='UTF-8'?>\n<url><loc>4a</loc><lastmod>2013-01-02T00:00:00Z</lastmod><rs:md hash=\"md5:ab54de\" length=\"9999\" /><rs:ln href=\"http://mirror1.example.com/res1\" modified=\"2013-01-02T18:00:00Z\" pri=\"1\" rel=\"duplicate\" /><rs:ln rel=\"num2\" /><rs:ln rel=\"num3\" /></url>" )
-
-    def test_08_print(self):
-        r1 = Resource(uri='a',lastmod='2001-01-01',length=1234)
-        r2 = Resource(uri='b',lastmod='2002-02-02',length=56789)
-        r3 = Resource(uri='c',lastmod='2003-03-03',length=0)
-        m = ResourceList(md={'capability':'resourcelist','modified':None})
-        m.add(r1)
-        m.add(r2)
-        m.add(r3)
-        #print m
-        self.assertEqual( Sitemap().resources_as_xml(m), "<?xml version='1.0' encoding='UTF-8'?>\n<urlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\" xmlns:rs=\"http://www.openarchives.org/rs/terms/\"><rs:md capability=\"resourcelist\" /><url><loc>a</loc><lastmod>2001-01-01T00:00:00Z</lastmod><rs:md length=\"1234\" /></url><url><loc>b</loc><lastmod>2002-02-02T00:00:00Z</lastmod><rs:md length=\"56789\" /></url><url><loc>c</loc><lastmod>2003-03-03T00:00:00Z</lastmod><rs:md length=\"0\" /></url></urlset>")
-
-    def test_09_print_from_iter(self): 
-        r1 = Resource(uri='a',lastmod='2001-01-01',length=1234)
-        r2 = Resource(uri='b',lastmod='2002-02-02',length=56789)
-        m = ResourceList()
-        m.add(r1)
-        m.add(r2)
-        i = iter(m)
-        self.assertEqual( Sitemap().resources_as_xml(i), "<?xml version='1.0' encoding='UTF-8'?>\n<urlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\" xmlns:rs=\"http://www.openarchives.org/rs/terms/\"><url><loc>a</loc><lastmod>2001-01-01T00:00:00Z</lastmod><rs:md length=\"1234\" /></url><url><loc>b</loc><lastmod>2002-02-02T00:00:00Z</lastmod><rs:md length=\"56789\" /></url></urlset>")
-
-    def test_10_sitemap(self):
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
-<url><loc>http://e.com/a</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md hash="md5:Q2hlY2sgSW50ZWdyaXR5IQ==" length=\"12\" /></url>\
-</urlset>'
-        s=Sitemap()
-        i=s.parse_xml(fh=StringIO.StringIO(xml))
-        self.assertFalse( s.parsed_index, 'was a sitemap')
-        self.assertEqual( s.resources_created, 1, 'got 1 resources')
-        for r in i.resources:
-            self.assertTrue( r is not None, 'got the uri expected')
-            self.assertEqual( r.uri, 'http://e.com/a' )
-            self.assertEqual( r.lastmod, '2012-03-14T18:37:36Z' )
-            self.assertEqual( r.length, 12 )
-            self.assertEqual( r.md5, 'Q2hlY2sgSW50ZWdyaXR5IQ==' )
-
-    def test_11_parse_2(self):
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
-<url><loc>/tmp/rs_test/src/file_a</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length=\"12\" /></url>\
-<url><loc>/tmp/rs_test/src/file_b</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length=\"32\" /></url>\
-</urlset>'
-        s=Sitemap()
-        i=s.parse_xml(fh=StringIO.StringIO(xml))
-        self.assertFalse( s.parsed_index, 'was a sitemap')
-        self.assertEqual( s.resources_created, 2, 'got 2 resources')
-
-    def test_12_parse_multi_loc(self):
-        xml_start='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
-<url>'
-        xml_end='<lastmod>2012-03-14T18:37:36Z</lastmod></url></urlset>'
-        s=Sitemap()
-        two_loc='<loc>/tmp/rs_test/src/file_a</loc><loc>/tmp/rs_test/src/file_b</loc>'
-        self.assertRaises( SitemapParseError, s.parse_xml, 
-                           StringIO.StringIO(xml_start+two_loc+xml_end))
-        mt_loc='<loc></loc>'
-        self.assertRaises( SitemapParseError, s.parse_xml, 
-                           StringIO.StringIO(xml_start+mt_loc+xml_end))
-        mt_loc_att='<loc att="value"/>'
-        self.assertRaises( SitemapParseError, s.parse_xml, 
-                           StringIO.StringIO(xml_start+mt_loc_att+xml_end))
-
-    def test_13_parse_multi_lastmod(self):
-        xml_start='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
-<url><loc>uri:a</loc>'
-        xml_end='</url></urlset>'
-        s=Sitemap()
-        two_lastmod='<lastmod>2013-01-01</lastmod><lastmod>2013-01-02</lastmod>'
-        self.assertRaises( SitemapParseError, s.parse_xml, 
-                           StringIO.StringIO(xml_start+two_lastmod+xml_end))
-        # While it not ideal to omit, <lastmod> is not required and
-        # thus either empty lastmod or lastmod with just an attribute
-        # and no content are not ambiguous and thus should be accepted
-        # with resulting None for resource.lastmod
-        mt_lastmod='<lastmod></lastmod>'
-        i=s.parse_xml(fh=StringIO.StringIO(xml_start+mt_lastmod+xml_end))
-        self.assertEqual( s.resources_created, 1 )
-        self.assertEqual( i.resources[0].lastmod, None )
-        mt_lastmod_att='<lastmod att="value"/>'
-        i=s.parse_xml(fh=StringIO.StringIO(xml_start+mt_lastmod_att+xml_end))
-        self.assertEqual( s.resources_created, 1 )
-        self.assertEqual( i.resources[0].lastmod, None )
-
-    def test_14_parse_multi(self):
-        xml_start='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">'
-        xml_end='<url><loc>uri:a</loc></url></urlset>'
-        s=Sitemap()
-        two_md='<rs:md capability="resourcelist"/><rs:md capability="resourcelist"/>'
-        self.assertRaises( SitemapParseError, s.parse_xml, 
-                           StringIO.StringIO(xml_start+two_md+xml_end))
-
-    def test_15_parse_illformed(self):
-        s=Sitemap()
-        # ExpatError in python2.6, ParserError in 2.7
-        self.assertRaises( etree_error_class, s.parse_xml, StringIO.StringIO('not xml') )
-        self.assertRaises( etree_error_class, s.parse_xml, StringIO.StringIO('<urlset><url>something</urlset>') )
-
-    def test_16_parse_valid_xml_but_other(self):
-        s=Sitemap()
-        self.assertRaises( SitemapParseError, s.parse_xml, StringIO.StringIO('<urlset xmlns="http://example.org/other_namespace"> </urlset>') )
-        self.assertRaises( SitemapParseError, s.parse_xml, StringIO.StringIO('<other xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> </other>') )
-
-    def test_17_parse_sitemapindex_as_sitemap(self):
-        s=Sitemap()
-        self.assertRaises( SitemapIndexError, s.parse_xml, StringIO.StringIO('<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> </sitemapindex>'), sitemapindex=False )
-
-    def test_18_parse_with_rs_ln_on_resource(self):
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
-<rs:md capability=\"resourcelist\"/>\
-<url>\
-  <loc>http://example.com/file_a</loc><lastmod>2012-03-14T18:37:36Z</lastmod>\
-  <rs:md hash=\"md5:r2d2\" length=\"12345\" />\
-  <rs:ln rel=\"duplicate\" href=\"http://mirror1.example.com/res1\" modified=\"2013-01-02\" pri=\"1\" />\
-  <rs:ln rel=\"num2\" href=\"http://m2.example.com/res1\"/>\
-  <rs:ln rel=\"num3\" href=\"http://m3.example.com/res1\"/>\
-</url>\
-<url>\
-  <loc>http://example.com/file_b</loc><lastmod>2012-03-14T18:37:36Z</lastmod>\
-  <rs:md length=\"32\" />\
-</url>\
-</urlset>'
-        s=Sitemap()
-        rc=s.parse_xml(fh=StringIO.StringIO(xml))
-        self.assertFalse( s.parsed_index, 'was a sitemap')
-        self.assertEqual( s.resources_created, 2, 'got 2 resources')
-        i = iter(rc)
-        r1 = i.next()
-        r2 = i.next()
-        self.assertEqual( r1.uri, 'http://example.com/file_a' )
-        self.assertEqual( r1.ln[0]['rel'], 'duplicate' )
-        self.assertEqual( r1.ln[0]['href'], 'http://mirror1.example.com/res1' )
-        self.assertEqual( r1.ln[0]['modified'], '2013-01-02' )
-        self.assertEqual( r1.ln[0]['pri'], 1 )
-        self.assertEqual( r2.uri, 'http://example.com/file_b' )
-
-    def test_19_parse_with_bad_rs_ln(self):
-        xmlstart='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
-<rs:md capability="resourcelist"/>\
-<url><loc>http://example.com/file_a</loc>'
-        xmlend='</url></urlset>'
-        s=Sitemap()
-        #
-        # missing href
-        xml=xmlstart+'<rs:ln rel="duplicate"/>'+xmlend
-        self.assertRaises( SitemapParseError, s.parse_xml, fh=StringIO.StringIO(xml))
-        # missing rel
-        xml=xmlstart+'<rs:ln href="http://example.com/"/>'+xmlend
-        self.assertRaises( SitemapParseError, s.parse_xml, fh=StringIO.StringIO(xml))
-        # bad length
-        xml=xmlstart+'<rs:ln rel="duplicate" href="http://example.com/" length="a"/>'+xmlend
-        self.assertRaises( SitemapParseError, s.parse_xml, fh=StringIO.StringIO(xml))
-        # bad pri
-        xml=xmlstart+'<rs:ln rel="duplicate" href="http://example.com/" pri="fff"/>'+xmlend
-        self.assertRaises( SitemapParseError, s.parse_xml, fh=StringIO.StringIO(xml))
-        xml=xmlstart+'<rs:ln rel="duplicate" href="http://example.com/" pri="0"/>'+xmlend
-        self.assertRaises( SitemapParseError, s.parse_xml, fh=StringIO.StringIO(xml))
-        xml=xmlstart+'<rs:ln rel="duplicate" href="http://example.com/" pri="1000000"/>'+xmlend
-        self.assertRaises( SitemapParseError, s.parse_xml, fh=StringIO.StringIO(xml))
-        # and finally OK with errors fixes
-        xml=xmlstart+'<rs:ln rel="duplicate" href="http://example.com/" length="12345" pri="1" other="whatever"/>'+xmlend
-        rc = s.parse_xml(fh=StringIO.StringIO(xml))
-        self.assertEqual( len(rc.resources), 1, 'good at last, extra attribute ignored' )
-
-    def test_20_parse_sitemapindex_empty(self):
-        s=Sitemap()
-        si = s.parse_xml( fh=StringIO.StringIO('<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> </sitemapindex>'), sitemapindex=True )
-        self.assertTrue( s.parsed_index, 'was a sitemapindex')
-        self.assertEqual( len(si.resources), 0, '0 sitemaps')
-
-    def test_21_parse_sitemapindex(self):
-        s=Sitemap()
-        si = s.parse_xml( fh=StringIO.StringIO('<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><sitemap><loc>aaa</loc></sitemap><sitemap><loc>bbb</loc></sitemap></sitemapindex>'), sitemapindex=True )
-        self.assertEqual( len(si.resources), 2, '2 sitemaps')
-        sms = sorted(si.uris())
-        self.assertEqual( sms, ['aaa','bbb'] )
-        # add a couple more
-        s.parse_xml( fh=StringIO.StringIO('<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><sitemap><loc>cc</loc></sitemap><sitemap><loc>dd</loc></sitemap></sitemapindex>'), resources=si )
-        self.assertTrue( s.parsed_index, 'was a sitemapindex')
-        self.assertEqual( len(si.resources), 4, '4 sitemaps total')
-        sms = sorted(si.uris())
-        self.assertEqual( sms, ['aaa','bbb', 'cc', 'dd'] )
-
-    def test_22_parse_sitemapindex_file(self):
-        s=Sitemap()
-        fh=open('resync/test/testdata/sitemapindex1/sitemap.xml','r')
-        si = s.parse_xml( fh=fh, sitemapindex=True )
-        self.assertTrue( s.parsed_index, 'was a sitemapindex')
-        self.assertEqual( len(si.resources), 3, '3 sitemaps')
-        sms = sorted(si.uris())
-        self.assertEqual( sms, ['http://localhost:8888/sitemap00000.xml','http://localhost:8888/sitemap00001.xml','http://localhost:8888/sitemap00002.xml'] )
-        #self.assertEqual( si.resources['http://localhost:8888/sitemap00000.xml'].lastmod, '2012-06-13T18:09:13Z' )
-
-    def test_21_parse_multi_sitemapindex(self):
-        s = Sitemap()
-        fh=open('resync/test/testdata/sitemapindex2/sitemap.xml','r')
-        si = s.parse_xml( fh=fh, sitemapindex=True )
-        self.assertTrue( s.parsed_index, 'was a sitemapindex')
-        self.assertEqual( len(si.resources), 3, '3 sitemaps listed')
-
-    def test_30_parse_change_list(self):
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
-<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
-<url><loc>/tmp/rs_test/src/file_a</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md change="updated" length="12" /></url>\
-<url><loc>/tmp/rs_test/src/file_b</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="32" /></url>\
-</urlset>'
-        s=Sitemap()
-        s.resource_class=Resource
-        c=s.parse_xml(fh=StringIO.StringIO(xml))
-        self.assertEqual( s.resources_created, 2, 'got 2 resources')
-        i = iter(c)
-        r1 = i.next()
-        self.assertEqual( r1.uri, '/tmp/rs_test/src/file_a' )
-        self.assertEqual( r1.change, 'updated' )
-        r2 = i.next()
-        self.assertEqual( r2.uri, '/tmp/rs_test/src/file_b' )
-        self.assertEqual( r2.change, None )
-
-if  __name__ == '__main__':
-    suite = unittest.TestLoader().loadTestsFromTestCase(TestSitemap)
-    unittest.TextTestRunner(verbosity=2).run(suite)

resync/test/test_source_description.py

@@ -1,48 +0,0 @@
-import unittest
-import re
-from resync.resource import Resource
-from resync.source_description import SourceDescription
-
-class TestSourceDescription(unittest.TestCase):
-
-    def test01_empty(self):
-        rsd = SourceDescription()
-        rsd.describedby = "http://example.org/about"
-        self.assertEqual( len(rsd), 0 )
-        rsd.md_at = None
-        self.assertEqual( rsd.as_xml(), '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"><rs:ln href="http://example.org/about" rel="describedby" /><rs:md capability="description" /></urlset>' )
-
-    def test02_one_caplist(self):
-        rsd = SourceDescription()
-        rsd.describedby = "http://example.org/about"
-        self.assertEqual( len(rsd), 0 )
-        rsd.md_at = None
-        rsd.add_capability_list("http://example.org/ds1/cl.xml")
-        self.assertEqual( len(rsd), 1 )
-        self.assertEqual( rsd.as_xml(), '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"><rs:ln href="http://example.org/about" rel="describedby" /><rs:md capability="description" /><url><loc>http://example.org/ds1/cl.xml</loc><rs:md capability="capabilitylist" /></url></urlset>' )
-
-    def test03_a_bunch(self):
-        rsd = SourceDescription()
-        rsd.describedby = "http://example.org/about"
-        self.assertEqual( len(rsd), 0 )
-        rsd.add_capability_list("http://example.org/ds1/cl.xml")
-        rsd.add_capability_list("http://example.org/ds2/cl.xml")
-        rsd.add_capability_list("http://example.org/ds3/cl.xml")
-        self.assertEqual( len(rsd), 3 )
-        self.assertEqual( rsd.as_xml(), '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"><rs:ln href="http://example.org/about" rel="describedby" /><rs:md capability="description" /><url><loc>http://example.org/ds1/cl.xml</loc><rs:md capability="capabilitylist" /></url><url><loc>http://example.org/ds2/cl.xml</loc><rs:md capability="capabilitylist" /></url><url><loc>http://example.org/ds3/cl.xml</loc><rs:md capability="capabilitylist" /></url></urlset>' )
-
-    def test04_parse(self):
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"><rs:ln href="http://example.org/about" rel="describedby" /><rs:md capability="description" /><url><loc>http://example.org/ds1/cl.xml</loc><rs:md capability="capabilitylist" /></url><url><loc>http://example.org/ds2/cl.xml</loc><rs:md capability="capabilitylist" /></url><url><loc>http://example.org/ds3/cl.xml</loc><rs:md capability="capabilitylist" /></url></urlset>'
-        sd=SourceDescription()
-        sd.parse(str=xml)
-        self.assertEqual( sd.link_href('describedby'), 'http://example.org/about', 
-                          'describedby link' )
-        self.assertEqual( sd.capability, 'description' )
-        self.assertEqual( len(sd.resources), 3, 'got 3 capacility lists' )
-        [r1,r2,r3]=sd.resources
-        self.assertEqual( r1.uri, 'http://example.org/ds1/cl.xml' )
-        self.assertEqual( r1.capability, 'capabilitylist' )
-
-if __name__ == '__main__':
-    suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestSourceDescription)
-    unittest.TextTestRunner().run(suite)

resync/test/test_url_authority.py

@@ -1,62 +0,0 @@
-import unittest
-
-from resync.url_authority import UrlAuthority
-
-class TestUrlAuthority(unittest.TestCase):
-
-    def test01_strict_authority(self):
-        # Default is server check only
-        uauth = UrlAuthority( 'http://example.org/sitemap.xml', True )
-        self.assertTrue( uauth.has_authority_over( 'http://example.org/sitemap.xml' ) )
-        self.assertTrue( uauth.has_authority_over( 'http://example.org/sitemap.xml?anything' ) )
-        self.assertTrue( uauth.has_authority_over( 'http://example.org/sitemap.xml#frag' ) )
-        self.assertTrue( uauth.has_authority_over( 'http://example.org/same_level' ) )
-        self.assertTrue( uauth.has_authority_over( 'http://example.org/one/deeper' ) )
-        self.assertTrue( uauth.has_authority_over( 'http://example.org/one/two/deeper' ) )
-        self.assertTrue( uauth.has_authority_over( 'http://example.org/' ) )
-        self.assertTrue( uauth.has_authority_over( 'http://sub.example.org/subdomain' ) )
-        self.assertTrue( uauth.has_authority_over( 'http://sub.sub.example.org/subsubdomain' ) )
-
-    def test02_strict_no_authority(self):
-        uauth = UrlAuthority( 'http://example.org/dir/sitemap.xml', True )
-        self.assertFalse( uauth.has_authority_over( 'http://example.org/sitemap.xml' ) )
-        self.assertFalse( uauth.has_authority_over( 'http://sub.example.org/sitemap.xml' ) )
-        self.assertFalse( uauth.has_authority_over( 'https://example.org/dir/sitemap.xml' ) )
-        self.assertFalse( uauth.has_authority_over( 'unknown://example.org/dir/sitemap.xml' ) )
-
-    def test03_strict_domains(self):
-        uauth = UrlAuthority( 'http://a.example.org/sitemap.xml', True )
-        self.assertTrue( uauth.has_authority_over( 'http://a.example.org/sitemap.xml' ) )
-        self.assertTrue( uauth.has_authority_over( 'http://sub.a.example.org/sitemap.xml' ) )
-        self.assertFalse( uauth.has_authority_over( 'http://b.example.org/sitemap.xml' ) )
-        self.assertFalse( uauth.has_authority_over( 'http://sub.b.example.org/sitemap.xml' ) )
-
-    def test04_lax_authority(self):
-        # Default is server check only
-        uauth = UrlAuthority( 'http://example.org/dir/sitemap.xml' )
-        self.assertTrue( uauth.has_authority_over( 'http://example.org/sitemap.xml' ) )
-        self.assertTrue( uauth.has_authority_over( 'http://example.org/sitemap.xml?anything' ) )
-        self.assertTrue( uauth.has_authority_over( 'http://example.org/sitemap.xml#frag' ) )
-        self.assertTrue( uauth.has_authority_over( 'http://example.org/dir/same_level' ) )
-        self.assertTrue( uauth.has_authority_over( 'http://example.org/dir/one/deeper' ) )
-        self.assertTrue( uauth.has_authority_over( 'http://example.org/dir/one/two/deeper' ) )
-        self.assertTrue( uauth.has_authority_over( 'http://example.org/shallower' ) )
-        self.assertTrue( uauth.has_authority_over( 'http://example.org/' ) )
-        self.assertTrue( uauth.has_authority_over( 'http://sub.example.org/subdomain' ) )
-        self.assertTrue( uauth.has_authority_over( 'http://sub.sub.example.org/subsubdomain' ) )
-
-    def test05_lax_no_authority(self):
-        uauth = UrlAuthority( 'http://example.org/dir/sitemap.xml' )
-        self.assertFalse( uauth.has_authority_over( 'http://other.org/sitemap.xml' ) )
-        self.assertFalse( uauth.has_authority_over( 'unknown://example.org/dir/sitemap.xml' ) )
-
-    def test06_lax_domains(self):
-        uauth = UrlAuthority( 'http://a.example.org/sitemap.xml' )
-        self.assertTrue( uauth.has_authority_over( 'http://a.example.org/sitemap.xml' ) )
-        self.assertTrue( uauth.has_authority_over( 'http://sub.a.example.org/sitemap.xml' ) )
-        self.assertFalse( uauth.has_authority_over( 'http://b.example.org/sitemap.xml' ) )
-        self.assertFalse( uauth.has_authority_over( 'http://sub.b.example.org/sitemap.xml' ) )
-
-if __name__ == '__main__':
-    suite = unittest.TestLoader().loadTestsFromTestCase(TestUrlAuthority)
-    unittest.TextTestRunner(verbosity=2).run(suite)

resync/test/test_utils.py

@@ -1,16 +0,0 @@
-import unittest
-import resync.utils
-
-class TestUtill(unittest.TestCase):
-
-    def test1_string(self):
-        self.assertEqual( resync.utils.compute_md5_for_string('A file\n'),
-                          'j912liHgA/48DCHpkptJHg==')
-
-    def test2_file(self):
-        # Should be same as the string above
-        self.assertEqual( resync.utils.compute_md5_for_file('resync/test/testdata/a'),
-                          'j912liHgA/48DCHpkptJHg==')
-
-if __name__ == '__main__':
-    unittest.main()

resync/test/test_w3c_datetime.py

@@ -1,97 +0,0 @@
-import unittest
-import re
-from resync.w3c_datetime import str_to_datetime, datetime_to_str
-
-def rt(dts):
-    """ Do simple round-trip """
-    return( datetime_to_str(str_to_datetime( dts )) )
-
-class TestW3cDatetime(unittest.TestCase):
-
-    def test1_datetime_to_str(self):
-        """Writing..."""
-        self.assertEqual( datetime_to_str(0), "1970-01-01T00:00:00Z" )
-        self.assertEqual( datetime_to_str(0.000001), "1970-01-01T00:00:00.000001Z" )
-        self.assertEqual( datetime_to_str(0.1), "1970-01-01T00:00:00.100000Z" )
-        self.assertEqual( datetime_to_str(1), "1970-01-01T00:00:01Z" )
-        self.assertEqual( datetime_to_str(60), "1970-01-01T00:01:00Z" )
-        self.assertEqual( datetime_to_str(60*60), "1970-01-01T01:00:00Z" )
-        self.assertEqual( datetime_to_str(60*60*24), "1970-01-02T00:00:00Z" )
-        self.assertEqual( datetime_to_str(60*60*24*31), "1970-02-01T00:00:00Z" )
-        self.assertEqual( datetime_to_str(60*60*24*365), "1971-01-01T00:00:00Z" )
-        #
-        self.assertEqual( datetime_to_str(1234567890), "2009-02-13T23:31:30Z" )
-
-    def test2_str_to_datetime(self):
-        """Reading..."""
-        self.assertEqual( str_to_datetime("1970-01-01T00:00:00Z"), 0 )
-        self.assertEqual( str_to_datetime("1970-01-01T00:00:00.000Z"), 0 )
-        self.assertEqual( str_to_datetime("1970-01-01T00:00:00+00:00"), 0 )
-        self.assertEqual( str_to_datetime("1970-01-01T00:00:00-00:00"), 0 )
-        self.assertEqual( str_to_datetime("1970-01-01T00:00:00.000001Z"), 0.000001 )
-        self.assertEqual( str_to_datetime("1970-01-01T00:00:00.1Z"), 0.1 )
-        self.assertEqual( str_to_datetime("1970-01-01T00:00:00.100000Z"), 0.1 )
-        #
-        self.assertEqual( str_to_datetime("2009-02-13T23:31:30Z"), 1234567890 )
-
-    def test2_same(self):
-        """Datetime values that are the same"""
-        astr = '2012-01-01T00:00:00Z'
-        a = str_to_datetime( astr )
-        for bstr in ('2012',
-                     '2012-01',
-                     '2012-01-01',
-                     '2012-01-01T00:00Z',
-                     '2012-01-01T00:00:00Z',
-                     '2012-01-01T00:00:00.000000Z',
-                     '2012-01-01T00:00:00.000000000000Z',
-                     '2012-01-01T00:00:00.000000000001Z', #below resolution
-                     '2012-01-01T00:00:00.00+00:00',
-                     '2012-01-01T00:00:00.00-00:00',
-                     '2012-01-01T02:00:00.00-02:00',
-                     '2011-12-31T23:00:00.00+01:00'
-                     ):
-            b =  str_to_datetime( bstr )
-            self.assertEqual( a, b, ('%s (%f) == %s (%f)' % (astr,a,bstr,b)) )
-
-    def test4_bad_str(self):
-        # Bad formats
-        self.assertRaises( ValueError, str_to_datetime, "bad_lastmod" )
-        self.assertRaises( ValueError, str_to_datetime, "" )
-        self.assertRaises( ValueError, str_to_datetime, "2012-13-01" )
-        self.assertRaises( ValueError, str_to_datetime, "2012-12-32" )
-        self.assertRaises( ValueError, str_to_datetime, "2012-11-01T10:10:60" )
-        self.assertRaises( ValueError, str_to_datetime, "2012-11-01T10:10:59.9x" )
-        # Valid ISO8601 but not allowed in W3C Datetime
-        self.assertRaises( ValueError, str_to_datetime, "2012-11-01T01:01:01" )
-        self.assertRaises( ValueError, str_to_datetime, "2012-11-01 01:01:01Z" )
-        self.assertRaises( ValueError, str_to_datetime, "2012-11-01T01:01:01+0000" )
-        self.assertRaises( ValueError, str_to_datetime, "2012-11-01T01:01:01-1000" )
-        # Bad values
-        self.assertRaises( ValueError, str_to_datetime, "2012-00-01T01:01:01Z" )
-        self.assertRaises( ValueError, str_to_datetime, "2012-13-01T01:01:01Z" )
-        self.assertRaises( ValueError, str_to_datetime, "2012-11-00T01:01:01Z" )
-        self.assertRaises( ValueError, str_to_datetime, "2012-11-32T01:01:01Z" )
-        self.assertRaises( ValueError, str_to_datetime, "2012-13-01T24:01:01Z" )
-        self.assertRaises( ValueError, str_to_datetime, "2012-13-01T00:60:01Z" )
-        self.assertRaises( ValueError, str_to_datetime, "2012-13-01T00:00:60Z" )
-        self.assertRaises( ValueError, str_to_datetime, "2012-11-01T01:01:01+99:00" )
-        self.assertRaises( ValueError, str_to_datetime, "2012-11-01T01:01:01-25:00" )
-   
-    def test5_roundtrips(self):
-        self.assertEqual( rt('2012-03-14T00:00:00+00:00'),
-                             '2012-03-14T00:00:00Z')
-        self.assertEqual( rt('2012-03-14T00:00:00-00:00'),
-                             '2012-03-14T00:00:00Z')
-        self.assertEqual( rt('2012-03-14T11:00:00-11:00' ),
-                             '2012-03-14T00:00:00Z')
-        self.assertEqual( rt('2012-03-14T18:37:36Z' ),
-                             '2012-03-14T18:37:36Z' )
-        self.assertEqual( rt('2012-03-14T18:37:36+00:10' ),
-                             '2012-03-14T18:47:36Z' )
-        self.assertEqual( rt('2012-03-14T18:37:36-01:01' ),
-                             '2012-03-14T17:36:36Z' )
-
-if __name__ == '__main__':
-    suite = unittest.TestLoader().loadTestsFromTestCase(TestW3cDatetime)
-    unittest.TextTestRunner(verbosity=2).run(suite)

resync/test/testdata/sitemapindex2/sitemap.xml

@@ -1,6 +0,0 @@
-<?xml version='1.0' encoding='UTF-8'?>
-<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
-<sitemap><loc>resync/test/testdata/sitemapindex2/sitemap00000.xml</loc><lastmod>2012-06-13T18:09:13Z</lastmod></sitemap>
-<sitemap><loc>resync/test/testdata/sitemapindex2/sitemap00001.xml</loc><lastmod>2012-06-13T18:09:13Z</lastmod></sitemap>
-<sitemap><loc>resync/test/testdata/sitemapindex2/sitemap00002.xml</loc><lastmod>2012-06-13T18:09:13Z</lastmod></sitemap>
-</sitemapindex>

resync/url_authority.py

@@ -1,69 +1,73 @@
-"""Determine whether one resource can speak authoritatively
-about another based on DNS hierarchy of server names and
-path hierarchy within URLs.
+"""Determine URI authority based on DNS and paths."""
 
-Two modes are supported:
-
-strict=True: requires that a query URL has the same URI 
-scheme (e.g. http) as the master, is on the same server
-or one in a sub-domain, and that the path component is
-at the same level or below the master.
-
-strict=False (default): requires only that a query URL
-has the same URI scheme as the master, and is on the same 
-server or one in a sub-domain of the master.
-
-Example use:
-
-    from resync.url_authority import UrlAuthority
-
-    auth = UrlAuthority("http://example.org/master")
-    if (auth.has_authority_over("http://example.com/res1")):
-        # will be true
-    if (auth.has_authority_over("http://other.com/res1")):
-        # will be false
-"""
-
-import urlparse
+from urllib.parse import urlparse
 import os.path
 
+
 class UrlAuthority(object):
+    """Determine URI authority based on DNS and paths.
+
+    Determine whether one resource can speak authoritatively
+    about another based on DNS hierarchy of server names and
+    path hierarchy within URLs.
+
+    Two modes are supported:
+
+    strict=True: requires that a query URL has the same URI
+    scheme (e.g. http) as the master, is on the same server
+    or one in a sub-domain, and that the path component is
+    at the same level or below the master.
+
+    strict=False (default): requires only that a query URL
+    has the same URI scheme as the master, and is on the same
+    server or one in a sub-domain of the master.
+
+    Example use:
+
+        from resync.url_authority import UrlAuthority
+
+        auth = UrlAuthority("http://example.org/master")
+        if (auth.has_authority_over("http://example.com/res1")):
+            # will be true
+        if (auth.has_authority_over("http://other.com/res1")):
+            # will be false
+    """
 
     def __init__(self, url=None, strict=False):
-        """Create object and optionally set master url and/or strict mode"""
+        """Create object and optionally set master url and/or strict mode."""
         self.url = url
         self.strict = strict
         if (self.url is not None):
             self.set_master(self.url)
         else:
-            self.master_scheme='none'
-            self.master_netloc='none.none.none'
-            self.master_path='/not/very/likely'
+            self.master_scheme = 'none'
+            self.master_netloc = 'none.none.none'
+            self.master_path = '/not/very/likely'
 
     def set_master(self, url):
-        """Set the master url that this object works with"""
-        m = urlparse.urlparse(url)
-        self.master_scheme=m.scheme
-        self.master_netloc=m.netloc
-        self.master_path=os.path.dirname(m.path)
+        """Set the master url that this object works with."""
+        m = urlparse(url)
+        self.master_scheme = m.scheme
+        self.master_netloc = m.netloc
+        self.master_path = os.path.dirname(m.path)
 
     def has_authority_over(self, url):
-        """Returns True of the current master has authority over url
+        """Return True of the current master has authority over url.
 
         In strict mode checks scheme, server and path. Otherwise checks
-        just that the server names match or the query url is a 
-        sub-domain of the master
+        just that the server names match or the query url is a
+        sub-domain of the master.
         """
-        s = urlparse.urlparse(url)
+        s = urlparse(url)
         if (s.scheme != self.master_scheme):
             return(False)
         if (s.netloc != self.master_netloc):
-            if (not s.netloc.endswith('.'+self.master_netloc)):
+            if (not s.netloc.endswith('.' + self.master_netloc)):
                 return(False)
-            #Maybe should allow parallel for 3+ components, eg. a.example.org, b.example.org
+            # Maybe should allow parallel for 3+ components, eg. a.example.org,
+            # b.example.org
         path = os.path.dirname(s.path)
-        if (self.strict and
-            path != self.master_path and
-            not path.startswith(self.master_path)):
+        if (self.strict and path != self.master_path
+                and not path.startswith(self.master_path)):
             return(False)
         return(True)

resync/url_or_file_open.py

@@ -0,0 +1,49 @@
+"""Local version of urlopen that supports local files & web URLs, plus adds auth."""
+
+import re
+import time
+from urllib.request import Request, urlopen
+
+from . import __version__
+
+
+# Global configuration settings
+NUM_REQUESTS = 0
+CONFIG = {
+    'bearer_token': None,
+    'delay': None
+}
+
+
+def set_url_or_file_open_config(key, value):
+    """Set the global config."""
+    global CONFIG
+    CONFIG[key] = value
+
+
+def url_or_file_open(uri, method=None, timeout=None):
+    """Wrapper around urlopen() to prepend file: if no scheme provided.
+
+    Can be used as a context manager because the return value from urlopen(...)
+    supports both that and straightforwrd use as simple file handle object.
+
+    If timeout is exceeded then urlopen(..) will raise a socket.timeout exception. If
+    no timeout is specified then the global default will be used.
+    """
+    if (not re.match(r'''\w+:''', uri)):
+        uri = 'file:' + uri
+    headers = {'User-Agent': 'resync/' + __version__}
+    # Do we need to send an Authorization header?
+    # FIXME - This token will be added blindy to all requests. This is insecure
+    # if the --noauth setting is used allowing requests across different domains.
+    # It would be better to have some scheme where a token is tied to a particular
+    # domain, or domain pattern.
+    if CONFIG['bearer_token'] is not None:
+        headers['Authorization'] = 'Bearer ' + CONFIG['bearer_token']
+    # Have we got a delay set? Apply only to web requests after first
+    global NUM_REQUESTS
+    if NUM_REQUESTS != 0 and CONFIG['delay'] is not None and not uri.startswith('file:'):
+        time.sleep(CONFIG['delay'])
+    NUM_REQUESTS += 1
+    maybe_timeout = {} if timeout is None else {'timeout': timeout}
+    return urlopen(Request(url=uri, headers=headers, method=method), **maybe_timeout)

resync/utils.py

@@ -1,64 +0,0 @@
-"""
-util.py: A collection of common util functions used in source and/or client.
-
-"""
-
-from logging import Formatter
-from datetime import datetime
-
-class UTCFormatter(Formatter):
-    # based on http://bit.ly/T2n3Xk
-    def formatTime(self, record, datefmt=None):
-        timestamp = record.created
-        return datetime.utcfromtimestamp(timestamp).isoformat() + 'Z'
-
-"""Compute digests for ResourceSync
-
-These are all base64 encoded according to the rules of 
-http://www.ietf.org/rfc/rfc4648.txt
-
-MD5 
-
-ResourceSync defined to be the same as for Content-MD5 in HTTP,
-http://www.ietf.org/rfc/rfc2616.txt which, in turn, defined the
-digest string as the "base64 of 128 bit MD5 digest as per RFC 1864"
-http://www.ietf.org/rfc/rfc1864.txt
-
-Unfortunately, RFC1864 is rather vague and contains only and example
-which doesn't use encoding characters for 62 or 63. It points to
-RFC1521 to describe base64 which is explicit that the encoding alphabet
-is [A-Za-z0-9+/] with = to pad. 
-
-The above corresponds with the alphabet of "3. Base 64 Encoding" in RFC3548
-http://www.ietf.org/rfc/rfc3548.txt
-and not the url safe version, "Base 64 Encoding with URL and Filename Safe 
-Alphabet" which replaces + and / with - and _ respectively.
-
-This is the same as the alphabet of "4. Base 64 Encoding" in RFC4648
-http://www.ietf.org/rfc/rfc4648.txt.
-
-This algorithm is implemented by base64.standard_b64encode() or 
-base64.b64encode() with no altchars specified. Available in python2.4 and
-up [http://docs.python.org/library/base64.html]
-"""
-import base64
-import hashlib
-
-def compute_md5_for_string(string):
-    """Compute MD5 digest over some string payload"""
-    return base64.b64encode(hashlib.md5(string).digest())
-
-def compute_md5_for_file(file, block_size=2**14):
-    """Compute MD5 digest for a file
-
-    Optional block_size parameter controls memory used to do MD5 calculation.
-    This should be a multiple of 128 bytes.
-    """
-    f = open(file, 'r')
-    md5 = hashlib.md5()
-    while True:
-        data = f.read(block_size)
-        if not data:
-            break
-        md5.update(data)
-    return base64.b64encode(md5.digest())

resync/w3c_datetime.py

@@ -1,4 +1,4 @@
-"""Write and parse W3C datetime
+"""Write and parse W3C datetime.
 
 Each web resource is identified by a URI and may optionally have
 other metadata such as timestamp, size, md5. The lastmod property
@@ -13,17 +13,18 @@ from datetime import datetime
 from dateutil import parser as dateutil_parser
 import re
 
-def datetime_to_str(dt='now',no_fractions=False):
-    """The Last-Modified data in ISO8601 syntax, Z notation
+
+def datetime_to_str(dt='now', no_fractions=False):
+    """The Last-Modified data in ISO8601 syntax, Z notation.
 
     The lastmod is stored as unix timestamp which is already
     in UTC. At preesent this code will return 6 decimal digits
-    if any fraction of a second is given. It would perhaps be 
+    if any fraction of a second is given. It would perhaps be
     better to return only the number of decimal digits necessary,
-    up to a resultion of 1 microsecond.
-    
+    up to a resolution of 1 microsecond.
+
     Special cases:
-    - Returns datetime str for now if no parameter given.
+    - Returns datetime str for 'now' if no parameter given.
     - Returns None if None is supplied.
     """
     if (dt is None):
@@ -32,15 +33,18 @@ def datetime_to_str(dt='now',no_fractions=False):
         dt = time.time()
     if (no_fractions):
         dt = int(dt)
+    else:
+        dt += 0.0000001  # improve rounding to microseconds
     return datetime.utcfromtimestamp(dt).isoformat() + 'Z'
 
+
 def str_to_datetime(s, context='datetime'):
-    """Set timestamp from an W3C Datetime Last-Modified value
+    """Set timestamp from an W3C Datetime Last-Modified value.
 
     The sitemaps.org specification says that <lastmod> values
-    must comply with the W3C Datetime format 
+    must comply with the W3C Datetime format
     (http://www.w3.org/TR/NOTE-datetime). This is a restricted
-    subset of ISO8601. In particular, all forms that include a 
+    subset of ISO8601. In particular, all forms that include a
     time must include a timezone indication so there is no
     notion of local time (which would be tricky on the web). The
     forms allowed are:
@@ -55,7 +59,7 @@ def str_to_datetime(s, context='datetime'):
           YYYY-MM-DDThh:mmTZD (eg 1997-07-16T19:20+01:00)
         Complete date plus hours, minutes and seconds:
           YYYY-MM-DDThh:mm:ssTZD (eg 1997-07-16T19:20:30+01:00)
-        Complete date plus hours, minutes, seconds and a decimal fraction 
+        Complete date plus hours, minutes, seconds and a decimal fraction
         of a second
           YYYY-MM-DDThh:mm:ss.sTZD (eg 1997-07-16T19:20:30.45+01:00)
         where:
@@ -74,7 +78,7 @@ def str_to_datetime(s, context='datetime'):
     if (s == ''):
         raise ValueError('Attempt to set empty %s' % (context))
     # Make a date into a full datetime
-    m = re.match(r"\d\d\d\d(\-\d\d(\-\d\d)?)?$",s)
+    m = re.match(r"\d\d\d\d(\-\d\d(\-\d\d)?)?$", s)
     if (m is not None):
         if (m.group(1) is None):
             s += '-01-01'
@@ -82,7 +86,7 @@ def str_to_datetime(s, context='datetime'):
             s += '-01'
         s += 'T00:00:00Z'
     # Now have datetime with timezone info
-    m = re.match(r"(.*\d{2}:\d{2}:\d{2})(\.\d+)([^\d].*)?$",s)
+    m = re.match(r"(.*\d{2}:\d{2}:\d{2})(\.\d+)([^\d].*)?$", s)
     # Chop out fractional seconds if present
     fractional_seconds = 0
     if (m is not None):
@@ -97,20 +101,21 @@ def str_to_datetime(s, context='datetime'):
     # Seems that one should be able to handle timezone offset
     # with dt.tzinfo module but this has variation in behavior
     # between python 2.6 and 2.7... so do here for now
-    m = re.match(r"(\d\d\d\d\-\d\d\-\d\dT\d\d:\d\d(:\d\d)?)(Z|([+-])(\d\d):(\d\d))$",s)
+    m = re.match(r"(\d\d\d\d\-\d\d\-\d\dT\d\d:\d\d(:\d\d)?)(Z|([+-])"
+                 r"(\d\d):(\d\d))$", s)
     if (m is None):
         raise ValueError("Bad datetime format (%s)" % s)
     str = m.group(1) + 'Z'
     dt = dateutil_parser.parse(str)
     offset_seconds = 0
     if (m.group(3) != 'Z'):
-        hh = int(m.group(5));
-        mm = int(m.group(6));
-        if (hh>23 or mm>59):
-            raise ValueError("Bad timezone offset (%s)" % s) 
-        offset_seconds = hh*3600 + mm*60
+        hh = int(m.group(5))
+        mm = int(m.group(6))
+        if (hh > 23 or mm > 59):
+            raise ValueError("Bad timezone offset (%s)" % s)
+        offset_seconds = hh * 3600 + mm * 60
         if (m.group(4) == '-'):
             offset_seconds = -offset_seconds
     # timetuple() ignores timezone information so we have to add in
     # the offset here, and any fractional component of the seconds
-    return( timegm(dt.timetuple()) + offset_seconds + fractional_seconds )
+    return(timegm(dt.timetuple()) + offset_seconds + fractional_seconds)

setup.py

@@ -1,13 +1,13 @@
-from setuptools import setup
-# setuptools used instead of distutils.core so that 
-# dependencies can be handled automatically
+"""Setup for ResurceSync library and client implementation."""
+from setuptools import setup, Command
+import os
 
-# Extract version number from resync/_version.py. Here we 
-# are very strict about the format of the version string 
-# as an extra sanity check. (Thanks for comments in 
+# Extract version number from resync/__init__.py. Here we
+# are very strict about the format of the version string
+# as an extra sanity check. (Thanks for comments in
 # http://stackoverflow.com/questions/458550/standard-way-to-embed-version-into-python-package )
 import re
-VERSIONFILE="resync/_version.py"
+VERSIONFILE = "resync/__init__.py"
 verfilestr = open(VERSIONFILE, "rt").read()
 match = re.search(r"^__version__ = '(\d\.\d.\d+(\.\d+)?)'", verfilestr, re.MULTILINE)
 if match:
@@ -15,29 +15,61 @@ if match:
 else:
     raise RuntimeError("Unable to find version string in %s." % (VERSIONFILE))
 
+
+class Coverage(Command):
+    """Class to allow coverage run from setup."""
+
+    description = "run coverage"
+    user_options = []
+
+    def initialize_options(self):
+        """Empty initialize_options."""
+        pass
+
+    def finalize_options(self):
+        """Empty finalize_options."""
+        pass
+
+    def run(self):
+        """Run coverage program."""
+        os.system("coverage run --source=resync setup.py test")
+        os.system("coverage report")
+        os.system("coverage html")
+        print("See htmlcov/index.html for details.")
+
+
 setup(
     name='resync',
     version=version,
     packages=['resync'],
-    scripts=['bin/resync','bin/resync-explorer'],
-    classifiers=["Development Status :: 3 - Alpha",
+    scripts=['resync-sync', 'resync-build', 'resync-explorer'],
+    classifiers=["Development Status :: 4 - Beta",
                  "Intended Audience :: Developers",
                  "License :: OSI Approved :: Apache Software License",
-                 "Operating System :: OS Independent", #is this true? know Linux & OS X ok
+                 "Operating System :: OS Independent",  # is this true? know Linux & OS X ok
                  "Programming Language :: Python",
-                 "Programming Language :: Python :: 2.6",
-                 "Programming Language :: Python :: 2.7",
+                 "Programming Language :: Python :: 3.5",
+                 "Programming Language :: Python :: 3.6",
+                 "Programming Language :: Python :: 3.7",
+                 "Programming Language :: Python :: 3.8",
                  "Topic :: Internet :: WWW/HTTP",
                  "Topic :: Software Development :: Libraries :: Python Modules",
                  "Environment :: Web Environment"],
     author='Simeon Warner',
     author_email='simeon.warner@cornell.edu',
     description='ResourceSync library and client',
-    long_description=open('README').read(),
-    url='http://github.com/resync/resync',
+    long_description=open('README.md').read(),
+    long_description_content_type='text/markdown',
     install_requires=[
-        "requests",
-        "python-dateutil>=1.5"
+        "python-dateutil>=1.5",
+        "defusedxml>=0.4.1"
     ],
-    test_suite="resync.test",
+    test_suite="tests",
+    tests_require=[
+        "testfixtures"
+    ],
+    python_requires='>=3.5',
+    cmdclass={
+        'coverage': Coverage
+    }
 )

tests/__init__.py

@@ -0,0 +1 @@
+"""Tests for library code in test."""

tests/test_capability_list.py

@@ -0,0 +1,62 @@
+import unittest
+import re
+from resync.resource import Resource
+from resync.resource_list import ResourceList
+from resync.resource_set import ResourceSetDupeError
+from resync.change_list import ChangeList
+from resync.capability_list import CapabilityList
+
+
+class TestCapabilityList(unittest.TestCase):
+
+    def test01_add(self):
+        # one
+        caps = CapabilityList()
+        r1 = Resource(uri='http://example.org/r1')
+        caps.add(r1)
+        self.assertEqual(len(caps), 1)
+        # dupe
+        self.assertRaises(ResourceSetDupeError, caps.add, r1)
+        self.assertEqual(len(caps), 1)
+        # dupe with replace
+        caps = CapabilityList()
+        caps.add([r1, r1], replace=True)
+        self.assertEqual(len(caps), 1)
+        # diff
+        caps = CapabilityList()
+        r2 = ChangeList(uri='http://example.org/r2')
+        caps.add([r1, r2])
+        self.assertEqual(len(caps), 2)
+
+    def test02_resourcelist(self):
+        rl = ResourceList()
+        caps = CapabilityList()
+        caps.add_capability(rl, "http://example.org/resourcelist.xml")
+        caps.md['from'] = "2013-02-07T22:39:00"
+        self.assertEqual(len(caps), 1)
+        print(caps.as_xml())
+        self.assertEqual(caps.as_xml(), '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"><rs:md capability="capabilitylist" from="2013-02-07T22:39:00" /><url><loc>http://example.org/resourcelist.xml</loc><rs:md capability="resourcelist" /></url></urlset>')
+
+    def test03_multiple(self):
+        caps = CapabilityList()
+        rl = ResourceList()
+        caps.add_capability(rl, "rl.xml")
+        cl = ChangeList()
+        caps.add_capability(cl, "cl.xml")
+        self.assertEqual(len(caps), 2)
+        xml = caps.as_xml()
+        self.assertTrue(
+            re.search(r'<loc>rl.xml</loc><rs:md capability="resourcelist" />', xml))
+        self.assertTrue(
+            re.search(r'<loc>cl.xml</loc><rs:md capability="changelist" />', xml))
+
+    def test04_parse(self):
+        xml = '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"><rs:md capability="capabilitylist" from="2013-02-07T22:39:00" /><url><loc>http://example.org/resourcelist.xml</loc><rs:md capability="resourcelist" /></url></urlset>'
+        cl = CapabilityList()
+        cl.parse(str_data=xml)
+        self.assertEqual(cl.capability, 'capabilitylist')
+        self.assertEqual(len(cl.resources), 1, 'got 1 resource')
+        [r] = cl.resources
+        self.assertEqual(
+            r.uri, 'http://example.org/resourcelist.xml', 'resourcelist uri')
+        self.assertEqual(r.capability, 'resourcelist')

tests/test_change_dump.py

@@ -0,0 +1,63 @@
+import unittest
+try:  # python2
+    # Must try this first as io also exists in python2
+    # but in the wrong one!
+    import StringIO as io
+except ImportError:  # python3
+    import io
+import re
+
+from resync.resource import Resource
+from resync.change_dump import ChangeDump
+from resync.sitemap import SitemapParseError
+
+
+class TestChangeDump(unittest.TestCase):
+
+    def test01_as_xml(self):
+        rd = ChangeDump()
+        rd.add(Resource('a.zip', timestamp=1))
+        rd.add(Resource('b.zip', timestamp=2))
+        xml = rd.as_xml()
+        # print(xml)
+        self.assertTrue(
+            re.search(r'<rs:md .*capability="changedump"', xml), 'XML has capability')
+        self.assertTrue(re.search(
+            r'<url><loc>a.zip</loc><lastmod>1970-01-01T00:00:01Z</lastmod></url>', xml), 'XML has resource a')
+
+    def test10_parse(self):
+        xml = '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
+<rs:md capability="changedump" from="2013-01-01"/>\
+<url><loc>http://example.com/a.zip</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="12345" /></url>\
+<url><loc>http://example.com/b.zip</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="56789" /></url>\
+</urlset>'
+        rd = ChangeDump()
+        rd.parse(fh=io.StringIO(xml))
+        self.assertEqual(len(rd.resources), 2, 'got 2 resource dumps')
+        self.assertEqual(rd.md['capability'], 'changedump', 'capability set')
+        self.assertEqual(rd.md_from, '2013-01-01')
+        self.assertTrue('http://example.com/a.zip' in rd.resources)
+        self.assertTrue(rd.resources['http://example.com/a.zip'].length, 12345)
+        self.assertTrue('http://example.com/b.zip' in rd.resources)
+        self.assertTrue(rd.resources['http://example.com/b.zip'].length, 56789)
+
+    def test11_parse_no_capability(self):
+        # For a resource dump this should be an error
+        xml = '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
+<rs:md at="2013-01-01"/>\
+<url><loc>http://example.com/a.zip</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="12" /></url>\
+</urlset>'
+        rd = ChangeDump()
+        self.assertRaises(SitemapParseError, rd.parse, fh=io.StringIO(xml))
+
+    def test12_parse_bad_capability(self):
+        # the <rs:md capability="bad_capability".. should give error
+        xml = '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
+<rs:md capability="bad_capability" from="2013-01-01"/>\
+<url><loc>http://example.com/bad_res_1</loc><lastmod>2012-03-14T18:37:36Z</lastmod></url>\
+</urlset>'
+        rd = ChangeDump()
+        self.assertRaises(SitemapParseError, rd.parse, fh=io.StringIO(xml))

tests/test_change_list.py

@@ -0,0 +1,194 @@
+import unittest
+try:  # python2
+    # Must try this first as io also exists in python2
+    # but in the wrong one!
+    import StringIO as io
+except ImportError:  # python3
+    import io
+import re
+
+from resync.resource import Resource
+from resync.change_list import ChangeList, ChangeTypeError
+from resync.resource_list import ResourceList
+from resync.sitemap import SitemapParseError
+
+
+class TestChangeList(unittest.TestCase):
+
+    def test01_add_if_changed(self):
+        cl = ChangeList()
+        cl.add_if_changed(Resource('a', timestamp=1, change='updated'))
+        self.assertEqual(len(cl), 1)
+        self.assertRaises(ChangeTypeError, cl.add_if_changed,
+                          Resource('c', timestamp=3))
+
+    def test02_set_with_repeats(self):
+        cl = ChangeList()
+        cl.add(Resource('a', timestamp=1, change='updated'))
+        cl.add(Resource('b', timestamp=1, change='created'))
+        cl.add(Resource('c', timestamp=1, change='deleted'))
+        cl.add(Resource('a', timestamp=2, change='deleted'))
+        cl.add(Resource('b', timestamp=2, change='updated'))
+        self.assertEqual(len(cl), 5, "5 changes in change_list")
+
+    def test03_with_repeats_again(self):
+        r1 = Resource(uri='a', length=1, change='created')
+        r2 = Resource(uri='b', length=2, change='created')
+        i = ChangeList()
+        i.add(r1)
+        i.add(r2)
+        self.assertEqual(len(i), 2)
+        # Can add another Resource with same URI
+        r1d = Resource(uri='a', length=10, change='created')
+        i.add(r1d)
+        self.assertEqual(len(i), 3)
+
+    def test04_change_list(self):
+        cl = ChangeList()
+        cl.add(Resource('a', timestamp=1, change='created'))
+        cl.add(Resource('b', timestamp=2, change='created'))
+        cl.add(Resource('c', timestamp=3, change='created'))
+        cl.add(Resource('d', timestamp=4, change='created'))
+        cl.add(Resource('e', timestamp=5, change='created'))
+        self.assertEqual(len(cl), 5, "5 things in src")
+
+    def test05_iter(self):
+        i = ChangeList()
+        i.add(Resource('a', timestamp=1, change='created'))
+        i.add(Resource('b', timestamp=2, change='created'))
+        i.add(Resource('c', timestamp=3, change='created'))
+        i.add(Resource('d', timestamp=4, change='created'))
+        resources = []
+        for r in i:
+            resources.append(r)
+        self.assertEqual(len(resources), 4)
+        self.assertEqual(resources[0].uri, 'a')
+        self.assertEqual(resources[3].uri, 'd')
+
+    def test06_add_changed_resources(self):
+        added = ResourceList()
+        added.add(Resource('a', timestamp=1, change='created'))
+        added.add(Resource('d', timestamp=4, change='created'))
+        self.assertEqual(len(added), 2, "2 things in added resource_list")
+        changes = ChangeList()
+        changes.add_changed_resources(added, change='created')
+        self.assertEqual(len(changes), 2, "2 things added")
+        i = iter(changes)
+        first = next(i)
+        self.assertEqual(first.uri, 'a', "changes[0].uri=a")
+        self.assertEqual(first.timestamp, 1, "changes[0].timestamp=1")
+        # , "changes[0].change=createdd")
+        self.assertEqual(first.change, 'created')
+        second = next(i)
+        self.assertEqual(second.timestamp, 4, "changes[1].timestamp=4")
+        self.assertEqual(second.change, 'created',
+                         "changes[1].change=createdd")
+        # Now add some with updated (one same, one diff)
+        updated = ResourceList()
+        updated.add(Resource('a', timestamp=5, change='created'))
+        updated.add(Resource('b', timestamp=6, change='created'))
+        self.assertEqual(len(updated), 2, "2 things in updated resource_list")
+        changes.add_changed_resources(updated, change='updated')
+        self.assertEqual(len(changes), 4, "4 = 2 old + 2 things updated")
+        # Make new resource_list from the changes which should not have dupes
+        dst = ResourceList()
+        dst.add(changes, replace=True)
+        self.assertEqual(len(dst), 3, "3 unique resources")
+        # 5 was later in last the 1
+        self.assertEqual(dst.resources['a'].timestamp, 5)
+        self.assertEqual(dst.resources['a'].change, 'updated')
+        self.assertEqual(dst.resources['b'].timestamp, 6)
+        self.assertEqual(dst.resources['b'].change, 'updated')
+        self.assertEqual(dst.resources['d'].timestamp, 4)
+        self.assertEqual(dst.resources['d'].change, 'created')
+
+    def test07_as_xml(self):
+        cl = ChangeList()
+        cl.md_from = '1970-01-01T00:00:00Z'
+        cl.add(Resource('a', timestamp=1, change='updated'))
+        cl.add(Resource('b', timestamp=2, change='updated'))
+        xml = cl.as_xml()
+        self.assertTrue(
+            re.search(r'<rs:md .*capability="changelist"', xml), 'XML has capability')
+        self.assertTrue(re.search(r'<rs:md .*from="\d\d\d\d\-\d\d\-\d\dT\d\d:\d\d:\d\dZ"',
+                                  xml), 'XML has from to seconds precision (and not more)')
+        self.assertTrue(re.search(
+            r'<url><loc>a</loc><lastmod>1970-01-01T00:00:01Z</lastmod>', xml), 'XML has resource a')
+
+    def test08_parse(self):
+        xml = '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
+<rs:md capability="changelist" from="2013-01-01"/>\
+<url><loc>/tmp/rs_test/src/file_a</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md change="updated" length="12" /></url>\
+<url><loc>/tmp/rs_test/src/file_b</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md change="deleted" length="32" /></url>\
+</urlset>'
+        cl = ChangeList()
+        cl.parse(fh=io.StringIO(xml))
+        self.assertEqual(len(cl.resources), 2, 'got 2 resources')
+        self.assertEqual(cl.md['capability'], 'changelist', 'capability set')
+        self.assertEqual(cl.md['md_from'], '2013-01-01')
+
+    def test09_parse_no_capability(self):
+        # missing capability is an error for changelist
+        xml = '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
+<url><loc>http://example.com/res1</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md change="updated"/></url>\
+</urlset>'
+        cl = ChangeList()
+        self.assertRaises(SitemapParseError, cl.parse, fh=io.StringIO(xml))
+
+    def test10_parse_bad_capability(self):
+        # the <rs:md capability="bad_capability".. should give error
+        xml = '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
+<rs:md capability="bad_capability" from="2013-01-01"/>\
+<url><loc>http://example.com/bad_res_1</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md change="updated"/></url>\
+</urlset>'
+        cl = ChangeList()
+        self.assertRaises(SitemapParseError, cl.parse, fh=io.StringIO(xml))
+
+    def test11_prune_updates_before(self):
+        """Test prune_updates_before method."""
+        # v1.0
+        cl = ChangeList()
+        cl.resources.append(Resource('a', timestamp=1))
+        cl.resources.append(Resource('b', timestamp=2))
+        cl.resources.append(Resource('c', timestamp=3))
+        cl.resources.append(Resource('d', timestamp=4))
+        cl.prune_updates_before(3, spec_version='1.0')
+        self.assertEqual(len(cl.resources), 2)
+        i = iter(cl)
+        self.assertEqual(next(i).uri, 'c')
+        self.assertEqual(next(i).uri, 'd')
+        # put some more back out of order
+        cl.resources.append(Resource('a', timestamp=1))
+        cl.resources.append(Resource('b', timestamp=2))
+        cl.resources.append(Resource('e', timestamp=1000))
+        cl.prune_updates_before(3.5, spec_version='1.0')
+        self.assertEqual(len(cl.resources), 2)
+        i = iter(cl)
+        self.assertEqual(next(i).uri, 'd')
+        self.assertEqual(next(i).uri, 'e')
+        # without a timestamp
+        cl.resources.append(Resource('nt_1_0', ts_datetime=123))
+        self.assertRaises(Exception, cl.prune_updates_before, 3.5, spec_version='1.0')
+        # v1.1
+        cl = ChangeList()
+        cl.resources.append(Resource('aa', ts_datetime=1))
+        cl.resources.append(Resource('bb', ts_datetime=2))
+        cl.resources.append(Resource('cc', ts_datetime=3))
+        cl.prune_updates_before(3, spec_version='1.1')
+        self.assertEqual(len(cl.resources), 1)
+        i = iter(cl)
+        self.assertEqual(next(i).uri, 'cc')
+        # put some more back out of order
+        cl.resources.append(Resource('aa', ts_datetime=1))
+        cl.resources.append(Resource('bb', ts_datetime=2))
+        cl.resources.append(Resource('ee', ts_datetime=1000))
+        cl.prune_updates_before(3.5, spec_version='1.1')
+        self.assertEqual(len(cl.resources), 1)
+        i = iter(cl)
+        self.assertEqual(next(i).uri, 'ee')
+        # without a ts_datetime
+        cl.resources.append(Resource('nt_1_1', timestamp=456))
+        self.assertRaises(Exception, cl.prune_updates_before, 3.5, spec_version='1.1')

tests/test_client.py

@@ -0,0 +1,394 @@
+from .testlib import TestCase, capture_stdout, webserver
+
+import unittest
+import re
+import logging
+from testfixtures import LogCapture
+import sys
+import os.path
+
+from resync.client import Client, ClientError, ClientFatalError
+from resync.resource import Resource
+from resync.resource_list import ResourceList
+from resync.change_list import ChangeList
+
+logging.basicConfig(level=logging.INFO)
+
+
+class TestClient(TestCase):
+    """Test cases for resync.client."""
+
+    def test01_make_resource_list_empty(self):
+        c = Client()
+        # No mapping is an error
+        self.assertRaises(ClientFatalError, c.build_resource_list)
+
+    def test02_bad_source_uri(self):
+        c = Client()
+        self.assertRaises(ClientFatalError, c.baseline_or_audit)
+        c.set_mappings(['http://example.org/bbb', '/tmp/this_does_not_exist'])
+        self.assertRaises(ClientFatalError, c.baseline_or_audit)
+
+    def test03_sitemap_uri(self):
+        c = Client()
+        c.set_mappings(['http://example.org/c', '/tmp/not_there_at_all'])
+        self.assertEqual(c.sitemap_uri('abcd1'), 'http://example.org/c/abcd1')
+        self.assertEqual(c.sitemap_uri('/abcd2'), '/abcd2')
+        self.assertEqual(c.sitemap_uri('scheme:/abcd3'), 'scheme:/abcd3')
+
+    def test04_read_resource_list(self):
+        c = Client()
+        rl = c.read_resource_list('tests/testdata/client/dir1/resourcelist.xml')
+        self.assertEqual(len(rl), 3)
+        rl = c.read_resource_list('file:tests/testdata/client/dir1/resourcelist.xml')
+        self.assertEqual(len(rl), 3)
+        self.assertEqual(len(rl), 3)
+        self.assertRaises(ClientError, c.read_resource_list, 'file://tests/testdata/client/dir1/resourcelist.xml')
+        self.assertRaises(ClientError, c.read_resource_list, 'file:///tests/testdata/client/dir1/resourcelist.xml')
+        self.assertRaises(ClientError, c.read_resource_list, 'DOES_NOT_EXIST')
+
+    def test05_find_resource_list(self):
+        c = Client()
+        # Filesystem tests
+        c.set_mappings(['tests/testdata/find/find1', 'xxx'])
+        self.assertEqual(c.find_resource_list().up, 'find1')
+        c.set_mappings(['tests/testdata/find/find2', 'xxx'])
+        self.assertEqual(c.find_resource_list().up, 'find2')
+        c.set_mappings(['tests/testdata/find/find3', 'xxx'])
+        self.assertEqual(c.find_resource_list().up, 'find3')
+        # Tests requiring a server
+        with webserver('tests/testdata/find', 'localhost', 9999):
+            c.set_mappings(['http://localhost:9999/find1', 'xxx'])
+            self.assertEqual(c.find_resource_list().up, 'find1')
+            c.set_mappings(['http://localhost:9999/find2', 'xxx'])
+            self.assertEqual(c.find_resource_list().up, 'find2')
+            c.set_mappings(['http://localhost:9999/find3', 'xxx'])
+            self.assertEqual(c.find_resource_list().up, 'find3')
+        with webserver('tests/testdata/find/find1', 'localhost', 9999):
+            c.set_mappings(['http://localhost:9999/data', 'xxx'])
+            self.assertEqual(c.find_resource_list().up, 'find1')
+        with webserver('tests/testdata/find/find3', 'localhost', 9999):
+            c.set_mappings(['http://localhost:9999/data/data1', 'xxx'])
+            self.assertEqual(c.find_resource_list().up, 'find3')
+
+    def test10_baseline_or_audit(self):
+        # FIXME - this is the guts of the client, tough to test, need to work
+        # through more cases...
+        c = Client()
+        dst = os.path.join(self.tmpdir, 'dst_dir1')
+        with webserver('tests/testdata/client', 'localhost', 9999):
+            c.set_mappings(['http://localhost:9999/dir1', dst])
+            # audit with empty dst, should say 3 to create
+            with LogCapture() as lc:
+                c.baseline_or_audit(audit_only=True)
+                self.assertTrue(
+                    re.match(r'Status:\s+NOT IN SYNC.*to create=3', lc.records[-2].msg))
+                self.assertEqual(lc.records[-1].msg, 'Completed audit')
+            # now do the sync
+            with LogCapture() as lc:
+                c.baseline_or_audit()
+                self.assertTrue(
+                    re.match(r'Status:\s+SYNCED.*created=3', lc.records[-2].msg))
+                self.assertEqual(lc.records[-1].msg, 'Completed baseline sync')
+
+    def test18_update_resource(self):
+        c = Client()
+        resource = Resource(uri='http://example.org/dir/2')
+        filename = os.path.join(self.tmpdir, 'dir/resource2')
+        # dryrun
+        with LogCapture() as lc:
+            c.dryrun = True
+            c.logger = logging.getLogger('resync.client')
+            n = c.update_resource(resource, filename)
+            self.assertEqual(n, 0)
+            self.assertTrue(
+                lc.records[-1].msg.startswith('dryrun: would GET http://example.org/dir/2 '))
+        c.dryrun = False
+        with webserver('tests/testdata', 'localhost', 9999):
+            # get from file uri that does not exist
+            resource = Resource(uri='http://localhost:9999/i_do_not_exist')
+            self.assertRaises(ClientFatalError,
+                              c.update_resource, resource, filename)
+            # get from file uri that does not exist but with c.ignore_failures to
+            # log
+            resource = Resource(uri='http://localhost:9999/i_do_not_exist')
+            with LogCapture() as lc:
+                c.logger = logging.getLogger('resync.client')
+                c.ignore_failures = True
+                n = c.update_resource(resource, filename)
+                self.assertEqual(n, 0)
+                self.assertTrue(
+                    lc.records[-1].msg.startswith('Failed to GET http://localhost:9999/i_do_not_exist '))
+            # get from file uri
+            resource = Resource(uri='http://localhost:9999/examples_from_spec_v1_0/resourcesync_ex_1.xml',
+                                length=355, md5='abc',
+                                timestamp=10)
+            c.last_timestamp = 0
+            with LogCapture() as lc:
+                c.logger = logging.getLogger('resync.client')
+                n = c.update_resource(resource, filename)
+                self.assertEqual(n, 1)
+                self.assertTrue(lc.records[-1].msg.startswith('Event: {'))
+            # get from file uri with length and md5 warnings
+            resource = Resource(uri='http://localhost:9999/examples_from_spec_v1_0/resourcesync_ex_1.xml',
+                                length=111, md5='abc',
+                                timestamp=10)
+            c.last_timestamp = 0
+            with LogCapture() as lc:
+                c.logger = logging.getLogger('resync.client')
+                c.hashes = set(['md5'])
+                n = c.update_resource(resource, filename)
+                self.assertEqual(n, 1)
+                self.assertTrue(lc.records[-1].msg.startswith('MD5 mismatch '))
+                self.assertTrue(
+                    lc.records[-2].msg.startswith('Downloaded size for '))
+                self.assertTrue(lc.records[-3].msg.startswith('Event: {'))
+
+    def test19_delete_resource(self):
+        c = Client()
+        resource = Resource(uri='http://example.org/1')
+        filename = os.path.join(self.tmpdir, 'resource1')
+        c.last_timestamp = 5
+        # no delete, no timestamp update
+        with LogCapture() as lc:
+            c.logger = logging.getLogger('resync.client')
+            n = c.delete_resource(resource, filename)
+            self.assertEqual(n, 0)
+            self.assertEqual(lc.records[-1].msg,
+                             'nodelete: would delete http://example.org/1 (--delete to enable)')
+            self.assertEqual(c.last_timestamp, 5)
+        # no delete but timestamp update
+        resource.timestamp = 10
+        with LogCapture() as lc:
+            c.logger = logging.getLogger('resync.client')
+            n = c.delete_resource(resource, filename)
+            self.assertEqual(n, 0)
+            self.assertEqual(lc.records[-1].msg,
+                             'nodelete: would delete http://example.org/1 (--delete to enable)')
+            self.assertEqual(c.last_timestamp, 10)
+        # allow delete but dryrun
+        with LogCapture() as lc:
+            c.dryrun = True
+            c.logger = logging.getLogger('resync.client')
+            n = c.delete_resource(resource, filename, allow_deletion=True)
+            self.assertEqual(n, 0)
+            self.assertTrue(
+                lc.records[-1].msg.startswith('dryrun: would delete http://example.org/1'))
+        c.dryrun = False
+        # allow delete but no resource present
+        with LogCapture() as lc:
+            c.logger = logging.getLogger('resync.client')
+            n = c.delete_resource(resource, filename, allow_deletion=True)
+            self.assertEqual(n, 0)
+            self.assertTrue(
+                lc.records[-1].msg.startswith('Failed to DELETE http://example.org/1'))
+        # successful deletion, first make file...
+        with open(filename, 'w') as fh:
+            fh.write('delete me')
+            fh.close()
+        with LogCapture() as lc:
+            c.logger = logging.getLogger('resync.client')
+            n = c.delete_resource(resource, filename, allow_deletion=True)
+            self.assertEqual(n, 1)
+            self.assertTrue(lc.records[-1].msg.startswith('Event: {'))
+            self.assertTrue(
+                lc.records[-2].msg.startswith('deleted: http://example.org/1 ->'))
+
+    def test20_parse_document(self):
+        # Key property of the parse_document() method is that it parses the
+        # document and identifies its type
+        c = Client()
+        with capture_stdout() as capturer:
+            c.sitemap_name = 'tests/testdata/examples_from_spec_v1_0/resourcesync_ex_1.xml'
+            c.parse_document()
+        self.assertRegex(capturer.result, r'Parsed resourcelist document with 2 entries')
+        with capture_stdout() as capturer:
+            c.sitemap_name = 'tests/testdata/examples_from_spec_v1_0/resourcesync_ex_17.xml'
+            c.parse_document()
+        self.assertRegex(capturer.result, r'Parsed resourcedump document with 3 entries')
+        with capture_stdout() as capturer:
+            c.sitemap_name = 'tests/testdata/examples_from_spec_v1_0/resourcesync_ex_19.xml'
+            c.parse_document()
+        self.assertRegex(capturer.result, r'Parsed changelist document with 4 entries')
+        with capture_stdout() as capturer:
+            c.sitemap_name = 'tests/testdata/examples_from_spec_v1_0/resourcesync_ex_22.xml'
+            c.parse_document()
+        self.assertRegex(capturer.result, r'Parsed changedump document with 3 entries')
+        # Document that doesn't exist
+        c.sitemap_name = '/does_not_exist'
+        self.assertRaises(ClientFatalError, c.parse_document)
+        # and verbose with truncation...
+        with capture_stdout() as capturer:
+            c.verbose = True
+            c.sitemap_name = 'tests/testdata/examples_from_spec_v1_0/resourcesync_ex_1.xml'
+            c.max_sitemap_entries = 1
+            c.parse_document()
+        self.assertRegex(capturer.result, r'Showing first 1 entries')
+        self.assertRegex(capturer.result, r'\[0\] ')
+        self.assertNotRegex(capturer.result, r'\[1\] ')
+
+    def test40_write_resource_list_mappings(self):
+        c = Client()
+        c.set_mappings(['http://example.org/d1=tests/testdata/dir1',
+                        'http://example.org/d2=tests/testdata/dir2'])
+        # with no explicit paths seting the mapping will be used
+        with capture_stdout() as capturer:
+            c.write_resource_list()
+        # sys.stderr.write(capturer.result)
+        self.assertRegex(capturer.result, r'<rs:md at="\S+" capability="resourcelist"')
+        self.assertRegex(capturer.result, r'<url><loc>http://example.org/d1/file_a</loc>')
+        self.assertRegex(capturer.result, r'<url><loc>http://example.org/d1/file_b</loc>')
+        self.assertRegex(capturer.result, r'<url><loc>http://example.org/d2/file_x</loc>')
+
+    def test41_write_resource_list_path(self):
+        c = Client()
+        c.set_mappings(['http://example.org/', 'tests/testdata/'])
+        links = [{'rel': 'uri_c', 'href': 'uri_d'}]
+        # with an explicit paths setting only the specified paths will be
+        # included
+        with capture_stdout() as capturer:
+            c.write_resource_list(paths='tests/testdata/dir1', links=links)
+        print("capturer.result: " + capturer.result)
+        self.assertRegex(capturer.result, r'<rs:md at="\S+" capability="resourcelist"')
+        self.assertRegex(capturer.result, r'<url><loc>http://example.org/dir1/file_a</loc>')
+        self.assertRegex(capturer.result, r'<url><loc>http://example.org/dir1/file_b</loc>')
+        self.assertNotRegex(capturer.result, r'<url><loc>http://example.org/dir2/file_x</loc>')
+        # check link present
+        self.assertRegex(capturer.result, r'rel="uri_c"')
+        self.assertRegex(capturer.result, r'href="uri_d"')
+        # Travis CI does not preserve timestamps from github so test here for the file
+        # size but not the datestamp
+        # self.assertRegex(capturer.result, r'<url><loc>http://example.org/dir1/file_a</loc><lastmod>[\w\-:]+</lastmod><rs:md length="20" /></url>')
+        # self.assertRegex(capturer.result, r'<url><loc>http://example.org/dir1/file_b</loc><lastmod>[\w\-:]+</lastmod><rs:md length="45" /></url>')
+        # to file
+        outfile = os.path.join(self.tmpdir, 'rl_out.xml')
+        c.write_resource_list(paths='tests/testdata/dir1', outfile=outfile)
+        self.assertTrue(os.path.getsize(outfile) > 100)
+        # dump instead (default file)
+        c.default_resource_dump = os.path.join(self.tmpdir, 'rl_out_dump_def')
+        outfile = c.default_resource_dump + '00000.zip'
+        self.assertFalse(os.path.exists(outfile))
+        c.write_resource_list(paths='tests/testdata/dir1', dump=True)
+        self.assertTrue(os.path.getsize(outfile) > 100)
+        # (specific file)
+        outbase = os.path.join(self.tmpdir, 'rl_out_dump')
+        outfile = outbase + '00000.zip'
+        self.assertFalse(os.path.exists(outfile))
+        c.write_resource_list(paths='tests/testdata/dir1',
+                              dump=True, outfile=outbase)
+        self.assertTrue(os.path.getsize(outfile) > 100)
+
+    def test45_write_change_list(self):
+        c = Client()
+        ex1 = 'tests/testdata/examples_from_spec_v1_0/resourcesync_ex_1.xml'
+        with capture_stdout() as capturer:
+            c.write_change_list(ref_sitemap=ex1, newref_sitemap=ex1)
+        self.assertRegex(capturer.result, r'<rs:md capability="changelist"')
+        # compare ex1 with testdata on disk
+        c.set_mappings(['http://example.org/', 'tests/testdata/'])
+        with capture_stdout() as capturer:
+            c.write_change_list(ref_sitemap=ex1, paths='tests/testdata/dir1')
+        self.assertRegex(capturer.result, r'<rs:md capability="changelist"')
+        self.assertRegex(capturer.result, r'<url><loc>http://example.com/res1</loc><rs:md change="deleted" /></url>')
+        # to file
+        outfile = os.path.join(self.tmpdir, 'cl_out.xml')
+        c.write_change_list(
+            ref_sitemap=ex1, newref_sitemap=ex1, outfile=outfile)
+        self.assertTrue(os.path.getsize(outfile) > 100)
+
+    def test46_write_capability_list(self):
+        c = Client()
+        caps = {'a': 'uri_a', 'b': 'uri_b'}
+        # simple case to STDOUT
+        with capture_stdout() as capturer:
+            c.write_capability_list(caps)
+        self.assertRegex(capturer.result, r'<urlset ')
+        self.assertRegex(capturer.result, r'<rs:md capability="capabilitylist" />')
+        self.assertRegex(capturer.result, r'<url><loc>uri_a</loc><rs:md capability="a"')
+        self.assertRegex(capturer.result, r'<url><loc>uri_b</loc><rs:md capability="b"')
+        # to file (just check that something is written)
+        outfile = os.path.join(self.tmpdir, 'caps_out.xml')
+        c.write_capability_list(capabilities=caps, outfile=outfile)
+        self.assertTrue(os.path.getsize(outfile) > 100)
+
+    def test47_write_source_description(self):
+        c = Client()
+        # simple case to STDOUT
+        with capture_stdout() as capturer:
+            c.write_source_description(['a', 'b', 'c'])
+        self.assertRegex(capturer.result, r'<urlset ')
+        self.assertRegex(capturer.result, r'<rs:md capability="description" />')
+        self.assertRegex(capturer.result, r'<url><loc>a</loc><rs:md capability="capabilitylist" /></url>')
+        self.assertRegex(capturer.result, r'<url><loc>b</loc><rs:md capability="capabilitylist" /></url>')
+        # more complex case to STDOUT
+        with capture_stdout() as capturer:
+            c.write_source_description(
+                capability_lists=['http://a.b/'], links=[{'rel': 'c', 'href': 'd'}])
+        self.assertRegex(capturer.result, r'http://a.b/')
+        # to file (just check that something is written)
+        outfile = os.path.join(self.tmpdir, 'sd_out.xml')
+        c.write_source_description(capability_lists=[
+                                   'http://a.b/'], outfile=outfile, links=[{'rel': 'c', 'href': 'd'}])
+        self.assertTrue(os.path.getsize(outfile) > 100)
+
+    def test48_write_dump_if_requested(self):
+        c = Client()
+        # no dump file
+        self.assertFalse(c.write_dump_if_requested(ChangeList(), None))
+        # with dump file
+        with capture_stdout() as capturer:
+            c.write_dump_if_requested(ChangeList(), '/tmp/a_file')
+        self.assertRegex(capturer.result, r'FIXME')
+
+    def test49_read_reference_resource_list(self):
+        c = Client()
+        with capture_stdout() as capturer:
+            rl = c.read_reference_resource_list(
+                'tests/testdata/examples_from_spec_v1_0/resourcesync_ex_1.xml')
+        self.assertEqual(len(rl), 2)
+        self.assertEqual('', capturer.result)
+        c.verbose = True
+        with capture_stdout() as capturer:
+            rl = c.read_reference_resource_list(
+                'tests/testdata/examples_from_spec_v1_0/resourcesync_ex_1.xml')
+        self.assertEqual(len(rl), 2)
+        self.assertRegex(capturer.result, r'http://example.com/res2')
+        c.verbose = True
+        c.max_sitemap_entries = 1
+        with capture_stdout() as capturer:
+            rl = c.read_reference_resource_list(
+                'tests/testdata/examples_from_spec_v1_0/resourcesync_ex_1.xml')
+        self.assertEqual(len(rl), 2)
+        self.assertRegex(capturer.result, r'http://example.com/res1')
+        self.assertRegex(capturer.result, r'Showing first 1 entries')
+        self.assertNotRegex(capturer.result, r'http://example.com/res2')
+
+    def test50_log_status(self):
+        c = Client()
+        with LogCapture() as lc:
+            c.logger = logging.getLogger('resync.client')
+            c.log_status()
+            self.assertEqual(lc.records[-1].msg,
+                             'Status:         IN SYNC (created=0, updated=0, deleted=0)')
+            c.log_status(created=123, updated=456, deleted=789)
+            self.assertEqual(lc.records[-1].msg,
+                             'Status:         IN SYNC (created=123, updated=456, deleted=789)')
+            c.log_status(incremental=True)
+            self.assertEqual(lc.records[-1].msg,
+                             'Status:      NO CHANGES (created=0, updated=0, deleted=0)')
+            c.log_status(audit=True)
+            self.assertEqual(lc.records[-1].msg,
+                             'Status:         IN SYNC (to create=0, to update=0, to delete=0)')
+            c.log_status(in_sync=False, audit=True)
+            self.assertEqual(lc.records[-1].msg,
+                             'Status:     NOT IN SYNC (to create=0, to update=0, to delete=0)')
+            c.log_status(in_sync=False, audit=False, to_delete=1)
+            self.assertEqual(lc.records[-1].msg,
+                             'Status:     PART SYNCED (created=0, updated=0, to delete (--delete)=1)')
+            c.log_status(in_sync=False, audit=False,
+                         to_delete=1, incremental=1)
+            self.assertEqual(lc.records[-1].msg,
+                             'Status:    PART APPLIED (created=0, updated=0, to delete (--delete)=1)')
+            c.log_status(in_sync=False)
+            self.assertEqual(lc.records[-1].msg,
+                             'Status:          SYNCED (created=0, updated=0, deleted=0)')

tests/test_client_state.py

@@ -0,0 +1,21 @@
+from .testlib import TestCase
+
+import os.path
+from resync.client_state import ClientState
+
+
+class TestClientState(TestCase):
+
+    def test01_set_and_get(self):
+        cs = ClientState()
+        self.assertEqual(cs.status_file, '.resync-client-status.cfg')
+        cs.status_file = os.path.join(self.tmpdir, cs.status_file)
+        # Get state
+        site = 'https://this.site/'
+        self.assertEqual(cs.get_state(site), None)
+        cs.set_state(site, 123)
+        self.assertEqual(cs.get_state(site), 123)
+        cs.set_state(site, 456)
+        self.assertEqual(cs.get_state(site), 456)
+        cs.set_state(site)
+        self.assertEqual(cs.get_state(site), None)

tests/test_client_utils.py

@@ -0,0 +1,77 @@
+from .testlib import TestCase
+
+import logging
+import os.path
+import re
+import unittest
+from resync.client_utils import init_logging, count_true_args, parse_links, parse_link, parse_capabilities, parse_capability_lists
+from resync.client import ClientFatalError
+
+
+class TestClientUtils(TestCase):
+
+    def test01_init_logging(self):
+        # to_file=False, logfile=None, default_logfile='/tmp/resync.log',
+        #         human=True, verbose=False, eval_mode=False,
+        #         default_logger='client', extra_loggers=None):
+        tmplog = os.path.join(self.tmpdir, 'tmp.log')
+        init_logging(to_file=True, default_logfile=tmplog,
+                     extra_loggers=['x1', 'x2'])
+        # check x1 and x2 set, not x3 (can tell by level)
+        self.assertTrue(logging.getLogger('x1').level, logging.DEBUG)
+        self.assertTrue(logging.getLogger('x2').level, logging.DEBUG)
+        self.assertEqual(logging.getLogger('x3').level, 0)
+        # write something, check goes to file
+        log = logging.getLogger('resync')
+        log.warning('PIGS MIGHT FLY')
+        logtxt = open(tmplog, 'r').read()
+        self.assertTrue(re.search(r'WARNING \| PIGS MIGHT FLY', logtxt))
+
+    def test02_count_true_args(self):
+        self.assertEqual(count_true_args(), 0)
+        self.assertEqual(count_true_args(True), 1)
+        self.assertEqual(count_true_args(False), 0)
+        self.assertEqual(count_true_args(0, 1, 2, 3), 3)
+
+    def test03_parse_links(self):
+        self.assertEqual(parse_links([]), [])
+        self.assertEqual(parse_links(['u,h']), [{'href': 'h', 'rel': 'u'}])
+        self.assertEqual(parse_links(['u,h', 'v,i']), [
+                         {'href': 'h', 'rel': 'u'}, {'href': 'i', 'rel': 'v'}])
+        self.assertRaises(ClientFatalError, parse_links, 'xx')
+        self.assertRaises(ClientFatalError, parse_links, ['u'])
+        self.assertRaises(ClientFatalError, parse_links, ['u,h', 'u'])
+
+    def test04_parse_link(self):
+        # Input string of the form: rel,href,att1=val1,att2=val2
+        self.assertEqual(parse_link('u,h'), {'href': 'h', 'rel': 'u'})
+        self.assertEqual(parse_link('u,h,a=b'), {
+                         'a': 'b', 'href': 'h', 'rel': 'u'})
+        self.assertEqual(parse_link('u,h,a=b,c=d'), {
+                         'a': 'b', 'c': 'd', 'href': 'h', 'rel': 'u'})
+        self.assertEqual(parse_link('u,h,a=b,a=d'), {
+                         'a': 'd', 'href': 'h', 'rel': 'u'})  # desired??
+        self.assertRaises(ClientFatalError, parse_link, '')
+        self.assertRaises(ClientFatalError, parse_link, 'u')
+        self.assertRaises(ClientFatalError, parse_link, 'u,')
+        self.assertRaises(ClientFatalError, parse_link, 'u,h,,')
+        self.assertRaises(ClientFatalError, parse_link, 'u,h,a')
+        self.assertRaises(ClientFatalError, parse_link, 'u,h,a=')
+        self.assertRaises(ClientFatalError, parse_link, 'u,h,a=b,=c')
+
+    def test05_parse_capabilities(self):
+        # Input string of the form: cap_name=uri,cap_name=uri
+        # good
+        c = parse_capabilities('a=')
+        self.assertEqual(len(c), 1)
+        self.assertEqual(c['a'], '')
+        c = parse_capabilities('a=b,c=')
+        self.assertEqual(len(c), 2)
+        self.assertEqual(c['a'], 'b')
+        # bad
+        self.assertRaises(ClientFatalError, parse_capabilities, 'a')
+        self.assertRaises(ClientFatalError, parse_capabilities, 'a=b,')
+
+    def test06_parse_capability_lists(self):
+        # Input string of the form: uri,uri
+        self.assertEqual(parse_capability_lists('a,b'), ['a', 'b'])

tests/test_dump.py

@@ -0,0 +1,164 @@
+from .testlib import TestCase
+
+import os.path
+import zipfile
+
+from resync.dump import Dump, DumpError
+from resync.resource_list import ResourceList
+from resync.change_list import ChangeList
+from resync.resource import Resource
+from resync.resource_dump_manifest import ResourceDumpManifest
+from resync.change_dump_manifest import ChangeDumpManifest
+
+
+class TestDump(TestCase):
+
+    def test00_dump_zip_resource_list(self):
+        rl = ResourceDumpManifest()
+        rl.add(Resource('http://ex.org/a', length=7, path='tests/testdata/a'))
+        rl.add(Resource('http://ex.org/b', length=21, path='tests/testdata/b'))
+        d = Dump()
+        zipf = os.path.join(self.tmpdir, "test00_dump.zip")
+        d.write_zip(resources=rl, dumpfile=zipf)  # named args
+        self.assertTrue(os.path.exists(zipf))
+        self.assertTrue(zipfile.is_zipfile(zipf))
+        zo = zipfile.ZipFile(zipf, 'r')
+        self.assertEqual(len(zo.namelist()), 3)
+        zo.close()
+        os.unlink(zipf)
+
+    def test01_dump_zip_change_list(self):
+        cl = ChangeDumpManifest()
+        cl.add(Resource('http://ex.org/a', length=7,
+                        path='tests/testdata/a', change="updated"))
+        cl.add(Resource('http://ex.org/b', length=21,
+                        path='tests/testdata/b', change="updated"))
+        d = Dump()
+        zipf = os.path.join(self.tmpdir, "test01_dump.zip")
+        d.write_zip(cl, zipf)  # positional args
+        self.assertTrue(os.path.exists(zipf))
+        self.assertTrue(zipfile.is_zipfile(zipf))
+        zo = zipfile.ZipFile(zipf, 'r')
+        self.assertEqual(len(zo.namelist()), 3)
+        zo.close()
+        os.unlink(zipf)
+
+    def test02_dump_check_files(self):
+        cl = ChangeList()
+        cl.add(Resource('http://ex.org/a', length=7,
+                        path='tests/testdata/a', change="updated"))
+        cl.add(Resource('http://ex.org/b', length=21,
+                        path='tests/testdata/b', change="updated"))
+        d = Dump(resources=cl)
+        self.assertTrue(d.check_files())
+        self.assertEqual(d.total_size, 28)
+
+    def test03_dump_multi_file_max_size(self):
+        rl = ResourceList()
+        for letter in map(chr, range(ord('a'), ord('l') + 1)):
+            uri = 'http://ex.org/%s' % (letter)
+            fname = 'tests/testdata/a_to_z/%s' % (letter)
+            rl.add(Resource(uri, path=fname))
+        self.assertEqual(len(rl), 12)
+        # d=Dump(rl)
+        # tmpdir=tempfile.mkdtemp()
+        # tmpbase=os.path.join(tmpdir,'base')
+        # d.max_size=2000 # start new zip after size exceeds 2000 bytes
+        # n=d.write(tmpbase)
+        # self.assertEqual( n, 2, 'expect to write 2 dump files' )
+        #
+        # Now repeat with large size limit but small number of files limit
+        d2 = Dump(rl)
+        tmpbase = os.path.join(self.tmpdir, 'test03_')
+        d2.max_files = 4
+        n = d2.write(tmpbase)
+        self.assertEqual(n, 3, 'expect to write 3 dump files')
+        self.assertTrue(os.path.isfile(tmpbase + '00000.zip'))
+        self.assertTrue(os.path.isfile(tmpbase + '00001.zip'))
+        self.assertTrue(os.path.isfile(tmpbase + '00002.zip'))
+        # Look at the first file in detail
+        zipf = tmpbase + '00000.zip'
+        zo = zipfile.ZipFile(zipf, 'r')
+        self.assertEqual(zo.namelist(), ['manifest.xml', 'a', 'b', 'c', 'd'])
+        # self.assertEqual( zo.getinfo('manifest.xml').file_size, 470 )
+        self.assertEqual(zo.getinfo('a').file_size, 9)
+        self.assertEqual(zo.getinfo('b').file_size, 1116)
+        self.assertEqual(zo.getinfo('c').file_size, 32)
+        self.assertEqual(zo.getinfo('d').file_size, 13)
+        zo.close()
+        os.unlink(zipf)
+        # Check second and third files have expected contents
+        zipf = tmpbase + '00001.zip'
+        zo = zipfile.ZipFile(zipf, 'r')
+        self.assertEqual(zo.namelist(), ['manifest.xml', 'e', 'f', 'g', 'h'])
+        zo.close()
+        os.unlink(zipf)
+        zipf = tmpbase + '00002.zip'
+        zo = zipfile.ZipFile(zipf, 'r')
+        self.assertEqual(zo.namelist(), ['manifest.xml', 'i', 'j', 'k', 'l'])
+        zo.close()
+        os.unlink(zipf)
+
+    def test04_dump_multi_file_max_size(self):
+        rl = ResourceList()
+        for letter in map(chr, range(ord('a'), ord('l') + 1)):
+            uri = 'http://ex.org/%s' % (letter)
+            fname = 'tests/testdata/a_to_z/%s' % (letter)
+            rl.add(Resource(uri, path=fname))
+        self.assertEqual(len(rl), 12)
+        d2 = Dump(rl)
+        tmpbase = os.path.join(self.tmpdir, 'test0f_')
+        d2.max_size = 2000
+        n = d2.write(tmpbase)
+        self.assertEqual(n, 2, 'expect to write 2 dump files')
+        self.assertTrue(os.path.isfile(tmpbase + '00000.zip'))
+        self.assertTrue(os.path.isfile(tmpbase + '00001.zip'))
+        # Look at the first file in detail
+        zipf = tmpbase + '00000.zip'
+        zo = zipfile.ZipFile(zipf, 'r')
+        self.assertEqual(
+            zo.namelist(), ['manifest.xml', 'a', 'b', 'c', 'd', 'e', 'f'])
+        # self.assertEqual( zo.getinfo('manifest.xml').file_size, 470 )
+        self.assertEqual(zo.getinfo('a').file_size, 9)
+        self.assertEqual(zo.getinfo('b').file_size, 1116)
+        self.assertEqual(zo.getinfo('c').file_size, 32)
+        self.assertEqual(zo.getinfo('d').file_size, 13)
+        self.assertEqual(zo.getinfo('e').file_size, 20)
+        self.assertEqual(zo.getinfo('f').file_size, 1625)
+        zo.close()
+        os.unlink(zipf)
+        # Check second and third files have expected contents
+        zipf = tmpbase + '00001.zip'
+        zo = zipfile.ZipFile(zipf, 'r')
+        self.assertEqual(
+            zo.namelist(), ['manifest.xml', 'g', 'h', 'i', 'j', 'k', 'l'])
+        zo.close()
+        os.unlink(zipf)
+
+    def test10_no_path(self):
+        rl = ResourceList()
+        rl.add(Resource('http://ex.org/a', length=7, path='tests/testdata/a'))
+        rl.add(Resource('http://ex.org/b', length=21))
+        d = Dump(rl)
+        self.assertRaises(DumpError, d.check_files)
+
+    def test11_bad_size(self):
+        rl = ResourceList()
+        rl.add(Resource('http://ex.org/a', length=9999, path='tests/testdata/a'))
+        d = Dump(rl)
+        self.assertTrue(d.check_files(check_length=False))
+        self.assertRaises(DumpError, d.check_files)
+
+    def test10_no_path(self):
+        rl = ResourceList()
+        rl.add(Resource('http://ex.org/a', length=7, path='tests/testdata/a'))
+        rl.add(Resource('http://ex.org/b', length=21))
+        d = Dump(rl)
+        self.assertRaises(DumpError, d.check_files)
+
+    def test11_bad_size(self):
+        rl = ResourceList()
+        rl.add(Resource('http://ex.org/a', length=9999, path='tests/testdata/a'))
+        d = Dump(rl)
+        self.assertTrue(d.check_files(check_length=False))
+        self.assertRaises(DumpError, d.check_files)

tests/test_examples_from_spec_v1_0.py

@@ -0,0 +1,833 @@
+"""Tests on the examples in the specification documents
+"""
+import re
+import sys
+
+from resync.resource import Resource
+from resync.source_description import SourceDescription
+from resync.capability_list import CapabilityList
+from resync.resource_list import ResourceList, ResourceListOrdered
+from resync.resource_dump import ResourceDump
+from resync.resource_dump_manifest import ResourceDumpManifest
+from resync.change_list import ChangeList
+from resync.change_dump import ChangeDump
+from resync.change_dump_manifest import ChangeDumpManifest
+from resync.archives import ResourceListArchive, ResourceDumpArchive, ChangeListArchive, ChangeDumpArchive
+from resync.sitemap import Sitemap
+
+from .testlib.testcase_with_xml_comparison import TestCase
+
+
+class TestExamplesFromSpec(TestCase):
+
+    def _ex_path(self, ex):
+        return 'tests/testdata/examples_from_spec_v1_0/%s.xml' % (ex)
+
+    def test_all_simple_read(self):
+        """Just try to read each one"""
+        for ex in ('archives_ex_2_1', 'archives_ex_2_2',
+                   'archives_ex_3_1', 'archives_ex_3_2',
+                   'archives_ex_4_1',
+                   'archives_ex_5_1',
+                   'archives_ex_6_1',
+                   'resourcesync_ex_1', 'resourcesync_ex_2', 'resourcesync_ex_3',
+                   'resourcesync_ex_4', 'resourcesync_ex_5', 'resourcesync_ex_6',
+                   'resourcesync_ex_7', 'resourcesync_ex_8', 'resourcesync_ex_12',
+                   'resourcesync_ex_13', 'resourcesync_ex_14', 'resourcesync_ex_15',
+                   'resourcesync_ex_16', 'resourcesync_ex_17', 'resourcesync_ex_18',
+                   'resourcesync_ex_19', 'resourcesync_ex_20', 'resourcesync_ex_21',
+                   'resourcesync_ex_22', 'resourcesync_ex_23', 'resourcesync_ex_24',
+                   'resourcesync_ex_25', 'resourcesync_ex_26', 'resourcesync_ex_27',
+                   'resourcesync_ex_28', 'resourcesync_ex_29', 'resourcesync_ex_30',
+                   'resourcesync_ex_31', 'resourcesync_ex_32', 'resourcesync_ex_33'):
+            s = Sitemap()
+            with open(self._ex_path(ex), 'r') as fh:
+                si = s.parse_xml(fh=fh)
+
+    def test_ex_01(self):
+        """resourcesync_ex_1 is a simple resource_list with 2 resources, no metadata"""
+        rl = ResourceList(spec_version='1.0')
+        rl.parse(uri='tests/testdata/examples_from_spec_v1_0/resourcesync_ex_1.xml')
+        self.assertEqual(rl.capability, 'resourcelist')
+        self.assertEqual(len(rl.resources), 2, '2 resources')
+        sms = sorted(rl.uris())
+        self.assertEqual(
+            sms, ['http://example.com/res1', 'http://example.com/res2'])
+        self.assertEqual(rl.resources['http://example.com/res1'].lastmod, None)
+
+    def test_ex_02(self):
+        """resourcesync_ex_2 is a simple resource_list with 2 resources, some metadata"""
+        rl = ResourceList(spec_version='1.0')
+        rl.parse(uri='tests/testdata/examples_from_spec_v1_0/resourcesync_ex_2.xml')
+        self.assertEqual(len(rl.resources), 2, '2 resources')
+        sms = sorted(rl.uris())
+        self.assertEqual(
+            sms, ['http://example.com/res1', 'http://example.com/res2'])
+        self.assertEqual(
+            rl.resources['http://example.com/res1'].lastmod, '2013-01-02T13:00:00Z')
+        self.assertEqual(
+            rl.resources['http://example.com/res2'].lastmod, '2013-01-02T14:00:00Z')
+        self.assertEqual(
+            rl.resources['http://example.com/res1'].md5, '1584abdf8ebdc9802ac0c6a7402c03b6')
+        self.assertEqual(
+            rl.resources['http://example.com/res2'].md5, '1e0d5cb8ef6ba40c99b14c0237be735e')
+
+    def test_ex_03(self):
+        """resourcesync_ex_3 is a simple change_list with 2 resources"""
+        cl = ChangeList(spec_version='1.0')
+        cl.parse('tests/testdata/examples_from_spec_v1_0/resourcesync_ex_3.xml')
+        self.assertEqual(len(cl.resources), 2, '2 resources')
+        sms = sorted(cl.uris())
+        self.assertEqual(sms, ['http://example.com/res2.pdf', 'http://example.com/res3.tiff'])
+        self.assertEqual(cl.resources[0].lastmod, '2013-01-02T13:00:00Z')
+        self.assertEqual(cl.resources[1].lastmod, '2013-01-02T18:00:00Z')
+        self.assertEqual(cl.resources[0].change, 'updated')
+        self.assertEqual(cl.resources[1].change, 'deleted')
+
+    def test_ex_04(self):
+        """resourcesync_ex_4 is a simple resource dump with one ZIP listed"""
+        rd = ResourceDump(spec_version='1.0')
+        rd.parse('tests/testdata/examples_from_spec_v1_0/resourcesync_ex_4.xml')
+        self.assertEqual(len(rd.resources), 1, '1 resources')
+        self.assertTrue('http://example.com/resourcedump.zip' in rd.resources)
+        self.assertEqual(rd.resources[
+                         'http://example.com/resourcedump.zip'].lastmod, '2013-01-03T09:00:00Z')
+
+    def test_ex_05(self):
+        """resourcesync_ex_5 is a simple resource dump manifest with two files listed"""
+        rdm = ResourceDumpManifest(spec_version='1.0')
+        rdm.parse('tests/testdata/examples_from_spec_v1_0/resourcesync_ex_5.xml')
+        self.assertEqual(len(rdm.resources), 2, '2 resources')
+        sms = sorted(rdm.uris())
+        self.assertEqual(
+            sms, ['http://example.com/res1', 'http://example.com/res2'])
+        self.assertEqual(
+            rdm.resources['http://example.com/res1'].lastmod, '2013-01-03T03:00:00Z')
+        self.assertEqual(rdm.resources[
+                         'http://example.com/res1'].md5, '1584abdf8ebdc9802ac0c6a7402c03b6')
+        self.assertEqual(
+            rdm.resources['http://example.com/res1'].path, '/resources/res1')
+        self.assertEqual(
+            rdm.resources['http://example.com/res2'].lastmod, '2013-01-03T04:00:00Z')
+        self.assertEqual(rdm.resources[
+                         'http://example.com/res2'].md5, '1e0d5cb8ef6ba40c99b14c0237be735e')
+        self.assertEqual(
+            rdm.resources['http://example.com/res2'].path, '/resources/res2')
+
+    def test_ex_06(self):
+        """resourcesync_ex_6 is a simple capability list with three capabilities"""
+        capl = CapabilityList(spec_version='1.0')
+        capl.parse('tests/testdata/examples_from_spec_v1_0/resourcesync_ex_6.xml')
+        self.assertEqual(len(capl.resources), 3, '3 capabilities')
+        # What capabilities are present?
+        self.assertTrue(capl.has_capability('resourcelist'))
+        self.assertEqual(capl.capability_info('resourcelist').uri,
+                         'http://example.com/dataset1/resourcelist.xml')
+        self.assertTrue(capl.has_capability('resourcedump'))
+        self.assertEqual(capl.capability_info('resourcedump').uri,
+                         'http://example.com/dataset1/resourcedump.xml')
+        self.assertTrue(capl.has_capability('changelist'))
+        self.assertEqual(capl.capability_info('changelist').uri,
+                         'http://example.com/dataset1/changelist.xml')
+        # Check some that aren't
+        self.assertFalse(capl.has_capability())
+        self.assertFalse(capl.has_capability('bogus'))
+        self.assertFalse(capl.has_capability('capabilitylist'))
+
+    def text_ex_07(self):
+        """resourcesync_ex_7 is a source description that list a single Capability List"""
+        sd = SourceDescription(spec_version='1.0')
+        sd.read(uri='tests/testdata/examples_from_spec_v1_0/resourcesync_ex_7.xml')
+        self.assertEqual(len(sd.resources), 1, '1 capability list')
+        cl = sd.resources[0]
+        self.assertEqual(
+            cl.uri, 'http://example.com/dataset1/capabilitylist.xml')
+        self.assertEqual(cl.capability, 'resourcelist')
+        self.assertEqual(
+            cl.describedby, 'http://example.com/info_about_set1_of_resources.xml')
+
+    def test_ex_08(self):
+        """resourcesync_ex_8 is a simple Resource List Index with 2 Resource Lists"""
+        rl = ResourceList(spec_version='1.0')
+        rl.read(
+            uri='tests/testdata/examples_from_spec_v1_0/resourcesync_ex_8.xml', index_only=True)
+        self.assertEqual(rl.capability, 'resourcelist')
+        self.assertEqual(rl.md_at, '2013-01-03T09:00:00Z')
+        self.assertEqual(len(rl.resources), 2, '2 resources')
+        sms = sorted(rl.uris())
+        self.assertEqual(sms, ['http://example.com/resourcelist-part1.xml',
+                               'http://example.com/resourcelist-part2.xml'])
+
+    # Examples 9, 10, 11 in the spec are not XML documents
+
+    def test_ex_12(self):
+        """resourcesync_ex_12 is a Source Description that talks about 3 sets of resources"""
+        sd = SourceDescription(spec_version='1.0')
+        sd.read(uri='tests/testdata/examples_from_spec_v1_0/resourcesync_ex_12.xml')
+        self.assertEqual(len(sd), 3)
+        self.assertEqual(sd.uris(), ['http://example.com/capabilitylist1.xml',
+                                     'http://example.com/capabilitylist2.xml',
+                                     'http://example.com/capabilitylist3.xml'])
+        cl1 = sd['http://example.com/capabilitylist1.xml']
+        self.assertEqual(cl1.capability, 'capabilitylist')
+        self.assertEqual(
+            cl1.describedby, 'http://example.com/info_about_set1_of_resources.xml')
+
+    def test_build_ex_01(self):
+        """Simple Resource List document"""
+        rl = ResourceList(spec_version='1.0')
+        rl.md_at = '2013-01-03T09:00:00Z'
+        rl.add(Resource('http://example.com/res1'))
+        rl.add(Resource('http://example.com/res2'))
+        ex_xml = self._read_ex('resourcesync_ex_1')
+        self._assert_xml_equal(rl.as_xml(), ex_xml)
+
+    def test_build_ex_02(self):
+        """Slightly more complex Resource List document"""
+        rl = ResourceList(spec_version='1.0')
+        rl.md_at = '2013-01-03T09:00:00Z'
+        rl.add(Resource(uri='http://example.com/res1',
+                        lastmod='2013-01-02T13:00:00Z',
+                        md5='1584abdf8ebdc9802ac0c6a7402c03b6'))
+        r2 = Resource(uri='http://example.com/res2',
+                      lastmod='2013-01-02T14:00:00Z',
+                      md5='1e0d5cb8ef6ba40c99b14c0237be735e')
+        r2.link_set(rel="duplicate", href="http://mirror.example.com/res2")
+        rl.add(r2)
+        ex_xml = self._read_ex('resourcesync_ex_2')
+        self._assert_xml_equal(rl.as_xml(), ex_xml)
+
+    def test_build_ex_03(self):
+        """Simple Change List document """
+        cl = ChangeList(spec_version='1.0')
+        cl.md_from = '2013-01-02T00:00:00Z'
+        cl.md_until = '2013-01-03T00:00:00Z'
+        cl.add(Resource(uri='http://example.com/res2.pdf',
+                        lastmod='2013-01-02T13:00:00Z',
+                        change="updated"))
+        cl.add(Resource(uri='http://example.com/res3.tiff',
+                        lastmod='2013-01-02T18:00:00Z',
+                        change='deleted'))
+        ex_xml = self._read_ex('resourcesync_ex_3')
+        self._assert_xml_equal(cl.as_xml(), ex_xml)
+
+    def test_build_ex_04(self):
+        """Simple Resource Dump document """
+        rd = ResourceDump(spec_version='1.0')
+        rd.md_at = '2013-01-03T09:00:00Z'
+        rd.add(Resource(uri='http://example.com/resourcedump.zip',
+                        lastmod='2013-01-03T09:00:00Z'))
+        ex_xml = self._read_ex('resourcesync_ex_4')
+        self._assert_xml_equal(rd.as_xml(), ex_xml)
+
+    def test_build_ex_05(self):
+        """Simple Resource Dump Manifest document """
+        rdm = ResourceDumpManifest(spec_version='1.0')
+        rdm.md_at = '2013-01-03T09:00:00Z'
+        rdm.add(Resource(uri='http://example.com/res1',
+                         lastmod='2013-01-03T03:00:00Z',
+                         md5='1584abdf8ebdc9802ac0c6a7402c03b6',
+                         path='/resources/res1'))
+        rdm.add(Resource(uri='http://example.com/res2',
+                         lastmod='2013-01-03T04:00:00Z',
+                         md5='1e0d5cb8ef6ba40c99b14c0237be735e',
+                         path='/resources/res2'))
+        ex_xml = self._read_ex('resourcesync_ex_5')
+        self._assert_xml_equal(rdm.as_xml(), ex_xml)
+
+    def test_build_ex_06(self):
+        """Simple Capability List document """
+        cl = CapabilityList(spec_version='1.0')
+        cl.describedby = 'http://example.com/info_about_set1_of_resources.xml'
+        cl.up = 'http://example.com/resourcesync_description.xml'
+        cl.add_capability(
+            uri='http://example.com/dataset1/resourcelist.xml', name='resourcelist')
+        cl.add_capability(
+            uri='http://example.com/dataset1/resourcedump.xml', name='resourcedump')
+        cl.add_capability(
+            uri='http://example.com/dataset1/changelist.xml', name='changelist')
+        ex_xml = self._read_ex('resourcesync_ex_6')
+        self._assert_xml_equal(cl.as_xml(), ex_xml)
+
+    def test_build_ex_07(self):
+        """A Source Description document """
+        sd = SourceDescription(spec_version='1.0')
+        sd.describedby = 'http://example.com/info-about-source.xml'
+        r = Resource(uri='http://example.com/dataset1/capabilitylist.xml',
+                     capability='capabilitylist')
+        r.link_set(rel='describedby',
+                   href='http://example.com/info_about_set1_of_resources.xml')
+        sd.add(r)
+        ex_xml = self._read_ex('resourcesync_ex_7')
+        self._assert_xml_equal(sd.as_xml(), ex_xml)
+
+    def test_build_ex_08(self):
+        """Simple Resource List Index document
+
+        This is not something that would usually be created directly
+        but instead would be created as part of the process of
+        writing a large Resource List in multiple files. However,
+        it is possible to create manually.
+        """
+        rli = ResourceList(spec_version='1.0')
+        rli.sitemapindex = True
+        rli.md_at = '2013-01-03T09:00:00Z'
+        rli.add(Resource(uri='http://example.com/resourcelist-part1.xml'))
+        rli.add(Resource(uri='http://example.com/resourcelist-part2.xml'))
+        ex_xml = self._read_ex('resourcesync_ex_8')
+        self._assert_xml_equal(rli.as_xml(), ex_xml)
+
+    # Examples 9, 10, 11 in the spec are not XML documents
+
+    def test_build_ex_12(self):
+        """Source Description document with describedby links"""
+        sd = SourceDescription(spec_version='1.0')
+        sd.describedby = 'http://example.com/info_about_source.xml'
+        cl1 = CapabilityList(uri='http://example.com/capabilitylist1.xml')
+        cl1.describedby = 'http://example.com/info_about_set1_of_resources.xml'
+        sd.add_capability_list(cl1)
+        cl2 = CapabilityList(uri='http://example.com/capabilitylist2.xml')
+        cl2.describedby = 'http://example.com/info_about_set2_of_resources.xml'
+        sd.add_capability_list(cl2)
+        cl3 = CapabilityList(uri='http://example.com/capabilitylist3.xml')
+        cl3.describedby = 'http://example.com/info_about_set3_of_resources.xml'
+        sd.add_capability_list(cl3)
+        ex_xml = self._read_ex('resourcesync_ex_12')
+        self._assert_xml_equal(sd.as_xml(), ex_xml)
+
+    def test_build_ex_13(self):
+        """Capability List document with 4 entries"""
+        cl = CapabilityList(spec_version='1.0')
+        cl.describedby = 'http://example.com/info_about_set1_of_resources.xml'
+        cl.up = 'http://example.com/resourcesync_description.xml'
+        cl.add_capability(capability=ResourceList(
+            uri='http://example.com/dataset1/resourcelist.xml'))
+        cl.add_capability(capability=ResourceDump(
+            uri='http://example.com/dataset1/resourcedump.xml'))
+        cl.add_capability(capability=ChangeList(
+            uri='http://example.com/dataset1/changelist.xml'))
+        cl.add_capability(capability=ChangeDump(
+            uri='http://example.com/dataset1/changedump.xml'))
+        ex_xml = self._read_ex('resourcesync_ex_13')
+        self._assert_xml_equal(cl.as_xml(), ex_xml)
+
+    def test_build_ex_14(self):
+        """Resource List with 2 entries and some metadata"""
+        rl = ResourceList(spec_version='1.0')
+        rl.up = 'http://example.com/dataset1/capabilitylist.xml'
+        rl.md_at = "2013-01-03T09:00:00Z"
+        rl.md_completed = "2013-01-03T09:01:00Z"
+        rl.add(Resource(uri='http://example.com/res1',
+                        lastmod='2013-01-02T13:00:00Z',
+                        md5='1584abdf8ebdc9802ac0c6a7402c03b6',
+                        length=8876,
+                        mime_type="text/html"))
+        rl.add(Resource(uri='http://example.com/res2',
+                        lastmod='2013-01-02T14:00:00Z',
+                        md5='1e0d5cb8ef6ba40c99b14c0237be735e',
+                        sha256='854f61290e2e197a11bc91063afce22e43f8ccc655237050ace766adc68dc784',
+                        length=14599,
+                        mime_type="application/pdf"))
+        ex_xml = self._read_ex('resourcesync_ex_14')
+        self._assert_xml_equal(rl.as_xml(), ex_xml)
+
+    def test_build_ex_15(self):
+        """Resource List Index with metadata"""
+        rl = ResourceList(
+            resources_class=ResourceListOrdered,
+            spec_version='1.0')  # order in example is non-canonical
+        rl.sitemapindex = True
+        rl.up = 'http://example.com/dataset1/capabilitylist.xml'
+        rl.md_at = "2013-01-03T09:00:00Z"
+        rl.md_completed = "2013-01-03T09:10:00Z"
+        rl.add(Resource(uri='http://example.com/resourcelist1.xml',
+                        md_at='2013-01-03T09:00:00Z'))
+        rl.add(Resource(uri='http://example.com/resourcelist2.xml',
+                        md_at='2013-01-03T09:03:00Z'))
+        rl.add(Resource(uri='http://example.com/resourcelist3.xml',
+                        md_at='2013-01-03T09:07:00Z'))
+        ex_xml = self._read_ex('resourcesync_ex_15')
+        self._assert_xml_equal(rl.as_xml(), ex_xml)
+
+    def test_build_ex_16(self):
+        rl = ResourceList(spec_version='1.0')
+        rl.up = 'http://example.com/dataset1/capabilitylist.xml'
+        rl.index = 'http://example.com/dataset1/resourcelist-index.xml'
+        rl.md_at = "2013-01-03T09:00:00Z"
+        rl.add(Resource(uri='http://example.com/res3',
+                        lastmod='2013-01-02T13:00:00Z',
+                        md5='1584abdf8ebdc9802ac0c6a7402c8753',
+                        length=4385,
+                        mime_type="application/pdf"))
+        rl.add(Resource(uri='http://example.com/res4',
+                        lastmod='2013-01-02T14:00:00Z',
+                        md5='4556abdf8ebdc9802ac0c6a7402c9881',
+                        length=883,
+                        mime_type="image/png"))
+        ex_xml = self._read_ex('resourcesync_ex_16')
+        self._assert_xml_equal(rl.as_xml(), ex_xml)
+
+    def test_build_ex_17(self):
+        """Resource Dump with 3 entries and some metadata"""
+        rd = ResourceDump(spec_version='1.0')
+        rd.up = 'http://example.com/dataset1/capabilitylist.xml'
+        rd.md_at = "2013-01-03T09:00:00Z"
+        rd.md_completed = "2013-01-03T09:04:00Z"
+        z1 = Resource(uri='http://example.com/resourcedump-part1.zip',
+                      mime_type="application/zip",
+                      length=4765,
+                      md_at="2013-01-03T09:00:00Z",
+                      md_completed="2013-01-03T09:02:00Z")
+        z1.link_set(rel="contents",
+                    href="http://example.com/resourcedump_manifest-part1.xml",
+                    mime_type="application/xml")
+        rd.add(z1)
+        z2 = Resource(uri='http://example.com/resourcedump-part2.zip',
+                      mime_type="application/zip",
+                      length=9875,
+                      md_at="2013-01-03T09:01:00Z",
+                      md_completed="2013-01-03T09:03:00Z")
+        z2.link_set(rel="contents",
+                    href="http://example.com/resourcedump_manifest-part2.xml",
+                    mime_type="application/xml")
+        rd.add(z2)
+        z3 = Resource(uri='http://example.com/resourcedump-part3.zip',
+                      mime_type="application/zip",
+                      length=2298,
+                      md_at="2013-01-03T09:03:00Z",
+                      md_completed="2013-01-03T09:04:00Z")
+        z3.link_set(rel="contents",
+                    href="http://example.com/resourcedump_manifest-part3.xml",
+                    mime_type="application/xml")
+        rd.add(z3)
+        ex_xml = self._read_ex('resourcesync_ex_17')
+        self._assert_xml_equal(rd.as_xml(), ex_xml)
+
+    def test_build_ex_18(self):
+        """Resource Dump Manifest with 2 entries and some metadata"""
+        rdm = ResourceDumpManifest(spec_version='1.0')
+        rdm.up = 'http://example.com/dataset1/capabilitylist.xml'
+        rdm.md_at = "2013-01-03T09:00:00Z"
+        rdm.md_completed = "2013-01-03T09:02:00Z"
+        rdm.add(Resource(uri='http://example.com/res1',
+                         lastmod='2013-01-02T13:00:00Z',
+                         md5='1584abdf8ebdc9802ac0c6a7402c03b6',
+                         length=8876,
+                         mime_type='text/html',
+                         path='/resources/res1'))
+        rdm.add(Resource(uri='http://example.com/res2',
+                         lastmod='2013-01-02T14:00:00Z',
+                         md5='1e0d5cb8ef6ba40c99b14c0237be735e',
+                         sha256='854f61290e2e197a11bc91063afce22e43f8ccc655237050ace766adc68dc784',
+                         length=14599,
+                         mime_type='application/pdf',
+                         path='/resources/res2'))
+        ex_xml = self._read_ex('resourcesync_ex_18')
+        self._assert_xml_equal(rdm.as_xml(), ex_xml)
+
+    def test_build_ex_19(self):
+        """Change List with 4 changes, 'open' as no until"""
+        cl = ChangeList(spec_version='1.0')
+        cl.up = 'http://example.com/dataset1/capabilitylist.xml'
+        cl.md_from = "2013-01-03T00:00:00Z"
+        cl.add(Resource(uri='http://example.com/res1.html',
+                        lastmod='2013-01-03T11:00:00Z',
+                        change='created'))
+        cl.add(Resource(uri='http://example.com/res2.pdf',
+                        lastmod='2013-01-03T13:00:00Z',
+                        change='updated'))
+        cl.add(Resource(uri='http://example.com/res3.tiff',
+                        lastmod='2013-01-03T18:00:00Z',
+                        change='deleted'))
+        cl.add(Resource(uri='http://example.com/res2.pdf',
+                        lastmod='2013-01-03T21:00:00Z',
+                        change='updated'))
+        ex_xml = self._read_ex('resourcesync_ex_19')
+        self._assert_xml_equal(cl.as_xml(), ex_xml)
+
+    def test_build_ex_20(self):
+        """Change List Index listing 3 Change Lists, the last one 'open'"""
+        cl = ChangeListArchive(
+            resources_class=ResourceListOrdered,   # order in example is non-canonical
+            spec_version='1.0')
+        cl.sitemapindex = True
+        cl.capability_name = 'changelist'
+        cl.up = 'http://example.com/dataset1/capabilitylist.xml'
+        cl.md_from = "2013-01-01T00:00:00Z"
+        cl.add(Resource(uri='http://example.com/20130101-changelist.xml',
+                        md_from='2013-01-01T00:00:00Z',
+                        md_until='2013-01-02T00:00:00Z'))
+        cl.add(Resource(uri='http://example.com/20130102-changelist.xml',
+                        md_from='2013-01-02T00:00:00Z',
+                        md_until='2013-01-03T00:00:00Z'))
+        cl.add(Resource(uri='http://example.com/20130103-changelist.xml',
+                        md_from='2013-01-03T00:00:00Z'))
+        ex_xml = self._read_ex('resourcesync_ex_20')
+        self._assert_xml_equal(cl.as_xml(), ex_xml)
+
+    def test_build_ex_21(self):
+        """Change List which points back to index"""
+        cl = ChangeList(spec_version='1.0')
+        cl.up = 'http://example.com/dataset1/capabilitylist.xml'
+        cl.index = 'http://example.com/dataset1/changelist.xml'
+        cl.md_from = "2013-01-02T00:00:00Z"
+        cl.md_until = "2013-01-03T00:00:00Z"
+        cl.add(Resource(uri='http://example.com/res7.html',
+                        lastmod='2013-01-02T12:00:00Z',
+                        change='created'))
+        cl.add(Resource(uri='http://example.com/res9.pdf',
+                        lastmod='2013-01-02T13:00:00Z',
+                        change='updated'))
+        cl.add(Resource(uri='http://example.com/res5.tiff',
+                        lastmod='2013-01-02T19:00:00Z',
+                        change='deleted'))
+        cl.add(Resource(uri='http://example.com/res7.html',
+                        lastmod='2013-01-02T20:00:00Z',
+                        change='updated'))
+        ex_xml = self._read_ex('resourcesync_ex_21')
+        self._assert_xml_equal(cl.as_xml(), ex_xml)
+
+    def test_build_ex_22(self):
+        """Change Dump with three dump files"""
+        cd = ChangeDump(spec_version='1.0')
+        cd.up = 'http://example.com/dataset1/capabilitylist.xml'
+        cd.md_from = "2013-01-01T00:00:00Z"
+        z1 = Resource(uri='http://example.com/20130101-changedump.zip',
+                      lastmod='2013-01-01T23:59:59Z',
+                      length=3109,
+                      md_from="2013-01-01T00:00:00Z",
+                      md_until="2013-01-02T00:00:00Z",
+                      mime_type="application/zip")
+        z1.contents = 'http://example.com/20130101-changedump-manifest.xml'
+        z2 = Resource(uri='http://example.com/20130102-changedump.zip',
+                      lastmod='2013-01-02T23:59:59Z',
+                      length=6629,
+                      md_from="2013-01-02T00:00:00Z",
+                      md_until="2013-01-03T00:00:00Z",
+                      mime_type="application/zip")
+        z2.contents = 'http://example.com/20130102-changedump-manifest.xml'
+        z3 = Resource(uri='http://example.com/20130103-changedump.zip',
+                      lastmod='2013-01-03T23:59:59Z',
+                      length=8124,
+                      md_from="2013-01-03T00:00:00Z",
+                      md_until="2013-01-04T00:00:00Z",
+                      mime_type="application/zip")
+        z3.contents = 'http://example.com/20130103-changedump-manifest.xml'
+        cd.add([z1, z2, z3])
+        ex_xml = self._read_ex('resourcesync_ex_22')
+        self._assert_xml_equal(cd.as_xml(), ex_xml)
+
+    def test_build_ex_23(self):
+        cdm = ChangeDumpManifest(spec_version='1.0')
+        cdm.up = "http://example.com/dataset1/capabilitylist.xml"
+        cdm.md_from = "2013-01-02T00:00:00Z"
+        cdm.md_until = "2013-01-03T00:00:00Z"
+        cdm.add(Resource(uri="http://example.com/res7.html",
+                         lastmod="2013-01-02T12:00:00Z",
+                         change="created",
+                         md5="1c1b0e264fa9b7e1e9aa6f9db8d6362b",
+                         length=4339,
+                         mime_type="text/html",
+                         path="/changes/res7.html"))
+        cdm.add(Resource(uri="http://example.com/res9.pdf",
+                         lastmod="2013-01-02T13:00:00Z",
+                         change="updated",
+                         md5="f906610c3d4aa745cb2b986f25b37c5a",
+                         length=38297,
+                         mime_type="application/pdf",
+                         path="/changes/res9.pdf"))
+        cdm.add(Resource(uri="http://example.com/res5.tiff",
+                         lastmod="2013-01-02T19:00:00Z",
+                         change="deleted"))
+        cdm.add(Resource(uri="http://example.com/res7.html",
+                         lastmod="2013-01-02T20:00:00Z",
+                         change="updated",
+                         md5="0988647082c8bc51778894a48ec3b576",
+                         length="5426",  # should also take string
+                         mime_type="text/html",
+                         path="/changes/res7-v2.html"))
+        self._assert_xml_equal_ex(cdm.as_xml(), 'resourcesync_ex_23')
+
+    def test_build_ex_24(self):
+        cl = ChangeList(spec_version='1.0')
+        cl.up = "http://example.com/dataset1/capabilitylist.xml"
+        cl.md_from = "2013-01-03T00:00:00Z"
+        c1 = Resource(uri="http://example.com/res1",
+                      lastmod="2013-01-03T18:00:00Z",
+                      change="updated",
+                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
+                      length=8876,
+                      mime_type="text/html")
+        # Resource.link_set with add or change link depending on one with
+        # the particular rel exists unless allow_duplicates=True.
+        # Resource.link_add will always add. Test both here...
+        c1.link_set(rel="duplicate",
+                    href="http://mirror1.example.com/res1",
+                    pri="1",
+                    modified="2013-01-03T18:00:00Z")
+        c1.link_set(rel="duplicate",
+                    href="http://mirror2.example.com/res1",
+                    pri="2",
+                    modified="2013-01-03T18:00:00Z",
+                    allow_duplicates=True)
+        c1.link_add(rel="duplicate",
+                    href="gsiftp://gridftp.example.com/res1",
+                    pri="3",
+                    modified="2013-01-03T18:00:00Z")
+        cl.add(c1)
+        self._assert_xml_equal_ex(cl.as_xml(), 'resourcesync_ex_24')
+
+    def test_build_ex_25(self):
+        cl = ChangeList(spec_version='1.0')
+        cl.up = "http://example.com/dataset1/capabilitylist.xml"
+        cl.md_from = "2013-01-03T11:00:00Z"
+        c1 = Resource(uri="http://example.com/res1",
+                      lastmod="2013-01-03T18:00:00Z",
+                      change="updated")
+        c1.link_add(rel="alternate",
+                    href="http://example.com/res1.html",
+                    modified="2013-01-03T18:00:00Z",
+                    type="text/html")  # FIXME - inconsistent
+        c1.link_add(rel="alternate",
+                    href="http://example.com/res1.pdf",
+                    modified="2013-01-03T18:00:00Z",
+                    type="application/pdf")
+        cl.add(c1)
+        self._assert_xml_equal_ex(cl.as_xml(), 'resourcesync_ex_25')
+
+    def test_build_ex_26(self):
+        cl = ChangeList(spec_version='1.0')
+        cl.up = "http://example.com/dataset1/capabilitylist.xml"
+        cl.md_from = "2013-01-03T00:00:00Z"
+        c1 = Resource(uri="http://example.com/res1.html",
+                      lastmod="2013-01-03T18:00:00Z",
+                      change="updated",
+                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
+                      length=8876)
+        c1.link_add(rel="canonical",
+                    href="http://example.com/res1",
+                    modified="2013-01-03T18:00:00Z")
+        cl.add(c1)
+        self._assert_xml_equal_ex(cl.as_xml(), 'resourcesync_ex_26')
+
+    def test_build_ex_27(self):
+        cl = ChangeList(spec_version='1.0')
+        cl.up = "http://example.com/dataset1/capabilitylist.xml"
+        cl.md_from = "2013-01-03T00:00:00Z"
+        c1 = Resource(uri="http://example.com/res4",
+                      lastmod="2013-01-03T17:00:00Z",
+                      change="updated",
+                      sha256="f4OxZX_x_DFGFDgghgdfb6rtSx-iosjf6735432nklj",
+                      length=56778,
+                      mime_type="application/json")
+        c1.link_set(rel="http://www.openarchives.org/rs/terms/patch",
+                    href="http://example.com/res4-json-patch",
+                    modified="2013-01-03T17:00:00Z",
+                    hash="sha-256:y66dER_t_HWEIKpesdkeb7rtSc-ippjf9823742opld",  # FIXME - inconsistent
+                    length=73,
+                    type="application/json-patch")
+        c2 = Resource(uri="http://example.com/res5-full.tiff",
+                      lastmod="2013-01-03T18:00:00Z",
+                      change="updated",
+                      sha256="f4OxZX_x_FO5LcGBSKHWXfwtSx-j1ncoSt3SABJtkGk",
+                      length="9788456778",
+                      mime_type="image/tiff")
+        c2.link_set(rel="http://www.openarchives.org/rs/terms/patch",
+                    href="http://example.com/res5-diff",
+                    modified="2013-01-03T18:00:00Z",
+                    hash="sha-256:h986gT_t_87HTkjHYE76G558hY-jdfgy76t55sadJUYT",
+                    length=4533,
+                    type="application/x-tiff-diff")
+        cl.add([c1, c2])
+        self._assert_xml_equal_ex(cl.as_xml(), 'resourcesync_ex_27')
+
+    def test_build_ex_28(self):
+        cl = ChangeList(spec_version='1.0')
+        cl.up = "http://example.com/dataset1/capabilitylist.xml"
+        cl.md_from = "2013-01-03T00:00:00Z"
+        c1 = Resource(uri="http://example.com/res2.pdf",
+                      lastmod="2013-01-03T18:00:00Z",
+                      change="updated",
+                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
+                      length=8876,
+                      mime_type="application/pdf")
+        c1.link_set(rel="describedby",
+                    href="http://example.com/res2_dublin-core_metadata.xml",
+                    modified="2013-01-01T12:00:00Z",
+                    type="application/xml")
+        c2 = Resource(uri="http://example.com/res2_dublin-core_metadata.xml",
+                      lastmod="2013-01-03T19:00:00Z",
+                      change="updated",
+                      mime_type="application/xml")
+        c2.link_set(rel="describes",
+                    href="http://example.com/res2.pdf",
+                    modified="2013-01-03T18:00:00Z",
+                    hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6",
+                    length="8876",
+                    type="application/pdf")
+        c2.link_set(rel="profile",
+                    href="http://purl.org/dc/elements/1.1/")
+        cl.add([c1, c2])
+        self._assert_xml_equal_ex(cl.as_xml(), 'resourcesync_ex_28')
+
+    def test_build_ex_29(self):
+        cl = ChangeList(spec_version='1.0')
+        cl.up = "http://example.com/dataset1/capabilitylist.xml"
+        cl.md_from = "2013-01-03T00:00:00Z"
+        c1 = Resource(uri="http://example.com/res1",
+                      lastmod="2013-01-03T18:00:00Z",
+                      change="updated",
+                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
+                      length=8876,
+                      mime_type="text/html")
+        c1.link_add(rel="memento",
+                    href="http://example.com/20130103070000/res1",
+                    modified="2013-01-02T18:00:00Z",
+                    hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6",
+                    length="8876",
+                    type="text/html")
+        c1.link_add(rel="timegate",
+                    href="http://example.com/timegate/http://example.com/res1")
+        c1.link_add(rel="timemap",
+                    href="http://example.com/timemap/http://example.com/res1",
+                    type="application/link-format")
+        cl.add(c1)
+        self._assert_xml_equal_ex(cl.as_xml(), 'resourcesync_ex_29')
+
+    def test_build_ex_30(self):
+        cl = ChangeList(spec_version='1.0')
+        cl.up = "http://example.com/dataset1/capabilitylist.xml"
+        cl.md_from = "2013-01-03T00:00:00Z"
+        c1 = Resource(uri="http://example.com/res1",
+                      lastmod="2013-01-03T07:00:00Z",
+                      change="updated",
+                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
+                      length=8876,
+                      mime_type="text/html")
+        c1.link_add(rel="collection",
+                    href="http://example.com/aggregation/0601007")
+        cl.add(c1)
+        self._assert_xml_equal_ex(cl.as_xml(), 'resourcesync_ex_30')
+
+    def test_build_ex_31(self):
+        cl = ChangeList(spec_version='1.0')
+        cl.up = "http://example.com/dataset1/capabilitylist.xml"
+        cl.md_from = "2013-01-03T00:00:00Z"
+        c1 = Resource(uri="http://original.example.com/res1.html",
+                      lastmod="2013-01-03T07:00:00Z",
+                      change="updated",
+                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
+                      length=8876,
+                      mime_type="text/html")
+        cl.add(c1)
+        self._assert_xml_equal_ex(cl.as_xml(), 'resourcesync_ex_31')
+
+    def test_build_ex_32(self):
+        cl = ChangeList(spec_version='1.0')
+        cl.up = "http://aggregator1.example.com/dataset1/capabilitylist.xml"
+        cl.md_from = "2013-01-03T11:00:00Z"
+        c1 = Resource(uri="http://aggregator1.example.com/res1.html",
+                      lastmod="2013-01-03T20:00:00Z",
+                      change="updated",
+                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
+                      length=8876,
+                      mime_type="text/html")
+        c1.link_add(rel="via",
+                    href="http://original.example.com/res1.html",
+                    modified="2013-01-03T07:00:00Z",
+                    hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6",
+                    length="8876",
+                    type="text/html")
+        cl.add(c1)
+        self._assert_xml_equal_ex(cl.as_xml(), 'resourcesync_ex_32')
+
+    def test_build_ex_33(self):
+        cl = ChangeList(spec_version='1.0')
+        cl.up = "http://aggregator2.example.com/dataset1/capabilitylist.xml"
+        cl.md_from = "2013-01-03T12:00:00Z"
+        c1 = Resource(uri="http://aggregator2.example.com/res1.html",
+                      lastmod="2013-01-04T09:00:00Z",
+                      change="updated",
+                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
+                      length=8876,
+                      mime_type="text/html")
+        c1.link_add(rel="via",
+                    href="http://original.example.com/res1.html",
+                    modified="2013-01-03T07:00:00Z",
+                    hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6",
+                    length="8876",
+                    type="text/html")
+        cl.add(c1)
+        self._assert_xml_equal_ex(cl.as_xml(), 'resourcesync_ex_33')
+
+    # Archives tests
+
+    def test_build_archives_ex_3_1(self):
+        """Resource List Archive listing 3 Resource Lists"""
+        rla = ResourceListArchive(spec_version='1.0')
+        rla.up = 'http://example.com/dataset1/capabilitylist.xml'
+        rla.add(Resource(uri='http://example.com/resourcelist1.xml',
+                         md_at='2012-11-03T09:00:00Z'))
+        rla.add(Resource(uri='http://example.com/resourcelist2.xml',
+                         md_at='2012-12-03T09:00:00Z'))
+        rla.add(Resource(uri='http://example.com/resourcelist3.xml',
+                         md_at='2013-01-03T09:00:00Z'))
+        ex_xml = self._read_ex('archives_ex_3_1')
+        self._assert_xml_equal(rla.as_xml(), ex_xml)
+
+    def test_build_archives_ex_3_2(self):
+        """Resource List Archive Index listing 2 component Resource List Archives"""
+        rlai = ResourceListArchive(spec_version='1.0')
+        rlai.sitemapindex = True
+        rlai.up = 'http://example.com/dataset1/capabilitylist.xml'
+        rlai.add(Resource(uri='http://example.com/resourcelistarchive00001.xml'))
+        rlai.add(Resource(uri='http://example.com/resourcelistarchive00002.xml'))
+        ex_xml = self._read_ex('archives_ex_3_2')
+        self._assert_xml_equal(rlai.as_xml(), ex_xml)
+
+    def test_build_archives_ex_4_1(self):
+        """Resource Dump Archive listing 2 Resource Dumps"""
+        rda = ResourceDumpArchive(spec_version='1.0')
+        rda.up = 'http://example.com/dataset1/capabilitylist.xml'
+        rda.add(Resource(uri='http://example.com/resourcedump1.xml',
+                         lastmod='2012-11-03T09:05:42Z',
+                         md_at="2012-11-03T09:00:00Z",
+                         md_completed="2012-11-03T09:05:01Z"))
+
+        rda.add(Resource(uri='http://example.com/resourcedump2.xml',
+                         lastmod='2012-12-03T09:06:12Z',
+                         md_at="2012-12-03T09:00:00Z",
+                         md_completed="2012-12-03T09:05:17Z"))
+        ex_xml = self._read_ex('archives_ex_4_1')
+        self._assert_xml_equal(rda.as_xml(), ex_xml)
+
+    def test_build_archives_ex_5_1(self):
+        """Change List Archive listing 3 Change Lists"""
+        cla = ChangeListArchive(spec_version='1.0')
+        cla.up = 'http://example.com/dataset1/capabilitylist.xml'
+        cla.add(Resource(uri='http://example.com/changelist1.xml',
+                         md_from='2013-01-01T09:00:00Z',
+                         md_until='2013-01-02T09:00:00Z'))
+        cla.add(Resource(uri='http://example.com/changelist2.xml',
+                         md_from='2013-01-02T09:00:00Z',
+                         md_until='2013-01-03T09:00:00Z'))
+        cla.add(Resource(uri='http://example.com/changelist3.xml',
+                         md_from='2013-01-03T09:00:00Z',
+                         md_until='2013-01-04T09:00:00Z'))
+        ex_xml = self._read_ex('archives_ex_5_1')
+        self._assert_xml_equal(cla.as_xml(), ex_xml)
+
+    def test_build_archives_ex_6_1(self):
+        """Change Dump Archive listing 2 Change Dumps"""
+        cda = ChangeDumpArchive(spec_version='1.0')
+        cda.up = 'http://example.com/dataset1/capabilitylist.xml'
+        cda.add(Resource(uri='http://example.com/changedump-w1.xml',
+                         lastmod='2012-12-20T09:02:43Z',
+                         md_from="2012-01-13T09:00:00Z",
+                         md_until="2013-01-20T09:00:00Z"))
+        cda.add(Resource(uri='http://example.com/changedump-w2.xml',
+                         lastmod='2012-12-27T09:01:57Z',
+                         md_from="2012-01-20T09:00:00Z",
+                         md_until="2013-01-27T09:00:00Z"))
+        ex_xml = self._read_ex('archives_ex_6_1')
+        self._assert_xml_equal(cda.as_xml(), ex_xml)

tests/test_examples_from_spec_v1_1.py

@@ -0,0 +1,752 @@
+"""Tests on the examples in the specification documents
+"""
+import re
+import sys
+
+from resync.resource import Resource
+from resync.source_description import SourceDescription
+from resync.capability_list import CapabilityList
+from resync.resource_list import ResourceList, ResourceListOrdered
+from resync.resource_dump import ResourceDump
+from resync.resource_dump_manifest import ResourceDumpManifest
+from resync.change_list import ChangeList
+from resync.change_dump import ChangeDump
+from resync.change_dump_manifest import ChangeDumpManifest
+from resync.archives import ResourceListArchive, ResourceDumpArchive, ChangeListArchive, ChangeDumpArchive
+from resync.sitemap import Sitemap
+
+from .testlib.testcase_with_xml_comparison import TestCase
+
+
+class TestExamplesFromSpec(TestCase):
+
+    def _ex_path(self, ex):
+        return 'tests/testdata/examples_from_spec_v1_1/%s.xml' % (ex)
+
+    def _read_ex(self, ex):
+        with open(self._ex_path(ex), 'r') as fh:
+            content = fh.read()
+        return content
+
+    def test_all_simple_read(self):
+        """Just try to read each one"""
+        for ex in ('resourcesync_ex_1', 'resourcesync_ex_2', 'resourcesync_ex_3',
+                   'resourcesync_ex_4', 'resourcesync_ex_5', 'resourcesync_ex_6',
+                   'resourcesync_ex_7', 'resourcesync_ex_8', 'resourcesync_ex_12',
+                   'resourcesync_ex_13', 'resourcesync_ex_14', 'resourcesync_ex_15',
+                   'resourcesync_ex_16', 'resourcesync_ex_17', 'resourcesync_ex_18',
+                   'resourcesync_ex_19', 'resourcesync_ex_20', 'resourcesync_ex_21',
+                   'resourcesync_ex_22', 'resourcesync_ex_23', 'resourcesync_ex_24',
+                   'resourcesync_ex_25', 'resourcesync_ex_26', 'resourcesync_ex_27',
+                   'resourcesync_ex_28', 'resourcesync_ex_29', 'resourcesync_ex_30',
+                   'resourcesync_ex_31', 'resourcesync_ex_32', 'resourcesync_ex_33'):
+            s = Sitemap()
+            with open(self._ex_path(ex), 'r') as fh:
+                si = s.parse_xml(fh=fh)
+
+    def test_ex_01(self):
+        """resourcesync_ex_1 is a simple resource_list with 2 resources, no metadata"""
+        rl = ResourceList()
+        rl.parse(uri='tests/testdata/examples_from_spec_v1_1/resourcesync_ex_1.xml')
+        self.assertEqual(rl.capability, 'resourcelist')
+        self.assertEqual(len(rl.resources), 2, '2 resources')
+        sms = sorted(rl.uris())
+        self.assertEqual(
+            sms, ['http://example.com/res1', 'http://example.com/res2'])
+        self.assertEqual(rl.resources['http://example.com/res1'].lastmod, None)
+
+    def test_ex_02(self):
+        """resourcesync_ex_2 is a simple resource_list with 2 resources, some metadata"""
+        rl = ResourceList()
+        rl.parse(uri='tests/testdata/examples_from_spec_v1_1/resourcesync_ex_2.xml')
+        self.assertEqual(len(rl.resources), 2, '2 resources')
+        sms = sorted(rl.uris())
+        self.assertEqual(
+            sms, ['http://example.com/res1', 'http://example.com/res2'])
+        self.assertEqual(
+            rl.resources['http://example.com/res1'].lastmod, '2013-01-02T13:00:00Z')
+        self.assertEqual(
+            rl.resources['http://example.com/res2'].lastmod, '2013-01-02T14:00:00Z')
+        self.assertEqual(
+            rl.resources['http://example.com/res1'].md5, '1584abdf8ebdc9802ac0c6a7402c03b6')
+        self.assertEqual(
+            rl.resources['http://example.com/res2'].md5, '1e0d5cb8ef6ba40c99b14c0237be735e')
+
+    def test_ex_03(self):
+        """resourcesync_ex_3 is a simple change_list with 3 resources"""
+        cl = ChangeList()
+        cl.parse('tests/testdata/examples_from_spec_v1_1/resourcesync_ex_3.xml')
+        self.assertEqual(len(cl.resources), 3, '3 resources')
+        sms = sorted(cl.uris())
+        self.assertEqual(sms, ['http://example.com/res1.pdf',
+                               'http://example.com/res2.pdf', 'http://example.com/res3.tiff'])
+        self.assertEqual(cl.resources[0].lastmod, '2013-01-02T13:00:00Z')
+        self.assertEqual(cl.resources[0].change, 'updated')
+        self.assertEqual(cl.resources[0].datetime, '2013-01-02T13:00:00Z')
+        self.assertEqual(cl.resources[1].change, 'deleted')
+        self.assertEqual(cl.resources[1].datetime, '2013-01-02T14:00:00Z')
+        self.assertEqual(cl.resources[2].lastmod, '2011-01-01T00:00:00Z')
+        self.assertEqual(cl.resources[2].change, 'created')
+        self.assertEqual(cl.resources[2].datetime, '2013-01-02T15:00:00Z')
+
+    def test_ex_04(self):
+        """resourcesync_ex_4 is a simple resource dump with one ZIP listed"""
+        rd = ResourceDump(spec_version='1.1')
+        rd.parse('tests/testdata/examples_from_spec_v1_1/resourcesync_ex_4.xml')
+        self.assertEqual(len(rd.resources), 1, '1 resources')
+        self.assertTrue('http://example.com/resourcedump.zip' in rd.resources)
+        self.assertEqual(rd.resources['http://example.com/resourcedump.zip'].lastmod, None)
+
+    def test_ex_05(self):
+        """resourcesync_ex_5 is a simple resource dump manifest with two files listed"""
+        rdm = ResourceDumpManifest(spec_version='1.1')
+        rdm.parse('tests/testdata/examples_from_spec_v1_1/resourcesync_ex_5.xml')
+        self.assertEqual(len(rdm.resources), 2, '2 resources')
+        sms = sorted(rdm.uris())
+        self.assertEqual(
+            sms, ['http://example.com/res1', 'http://example.com/res2'])
+        self.assertEqual(rdm.resources['http://example.com/res1'].lastmod, None)
+        self.assertEqual(rdm.resources['http://example.com/res1'].md5, '1584abdf8ebdc9802ac0c6a7402c03b6')
+        self.assertEqual(rdm.resources['http://example.com/res1'].path, '/resources/res1')
+        self.assertEqual(rdm.resources['http://example.com/res2'].lastmod, None)
+        self.assertEqual(rdm.resources['http://example.com/res2'].md5, '1e0d5cb8ef6ba40c99b14c0237be735e')
+        self.assertEqual(rdm.resources['http://example.com/res2'].path, '/resources/res2')
+
+    def test_ex_06(self):
+        """resourcesync_ex_6 is a simple capability list with three capabilities"""
+        capl = CapabilityList()
+        capl.parse('tests/testdata/examples_from_spec_v1_1/resourcesync_ex_6.xml')
+        self.assertEqual(len(capl.resources), 3, '3 capabilities')
+        # What capabilities are present?
+        self.assertTrue(capl.has_capability('resourcelist'))
+        self.assertEqual(capl.capability_info('resourcelist').uri,
+                         'http://example.com/dataset1/resourcelist.xml')
+        self.assertTrue(capl.has_capability('resourcedump'))
+        self.assertEqual(capl.capability_info('resourcedump').uri,
+                         'http://example.com/dataset1/resourcedump.xml')
+        self.assertTrue(capl.has_capability('changelist'))
+        self.assertEqual(capl.capability_info('changelist').uri,
+                         'http://example.com/dataset1/changelist.xml')
+        # Check some that aren't
+        self.assertFalse(capl.has_capability())
+        self.assertFalse(capl.has_capability('bogus'))
+        self.assertFalse(capl.has_capability('capabilitylist'))
+
+    def text_ex_07(self):
+        """resourcesync_ex_7 is a source description that list a single Capability List"""
+        sd = SourceDescription()
+        sd.read(uri='tests/testdata/examples_from_spec_v1_1/resourcesync_ex_7.xml')
+        self.assertEqual(len(sd.resources), 1, '1 capability list')
+        cl = sd.resources[0]
+        self.assertEqual(
+            cl.uri, 'http://example.com/dataset1/capabilitylist.xml')
+        self.assertEqual(cl.capability, 'resourcelist')
+        self.assertEqual(
+            cl.describedby, 'http://example.com/info_about_set1_of_resources.xml')
+
+    def test_ex_08(self):
+        """resourcesync_ex_8 is a simple Resource List Index with 2 Resource Lists"""
+        rl = ResourceList()
+        rl.read(
+            uri='tests/testdata/examples_from_spec_v1_1/resourcesync_ex_8.xml', index_only=True)
+        self.assertEqual(rl.capability, 'resourcelist')
+        self.assertEqual(rl.md_at, '2013-01-03T09:00:00Z')
+        self.assertEqual(len(rl.resources), 2, '2 resources')
+        sms = sorted(rl.uris())
+        self.assertEqual(sms, ['http://example.com/resourcelist-part1.xml',
+                               'http://example.com/resourcelist-part2.xml'])
+
+    # Examples 9, 10, 11 in the spec are not XML documents
+
+    def test_ex_12(self):
+        """resourcesync_ex_12 is a Source Description that talks about 3 sets of resources"""
+        sd = SourceDescription()
+        sd.read(uri='tests/testdata/examples_from_spec_v1_1/resourcesync_ex_12.xml')
+        self.assertEqual(len(sd), 3)
+        self.assertEqual(sd.uris(), ['http://example.com/capabilitylist1.xml',
+                                     'http://example.com/capabilitylist2.xml',
+                                     'http://example.com/capabilitylist3.xml'])
+        cl1 = sd['http://example.com/capabilitylist1.xml']
+        self.assertEqual(cl1.capability, 'capabilitylist')
+        self.assertEqual(
+            cl1.describedby, 'http://example.com/info_about_set1_of_resources.xml')
+
+    def test_build_ex_01(self):
+        """Simple Resource List document"""
+        rl = ResourceList()
+        rl.md_at = '2013-01-03T09:00:00Z'
+        rl.add(Resource('http://example.com/res1'))
+        rl.add(Resource('http://example.com/res2'))
+        ex_xml = self._read_ex('resourcesync_ex_1')
+        self._assert_xml_equal(rl.as_xml(), ex_xml)
+
+    def test_build_ex_02(self):
+        """Slightly more complex Resource List document"""
+        rl = ResourceList()
+        rl.md_at = '2013-01-03T09:00:00Z'
+        rl.add(Resource(uri='http://example.com/res1',
+                        lastmod='2013-01-02T13:00:00Z',
+                        md5='1584abdf8ebdc9802ac0c6a7402c03b6'))
+        r2 = Resource(uri='http://example.com/res2',
+                      lastmod='2013-01-02T14:00:00Z',
+                      md5='1e0d5cb8ef6ba40c99b14c0237be735e')
+        r2.link_set(rel="duplicate", href="http://mirror.example.com/res2")
+        rl.add(r2)
+        ex_xml = self._read_ex('resourcesync_ex_2')
+        self._assert_xml_equal(rl.as_xml(), ex_xml)
+
+    def test_build_ex_03(self):
+        """Simple Change List document """
+        cl = ChangeList()
+        cl.md_from = '2013-01-02T00:00:00Z'
+        cl.md_until = '2013-01-03T00:00:00Z'
+        cl.add(Resource(uri='http://example.com/res1.pdf',
+                        lastmod='2013-01-02T13:00:00Z',
+                        change='updated',
+                        datetime='2013-01-02T13:00:00Z'))
+        cl.add(Resource(uri='http://example.com/res2.pdf',
+                        change="deleted",
+                        datetime='2013-01-02T14:00:00Z'))
+        cl.add(Resource(uri='http://example.com/res3.tiff',
+                        lastmod='2011-01-01T00:00:00Z',
+                        change='created',
+                        datetime='2013-01-02T15:00:00Z'))
+        ex_xml = self._read_ex('resourcesync_ex_3')
+        self._assert_xml_equal(cl.as_xml(), ex_xml)
+
+    def test_build_ex_04(self):
+        """Simple Resource Dump document """
+        rd = ResourceDump()
+        rd.md_at = '2013-01-03T09:00:00Z'
+        rd.add(Resource(uri='http://example.com/resourcedump.zip'))
+        ex_xml = self._read_ex('resourcesync_ex_4')
+        self._assert_xml_equal(rd.as_xml(), ex_xml)
+
+    def test_build_ex_05(self):
+        """Simple Resource Dump Manifest document """
+        rdm = ResourceDumpManifest()
+        rdm.md_at = '2013-01-03T09:00:00Z'
+        rdm.add(Resource(uri='http://example.com/res1',
+                         md5='1584abdf8ebdc9802ac0c6a7402c03b6',
+                         path='/resources/res1'))
+        rdm.add(Resource(uri='http://example.com/res2',
+                         md5='1e0d5cb8ef6ba40c99b14c0237be735e',
+                         path='/resources/res2'))
+        ex_xml = self._read_ex('resourcesync_ex_5')
+        self._assert_xml_equal(rdm.as_xml(), ex_xml)
+
+    def test_build_ex_06(self):
+        """Simple Capability List document """
+        cl = CapabilityList()
+        cl.describedby = 'http://example.com/info_about_set1_of_resources.xml'
+        cl.up = 'http://example.com/resourcesync_description.xml'
+        cl.add_capability(
+            uri='http://example.com/dataset1/resourcelist.xml', name='resourcelist')
+        cl.add_capability(
+            uri='http://example.com/dataset1/resourcedump.xml', name='resourcedump')
+        cl.add_capability(
+            uri='http://example.com/dataset1/changelist.xml', name='changelist')
+        ex_xml = self._read_ex('resourcesync_ex_6')
+        self._assert_xml_equal(cl.as_xml(), ex_xml)
+
+    def test_build_ex_07(self):
+        """A Source Description document """
+        sd = SourceDescription()
+        sd.describedby = 'http://example.com/info-about-source.xml'
+        r = Resource(uri='http://example.com/dataset1/capabilitylist.xml',
+                     capability='capabilitylist')
+        r.link_set(rel='describedby',
+                   href='http://example.com/info_about_set1_of_resources.xml')
+        sd.add(r)
+        ex_xml = self._read_ex('resourcesync_ex_7')
+        self._assert_xml_equal(sd.as_xml(), ex_xml)
+
+    def test_build_ex_08(self):
+        """Simple Resource List Index document
+
+        This is not something that would usually be created directly
+        but instead would be created as part of the process of
+        writing a large Resource List in multiple files. However,
+        it is possible to create manually.
+        """
+        rli = ResourceList()
+        rli.sitemapindex = True
+        rli.md_at = '2013-01-03T09:00:00Z'
+        rli.add(Resource(uri='http://example.com/resourcelist-part1.xml'))
+        rli.add(Resource(uri='http://example.com/resourcelist-part2.xml'))
+        ex_xml = self._read_ex('resourcesync_ex_8')
+        self._assert_xml_equal(rli.as_xml(), ex_xml)
+
+    # Examples 9, 10, 11 in the spec are not XML documents
+
+    def test_build_ex_12(self):
+        """Source Description document with describedby links"""
+        sd = SourceDescription()
+        sd.describedby = 'http://example.com/info_about_source.xml'
+        cl1 = CapabilityList(uri='http://example.com/capabilitylist1.xml')
+        cl1.describedby = 'http://example.com/info_about_set1_of_resources.xml'
+        sd.add_capability_list(cl1)
+        cl2 = CapabilityList(uri='http://example.com/capabilitylist2.xml')
+        cl2.describedby = 'http://example.com/info_about_set2_of_resources.xml'
+        sd.add_capability_list(cl2)
+        cl3 = CapabilityList(uri='http://example.com/capabilitylist3.xml')
+        cl3.describedby = 'http://example.com/info_about_set3_of_resources.xml'
+        sd.add_capability_list(cl3)
+        ex_xml = self._read_ex('resourcesync_ex_12')
+        self._assert_xml_equal(sd.as_xml(), ex_xml)
+
+    def test_build_ex_13(self):
+        """Capability List document with 4 entries"""
+        cl = CapabilityList()
+        cl.describedby = 'http://example.com/info_about_set1_of_resources.xml'
+        cl.up = 'http://example.com/resourcesync_description.xml'
+        cl.add_capability(capability=ResourceList(
+            uri='http://example.com/dataset1/resourcelist.xml'))
+        cl.add_capability(capability=ResourceDump(
+            uri='http://example.com/dataset1/resourcedump.xml'))
+        cl.add_capability(capability=ChangeList(
+            uri='http://example.com/dataset1/changelist.xml'))
+        cl.add_capability(capability=ChangeDump(
+            uri='http://example.com/dataset1/changedump.xml'))
+        ex_xml = self._read_ex('resourcesync_ex_13')
+        self._assert_xml_equal(cl.as_xml(), ex_xml)
+
+    def test_build_ex_14(self):
+        """Resource List with 2 entries and some metadata"""
+        rl = ResourceList()
+        rl.up = 'http://example.com/dataset1/capabilitylist.xml'
+        rl.md_at = "2013-01-03T09:00:00Z"
+        rl.md_completed = "2013-01-03T09:01:00Z"
+        rl.add(Resource(uri='http://example.com/res1',
+                        lastmod='2013-01-02T13:00:00Z',
+                        md5='1584abdf8ebdc9802ac0c6a7402c03b6',
+                        length=8876,
+                        mime_type="text/html"))
+        rl.add(Resource(uri='http://example.com/res2',
+                        lastmod='2013-01-02T14:00:00Z',
+                        md5='1e0d5cb8ef6ba40c99b14c0237be735e',
+                        sha256='854f61290e2e197a11bc91063afce22e43f8ccc655237050ace766adc68dc784',
+                        length=14599,
+                        mime_type="application/pdf"))
+        ex_xml = self._read_ex('resourcesync_ex_14')
+        self._assert_xml_equal(rl.as_xml(), ex_xml)
+
+    def test_build_ex_15(self):
+        """Resource List Index with metadata"""
+        rl = ResourceList(
+            resources_class=ResourceListOrdered)  # order in example is non-canonical
+        rl.sitemapindex = True
+        rl.up = 'http://example.com/dataset1/capabilitylist.xml'
+        rl.md_at = "2013-01-03T09:00:00Z"
+        rl.md_completed = "2013-01-03T09:10:00Z"
+        rl.add(Resource(uri='http://example.com/resourcelist1.xml',
+                        md_at='2013-01-03T09:00:00Z'))
+        rl.add(Resource(uri='http://example.com/resourcelist2.xml',
+                        md_at='2013-01-03T09:03:00Z'))
+        rl.add(Resource(uri='http://example.com/resourcelist3.xml',
+                        md_at='2013-01-03T09:07:00Z'))
+        ex_xml = self._read_ex('resourcesync_ex_15')
+        self._assert_xml_equal(rl.as_xml(), ex_xml)
+
+    def test_build_ex_16(self):
+        rl = ResourceList()
+        rl.up = 'http://example.com/dataset1/capabilitylist.xml'
+        rl.index = 'http://example.com/dataset1/resourcelist-index.xml'
+        rl.md_at = "2013-01-03T09:00:00Z"
+        rl.add(Resource(uri='http://example.com/res3',
+                        md5='1584abdf8ebdc9802ac0c6a7402c8753',
+                        length=4385,
+                        mime_type="application/pdf"))
+        rl.add(Resource(uri='http://example.com/res4',
+                        md5='4556abdf8ebdc9802ac0c6a7402c9881',
+                        length=883,
+                        mime_type="image/png"))
+        ex_xml = self._read_ex('resourcesync_ex_16')
+        self._assert_xml_equal(rl.as_xml(), ex_xml)
+
+    def test_build_ex_17(self):
+        """Resource Dump with 3 entries and some metadata"""
+        rd = ResourceDump()
+        rd.up = 'http://example.com/dataset1/capabilitylist.xml'
+        rd.md_at = "2013-01-03T09:00:00Z"
+        rd.md_completed = "2013-01-03T09:04:00Z"
+        z1 = Resource(uri='http://example.com/resourcedump-part1.zip',
+                      mime_type="application/zip",
+                      length=4765,
+                      md_at="2013-01-03T09:00:00Z",
+                      md_completed="2013-01-03T09:02:00Z")
+        z1.link_set(rel="contents",
+                    href="http://example.com/resourcedump_manifest-part1.xml",
+                    mime_type="application/xml")
+        rd.add(z1)
+        z2 = Resource(uri='http://example.com/resourcedump-part2.zip',
+                      mime_type="application/zip",
+                      length=9875,
+                      md_at="2013-01-03T09:01:00Z",
+                      md_completed="2013-01-03T09:03:00Z")
+        z2.link_set(rel="contents",
+                    href="http://example.com/resourcedump_manifest-part2.xml",
+                    mime_type="application/xml")
+        rd.add(z2)
+        z3 = Resource(uri='http://example.com/resourcedump-part3.zip',
+                      mime_type="application/zip",
+                      length=2298,
+                      md_at="2013-01-03T09:03:00Z",
+                      md_completed="2013-01-03T09:04:00Z")
+        z3.link_set(rel="contents",
+                    href="http://example.com/resourcedump_manifest-part3.xml",
+                    mime_type="application/xml")
+        rd.add(z3)
+        ex_xml = self._read_ex('resourcesync_ex_17')
+        self._assert_xml_equal(rd.as_xml(), ex_xml)
+
+    def test_build_ex_18(self):
+        """Resource Dump Manifest with 2 entries and some metadata"""
+        rdm = ResourceDumpManifest()
+        rdm.up = 'http://example.com/dataset1/capabilitylist.xml'
+        rdm.md_at = "2013-01-03T09:00:00Z"
+        rdm.md_completed = "2013-01-03T09:02:00Z"
+        rdm.add(Resource(uri='http://example.com/res1',
+                         lastmod='2013-01-02T13:00:00Z',
+                         md5='1584abdf8ebdc9802ac0c6a7402c03b6',
+                         length=8876,
+                         mime_type='text/html',
+                         path='/resources/res1'))
+        rdm.add(Resource(uri='http://example.com/res2',
+                         lastmod='2013-01-02T14:00:00Z',
+                         md5='1e0d5cb8ef6ba40c99b14c0237be735e',
+                         sha256='854f61290e2e197a11bc91063afce22e43f8ccc655237050ace766adc68dc784',
+                         length=14599,
+                         mime_type='application/pdf',
+                         path='/resources/res2'))
+        ex_xml = self._read_ex('resourcesync_ex_18')
+        self._assert_xml_equal(rdm.as_xml(), ex_xml)
+
+    def test_build_ex_19(self):
+        """Change List with 4 changes, 'open' as no until"""
+        cl = ChangeList()
+        cl.up = 'http://example.com/dataset1/capabilitylist.xml'
+        cl.md_from = "2013-01-03T00:00:00Z"
+        cl.add(Resource(uri='http://example.com/res1.html',
+                        lastmod='2000-01-01T01:01:00Z',
+                        change='created',
+                        datetime='2013-01-03T11:00:00Z'))
+        cl.add(Resource(uri='http://example.com/res2.pdf',
+                        lastmod='2013-01-03T13:00:00Z',
+                        change='updated',
+                        datetime='2013-01-03T13:00:00Z'))
+        cl.add(Resource(uri='http://example.com/res3.tiff',
+                        change='deleted',
+                        datetime='2013-01-03T18:00:00Z'))
+        cl.add(Resource(uri='http://example.com/res2.pdf',
+                        change='updated'))
+        ex_xml = self._read_ex('resourcesync_ex_19')
+        self._assert_xml_equal(cl.as_xml(), ex_xml)
+
+    def test_build_ex_20(self):
+        """Change List Index listing 3 Change Lists, the last one 'open'"""
+        cl = ChangeListArchive(
+            resources_class=ResourceListOrdered)  # order in example is non-canonical
+        cl.sitemapindex = True
+        cl.capability_name = 'changelist'
+        cl.up = 'http://example.com/dataset1/capabilitylist.xml'
+        cl.md_from = "2013-01-01T00:00:00Z"
+        cl.add(Resource(uri='http://example.com/20130101-changelist.xml',
+                        md_from='2013-01-01T00:00:00Z',
+                        md_until='2013-01-02T00:00:00Z'))
+        cl.add(Resource(uri='http://example.com/20130102-changelist.xml',
+                        md_from='2013-01-02T00:00:00Z',
+                        md_until='2013-01-03T00:00:00Z'))
+        cl.add(Resource(uri='http://example.com/20130103-changelist.xml',
+                        md_from='2013-01-03T00:00:00Z'))
+        ex_xml = self._read_ex('resourcesync_ex_20')
+        self._assert_xml_equal(cl.as_xml(), ex_xml)
+
+    def test_build_ex_21(self):
+        """Change List which points back to index"""
+        cl = ChangeList()
+        cl.up = 'http://example.com/dataset1/capabilitylist.xml'
+        cl.index = 'http://example.com/dataset1/changelist.xml'
+        cl.md_from = "2013-01-02T00:00:00Z"
+        cl.md_until = "2013-01-03T00:00:00Z"
+        cl.add(Resource(uri='http://example.com/res7.html',
+                        change='created',
+                        datetime='2013-01-02T12:00:00Z',))
+        cl.add(Resource(uri='http://example.com/res9.pdf',
+                        change='updated',
+                        datetime='2013-01-02T13:00:00Z',))
+        cl.add(Resource(uri='http://example.com/res5.tiff',
+                        change='deleted',
+                        datetime='2013-01-02T19:00:00Z'))
+        cl.add(Resource(uri='http://example.com/res7.html',
+                        change='updated',
+                        datetime='2013-01-02T20:00:00Z'))
+        ex_xml = self._read_ex('resourcesync_ex_21')
+        self._assert_xml_equal(cl.as_xml(), ex_xml)
+
+    def test_build_ex_22(self):
+        """Change Dump with three dump files"""
+        cd = ChangeDump()
+        cd.up = 'http://example.com/dataset1/capabilitylist.xml'
+        cd.md_from = "2013-01-01T00:00:00Z"
+        z1 = Resource(uri='http://example.com/20130101-changedump.zip',
+                      lastmod='2013-01-01T23:59:59Z',
+                      length=3109,
+                      md_from="2013-01-01T00:00:00Z",
+                      md_until="2013-01-02T00:00:00Z",
+                      mime_type="application/zip")
+        z1.contents = 'http://example.com/20130101-changedump-manifest.xml'
+        z2 = Resource(uri='http://example.com/20130102-changedump.zip',
+                      lastmod='2013-01-02T23:59:59Z',
+                      length=6629,
+                      md_from="2013-01-02T00:00:00Z",
+                      md_until="2013-01-03T00:00:00Z",
+                      mime_type="application/zip")
+        z2.contents = 'http://example.com/20130102-changedump-manifest.xml'
+        z3 = Resource(uri='http://example.com/20130103-changedump.zip',
+                      lastmod='2013-01-03T23:59:59Z',
+                      length=8124,
+                      md_from="2013-01-03T00:00:00Z",
+                      md_until="2013-01-04T00:00:00Z",
+                      mime_type="application/zip")
+        z3.contents = 'http://example.com/20130103-changedump-manifest.xml'
+        cd.add([z1, z2, z3])
+        ex_xml = self._read_ex('resourcesync_ex_22')
+        self._assert_xml_equal(cd.as_xml(), ex_xml)
+
+    def test_build_ex_23(self):
+        cdm = ChangeDumpManifest()
+        cdm.up = "http://example.com/dataset1/capabilitylist.xml"
+        cdm.md_from = "2013-01-02T00:00:00Z"
+        cdm.md_until = "2013-01-03T00:00:00Z"
+        cdm.add(Resource(uri="http://example.com/res7.html",
+                         lastmod="2013-01-02T12:00:00Z",
+                         change="created",
+                         datetime="2013-01-02T12:00:00Z",
+                         md5="1c1b0e264fa9b7e1e9aa6f9db8d6362b",
+                         length=4339,
+                         mime_type="text/html",
+                         path="/changes/res7.html"))
+        cdm.add(Resource(uri="http://example.com/res9.pdf",
+                         lastmod="2013-01-02T13:00:00Z",
+                         change="updated",
+                         datetime="2013-01-02T13:00:00Z",
+                         md5="f906610c3d4aa745cb2b986f25b37c5a",
+                         length=38297,
+                         mime_type="application/pdf",
+                         path="/changes/res9.pdf"))
+        cdm.add(Resource(uri="http://example.com/res5.tiff",
+                         change="deleted",
+                         datetime="2013-01-02T19:00:00Z"))
+        cdm.add(Resource(uri="http://example.com/res7.html",
+                         lastmod="2013-01-02T20:00:00Z",
+                         change="updated",
+                         datetime="2013-01-02T20:00:00Z",
+                         md5="0988647082c8bc51778894a48ec3b576",
+                         length="5426",  # should also take string
+                         mime_type="text/html",
+                         path="/changes/res7-v2.html"))
+        self._assert_xml_equal_ex(cdm.as_xml(), 'resourcesync_ex_23')
+
+    def test_build_ex_24(self):
+        cl = ChangeList()
+        cl.up = "http://example.com/dataset1/capabilitylist.xml"
+        cl.md_from = "2013-01-03T00:00:00Z"
+        c1 = Resource(uri="http://example.com/res1",
+                      lastmod="2013-01-03T18:00:00Z",  # Example has lastmod and no datetime
+                      change="updated",
+                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
+                      length=8876,
+                      mime_type="text/html")
+        # Resource.link_set with add or change link depending on one with
+        # the particular rel exists unless allow_duplicates=True.
+        # Resource.link_add will always add. Test both here...
+        c1.link_set(rel="duplicate",
+                    href="http://mirror1.example.com/res1",
+                    pri="1",
+                    modified="2013-01-03T18:00:00Z")
+        c1.link_set(rel="duplicate",
+                    href="http://mirror2.example.com/res1",
+                    pri="2",
+                    modified="2013-01-03T18:00:00Z",
+                    allow_duplicates=True)
+        c1.link_add(rel="duplicate",
+                    href="gsiftp://gridftp.example.com/res1",
+                    pri="3",
+                    modified="2013-01-03T18:00:23Z")
+        cl.add(c1)
+        self._assert_xml_equal_ex(cl.as_xml(), 'resourcesync_ex_24')
+
+    def test_build_ex_25(self):
+        cl = ChangeList()
+        cl.up = "http://example.com/dataset1/capabilitylist.xml"
+        cl.md_from = "2013-01-03T11:00:00Z"
+        c1 = Resource(uri="http://example.com/res1",
+                      change="updated",
+                      datetime="2013-01-03T18:00:00Z")
+        c1.link_add(rel="alternate",
+                    href="http://example.com/res1.html",
+                    type="text/html")
+        c1.link_add(rel="alternate",
+                    href="http://example.com/res1.pdf",
+                    type="application/pdf")
+        cl.add(c1)
+        self._assert_xml_equal_ex(cl.as_xml(), 'resourcesync_ex_25')
+
+    def test_build_ex_26(self):
+        cl = ChangeList()
+        cl.up = "http://example.com/dataset1/capabilitylist.xml"
+        cl.md_from = "2013-01-03T00:00:00Z"
+        c1 = Resource(uri="http://example.com/res1.html",
+                      change="updated",
+                      datetime="2013-01-03T18:00:00Z",
+                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
+                      length=8876)
+        c1.link_add(rel="canonical",
+                    href="http://example.com/res1",
+                    modified="2013-01-03T18:00:00Z")
+        cl.add(c1)
+        self._assert_xml_equal_ex(cl.as_xml(), 'resourcesync_ex_26')
+
+    def test_build_ex_27(self):
+        cl = ChangeList()
+        cl.up = "http://example.com/dataset1/capabilitylist.xml"
+        cl.md_from = "2013-01-03T00:00:00Z"
+        c1 = Resource(uri="http://example.com/res4",
+                      change="updated",
+                      sha256="f4OxZX_x_DFGFDgghgdfb6rtSx-iosjf6735432nklj",
+                      length=56778,
+                      mime_type="application/json")
+        c1.link_set(rel="http://www.openarchives.org/rs/terms/patch",
+                    href="http://example.com/res4-json-patch",
+                    hash="sha-256:y66dER_t_HWEIKpesdkeb7rtSc-ippjf9823742opld",  # FIXME - inconsistent
+                    length=73,
+                    type="application/json-patch")
+        c2 = Resource(uri="http://example.com/res5-full.tiff",
+                      change="updated",
+                      sha256="f4OxZX_x_FO5LcGBSKHWXfwtSx-j1ncoSt3SABJtkGk",
+                      length="9788456778",
+                      mime_type="image/tiff")
+        c2.link_set(rel="http://www.openarchives.org/rs/terms/patch",
+                    href="http://example.com/res5-diff",
+                    hash="sha-256:h986gT_t_87HTkjHYE76G558hY-jdfgy76t55sadJUYT",
+                    length=4533,
+                    type="application/x-tiff-diff")
+        cl.add([c1, c2])
+        self._assert_xml_equal_ex(cl.as_xml(), 'resourcesync_ex_27')
+
+    def test_build_ex_28(self):
+        cl = ChangeList()
+        cl.up = "http://example.com/dataset1/capabilitylist.xml"
+        cl.md_from = "2013-01-03T00:00:00Z"
+        c1 = Resource(uri="http://example.com/res2.pdf",
+                      change="updated",
+                      datetime="2013-01-03T18:00:00Z",
+                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
+                      length=8876,
+                      mime_type="application/pdf")
+        c1.link_set(rel="describedby",
+                    href="http://example.com/res2_dublin-core_metadata.xml",
+                    type="application/xml")
+        c2 = Resource(uri="http://example.com/res2_dublin-core_metadata.xml",
+                      change="updated",
+                      datetime="2013-01-03T19:00:00Z",
+                      mime_type="application/xml")
+        c2.link_set(rel="describes",
+                    href="http://example.com/res2.pdf",
+                    hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6",
+                    length="8876",
+                    type="application/pdf")
+        c2.link_set(rel="profile",
+                    href="http://purl.org/dc/elements/1.1/")
+        cl.add([c1, c2])
+        self._assert_xml_equal_ex(cl.as_xml(), 'resourcesync_ex_28')
+
+    def test_build_ex_29(self):
+        cl = ChangeList()
+        cl.up = "http://example.com/dataset1/capabilitylist.xml"
+        cl.md_from = "2013-01-03T00:00:00Z"
+        c1 = Resource(uri="http://example.com/res1",
+                      lastmod="2013-01-03T18:00:00Z",
+                      change="updated",
+                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
+                      length=8876,
+                      mime_type="text/html")
+        c1.link_add(rel="memento",
+                    href="http://example.com/20130103070000/res1",
+                    modified="2013-01-02T18:00:00Z",
+                    hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6",
+                    length="8876",
+                    type="text/html")
+        c1.link_add(rel="timegate",
+                    href="http://example.com/timegate/http://example.com/res1")
+        c1.link_add(rel="timemap",
+                    href="http://example.com/timemap/http://example.com/res1",
+                    type="application/link-format")
+        cl.add(c1)
+        self._assert_xml_equal_ex(cl.as_xml(), 'resourcesync_ex_29')
+
+    def test_build_ex_30(self):
+        cl = ChangeList()
+        cl.up = "http://example.com/dataset1/capabilitylist.xml"
+        cl.md_from = "2013-01-03T00:00:00Z"
+        c1 = Resource(uri="http://example.com/res1",
+                      change="updated",
+                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
+                      length=8876,
+                      mime_type="text/html")
+        c1.link_add(rel="collection",
+                    href="http://example.com/aggregation/0601007")
+        cl.add(c1)
+        self._assert_xml_equal_ex(cl.as_xml(), 'resourcesync_ex_30')
+
+    def test_build_ex_31(self):
+        cl = ChangeList()
+        cl.up = "http://example.com/dataset1/capabilitylist.xml"
+        cl.md_from = "2013-01-03T00:00:00Z"
+        c1 = Resource(uri="http://original.example.com/res1.html",
+                      lastmod="2013-01-03T07:00:00Z",
+                      change="updated",
+                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
+                      length=8876,
+                      mime_type="text/html")
+        cl.add(c1)
+        self._assert_xml_equal_ex(cl.as_xml(), 'resourcesync_ex_31')
+
+    def test_build_ex_32(self):
+        cl = ChangeList()
+        cl.up = "http://aggregator1.example.com/dataset1/capabilitylist.xml"
+        cl.md_from = "2013-01-03T11:00:00Z"
+        c1 = Resource(uri="http://aggregator1.example.com/res1.html",
+                      lastmod="2013-01-03T20:00:00Z",
+                      change="updated",
+                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
+                      length=8876,
+                      mime_type="text/html")
+        c1.link_add(rel="via",
+                    href="http://original.example.com/res1.html",
+                    modified="2013-01-03T07:00:00Z",
+                    hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6",
+                    length="8876",
+                    type="text/html")
+        cl.add(c1)
+        self._assert_xml_equal_ex(cl.as_xml(), 'resourcesync_ex_32')
+
+    def test_build_ex_33(self):
+        cl = ChangeList()
+        cl.up = "http://aggregator2.example.com/dataset1/capabilitylist.xml"
+        cl.md_from = "2013-01-03T12:00:00Z"
+        c1 = Resource(uri="http://aggregator2.example.com/res1.html",
+                      lastmod="2013-01-04T09:00:00Z",
+                      change="updated",
+                      md5="1584abdf8ebdc9802ac0c6a7402c03b6",
+                      length=8876,
+                      mime_type="text/html")
+        c1.link_add(rel="via",
+                    href="http://original.example.com/res1.html",
+                    modified="2013-01-03T07:00:00Z",
+                    hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6",
+                    length="8876",
+                    type="text/html")
+        cl.add(c1)
+        self._assert_xml_equal_ex(cl.as_xml(), 'resourcesync_ex_33')

tests/test_explorer.py

@@ -0,0 +1,128 @@
+from .testlib import capture_stdout
+
+import unittest
+import re
+import logging
+import sys
+
+from resync.client import Client
+from resync.client_utils import ClientFatalError
+from resync.capability_list import CapabilityList
+from resync.explorer import Explorer, XResource, ExplorerQuit
+from resync.resource import Resource
+
+
+class TestExplorer(unittest.TestCase):
+
+    @classmethod
+    def setUpClass(cls):
+        logging.basicConfig(level=logging.INFO)
+
+    def test01_create(self):
+        # dumb test that we can create Exporer object
+        e = Explorer()
+        self.assertTrue(e)
+
+    def test01_xresource(self):
+        x = XResource(uri='abc', acceptable_capabilities=[1, 2],
+                      checks=[3, 4], context='http://example.org/def')
+        self.assertEqual(x.uri, 'http://example.org/abc')
+        self.assertEqual(x.acceptable_capabilities, [1, 2])
+        self.assertEqual(x.checks, [3, 4])
+
+    def test03_explorer_quit(self):
+        eq = ExplorerQuit()
+        self.assertTrue(isinstance(eq, Exception))
+
+    def test04_explore(self):
+        e = Explorer()
+        e.sitemap_name = 'tests/testdata/explore1/caps1.xml'
+        with capture_stdout() as capturer:
+            # IOError generated from attempt to read stdin
+            e.fake_input = 'q'
+            e.explore()
+        self.assertTrue(re.search(r'resync-explorer done', capturer.result))
+
+    def test05_explore_uri(self):
+        e = Explorer()
+        with capture_stdout() as capturer:
+            e.fake_input = 'q'
+            self.assertRaises(ExplorerQuit, e.explore_uri,
+                              XResource('tests/testdata/explore1/caps1.xml'))
+        self.assertTrue(re.search(r'Reading tests/testdata/explore1/caps1.xml',
+                                  capturer.result))
+        self.assertTrue(re.search(r'Parsed capabilitylist document with 4 entries:',
+                                  capturer.result))
+
+    def test06_explore_show_summary(self):
+        e = Explorer()
+        # file that exists with matching
+        with capture_stdout() as capturer:
+            e.explore_show_summary(list=CapabilityList())
+        self.assertTrue(re.search(r'Parsed \(unknown capability\) document with 0 entries:',
+                                  capturer.result))
+        # dummy capabilities object and display
+        cl = CapabilityList()
+        cl.add(Resource('uri:resourcelist'))
+        cl.add(Resource('uri:changelist'))
+        with capture_stdout() as capturer:
+            e.explore_show_summary(cl, False, [])
+        self.assertTrue(re.search(
+            r'Parsed \(unknown capability\) document with 2 entries:', capturer.result))
+        self.assertTrue(re.search(r'\[1\] uri:changelist', capturer.result))
+        self.assertTrue(re.search(r'\[2\] uri:resourcelist', capturer.result))
+
+    def test07_explore_show_head(self):
+        e = Explorer()
+        # file that exists with matching
+        with capture_stdout() as capturer:
+            e.explore_show_head(uri='tests/testdata/dir1/file_a',
+                                check_headers={'content-length': 20,
+                                               'unknown': 'abc'})
+        self.assertTrue(
+            re.search(r'HEAD tests/testdata/dir1/file_a', capturer.result))
+        self.assertTrue(
+            re.search(r'content-length: 20 MATCHES EXPECTED VALUE', capturer.result))
+        # same file, bad header check
+        with capture_stdout() as capturer:
+            e.explore_show_head('tests/testdata/dir1/file_a',
+                                check_headers={'content-length': 99})
+        self.assertTrue(
+            re.search(r'HEAD tests/testdata/dir1/file_a', capturer.result))
+        self.assertTrue(
+            re.search(r'content-length: 20 EXPECTED 99', capturer.result))
+        # file that does not exist
+        with capture_stdout() as capturer:
+            e.explore_show_head('tests/testdata/does_not_exist')
+        self.assertTrue(
+            re.search(r'HEAD tests/testdata/does_not_exist', capturer.result))
+        self.assertTrue(re.search(r'status: 404', capturer.result))
+
+    def test08_head_on_file(self):
+        e = Explorer()
+        (status_code, headers) = e.head_on_file('tests/testdata/does_not_exist')
+        self.assertEqual(status_code, '404')
+        self.assertEqual(headers, {})
+        (status_code, headers) = e.head_on_file('tests/testdata/dir1/file_a')
+        self.assertEqual(status_code, '200')
+        self.assertTrue(re.match(r'^\d\d\d\d\-\d\d\-\d\d',
+                        headers['last-modified']))
+        self.assertEqual(headers['content-length'], 20)
+
+    def test09_allowed_entries(self):
+        e = Explorer()
+        self.assertEqual(e.allowed_entries('capabilitylistindex'), [])
+        self.assertEqual(e.allowed_entries(
+            'resourcelistindex'), ['resourcelist'])
+        self.assertEqual(e.allowed_entries(
+            'changelist-archive'), ['changelist'])
+        self.assertEqual(e.allowed_entries('description'), ['capabilitylist'])
+        self.assertEqual(len(e.allowed_entries('capabilitylist')), 8)
+        self.assertEqual(e.allowed_entries('unknown'), [])
+
+    def test10_expand_relative_uri(self):
+        e = Explorer()
+        with capture_stdout() as capturer:
+            uri = e.expand_relative_uri('https://example.org/ctx', 'abc')
+        self.assertEqual(uri, 'https://example.org/abc')
+        self.assertTrue(re.search(r'expanded relative URI', capturer.result))

tests/test_hashes.py

@@ -0,0 +1,23 @@
+import unittest
+import resync.hashes
+
+
+class TestUtill(unittest.TestCase):
+
+    def test01_md5_sha1_file(self):
+        h = resync.hashes.Hashes(['md5', 'sha-1'])
+        h.compute_for_file('tests/testdata/a')
+        self.assertEqual(h.md5, '8fdd769621e003fe3c0c21e9929b491e')
+        self.assertEqual(h.sha1, '49844dd211aa33071a252d7cdc250a52cf39af33')
+        self.assertEqual(h.sha256, None)
+        h = resync.hashes.Hashes(['sha-256', 'sha-1'])
+        h.compute_for_file('tests/testdata/a')
+        self.assertEqual(h.md5, None)
+        self.assertEqual(h.sha1, '49844dd211aa33071a252d7cdc250a52cf39af33')
+        self.assertEqual(h.sha256, '69fe6314a94800456af959d380f5d6932052478ea03d5ccac7ba0a14bd5e67c6')
+        h = resync.hashes.Hashes(['sha-256'])
+        h.compute_for_file('tests/testdata/a')
+        self.assertEqual(h.sha1, None)
+
+    def test02_bad_type(self):
+        self.assertRaises(Exception, resync.hashes.Hashes, ['md5', 'xyz'])

tests/test_list_base.py

@@ -0,0 +1,91 @@
+from .testlib import TestCase
+
+import sys
+import os.path
+import unittest
+try:  # python2
+    # Must try this first as io also exists in python2
+    # but in the wrong one!
+    import StringIO as io
+except ImportError:  # python3
+    import io
+
+from resync.resource import Resource
+from resync.list_base import ListBase
+from resync.sitemap import Sitemap, SitemapIndexError
+
+# etree gives ParseError in 2.7, ExpatError in 2.6
+etree_error_class = None
+if (sys.version_info < (2, 7)):
+    from xml.parsers.expat import ExpatError
+    etree_error_class = ExpatError
+else:
+    from xml.etree.ElementTree import ParseError
+    etree_error_class = ParseError
+
+
+class TestListBase(TestCase):
+
+    def test01_len_count(self):
+        # count sets explicit number of resources, len(resources) not used
+        lb = ListBase()
+        self.assertEqual(len(lb), 0)
+        lb.add(Resource('a'))
+        self.assertEqual(len(lb), 1)
+        lb = ListBase(count=100)
+        self.assertEqual(len(lb), 100)
+
+    def test02_print(self):
+        lb = ListBase()
+        lb.add(Resource(uri='a', lastmod='2001-01-01', length=1234))
+        lb.add(Resource(uri='b', lastmod='2002-02-02', length=56789))
+        lb.add(Resource(uri='c', lastmod='2003-03-03', length=0))
+        lb.md['from'] = None  # avoid now being added
+        x = lb.as_xml()
+        self.assertEqual(x, '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"><rs:md capability="unknown" /><url><loc>a</loc><lastmod>2001-01-01T00:00:00Z</lastmod><rs:md length="1234" /></url><url><loc>b</loc><lastmod>2002-02-02T00:00:00Z</lastmod><rs:md length="56789" /></url><url><loc>c</loc><lastmod>2003-03-03T00:00:00Z</lastmod><rs:md length="0" /></url></urlset>')
+
+    def test03_read(self):
+        # just does parse
+        lb = ListBase()
+        lb.read(uri='tests/testdata/unknown/unknown1.xml')
+        self.assertEqual(len(lb.resources), 2)
+
+    def test04_parse(self):
+        xml = '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
+<rs:md capability="unknown" from="2013-02-12T14:09:00Z" />\
+<url><loc>/tmp/rs_test/src/file_a</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="12" /></url>\
+<url><loc>/tmp/rs_test/src/file_b</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="32" /></url>\
+</urlset>'
+        # parse from string
+        lb1 = ListBase()
+        lb1.parse(str_data=xml)
+        self.assertEqual(len(lb1.resources), 2, 'got 2 resources')
+        # parse from fh
+        lb2 = ListBase()
+        lb2.parse(fh=io.StringIO(xml))
+        self.assertEqual(len(lb2.resources), 2, 'got 2 resources')
+        # parse from string (LEGACY parameter name, to be removed)
+        lb3 = ListBase()
+        lb3.parse(str=xml)
+        self.assertEqual(len(lb3.resources), 2, 'got 2 resources')
+        # file that doesn't exits
+        lb4 = ListBase()
+        self.assertRaises(Exception, lb4.parse,
+                          uri='tests/testdata/does_not_exist')
+        # nothing
+        self.assertRaises(Exception, lb4.parse)
+
+    def test04_write(self):
+        lb = ListBase(capability_name='special')
+        lb.add(Resource(uri='http://example.org/lemon'))
+        lb.add(Resource(uri='http://example.org/orange'))
+        basename = os.path.join(self.tmpdir, 'lb.xml')
+        lb.write(basename=basename)
+        self.assertTrue(os.path.exists(basename))
+        # and now parse back
+        fh = open(basename, 'r')
+        lb2 = ListBase(capability_name='special')
+        lb2.parse(fh=fh)
+        self.assertEqual(lb2.capability, 'special')
+        self.assertEqual(len(lb2), 2)

tests/test_list_base_with_index.py

@@ -0,0 +1,131 @@
+import re
+import sys
+import unittest
+try:  # python2
+    # Must try this first as io also exists in python2
+    # but in the wrong one!
+    import StringIO as io
+except ImportError:  # python3
+    import io
+
+from resync.resource import Resource
+from resync.list_base_with_index import ListBaseWithIndex, ListBaseIndexError
+from resync.sitemap import Sitemap, SitemapIndexError
+
+# etree gives ParseError in 2.7, ExpatError in 2.6
+etree_error_class = None
+if (sys.version_info < (2, 7)):
+    from xml.parsers.expat import ExpatError
+    etree_error_class = ExpatError
+else:
+    from xml.etree.ElementTree import ParseError
+    etree_error_class = ParseError
+
+
+class TestListBaseWithIndex(unittest.TestCase):
+
+    def test01_print(self):
+        lb = ListBaseWithIndex()
+        lb.add(Resource(uri='a', lastmod='2001-01-01', length=1234))
+        lb.add(Resource(uri='b', lastmod='2002-02-02', length=56789))
+        lb.add(Resource(uri='c', lastmod='2003-03-03', length=0))
+        lb.md['from'] = None  # avoid now being added
+        self.assertEqual(lb.as_xml(), '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"><rs:md capability="unknown" /><url><loc>a</loc><lastmod>2001-01-01T00:00:00Z</lastmod><rs:md length="1234" /></url><url><loc>b</loc><lastmod>2002-02-02T00:00:00Z</lastmod><rs:md length="56789" /></url><url><loc>c</loc><lastmod>2003-03-03T00:00:00Z</lastmod><rs:md length="0" /></url></urlset>')
+
+    def test02_print_iter(self):
+        r = [Resource(uri='a', lastmod='2001-01-01', length=1234),
+             Resource(uri='b', lastmod='2002-02-02', length=56789),
+             Resource(uri='c', lastmod='2003-03-03', length=0)]
+        # without setting count will barf on len() attempt
+        lb = ListBaseWithIndex(resources=iter(r))
+        self.assertRaises(TypeError, lb.as_xml)
+        # set explicit count larger than max_sitemap_entiries and as_xml will
+        # throw exception
+        lb = ListBaseWithIndex(resources=iter(r), count=3)
+        lb.max_sitemap_entries = 2
+        self.assertRaises(ListBaseIndexError, lb.as_xml)
+        # set explicit count and all will be OK
+        lb = ListBaseWithIndex(resources=iter(r), count=3)
+        lb.md['from'] = None  # avoid now being added
+        self.assertEqual(lb.as_xml(), '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"><rs:md capability="unknown" /><url><loc>a</loc><lastmod>2001-01-01T00:00:00Z</lastmod><rs:md length="1234" /></url><url><loc>b</loc><lastmod>2002-02-02T00:00:00Z</lastmod><rs:md length="56789" /></url><url><loc>c</loc><lastmod>2003-03-03T00:00:00Z</lastmod><rs:md length="0" /></url></urlset>')
+
+    def test03_parse_2(self):
+        xml = '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
+<rs:md capability="unknown" from="2013-02-12T14:09:00Z" />\
+<url><loc>/tmp/rs_test/src/file_a</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="12" /></url>\
+<url><loc>/tmp/rs_test/src/file_b</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="32" /></url>\
+</urlset>'
+        lb = ListBaseWithIndex()
+        lb.parse(fh=io.StringIO(xml))
+        self.assertEqual(len(lb.resources), 2, 'got 2 resources')
+
+    def test16_as_xml(self):
+        r = [Resource(uri='a', lastmod='2014-04-14', length=14),
+             Resource(uri='b', lastmod='2015-05-15', length=15),
+             Resource(uri='c', lastmod='2016-06-16', length=16)]
+        lb = ListBaseWithIndex(resources=r)
+        lb.max_sitemap_entries = 3
+        # One file
+        xml = lb.as_xml()
+        self.assertTrue(re.search(r'<urlset ', xml))
+        self.assertTrue(re.search(r'<loc>a</loc>', xml))
+        self.assertTrue(re.search(r'<loc>b</loc>', xml))
+        self.assertTrue(re.search(r'<loc>c</loc>', xml))
+        # Needs multifile bit not allowed
+        lb.max_sitemap_entries = 1
+        self.assertRaises(ListBaseIndexError, lb.as_xml)
+        # Allow multifile...
+        xml = lb.as_xml(allow_multifile=True)
+        self.assertTrue(re.search(r'<sitemapindex', xml))
+        self.assertTrue(re.search(r'<loc>/tmp/sitemap00001.xml</loc>', xml))
+        self.assertTrue(re.search(r'<loc>/tmp/sitemap00002.xml</loc>', xml))
+        self.assertFalse(re.search(r'<loc>/tmp/sitemap00003.xml</loc>', xml))
+
+    def test17_as_xml_index(self):
+        r = [Resource(uri='a', lastmod='2006-01-01', length=12),
+             Resource(uri='b', lastmod='2007-02-02', length=34),
+             Resource(uri='c', lastmod='2008-03-03', length=56)]
+        lb = ListBaseWithIndex(resources=r)
+        lb.max_sitemap_entries = 2
+        xml = lb.as_xml_index()
+        self.assertTrue(re.search(r'<loc>/tmp/sitemap00001.xml</loc>', xml))
+        self.assertFalse(re.search(r'<loc>/tmp/sitemap00002.xml</loc>', xml))
+        # Index not required
+        lb.max_sitemap_entries = 3
+        self.assertRaises(ListBaseIndexError, lb.as_xml_index)
+
+    def test18_as_xml_part(self):
+        r = [Resource(uri='a', lastmod='2006-01-01', length=12),
+             Resource(uri='b', lastmod='2007-02-02', length=34),
+             Resource(uri='c', lastmod='2008-03-03', length=56)]
+        lb = ListBaseWithIndex(resources=r)
+        # Allow unlimited entries, part makes no sense
+        lb.max_sitemap_entries = None
+        self.assertRaises(ListBaseIndexError, lb.as_xml_part)
+        # Request after end
+        lb.max_sitemap_entries = 1
+        self.assertRaises(ListBaseIndexError, lb.as_xml_part, part_number=9)
+        # Allow only 1 entry
+        lb.max_sitemap_entries = 1
+        xml = lb.as_xml_part(part_number=1)
+        self.assertFalse(re.search(r'<loc>a</loc>', xml))
+        self.assertTrue(re.search(r'<loc>b</loc>', xml))
+        self.assertFalse(re.search(r'<loc>c</loc>', xml))
+        # Request truncated
+        lb.max_sitemap_entries = 2
+        xml = lb.as_xml_part(part_number=1)
+        self.assertFalse(re.search(r'<loc>a</loc>', xml))
+        self.assertFalse(re.search(r'<loc>b</loc>', xml))
+        self.assertTrue(re.search(r'<loc>c</loc>', xml))
+
+    def test20_index_as_xml(self):
+        # Check XML for empty case
+        lb = ListBaseWithIndex()
+        self.assertEqual(lb.index_as_xml(
+        ), '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"><rs:md capability="unknown" /></sitemapindex>')
+        # Add a resource and make sure we find that
+        lb.add(Resource(uri='a', lastmod='2001-01-01', length=1234))
+        xml = lb.index_as_xml()
+        self.assertTrue(
+            re.search(r'<loc>a</loc><lastmod>2001-01-01T00:00:00Z</lastmod>', xml))

tests/test_mapper.py

@@ -0,0 +1,138 @@
+"""Tests for resync.mapper."""
+import unittest
+from resync.mapper import Mapper, MapperError, Map
+
+
+class TestMapper(unittest.TestCase):
+
+    def test00_mapper_creation(self):
+        m1 = Mapper(['http://e.org/p/', '/tmp/q/'])
+        self.assertEqual(len(m1), 1)
+        m2 = Mapper(mappings=['http://e.org/p', '/tmp/q'])
+        self.assertEqual(len(m2), 1)
+        self.assertEqual(str(m1), str(m2))
+        m3 = Mapper(['http://e.org/p/=/tmp/q/'])
+        self.assertEqual(len(m3), 1)
+        self.assertEqual(str(m1), str(m3))
+        m4 = Mapper(['http://e.org/p/=/tmp/q/', 'http://e.org/r/=/tmp/s/'])
+        m5 = Mapper(['http://e.org/r/=/tmp/s/', 'http://e.org/p/=/tmp/q/'])
+        self.assertEqual(len(m4), 2)
+        self.assertEqual(len(m5), 2)
+        self.assertNotEqual(str(m4), str(m5))
+        # error cases
+        m6 = Mapper()
+        # too many equals
+        self.assertRaises(MapperError, m6.parse, ['a=b=c'])
+        self.assertRaises(MapperError, m6.parse, ['a=b=c=d'])
+        # dupes
+        self.assertRaises(MapperError, m6.parse, ['a=b', 'a=c'])
+        self.assertRaises(MapperError, m6.parse, ['x=z', 'y=z'])
+
+    def test01_mapper_src_to_dst(self):
+        m = Mapper(['http://e.org/p/', '/tmp/q/'])
+        self.assertEqual(m.src_to_dst('http://e.org/p/'), '/tmp/q/')
+        self.assertEqual(m.src_to_dst('http://e.org/p/aa'), '/tmp/q/aa')
+        self.assertEqual(m.src_to_dst('http://e.org/p/aa/bb'), '/tmp/q/aa/bb')
+        self.assertEqual(m.src_to_dst('http://e.org/p/aa/bb/'), '/tmp/q/aa/bb/')
+        self.assertEqual(m.src_to_dst('http://e.org/p'), '/tmp/q/')
+        self.assertRaises(MapperError, m.src_to_dst, 'http://e.org/pa')
+        self.assertRaises(MapperError, m.src_to_dst, 'nomatch')
+
+    def test02_mapper_dst_to_src(self):
+        m = Mapper(['http://e.org/p/', '/tmp/q/'])
+        self.assertEqual(m.dst_to_src('/tmp/q/'), 'http://e.org/p/')
+        self.assertEqual(m.dst_to_src('/tmp/q/bb'), 'http://e.org/p/bb')
+        self.assertEqual(m.dst_to_src('/tmp/q/bb/cc'), 'http://e.org/p/bb/cc')
+        self.assertEqual(m.dst_to_src('/tmp/q'), 'http://e.org/p/')
+        self.assertRaises(MapperError, m.dst_to_src, '/tmp/qa')
+        self.assertRaises(MapperError, m.dst_to_src, 'nomatch')
+
+    def test03_mapper2_src_to_dst(self):
+        m = Mapper(['http://e.org/p=/tmp/q', 'http://e.org/r=/tmp/s'])
+        self.assertEqual(m.src_to_dst('http://e.org/p/'), '/tmp/q/')
+        self.assertEqual(m.src_to_dst('http://e.org/p/aa'), '/tmp/q/aa')
+        self.assertEqual(m.src_to_dst('http://e.org/r/'), '/tmp/s/')
+        self.assertEqual(m.src_to_dst('http://e.org/r/aa'), '/tmp/s/aa')
+
+    def test04_mapper2_dst_to_src(self):
+        m = Mapper(['http://e.org/p=/tmp/q', 'http://e.org/r=/tmp/s'])
+        self.assertEqual(m.dst_to_src('/tmp/q/'), 'http://e.org/p/')
+        self.assertEqual(m.dst_to_src('/tmp/q/bb'), 'http://e.org/p/bb')
+        self.assertEqual(m.dst_to_src('/tmp/s/'), 'http://e.org/r/')
+        self.assertEqual(m.dst_to_src('/tmp/s/bb'), 'http://e.org/r/bb')
+
+    def test05_path_from_uri(self):
+        m = Mapper()
+        self.assertEqual(m.path_from_uri('a_file'), 'a_file')
+        self.assertEqual(m.path_from_uri(
+            'some_path/a_file'), 'some_path/a_file')
+        self.assertEqual(m.path_from_uri('http://localhost/p'), 'localhost_p')
+        self.assertEqual(m.path_from_uri(
+            'http://localhost:8888/p'), 'localhost_8888_p')
+        self.assertEqual(m.path_from_uri(
+            'https://localhost:8888/p'), 'localhost_8888_p')
+        self.assertEqual(m.path_from_uri('http://example.com'), 'example.com')
+        self.assertEqual(m.path_from_uri('http://example.com/'), 'example.com')
+        self.assertEqual(m.path_from_uri(
+            'http://example.com/ex1'), 'example.com_ex1')
+        self.assertEqual(m.path_from_uri(
+            'http://example.com/ex1/'), 'example.com_ex1')
+
+    def test06_mapper_unsafe(self):
+        self.assertFalse(Mapper(['http://example.com/=/tmp/a']).unsafe())
+        self.assertFalse(
+            Mapper(['http://example.com/=http://example.com/']).unsafe())
+        self.assertFalse(Mapper(['http://example.com/'],
+                                use_default_path=True).unsafe())
+        # Following hits case of single local arg supplied
+        self.assertTrue(Mapper(['/tmp/a'], use_default_path=True).unsafe())
+        # One good, one bad -> bad
+        self.assertTrue(
+            Mapper(['http://example.com/=/tmp/a', '/tmp/a=/tmp']).unsafe())
+
+    def test07_default_src_uri(self):
+        self.assertEqual(Mapper(['a=b']).default_src_uri(), 'a')
+        self.assertEqual(Mapper(['a=b', 'b=c']).default_src_uri(), 'a')
+        self.assertRaises(MapperError, Mapper().default_src_uri)
+
+
+class TestMap(unittest.TestCase):
+
+    def test01_map_unsafe(self):
+        """Test unsafe method."""
+        self.assertFalse(Map('http://example.com/', 'path').unsafe())
+        # Note the first is a URI and the second is a (silly) path in the
+        # following
+        self.assertFalse(Map('http://example.com/', 'http://example.com/').unsafe())
+        self.assertFalse(Map('a', 'b').unsafe())
+        self.assertFalse(Map('path/a', 'path/b').unsafe())
+        # The following are unsafe
+        self.assertTrue(Map('path', 'path').unsafe())
+        self.assertTrue(Map('path/a', 'path').unsafe())
+        self.assertTrue(Map('path', 'path/b').unsafe())
+
+    def test02_dst_to_src(self):
+        """Test dst_to_src method."""
+        m = Map('uri:a/b', 'path/c')
+        self.assertEqual(m.dst_to_src('path/c/file'), 'uri:a/b/file')
+        self.assertEqual(m.dst_to_src('path/c/file/'), 'uri:a/b/file/')
+        self.assertEqual(m.dst_to_src('path/c'), 'uri:a/b/')
+        self.assertEqual(m.dst_to_src('path/c/'), 'uri:a/b/')
+        self.assertEqual(m.dst_to_src('path/cliff'), None)
+        # Note that trailing slashes on the map elements have no effect
+        m = Map('uri:a/b/', 'path/c/')
+        self.assertEqual(m.dst_to_src('path/c'), 'uri:a/b/')
+        self.assertEqual(m.dst_to_src('path/c/'), 'uri:a/b/')
+        self.assertEqual(m.dst_to_src('path/cliff'), None)
+        # Failed maps return None
+        self.assertEqual(m.dst_to_src('c/file'), None)
+        self.assertEqual(m.dst_to_src(''), None)
+
+    def test03_src_to_dst(self):
+        """Test src_to_dst method."""
+        m = Map('uri:x/', '/tmp/gg/')
+        self.assertEqual(m.src_to_dst('uri:other'), None)
+        self.assertEqual(m.src_to_dst('uri:x'), '/tmp/gg/')
+        self.assertEqual(m.src_to_dst('uri:x/'), '/tmp/gg/')
+        self.assertEqual(m.src_to_dst('uri:x/lizard'), '/tmp/gg/lizard')
+        self.assertEqual(m.src_to_dst('uri:x/lizard/'), '/tmp/gg/lizard/')

tests/test_resource.py

@@ -0,0 +1,318 @@
+"""Tests for resync.resource."""
+import unittest
+import re
+from resync.resource import Resource, ChangeTypeError
+
+
+class TestResource(unittest.TestCase):
+
+    def test01_init(self):
+        """Test __init__ method."""
+        # No uri = error
+        self.assertRaises(ValueError, Resource)
+        self.assertRaises(ValueError, Resource, timestamp=12)
+        # Create with many params
+        r1 = Resource(uri='a', timestamp=0, length=123,
+                      md5='aaa', sha1='bbb', sha256='ccc',
+                      mime_type='text/plain', change='updated',
+                      ts_datetime=1000, path='/a/b/c', ln={'x': 'y'},
+                      ts_at=2000, ts_completed=3000, ts_from=4000,
+                      ts_until=5000)
+        self.assertEqual(r1.uri, 'a')
+        self.assertEqual(r1.timestamp, 0)
+        self.assertEqual(r1.length, 123)
+        self.assertEqual(r1.md5, 'aaa')
+        self.assertEqual(r1.sha1, 'bbb')
+        self.assertEqual(r1.sha256, 'ccc')
+        self.assertEqual(r1.mime_type, 'text/plain')
+        self.assertEqual(r1.change, 'updated')
+        self.assertEqual(r1.ts_datetime, 1000)
+        self.assertEqual(r1.path, '/a/b/c')
+        self.assertEqual(r1.ln, {'x': 'y'})
+        self.assertEqual(r1.ts_at, 2000)
+        self.assertEqual(r1.ts_completed, 3000)
+        self.assertEqual(r1.ts_from, 4000)
+        self.assertEqual(r1.ts_until, 5000)
+        # Create r2 like r1
+        r2 = Resource(resource=r1)
+        self.assertEqual(r2.uri, 'a')
+        self.assertEqual(r2.timestamp, 0)
+        self.assertEqual(r2.length, 123)
+        self.assertEqual(r2.md5, 'aaa')
+        self.assertEqual(r2.sha1, 'bbb')
+        self.assertEqual(r2.sha256, 'ccc')
+        self.assertEqual(r2.mime_type, 'text/plain')
+        self.assertEqual(r2.change, 'updated')
+        self.assertEqual(r2.ts_datetime, 1000)
+        self.assertEqual(r2.path, '/a/b/c')
+        self.assertEqual(r2.ln, {'x': 'y'})
+
+    def test02_equal(self):
+        """Test equal method via = operator."""
+        # just uri
+        r1 = Resource('a')
+        r2 = Resource('a')
+        self.assertEqual(r1, r1)
+        self.assertEqual(r1, r2)
+        # with timestamps
+        r1 = Resource(uri='a', timestamp=1234.0)
+        r2 = Resource(uri='a', timestamp=1234.0)
+        self.assertEqual(r1, r1)
+        self.assertEqual(r1, r2)
+        # with lastmod instead of direct timestamp
+        r1 = Resource('a')
+        r1lm = '2012-01-01T00:00:00Z'
+        r1.lastmod = r1lm
+        r2 = Resource('a')
+        for r2lm in ('2012',
+                     '2012-01',
+                     '2012-01-01',
+                     '2012-01-01T00:00Z',
+                     '2012-01-01T00:00:00Z',
+                     '2012-01-01T00:00:00.000000Z',
+                     '2012-01-01T00:00:00.000000000000Z',
+                     '2012-01-01T00:00:00.000000000001Z',  # below resolution
+                     '2012-01-01T00:00:00.00+00:00',
+                     '2012-01-01T00:00:00.00-00:00',
+                     '2012-01-01T02:00:00.00-02:00',
+                     '2011-12-31T23:00:00.00+01:00'
+                     ):
+            r2.lastmod = r2lm
+            self.assertEqual(r1.timestamp, r2.timestamp, ('%s (%f) == %s (%f)' % (
+                r1lm, r1.timestamp, r2lm, r2.timestamp)))
+            self.assertEqual(r1, r2)
+        # with slight timestamp diff
+        r1 = Resource('a')
+        r1.lastmod = '2012-01-02T01:02:03Z'
+        r2 = Resource('a')
+        r2.lastmod = '2012-01-02T01:02:03.99Z'
+        self.assertNotEqual(r1.timestamp, r2.timestamp)
+        self.assertEqual(r1, r2)
+        # now with too much time diff
+        r1 = Resource('a', lastmod='2012-01-11')
+        r2 = Resource('a', lastmod='2012-01-22')
+        self.assertNotEqual(r1, r2)
+        # different uris
+        r1 = Resource('a')
+        r2 = Resource('b')
+        self.assertNotEqual(r1, r2)
+        # same and different lengths
+        r1 = Resource('a', length=1234)
+        r2 = Resource('a', length=4321)
+        self.assertNotEqual(r1, r2)
+        r2.length = r1.md5
+        self.assertEqual(r1, r2)
+        # same and different md5
+        r1.md5 = "3006f84272f2653a6cf5ec3af8f0d773"
+        r2.md5 = "3006f84272f2653a6cf5ec3af8f00000"
+        self.assertNotEqual(r1, r2)
+        r2.md5 = r1.md5
+        self.assertEqual(r1, r2)
+        # same and different sha1
+        r1.sha1 = "3be0f3af2aa4656ce38e0cef305c6eb2af4385d4"
+        r2.sha1 = "555"
+        self.assertNotEqual(r1, r2)
+        r2.sha1 = r1.sha1
+        self.assertEqual(r1, r2)
+        # same and different sha256
+        r1.sha256 = "f41094ad47ef3e93ec1021bfa40f4bf0185f1bf897533638ae5358b61713f84a"
+        r2.sha256 = "fab"
+        self.assertNotEqual(r1, r2)
+        r2.sha256 = r1.sha256
+        self.assertEqual(r1, r2)
+
+    def test04_bad_lastmod(self):
+        def setlastmod(r, v):
+            r.lastmod = v
+        r = Resource('4')
+        # Bad formats
+        self.assertRaises(ValueError, setlastmod, r, "bad_lastmod")
+        self.assertRaises(ValueError, setlastmod, r, "")
+        self.assertRaises(ValueError, setlastmod, r, "2012-13-01")
+        self.assertRaises(ValueError, setlastmod, r, "2012-12-32")
+        self.assertRaises(ValueError, setlastmod, r, "2012-11-01T10:10:60")
+        self.assertRaises(ValueError, setlastmod, r, "2012-11-01T10:10:59.9x")
+        # Valid ISO8601 but not alloed in W3C Datetime
+        self.assertRaises(ValueError, setlastmod, r, "2012-11-01T01:01:01")
+        self.assertRaises(ValueError, setlastmod, r, "2012-11-01 01:01:01Z")
+        self.assertRaises(ValueError, setlastmod, r, "2012-11-01T01:01:01+0000")
+        self.assertRaises(ValueError, setlastmod, r, "2012-11-01T01:01:01-1000")
+
+    def test05_lastmod_roundtrips(self):
+        r = Resource('a')
+        r.lastmod = '2012-03-14'
+        self.assertEqual(r.lastmod, '2012-03-14T00:00:00Z')
+        r.lastmod = '2012-03-14T00:00:00+00:00'
+        # print r.timestamp
+        self.assertEqual(r.lastmod, '2012-03-14T00:00:00Z')
+        r.lastmod = '2012-03-14T00:00:00-00:00'
+        # print r.timestamp
+        self.assertEqual(r.lastmod, '2012-03-14T00:00:00Z')
+        r.lastmod = '2012-03-14T18:37:36Z'
+        # print r.timestamp
+        self.assertEqual(r.lastmod, '2012-03-14T18:37:36Z')
+
+    def test06_str(self):
+        r1 = Resource('abc', lastmod='2012-01-01')
+        self.assertTrue(re.match(r"\[ abc \| 2012-01-01T", str(r1)))
+
+    def test07_repr(self):
+        r1 = Resource('def', lastmod='2012-01-01')
+        self.assertEqual(repr(r1), "{'uri': 'def', 'timestamp': 1325376000}")
+
+    def test08_hash(self):
+        """Test hash getter and setters."""
+        r1 = Resource('abcd')
+        r1.md5 = "some_md5"
+        r1.sha1 = "some_sha1"
+        r1.sha256 = "some_sha256"
+        self.assertEqual(r1.md5, "some_md5")
+        self.assertEqual(r1.sha1, "some_sha1")
+        self.assertEqual(r1.sha256, "some_sha256")
+        self.assertEqual(r1.hash, "md5:some_md5 sha-1:some_sha1 sha-256:some_sha256")
+        r2 = Resource('def')
+        r2.hash = "md5:ddd"
+        self.assertEqual(r2.md5, 'ddd')
+        self.assertEqual(r2.sha1, None)
+        r2.hash = "sha-1:eee"
+        self.assertEqual(r2.md5, None)
+        self.assertEqual(r2.sha1, 'eee')
+        r2.hash = "md5:fff sha-1:eee sha-256:ggg"
+        self.assertEqual(r2.md5, 'fff')
+        self.assertEqual(r2.sha1, 'eee')
+        self.assertEqual(r2.sha256, 'ggg')
+        # bogus value will reset
+        r2.hash = 11
+        self.assertEqual(r2.md5, None)
+        self.assertEqual(r2.sha1, None)
+        self.assertEqual(r2.sha256, None)
+        # string withough : will raise error
+        with self.assertRaises(ValueError):
+            r2.hash = "no-colon"
+        # dupe
+        with self.assertRaises(ValueError):
+            r2.hash = "md5:aaa md5:bbb"
+        # unknown
+        with self.assertRaises(ValueError):
+            r2.hash = "sha999:aaa"
+
+    def test09_changetypeerror(self):
+        r1 = Resource('a')
+        self.assertEqual(r1.change, None)
+        r1.change = 'deleted'
+        self.assertEqual(r1.change, 'deleted')
+        self.assertRaises(ChangeTypeError, Resource, 'a', change="bad")
+        # disable checking
+        ct = Resource.CHANGE_TYPES
+        Resource.CHANGE_TYPES = False
+        r1 = Resource('a', change="bad")
+        self.assertEqual(r1.change, 'bad')
+        Resource.CHANGE_TYPES = ct
+
+    def test10_md_at_roundtrips(self):
+        r = Resource('a')
+        r.md_at = '2013-03-14'
+        self.assertEqual(r.md_at, '2013-03-14T00:00:00Z')
+        r.md_at = '2013-03-14T00:00:00+00:00'
+        self.assertEqual(r.md_at, '2013-03-14T00:00:00Z')
+        r.md_at = '2013-03-14T00:00:00-00:00'
+        self.assertEqual(r.md_at, '2013-03-14T00:00:00Z')
+        r.md_at = '2013-03-14T18:37:36Z'
+        self.assertEqual(r.md_at, '2013-03-14T18:37:36Z')
+
+    def test11_md_completed_roundtrips(self):
+        r = Resource('a')
+        r.md_completed = '2013-04-14'
+        self.assertEqual(r.md_completed, '2013-04-14T00:00:00Z')
+        r.md_completed = '2013-04-14T00:00:00+00:00'
+        self.assertEqual(r.md_completed, '2013-04-14T00:00:00Z')
+        r.md_completed = '2013-04-14T00:00:00-00:00'
+        self.assertEqual(r.md_completed, '2013-04-14T00:00:00Z')
+        r.md_completed = '2013-04-14T18:37:36Z'
+        self.assertEqual(r.md_completed, '2013-04-14T18:37:36Z')
+
+    def test12_timevalues(self):
+        r = Resource(uri='tv',
+                     lastmod="2000-01-01",
+                     md_at="2000-01-02",
+                     md_completed="2000-01-03",
+                     md_from="2000-01-04",
+                     md_until="2000-01-05")
+        self.assertEqual(r.lastmod, '2000-01-01T00:00:00Z')
+        self.assertEqual(r.md_at, '2000-01-02T00:00:00Z')
+        self.assertEqual(r.md_completed, '2000-01-03T00:00:00Z')
+        self.assertEqual(r.md_from, '2000-01-04T00:00:00Z')
+        self.assertEqual(r.md_until, '2000-01-05T00:00:00Z')
+
+    def test13_mime_type(self):
+        r = Resource(uri='tv1', mime_type='text/plain')
+        self.assertEqual(r.mime_type, 'text/plain')
+        r.mime_type = None
+        self.assertEqual(r.mime_type, None)
+        r = Resource(uri='tv2')
+        self.assertEqual(r.mime_type, None)
+
+    def test14_datetime(self):
+        # v1.1 adds a datetime attribute
+        r = Resource(uri='dt1')
+        self.assertEqual(r.datetime, None)
+        r = Resource(uri='dt2', datetime='2000-01-04T00:00:00Z')
+        self.assertEqual(r.datetime, '2000-01-04T00:00:00Z')
+
+    def test15_link(self):
+        """Test link link_href and link_set methods."""
+        r = Resource(uri='ln1')
+        self.assertEqual(r.link('up'), None)
+        self.assertEqual(r.link_href('up'), None)
+        r.link_set('up', 'uri:up')
+        self.assertEqual(r.link('up'), {'rel': 'up', 'href': 'uri:up'})
+        self.assertEqual(r.link_href('up'), 'uri:up')
+        r.link_set('down', 'uri:down')
+        self.assertEqual(r.link('down'), {'rel': 'down', 'href': 'uri:down'})
+        self.assertEqual(r.link_href('down'), 'uri:down')
+        r.link_set('up', 'uri:up2')
+        self.assertEqual(r.link('up'), {'rel': 'up', 'href': 'uri:up2'})
+        r.link_add('up', 'uri:up3')
+        self.assertEqual(r.link('up'), {'rel': 'up', 'href': 'uri:up2'})  # still get first
+        self.assertEqual(r.ln, [{'rel': 'up', 'href': 'uri:up2'},
+                                {'href': 'uri:down', 'rel': 'down'},
+                                {'rel': 'up', 'href': 'uri:up3'}])
+
+    def test16_specific_links(self):
+        """Test setters/getters for specific link types."""
+        r = Resource(uri='laughing')
+        r.describedby = 'uri:db'
+        self.assertEqual(r.describedby, 'uri:db')
+        r.up = 'uri:up'
+        self.assertEqual(r.up, 'uri:up')
+        r.index = 'uri:index'
+        self.assertEqual(r.index, 'uri:index')
+        r.contents = 'uri:ct'
+        self.assertEqual(r.contents, 'uri:ct')
+
+    def test17_basename(self):
+        """Test basename property derived from uri."""
+        r = Resource(uri='http://example.org/any/complex/path/file')
+        self.assertEqual(r.basename, 'file')
+        r.uri = 'http://example.org/any/complex/path/'
+        self.assertEqual(r.basename, '')
+        r.uri = 'http://example.org'
+        self.assertEqual(r.basename, '')
+
+    def test18_str(self):
+        """Test str method."""
+        self.assertEqual(str(Resource('uri:a')),
+                         '[ uri:a | None | None | None ]')
+        self.assertEqual(str(Resource('uri:a', timestamp=0, length=999)),
+                         '[ uri:a | 1970-01-01T00:00:00Z | 999 | None ]')
+        self.assertEqual(str(Resource('uri:a', timestamp=0, length=999, sha256='abcdef123')),
+                         '[ uri:a | 1970-01-01T00:00:00Z | 999 | abcdef123 ]')
+        self.assertEqual(str(Resource('uri:a', change='updated', ts_datetime=3661)),
+                         '[ uri:a | None | None | None | updated @ 1970-01-01T01:01:01Z ]')
+        self.assertEqual(str(Resource('uri:a', path='/a/b/c')),
+                         '[ uri:a | None | None | None | /a/b/c ]')
+
+    def test19_change_type_error(self):
+        """Test error from bad change type."""
+        cte = ChangeTypeError('unk')
+        self.assertIn('ChangeTypeError: got unk, expected one of ', str(cte))

tests/test_resource_container.py

@@ -0,0 +1,134 @@
+"""Tests for resync.resource_container."""
+
+import unittest
+from resync.resource import Resource
+from resync.resource_container import ResourceContainer, _str_datetime_now
+
+
+class TestResourceContainer(unittest.TestCase):
+
+    def test01_create_and_add(self):
+        rc = ResourceContainer(resources=[])
+        self.assertEqual(len(rc.resources), 0, "empty")
+        rc.add(Resource('a', timestamp=1))
+        rc.add(Resource('b', timestamp=2))
+        self.assertEqual(len(rc.resources), 2, "two resources")
+        # Add two more
+        rc.add([Resource('c'), Resource('d')])
+        self.assertEqual(len(rc.resources), 4, "four resources")
+
+    def test02_iter(self):
+        rc = ResourceContainer(resources=[])
+        rc.resources.append(Resource('a', timestamp=1))
+        rc.resources.append(Resource('b', timestamp=2))
+        rc.resources.append(Resource('c', timestamp=3))
+        rc.resources.append(Resource('d', timestamp=4))
+        resources = []
+        for r in rc:
+            resources.append(r)
+        self.assertEqual(len(resources), 4)
+        self.assertEqual(resources[0].uri, 'a')
+        self.assertEqual(resources[3].uri, 'd')
+
+    def test04_prune_dupes(self):
+        rc = ResourceContainer()
+        rc.resources.append(Resource('a', timestamp=1, change='created'))
+        rc.resources.append(Resource('b', timestamp=2, change='created'))
+        rc.resources.append(Resource('c', timestamp=3, change='updated'))
+        rc.resources.append(Resource('d', timestamp=4, change='deleted'))
+        rc.resources.append(Resource('a', timestamp=4, change='updated'))
+        rc.resources.append(Resource('b', timestamp=4, change='deleted'))
+        rc.resources.append(Resource('b', timestamp=5, change='created'))
+        rc.resources.append(Resource('b', timestamp=6, change='deleted'))
+        rc.resources.append(Resource('c', timestamp=7, change='deleted'))
+        rc.resources.append(Resource('d', timestamp=8, change='created'))
+        rc.prune_dupes()
+        self.assertEqual(len(rc.resources), 3)
+        i = iter(rc)
+        i1 = next(i)
+        self.assertEqual(i1.uri, 'a')
+        self.assertEqual(i1.change, 'updated')
+        i2 = next(i)
+        self.assertEqual(i2.uri, 'c')
+        self.assertEqual(i2.change, 'deleted')
+        i3 = next(i)
+        self.assertEqual(i3.uri, 'd')
+        self.assertEqual(i3.change, 'created')
+
+    def test05_from_until(self):
+        rc = ResourceContainer()
+        # via convenience methods
+        self.assertEqual(rc.md_from, None)
+        self.assertEqual(rc.md_until, None)
+        rc.md_from = "ftime"
+        rc.md_until = "utime"
+        self.assertEqual(rc.md_from, "ftime")
+        self.assertEqual(rc.md_until, "utime")
+        # via underlying dict
+        rc.md['md_from'] = "ftime2"
+        rc.md['md_until'] = "utime2too"
+        self.assertEqual(rc.md_from, "ftime2")
+        self.assertEqual(rc.md_until, "utime2too")
+
+    def test06_source_description(self):
+        # up links used to point to the source description
+        rc = ResourceContainer()
+        # via convenience methods
+        self.assertEqual(rc.up, None)
+        rc.up = "well-known"
+        self.assertEqual(rc.up, "well-known")
+        rc.up = "well-known2"
+        self.assertEqual(rc.up, "well-known2")
+        # via link
+        link = rc.link("up")
+        self.assertEqual(link['href'], "well-known2")
+        link['href'] = "wk3"
+        self.assertEqual(rc.up, "wk3")
+
+    def test07_describedby(self):
+        rc = ResourceContainer()
+        # via convenience methods
+        self.assertEqual(rc.describedby, None)
+        rc.describedby = "db_uri"
+        self.assertEqual(rc.describedby, "db_uri")
+
+    def test08_up(self):
+        rc = ResourceContainer()
+        # via convenience methods
+        self.assertEqual(rc.up, None)
+        rc.up = "up_uri"
+        self.assertEqual(rc.up, "up_uri")
+
+    def test09_capability(self):
+        """Test capability property."""
+        rc = ResourceContainer()
+        self.assertEqual(rc.capability, None)
+        rc.capability = "fixit"
+        self.assertEqual(rc.capability, 'fixit')
+
+    def test10_link_set(self):
+        """Test link_set method."""
+        rc = ResourceContainer()
+        self.assertEqual(rc.link('alink'), None)  # non-spec rel should be supported
+        rc.link_set('alink', 'uri:l', extra1='one', extra2='two')
+        ln = rc.link('alink')
+        self.assertEqual(rc.link('alink'), {'rel': 'alink', 'href': 'uri:l',
+                                            'extra1': 'one', 'extra2': 'two'})
+        # Can update existing link
+        rc.link_set('alink', 'uri:new', extra1='1')
+        self.assertEqual(rc.link('alink'), {'rel': 'alink', 'href': 'uri:new',
+                                            'extra1': '1', 'extra2': 'two'})
+
+    def test11_index(self):
+        """Test index property."""
+        rc = ResourceContainer()
+        self.assertEqual(rc.index, None)
+        rc.index = 'uri:i'
+        self.assertEqual(rc.index, 'uri:i')
+
+    def test12_str_datetime_now(self):
+        """Test _str_datetime_now function."""
+        self.assertTrue(isinstance(_str_datetime_now(), str))
+        self.assertTrue(isinstance(_str_datetime_now('now'), str))
+        self.assertEqual(_str_datetime_now(0), '1970-01-01T00:00:00Z')
+        self.assertEqual(_str_datetime_now('2020-12-24T16:01:00Z'), '2020-12-24T16:01:00Z')

tests/test_resource_dump.py

@@ -0,0 +1,84 @@
+"""Tests for resync.resource_dump."""
+from .testlib import TestCase
+
+import os.path
+try:  # python2
+    # Must try this first as io also exists in python2
+    # but in the wrong one!
+    import StringIO as io
+except ImportError:  # python3
+    import io
+import re
+
+from resync.resource import Resource
+from resync.resource_dump import ResourceDump
+from resync.sitemap import SitemapParseError
+
+
+class TestResourceDump(TestCase):
+
+    def test01_as_xml(self):
+        rd = ResourceDump()
+        rd.add(Resource('a.zip', timestamp=1))
+        rd.add(Resource('b.zip', timestamp=2))
+        xml = rd.as_xml()
+        self.assertTrue(
+            re.search(
+                r'<rs:md .*capability="resourcedump"',
+                xml),
+            'XML has capability')
+        self.assertTrue(
+            re.search(
+                r'<url><loc>a.zip</loc><lastmod>1970-01-01T00:00:01Z</lastmod></url>',
+                xml),
+            'XML has resource a')
+
+    def test02_parse(self):
+        xml = '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
+<rs:md at="2013-01-01" capability="resourcedump"/>\
+<url><loc>http://example.com/a.zip</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="12345" /></url>\
+<url><loc>http://example.com/b.zip</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="56789" /></url>\
+</urlset>'
+        rd = ResourceDump()
+        rd.parse(fh=io.StringIO(xml))
+        self.assertEqual(len(rd.resources), 2, 'got 2 resource dumps')
+        self.assertEqual(rd.capability, 'resourcedump', 'capability set')
+        self.assertEqual(rd.md_at, '2013-01-01')
+        self.assertTrue('http://example.com/a.zip' in rd.resources)
+        self.assertTrue(rd.resources['http://example.com/a.zip'].length, 12345)
+        self.assertTrue('http://example.com/b.zip' in rd.resources)
+        self.assertTrue(rd.resources['http://example.com/b.zip'].length, 56789)
+
+    def test03_parse_no_capability(self):
+        # For a resource dump this should be an error
+        xml = '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
+<rs:md from="2013-01-01"/>\
+<url><loc>http://example.com/a.zip</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="12" /></url>\
+</urlset>'
+        rd = ResourceDump()
+        self.assertRaises(SitemapParseError, rd.parse, fh=io.StringIO(xml))
+
+    def test04_parse_bad_capability(self):
+        # the <rs:md capability="bad_capability".. should give error
+        xml = '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
+<rs:md capability="bad_capability" from="2013-01-01"/>\
+<url><loc>http://example.com/bad_res_1</loc><lastmod>2012-03-14T18:37:36Z</lastmod></url>\
+</urlset>'
+        rd = ResourceDump()
+        self.assertRaises(SitemapParseError, rd.parse, fh=io.StringIO(xml))
+
+    def test05_write(self):
+        rd = ResourceDump()
+        rd.add(Resource('aa.zip', timestamp=1))
+        rd.add(Resource('bb.zip', timestamp=2))
+        dumpf = os.path.join(self.tmpdir, "test05_dump.xml")
+        rd.write(basename=dumpf)
+        self.assertTrue(os.path.exists(dumpf))
+        # Now read that back
+        rd2 = ResourceDump()
+        rd2.parse(dumpf)
+        self.assertEqual(len(rd2), 2)
+        self.assertEqual(rd2.uris(), ['aa.zip', 'bb.zip'])

tests/test_resource_dump_manifest.py

@@ -1,60 +1,65 @@
 import unittest
-import StringIO
+try:  # python2
+    # Must try this first as io also exists in python2
+    # but in the wrong one!
+    import StringIO as io
+except ImportError:  # python3
+    import io
 import re
+
 from resync.resource import Resource
 from resync.resource_dump_manifest import ResourceDumpManifest
 from resync.sitemap import SitemapParseError
 
+
 class TestResourceDumpManifest(unittest.TestCase):
 
     def test01_as_xml(self):
         rdm = ResourceDumpManifest()
-        rdm.add( Resource('a.zip',timestamp=1) )
-        rdm.add( Resource('b.zip',timestamp=2) )
+        rdm.add(Resource('a.zip', timestamp=1))
+        rdm.add(Resource('b.zip', timestamp=2))
         xml = rdm.as_xml()
-        print xml
-        self.assertTrue( re.search(r'<rs:md .*capability="resourcedump-manifest"', xml), 'XML has capability' )
-        self.assertTrue( re.search(r'<url><loc>a.zip</loc><lastmod>1970-01-01T00:00:01Z</lastmod></url>', xml), 'XML has resource a' ) 
+        self.assertTrue(re.search(
+            r'<rs:md .*capability="resourcedump-manifest"', xml), 'XML has capability')
+        self.assertTrue(re.search(
+            r'<url><loc>a.zip</loc><lastmod>1970-01-01T00:00:01Z</lastmod></url>', xml), 'XML has resource a')
 
     def test10_parse(self):
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
+        xml = '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
 <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
 <rs:md at="2013-08-08" capability="resourcedump-manifest"/>\
 <url><loc>http://example.com/res1</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="12" path="/res1" /></url>\
 <url><loc>http://example.com/res2</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="32" path="/res2"/></url>\
 </urlset>'
-        rdm=ResourceDumpManifest()
-        rdm.parse(fh=StringIO.StringIO(xml))
-        self.assertEqual( len(rdm.resources), 2, 'got 2 resource dumps')
-        self.assertEqual( rdm.capability, 'resourcedump-manifest', 'capability set' )
-        self.assertEqual( rdm.md_at, '2013-08-08' )
-        self.assertTrue( 'http://example.com/res1' in rdm.resources )
-        self.assertTrue( rdm.resources['http://example.com/res1'].length, 12 )
-        self.assertTrue( rdm.resources['http://example.com/res1'].path, '/res1' )
-        self.assertTrue( 'http://example.com/res2' in rdm.resources )
-        self.assertTrue( rdm.resources['http://example.com/res2'].path, '/res2' )
+        rdm = ResourceDumpManifest()
+        rdm.parse(fh=io.StringIO(xml))
+        self.assertEqual(len(rdm.resources), 2, 'got 2 resource dumps')
+        self.assertEqual(
+            rdm.capability, 'resourcedump-manifest', 'capability set')
+        self.assertEqual(rdm.md_at, '2013-08-08')
+        self.assertTrue('http://example.com/res1' in rdm.resources)
+        self.assertTrue(rdm.resources['http://example.com/res1'].length, 12)
+        self.assertTrue(rdm.resources['http://example.com/res1'].path, '/res1')
+        self.assertTrue('http://example.com/res2' in rdm.resources)
+        self.assertTrue(rdm.resources['http://example.com/res2'].path, '/res2')
 
     def test11_parse_no_capability(self):
         # For a resource dump this should be an error
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
+        xml = '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
 <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
 <rs:md from="2013-01-01"/>\
 <url><loc>http://example.com/res1</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="12" path="/res1" /></url>\
 <url><loc>http://example.com/res2</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="32" path="/res2"/></url>\
 </urlset>'
-        rdm=ResourceDumpManifest()
-        self.assertRaises( SitemapParseError, rdm.parse, fh=StringIO.StringIO(xml) )
+        rdm = ResourceDumpManifest()
+        self.assertRaises(SitemapParseError, rdm.parse, fh=io.StringIO(xml))
 
     def test12_parse_bad_capability(self):
         # the <rs:md capability="bad_capability".. should give error
-        xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
+        xml = '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
 <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
 <rs:md capability="bad_capability" from="2013-01-01"/>\
 <url><loc>http://example.com/a.zip</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="12" /></url>\
 </urlset>'
-        rdm=ResourceDumpManifest()
-        self.assertRaises( SitemapParseError, rdm.parse, fh=StringIO.StringIO(xml) )
-
-if __name__ == '__main__':
-    suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestResourceDumpManifest)
-    unittest.TextTestRunner().run(suite)
+        rdm = ResourceDumpManifest()
+        self.assertRaises(SitemapParseError, rdm.parse, fh=io.StringIO(xml))

tests/test_resource_list.py

@@ -0,0 +1,227 @@
+"""Tests for resync.resource_list."""
+
+from .testlib import TestCase
+
+import os.path
+import io
+import re
+
+from resync.resource import Resource
+from resync.resource_list import ResourceList, ResourceListOrdered, ResourceListDupeError
+from resync.sitemap import Sitemap, SitemapParseError
+
+
+class TestResourceList(TestCase):
+
+    def test01_same(self):
+        src = ResourceList()
+        src.add(Resource('a', timestamp=1))
+        src.add(Resource('b', timestamp=2))
+        dst = ResourceList()
+        dst.add(Resource('a', timestamp=1))
+        dst.add(Resource('b', timestamp=2))
+        (same, changed, deleted, added) = dst.compare(src)
+        self.assertEqual(len(same), 2, "2 things unchanged")
+        i = iter(same)
+        self.assertEqual(next(i).uri, 'a', "first was a")
+        self.assertEqual(next(i).uri, 'b', "second was b")
+        self.assertEqual(len(changed), 0, "nothing changed")
+        self.assertEqual(len(deleted), 0, "nothing deleted")
+        self.assertEqual(len(added), 0, "nothing added")
+
+    def test02_changed(self):
+        src = ResourceList()
+        src.add(Resource('a', timestamp=1))
+        src.add(Resource('b', timestamp=2))
+        dst = ResourceList()
+        dst.add(Resource('a', timestamp=3))
+        dst.add(Resource('b', timestamp=4))
+        (same, changed, deleted, added) = dst.compare(src)
+        self.assertEqual(len(same), 0, "0 things unchanged")
+        self.assertEqual(len(changed), 2, "2 things changed")
+        i = iter(changed)
+        self.assertEqual(next(i).uri, 'a', "first was a")
+        self.assertEqual(next(i).uri, 'b', "second was b")
+        self.assertEqual(len(deleted), 0, "nothing deleted")
+        self.assertEqual(len(added), 0, "nothing added")
+
+    def test03_deleted(self):
+        src = ResourceList()
+        src.add(Resource('a', timestamp=1))
+        src.add(Resource('b', timestamp=2))
+        dst = ResourceList()
+        dst.add(Resource('a', timestamp=1))
+        dst.add(Resource('b', timestamp=2))
+        dst.add(Resource('c', timestamp=3))
+        dst.add(Resource('d', timestamp=4))
+        (same, changed, deleted, added) = dst.compare(src)
+        self.assertEqual(len(same), 2, "2 things unchanged")
+        self.assertEqual(len(changed), 0, "nothing changed")
+        self.assertEqual(len(deleted), 2, "c and d deleted")
+        i = iter(deleted)
+        self.assertEqual(next(i).uri, 'c', "first was c")
+        self.assertEqual(next(i).uri, 'd', "second was d")
+        self.assertEqual(len(added), 0, "nothing added")
+
+    def test04_added(self):
+        src = ResourceList()
+        src.add(Resource('a', timestamp=1))
+        src.add(Resource('b', timestamp=2))
+        src.add(Resource('c', timestamp=3))
+        src.add(Resource('d', timestamp=4))
+        dst = ResourceList()
+        dst.add(Resource('a', timestamp=1))
+        dst.add(Resource('c', timestamp=3))
+        (same, changed, deleted, added) = dst.compare(src)
+        self.assertEqual(len(same), 2, "2 things unchanged")
+        self.assertEqual(len(changed), 0, "nothing changed")
+        self.assertEqual(len(deleted), 0, "nothing deleted")
+        self.assertEqual(len(added), 2, "b and d added")
+        i = iter(added)
+        self.assertEqual(next(i).uri, 'b', "first was b")
+        self.assertEqual(next(i).uri, 'd', "second was d")
+
+    def test05_add(self):
+        r1 = Resource(uri='a', length=1)
+        r2 = Resource(uri='b', length=2)
+        i = ResourceList()
+        i.add(r1)
+        self.assertRaises(ResourceListDupeError, i.add, r1)
+        i.add(r2)
+        self.assertRaises(ResourceListDupeError, i.add, r2)
+        # allow dupes
+        r1d = Resource(uri='a', length=10)
+        i.add(r1d, replace=True)
+        self.assertEqual(len(i), 2)
+        self.assertEqual(i.resources['a'].length, 10)
+
+    def test06_add_iterable(self):
+        r1 = Resource(uri='a', length=1)
+        r2 = Resource(uri='b', length=2)
+        i = ResourceList()
+        i.add([r1, r2])
+        self.assertRaises(ResourceListDupeError, i.add, r1)
+        self.assertRaises(ResourceListDupeError, i.add, r2)
+        # allow dupes
+        r1d = Resource(uri='a', length=10)
+        i.add([r1d], replace=True)
+        self.assertEqual(len(i), 2)
+        self.assertEqual(i.resources['a'].length, 10)
+
+    def test07_hashes(self):
+        r1 = Resource(uri='a')
+        r2 = Resource(uri='b')
+        i = ResourceList()
+        self.assertEqual(i.hashes(), set())
+        i.add(r1)
+        i.add(r2)
+        self.assertEqual(i.hashes(), set())
+        r1.md5 = "aabbcc"
+        self.assertEqual(i.hashes(), set(['md5']))
+        r2.sha1 = "ddeeff"
+        r2.sha256 = "hhiijj"
+        self.assertEqual(i.hashes(), set(['md5', 'sha-1', 'sha-256']))
+
+    def test08_iter(self):
+        i = ResourceList()
+        i.add(Resource('a', timestamp=1))
+        i.add(Resource('b', timestamp=2))
+        i.add(Resource('c', timestamp=3))
+        i.add(Resource('d', timestamp=4))
+        resources = list(i.resources)
+        self.assertEqual(len(resources), 4)
+        self.assertEqual(resources[0].uri, 'a')
+        self.assertEqual(resources[3].uri, 'd')
+
+    def test09_resource_list_ordered(self):
+        """Tests for ResourceList with ResourceListOrdered."""
+        i = ResourceList(resources_class=ResourceListOrdered)
+        i.add(Resource('a', timestamp=1))
+        i.add(Resource('d', timestamp=4))
+        i.add(Resource('c', timestamp=3))
+        self.assertEqual(list(i.resources.uris()), ['a', 'd', 'c'])
+        self.assertRaises(ResourceListDupeError, i.add, Resource('a', timestamp=11))
+        self.assertEqual(i.resources['a'].uri, 'a')
+        self.assertEqual(i.resources['a'].timestamp, 1)
+        # With replacement
+        i.add(Resource('a', timestamp=11), replace=True)
+        self.assertEqual(i.resources['a'].uri, 'a')
+        self.assertEqual(i.resources['a'].timestamp, 11)
+
+    def test20_as_xml(self):
+        rl = ResourceList()
+        rl.add(Resource('a', timestamp=1))
+        rl.add(Resource('b', timestamp=2))
+        xml = rl.as_xml()
+        self.assertTrue(
+            re.search(r'<rs:md .*capability="resourcelist"', xml), 'XML has capability')
+        self.assertTrue(re.search(
+            r'<url><loc>a</loc><lastmod>1970-01-01T00:00:01Z</lastmod></url>', xml), 'XML has resource a')
+
+    def test30_parse(self):
+        xml = '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
+<rs:md at="2013-08-07" capability="resourcelist" completed="2013-08-08" />\
+<url><loc>/tmp/rs_test/src/file_a</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md change="updated" length="12" /></url>\
+<url><loc>/tmp/rs_test/src/file_b</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="32" /></url>\
+</urlset>'
+        rl = ResourceList()
+        rl.parse(fh=io.StringIO(xml))
+        self.assertEqual(len(rl.resources), 2, 'got 2 resources')
+        self.assertEqual(rl.md['capability'], 'resourcelist', 'capability set')
+        self.assertEqual(rl.md_at, '2013-08-07')
+        self.assertEqual(rl.md_completed, '2013-08-08')
+
+    def test31_parse_no_capability(self):
+        xml = '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">\
+<url><loc>http://example.com/res1</loc><lastmod>2012-03-14T18:37:36Z</lastmod></url>\
+</urlset>'
+        rl = ResourceList()
+        rl.parse(fh=io.StringIO(xml))
+        self.assertEqual(len(rl.resources), 1, 'got 1 resource')
+        self.assertEqual(rl.md['capability'], 'resourcelist',
+                         'capability set by reading routine')
+        self.assertFalse('from' in rl.md)
+
+    def test32_parse_bad_capability(self):
+        # the <rs:md capability="bad_capability".. should give error
+        xml = '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
+<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
+<rs:md capability="bad_capability" from="2013-01-01"/>\
+<url><loc>http://example.com/bad_res_1</loc><lastmod>2012-03-14T18:37:36Z</lastmod></url>\
+</urlset>'
+        rl = ResourceList()
+        self.assertRaises(SitemapParseError, rl.parse, fh=io.StringIO(xml))
+
+    def test33_write(self):
+        # ResourceList
+        rl = ResourceList()
+
+        rl.add(Resource(uri='http://example.com/test/a', timestamp=1))
+        rl.add(Resource(uri='http://example.com/test/b', timestamp=1))
+        rl.add(Resource(uri='http://example.com/test/c', timestamp=1))
+
+        rl_filename = os.path.join(self.tmpdir, 'test33_write_resourcelist.xml')
+        rl.write(basename=rl_filename)
+
+        with open(rl_filename, 'r') as f:
+            s = Sitemap()
+            s.parse_xml(fh=f)
+            self.assertFalse(s.parsed_index)
+
+        # ResourceListIndex
+        rli = ResourceList()
+
+        rli.add(Resource(uri='http://example.com/test/resourcelist00000.xml', timestamp=1))
+        rli.add(Resource(uri='http://example.com/test/resourcelist00001.xml', timestamp=1))
+        rli.add(Resource(uri='http://example.com/test/resourcelist00002.xml', timestamp=1))
+        rli.sitemapindex = True
+
+        rli_filename = os.path.join(self.tmpdir, 'test33_write_resourcelist-index.xml')
+        rli.write(basename=rli_filename)
+
+        with open(rli_filename, 'r') as f:
+            s = Sitemap()
+            s.parse_xml(fh=f)
+            self.assertTrue(s.parsed_index)

tests/test_resource_list_builder.py

@@ -0,0 +1,223 @@
+import unittest
+import re
+import os
+from testfixtures import LogCapture
+import time
+
+from resync.resource_list_builder import ResourceListBuilder
+from resync.resource_list import ResourceList
+from resync.resource import Resource
+from resync.mapper import Mapper, MapperError
+
+
+class TestResourceListBuilder(unittest.TestCase):
+
+    def setUp(self):
+        # Set timestamps (mtime) for test data. Timestamps on disk are
+        # in UTC so no conversion issues.
+        # Test case file_a: 1343236426 = 2012-07-25T17:13:46Z
+        # Test case file_b: 1000000000 = 2001-09-09T01:46:40Z
+        os.utime("tests/testdata/dir1/file_a", (0, 1343236426))
+        os.utime("tests/testdata/dir1/file_b", (0, 1000000000))
+
+    def test01_simple_scan(self):
+        rlb = ResourceListBuilder()
+        rlb.mapper = Mapper(['http://example.org/t', 'tests/testdata/dir1'])
+        rl = rlb.from_disk()
+        self.assertEqual(len(rl), 2)
+        rli = iter(rl)
+        r = next(rli)
+        self.assertEqual(r.uri, 'http://example.org/t/file_a')
+        self.assertEqual(r.lastmod, '2012-07-25T17:13:46Z')
+        self.assertEqual(r.md5, None)
+        self.assertEqual(r.length, 20)
+        self.assertEqual(r.path, None)
+        r = next(rli)
+        self.assertEqual(r.uri, 'http://example.org/t/file_b')
+        self.assertEqual(r.lastmod, '2001-09-09T01:46:40Z')
+        self.assertEqual(r.md5, None)
+        self.assertEqual(r.length, 45)
+        self.assertEqual(r.path, None)
+        # Make sure at and completed were set
+        self.assertTrue(rl.md_at is not None)
+        self.assertTrue(rl.md_completed is not None)
+
+    def test02_no_length(self):
+        rlb = ResourceListBuilder(set_length=False)
+        rlb.mapper = Mapper(['http://example.org/t', 'tests/testdata/dir1'])
+        rl = rlb.from_disk()
+        self.assertEqual(len(rl), 2)
+        rli = iter(rl)
+        r = next(rli)
+        self.assertEqual(r.uri, 'http://example.org/t/file_a')
+        self.assertEqual(r.lastmod, '2012-07-25T17:13:46Z')
+        self.assertEqual(r.md5, None)
+        self.assertEqual(r.length, None)
+        self.assertEqual(r.path, None)
+        r = next(rli)
+        self.assertEqual(r.uri, 'http://example.org/t/file_b')
+        self.assertEqual(r.lastmod, '2001-09-09T01:46:40Z')
+        self.assertEqual(r.md5, None)
+        self.assertEqual(r.length, None)
+        self.assertEqual(r.path, None)
+
+    def test03_set_hashes(self):
+        rlb = ResourceListBuilder(set_hashes=['md5'])
+        rlb.mapper = Mapper(['http://example.org/t', 'tests/testdata/dir1'])
+        rl = rlb.from_disk()
+        self.assertEqual(len(rl), 2)
+        rli = iter(rl)
+        r = next(rli)
+        self.assertEqual(r.uri, 'http://example.org/t/file_a')
+        self.assertEqual(r.lastmod, '2012-07-25T17:13:46Z')
+        self.assertEqual(r.md5, '6bf26fd66601b528d2e0b47eaa87edfd')
+        self.assertEqual(r.length, 20)
+        self.assertEqual(r.path, None)
+        r = next(rli)
+        self.assertEqual(r.uri, 'http://example.org/t/file_b')
+        self.assertEqual(r.lastmod, '2001-09-09T01:46:40Z')
+        self.assertEqual(r.md5, '452e54bdae1626ac5d6e7be81b39de21')
+        self.assertEqual(r.length, 45)
+        self.assertEqual(r.path, None)
+
+    def test04_data(self):
+        rlb = ResourceListBuilder(set_path=True, set_hashes=['md5'])
+        rlb.mapper = Mapper(['http://example.org/t', 'tests/testdata/dir1'])
+        rl = rlb.from_disk()
+        self.assertEqual(len(rl), 2)
+        r = rl.resources.get('http://example.org/t/file_a')
+        self.assertTrue(r is not None)
+        self.assertEqual(r.uri, 'http://example.org/t/file_a')
+        self.assertEqual(r.lastmod, '2012-07-25T17:13:46Z')
+        self.assertEqual(r.md5, '6bf26fd66601b528d2e0b47eaa87edfd')
+        self.assertEqual(r.path, 'tests/testdata/dir1/file_a')
+
+    def test05_from_disk_paths(self):
+        rlb = ResourceListBuilder()
+        rlb.mapper = Mapper(['http://example.org/t', 'tests/testdata/dir1'])
+        # no path, should get no resources
+        rl = rlb.from_disk(paths=[])
+        self.assertEqual(len(rl), 0)
+        # full path, 2 resources
+        rl = rlb.from_disk(paths=['tests/testdata/dir1'])
+        self.assertEqual(len(rl), 2)
+        # new object with mapper covering larger space of disk
+        rlb = ResourceListBuilder(set_path=True)
+        rlb.mapper = Mapper(['http://example.org/t', 'tests/testdata'])
+        # same path with 2 resources
+        rl = rlb.from_disk(paths=['tests/testdata/dir1'])
+        self.assertEqual(len(rl), 2)
+        # same path with 2 resources
+        rl = rlb.from_disk(
+            paths=['tests/testdata/dir1', 'tests/testdata/dir2'])
+        self.assertEqual(len(rl), 3)
+        # path that is just a single file
+        rl = rlb.from_disk(paths=['tests/testdata/dir1/file_a'])
+        self.assertEqual(len(rl), 1)
+        rli = iter(rl)
+        r = next(rli)
+        self.assertTrue(r is not None)
+        self.assertEqual(r.uri, 'http://example.org/t/dir1/file_a')
+        self.assertEqual(r.lastmod, '2012-07-25T17:13:46Z')
+        self.assertEqual(r.md5, None)
+        self.assertEqual(r.length, 20)
+        self.assertEqual(r.path, 'tests/testdata/dir1/file_a')
+
+    def test06_odd_file_names(self):
+        """Verfify we can read unicode file names properly."""
+        rlb = ResourceListBuilder()
+        rlb.mapper = Mapper(['x:', 'tests/testdata/odd_file_names'])
+        rl = rlb.from_disk(paths=['tests/testdata/odd_file_names'])
+        # Get list of URIs to test
+        uris = [x.uri for x in rl]
+        self.assertTrue('x:/not_odd.txt' in uris)
+        self.assertTrue('x:/with&ampersand.txt' in uris)
+        self.assertTrue('x:/with spaces.txt' in uris)
+        # File names for accented chars represented with combining chars
+        self.assertTrue(u'x:/Pi\u006e\u0303a_Colada.txt' in uris)
+        self.assertFalse(u'x:/Pi\u00f1a_Colada.txt' in uris)
+        self.assertTrue(u'x:/A_\u0041\u0303_tilde.txt' in uris)
+        self.assertFalse(u'x:/A_\u00c3_tilde.txt' in uris)
+        # Snowman is single char
+        self.assertFalse(u'x:snowman_\u2603.txt' in uris)
+
+    def test10_add_exclude_patterns(self):
+        """Test add_exclude_patterns method."""
+        rlb = ResourceListBuilder()
+        self.assertEqual(len(rlb.exclude_patterns), 0)
+        rlb.add_exclude_patterns(['aaa', 'bbb'])
+        self.assertIn('aaa', rlb.exclude_patterns)
+        self.assertIn('bbb', rlb.exclude_patterns)
+
+    def test11_compile_excludes(self):
+        """Test _compile_excludes method."""
+        rlb = ResourceListBuilder()
+        self.assertEqual(len(rlb.compiled_exclude_patterns), 0)
+        rlb.exclude_patterns = [r'aaa\d+', r'bbb']
+        rlb._compile_excludes()
+        self.assertEqual(len(rlb.compiled_exclude_patterns), 2)
+        # Error case
+        rlb.exclude_patterns.append('bad regex \\')
+        self.assertRaises(ValueError, rlb._compile_excludes)
+
+    def test12_exclude(self):
+        """Test _exclude method."""
+        rlb = ResourceListBuilder()
+        rlb.add_exclude_patterns(['.*frog.*'])
+        rlb._compile_excludes()
+        self.assertTrue(rlb._exclude('a frog'))
+        self.assertFalse(rlb._exclude('toad'))
+
+    def test13_from_disk_add_path(self):
+        """Test from_disk_add_path method."""
+        # Check sanity check - must have path, resource_list and mapper
+        rlb = ResourceListBuilder(mapper=Mapper())
+        self.assertRaises(ValueError, rlb.from_disk_add_path, path='aaa')
+        self.assertRaises(ValueError, rlb.from_disk_add_path, resource_list=ResourceList())
+        rlb = ResourceListBuilder()
+        self.assertRaises(ValueError, rlb.from_disk_add_path, path='aaa', resource_list=ResourceList())
+        # Check log message
+        rlb = ResourceListBuilder(mapper=Mapper(['http://example.org/', 'tests']))
+        rlb.log_count_increment = 2
+        rl = ResourceList()
+        with LogCapture() as lc:
+            rlb.from_disk_add_path(path='tests/testdata/dir1', resource_list=rl)
+            self.assertIn('from_disk_add_path: 2 files...', lc.records[-1].msg)
+        # text excluding dirs -- just one file under find2 not excluced
+        rlb = ResourceListBuilder(mapper=Mapper(['http://example.org/', 'tests']))
+        rl = ResourceList()
+        rlb.add_exclude_patterns(['find1', 'find3'])
+        rlb._compile_excludes()
+        rlb.from_disk_add_path(path='tests/testdata/find', resource_list=rl)
+        self.assertEqual(len(rl), 1)
+
+    def test14_add_file(self):
+        """Test add_file method."""
+        rlb = ResourceListBuilder(mapper=Mapper(['http://example.org/', 'tests']))
+        rl = ResourceList()
+        rlb.add_exclude_patterns(['.*ro'])
+        with LogCapture() as lc:
+            # escluded
+            rlb.add_file(resource_list=rl, file='frog')
+            self.assertIn("Excluding file 'frog'", lc.records[-1].msg)
+            # mapper error
+            self.assertRaises(MapperError, rlb.add_file, resource_list=rl, file='i-dont-exist')
+            # map OK but doesn't exist
+            rlb.add_file(resource_list=rl, file='tests/i-dont-exist')
+            self.assertIn("Ignoring file 'tests/i-dont-exist'", lc.records[-1].msg)
+            # ignore symlink by default
+            rlb.add_file(resource_list=rl, file='tests/testdata/symlink/dir2/a_file.txt')
+            self.assertIn("Ignoring symlink 'tests/testdata/symlink/dir2/a_file.txt'", lc.records[-1].msg)
+            # ...or not
+            rl = ResourceList()
+            rlb.include_symlinks = True
+            rlb.add_file(resource_list=rl, file='tests/testdata/symlink/dir2/a_file.txt')
+            self.assertEqual(len(rl), 1)
+        # Check hashing
+        rlb = ResourceListBuilder(mapper=Mapper(['http://example.org/', 'tests/testdata/dir1/']),
+                                  set_hashes=['md5', 'sha-1', 'sha-256'])
+        rl = ResourceList()
+        rlb.add_file(resource_list=rl, file='tests/testdata/dir1/file_a')
+        self.assertEqual(rl['http://example.org/file_a'].md5, '6bf26fd66601b528d2e0b47eaa87edfd')
+        self.assertEqual(rl['http://example.org/file_a'].sha1, 'c60a598a5d9e489cf50533eeead6d70f15eafcf8')
+        self.assertEqual(rl['http://example.org/file_a'].sha256, '1c6291bfac0322752c4632ebd69bf6d81d53985fbf5ee54de5cc1fefba6566b6')

tests/test_resource_list_multifile.py

@@ -0,0 +1,101 @@
+import sys
+import unittest
+import tempfile
+import os.path
+import shutil
+
+from resync.list_base_with_index import ListBaseIndexError
+from resync.resource import Resource
+from resync.resource_list import ResourceList
+from resync.mapper import Mapper
+
+# etree gives ParseError in 2.7, ExpatError in 2.6
+etree_error_class = None
+if (sys.version_info < (2, 7)):
+    from xml.parsers.expat import ExpatError
+    etree_error_class = ExpatError
+else:
+    from xml.etree.ElementTree import ParseError
+    etree_error_class = ParseError
+
+
+class TestResourceListMultifile(unittest.TestCase):
+
+    def test_01_read_local_filenames(self):
+        rl = ResourceList()
+        rl.read('tests/testdata/sitemapindex2/sitemap.xml')
+        self.assertEqual(len(rl.resources), 17,
+                         '17 resources from 3 sitemaps listed')
+        sr = sorted(rl.uris())
+        self.assertEqual(sr[0], 'http://localhost:8888/resources/1')
+        self.assertEqual(sr[1], 'http://localhost:8888/resources/10')
+        self.assertEqual(sr[2], 'http://localhost:8888/resources/100')
+        self.assertEqual(sr[3], 'http://localhost:8888/resources/1000')
+        self.assertEqual(sr[16], 'http://localhost:8888/resources/826')
+
+    def test_02_read_with_mapper(self):
+        rl = ResourceList()
+        rl.mapper = Mapper(['http://localhost/=tests/testdata/sitemapindex2/'])
+        rl.read('tests/testdata/sitemapindex2/sitemap_mapper.xml')
+        self.assertEqual(len(rl.resources), 17,
+                         '17 resources from 3 sitemaps listed')
+        sr = sorted(rl.uris())
+        self.assertEqual(sr[0], 'http://localhost:8888/resources/1')
+        self.assertEqual(sr[1], 'http://localhost:8888/resources/10')
+        self.assertEqual(sr[2], 'http://localhost:8888/resources/100')
+        self.assertEqual(sr[3], 'http://localhost:8888/resources/1000')
+        self.assertEqual(sr[16], 'http://localhost:8888/resources/826')
+
+    def test_11_write_multifile(self):
+        tempdir = tempfile.mkdtemp(prefix='test_resource_list_multifile_dir')
+        rl = ResourceList()
+        rl.mapper = Mapper(['http://localhost/=%s/' % (tempdir)])
+        rl.add(Resource(uri='http://localhost/a'))
+        rl.add(Resource(uri='http://localhost/b'))
+        rl.add(Resource(uri='http://localhost/c'))
+        rl.add(Resource(uri='http://localhost/d'))
+        rl.max_sitemap_entries = 2
+        # first try writing without mutlifile allowed
+        rl.allow_multifile = False
+        self.assertRaises(ListBaseIndexError, rl.write,
+                          basename=os.path.join(tempdir, 'sitemap.xml'))
+        # second actually do it
+        rl.allow_multifile = True
+        rl.write(basename=os.path.join(tempdir, 'sitemap.xml'))
+        # check the two component sitemaps
+        rl1 = ResourceList()
+        rl1.read(os.path.join(tempdir, 'sitemap00000.xml'))
+        self.assertEqual(len(rl1), 2)
+        self.assertEqual(rl1.capability, 'resourcelist')
+        self.assertFalse(rl1.sitemapindex)
+        i = iter(rl1)
+        self.assertEqual(next(i).uri, 'http://localhost/a')
+        self.assertEqual(next(i).uri, 'http://localhost/b')
+        rl2 = ResourceList()
+        rl2.read(os.path.join(tempdir, 'sitemap00001.xml'))
+        self.assertEqual(len(rl2), 2)
+        i = iter(rl2)
+        self.assertEqual(next(i).uri, 'http://localhost/c')
+        self.assertEqual(next(i).uri, 'http://localhost/d')
+        # check the sitemapindex (read just as index)
+        rli = ResourceList()
+        rli.read(os.path.join(tempdir, 'sitemap.xml'), index_only=True)
+        self.assertEqual(len(rli), 2)
+        i = iter(rli)
+        self.assertEqual(rli.capability, 'resourcelist')
+        self.assertTrue(rli.sitemapindex)
+        self.assertEqual(next(i).uri, 'http://localhost/sitemap00000.xml')
+        self.assertEqual(next(i).uri, 'http://localhost/sitemap00001.xml')
+        # check the sitemapindex and components
+        rli = ResourceList(mapper=rl.mapper)
+        rli.read(os.path.join(tempdir, 'sitemap.xml'))
+        self.assertEqual(len(rli), 4)
+        self.assertEqual(rli.capability, 'resourcelist')
+        self.assertFalse(rli.sitemapindex)
+        i = iter(rli)
+        self.assertEqual(next(i).uri, 'http://localhost/a')
+        self.assertEqual(next(i).uri, 'http://localhost/b')
+        self.assertEqual(next(i).uri, 'http://localhost/c')
+        self.assertEqual(next(i).uri, 'http://localhost/d')
+        # cleanup tempdir
+        shutil.rmtree(tempdir)