gian/resync
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: gian:main
Branches Tags
gian:main
gian:develop
gian:v1_1
gian:find-resource-list
gian:gzip-support
gian:v2.0.1
gian:v2.0.0
gian:v1.0.9
gian:v1.0.8
gian:v1.0.5
gian:v1.0.4
gian:v1.0.3
gian:v1.0.2
gian:v1.0.1
gian:v1.0.0
gian:v0.9.5
gian:v0.9.4
gian:v0.9.3
gian:v0.9.2
gian:v0.9.1
gian:v0.6.2
gian:v0.5.3
gian:v0.6.1
gian:v0.5.2
gian:v0.5.1
..
compare: gian:v0.6.1
Branches Tags
gian:develop
gian:main
gian:v1_1
gian:find-resource-list
gian:gzip-support
gian:v2.0.1
gian:v2.0.0
gian:v1.0.9
gian:v1.0.8
gian:v1.0.5
gian:v1.0.4
gian:v1.0.3
gian:v1.0.2
gian:v1.0.1
gian:v1.0.0
gian:v0.9.5
gian:v0.9.4
gian:v0.9.3
gian:v0.9.2
gian:v0.9.1
gian:v0.6.2
gian:v0.5.3
gian:v0.6.1
gian:v0.5.2
gian:v0.5.1

No commits in common. "main" and "v0.6.1" have entirely different histories.
main ... v0.6.1

240 changed files with 3971 additions and 11629 deletions
Show Stats Expand all files Collapse all files

23
.gitignore vendored
View File

@ -1,23 +1,6 @@
# Files and directories used by setup.py
build
dist
MANIFEST
.tox
.eggs
*.egg
# Testing
.coverage
htmlcov
# Files generated by client / local config
.resync-client-status.cfg
resync-client.log
localhost_8888
tmp
# Related to packaging
resync.egg-info
# Other...
.DS_store
*.pyc
*~
.cache
.resync-client-status.cfg
rc
MANIFEST

16
.travis.yml
View File

@ -1,16 +0,0 @@
language: python
python:
- "3.5"
- "3.6"
- "3.7"
- "3.8"
install:
- pip install coveralls pycodestyle pep257 testfixtures
- python setup.py install
script:
- python setup.py test
- pycodestyle --ignore=E501,W503 resync tests resync-sync resync-build resync-explorer
- pep257 resync bin tests resync-sync resync-build resync-explorer
- coverage run --source=resync setup.py test
after_success:
- coveralls

119
CHANGES.md
View File

@ -1,113 +1,24 @@
# resync change log
=================
resync change log
=================
v2.0.1 2021-03-23
* Route all URI and file requests through `resync/url_or_file_open.py` so that settings such as authentication headers can be consistently applied
* Do not exclude any directories from sync by default, specify with --exclude
* Fix comparison of using possibly multiple checksums
* Improve test coverage
* Remove Python 2 cruft
v2.0.0 2020-12-16
* Supports ResourceSync v1.1 ANSI/NISO Z39.99-2017 <http://www.openarchives.org/rs/1.1/toc> as default
* Support for the prior v1.0 ANSI/NISO Z39.99-2014 <http://www.openarchives.org/rs/1.0/toc> is retained with `spec-version='1.0'` option in scripts and `spec_version='1.0'` in various classes
* Split old `resync` script into `resync-sync` and `resync-build`
* Move scripts from `bin` dir to base dir for easier testing/development (now that there is no conflict with the module name)
* Switch from optparse to argparse, use exclusive argument group for commands
* Add --access_token option to pass bearer token with web requests
* Add --delay option to pause between successive web requests
* Drop Python 2.7, 3.3 & 3.4 from tests, add 3.7 & 3.8
* Fix various depracation warnings and errors for 3.7 & 3.8
* Switch from pep8 to pycodestyle in tests
* Change README from RDS to github flavored Markdown
* Move libraries to support tests into test/testlib
v1.0.9 2018-10-23
* Add -t/--tries and -T/--timeout options (https://github.com/resync/resync/issues/34)
* Includes partial replacement of urllib.request with requests
v1.0.8 2017-08-30
* Fix support for explicitly writing ResourceList objects as indexes (https://github.com/resync/resync/issues/31)
* Change argument names for local modes to have `--write-` prefix, e.g. --write-resourcelist
* Improve modes to detect capabilities
* Abandon support for Python 2.6 (https://github.com/resync/resync/issues/23)
v1.0.7 2017-03-27
* Fixed silly error in `resync-explorer` introduced in 1.0.6
v1.0.6 2017-03-20
* Fixed md5 hash format (https://github.com/resync/resync/issues/25)
* Added support for sha-1 and sha-256 hashes
v1.0.5 2017-01-30
* Fixed to support non-ASCII URIs/filenames (https://github.com/resync/resync/issues/22)
* Added tests for Python 3.6
* Temporarily disabled tests for Python 2.6 due to problems with coverage (https://github.com/resync/resync/issues/23)
v1.0.4 2017-01-27
* pep8 formatting
* Added contributing note
v1.0.2 2016-05-06
* Python 3 fixes in resync-explorer
* Improved test coverage
* pep257 formatting
v1.0.1 2016-03-22
* Work with Python 2 (2.6/2.7) and Python 3 (3.3/3.4/3.5)
* Improved test coverage
* Use defusedxml for safer XML parsing
* Changed parameter str to str_data in ListBase.parse(), str still supported
but deprecated. Issue pointed out by @uweschmitt
v1.0.0 2014-06-18
* Update for v1.0, Z39.99-2014 specification (http://www.openarchives.org/rs/1.0/)
* Partial implementation of dump output (no read yet)
v0.9.5 2013-11-06
* Still working toward v0.9.1 specification (http://www.openarchives.org/rs/0.9.1/)
* Set up for Travis-CI working from github
* Use /usr/bin/env to find python in executables
* Fix timezone handling in w3c_datetime.py
* Fix a number of documentation inconsistencies
v0.9.4 2013-09-06
* Work toward v0.9.1 specification (http://www.openarchives.org/rs/0.9.1/)
* ResourceSync Description becomes Source Description
v0.9.3 2013-07-31
* Add dependencies and tests to setup, no code changes
v0.9.2 2013-07-04
* Cosmetic only
v0.9.1 2013-07-04
* First release working toward v0.9 specification
* Use test examples from v0.9 specification, add code to build them too
* Added ResourceSync Description document
* Improved resync-explore binary replaces --explore mode
* Relax default notion of URL authority, add --strictauth
v0.6.3 2013-05...
* Improved --explore mode
v0.6.2 2013-05-14
* Fixed example code in README!
* Added --version flag
* Added --paths option to specify local paths to search rather than mappings
The first two components of the version tags are tied to the ResourceSync specification
version. Versions 0.6.x are intended to implement the v0.6 ResourceSync specification
(http://www.openarchives.org/rs/0.6/),
v0.6.1 2013-05-09
* First release working toward v0.6 specification
* Changed test examples to be those from v0.6 specification
* Added first stab at ResourceDump and ResourceDumpManifest objects
- First release working toward v0.6 specification
- Changed test examples to be those from v0.6 specification
- Added first stab at ResourceDump and ResourceDumpManifest objects
v0.5.3 2013-05-08
* Final release working with v0.5 specification
* Improved handling of assumed mappings for testing client on a local filesystem
* Fixed sitemapindex support for large resource lists, add rel="up" for component sitemaps
- Final release working with v0.5 specification
- Improve handling of assumed mappings for testing client on a local filesystem
- Fix sitemapindex support for large resource lists, add rel="up" for component sitemaps
v0.5.2 2013-03-26
* Fixed setup.py
- Fix setup.py
v0.5.1 2013-03-22
* Code reworked for 0.5 specification (http://www.openarchives.org/rs/0.5/)
* Client code handles --baseline, --audit and --incremental sync against the simulator
- Code reworked for 0.5 specification (http://www.openarchives.org/rs/0.5/)
- Client code handles --baseline, --audit and --incremental sync against the simulator

24
CONTRIBUTING.md
View File

@ -1,24 +0,0 @@
# Contributing to resync
Issues and pull requests are appreciated. I apologize if it takes me some time to respond to issues.
## Submitting issues
In the case of bugs please describe in detail, preferably with a recipe to repeat and demonstrate the problem. Although you should include test data as necessary, please be careful not to include secrets or passwords!
If you suggest a new feature, please give an example of expected behavior and use. For changes to modules and methods a tests case would be idea.
## Coding style
If submitting a pull request:
* Understand that this code and any merged contributions are covered by an Apache 2.0 license
* Please discuss in a issue before submitting a pull request for significant changes.
* Please submit pull requests against the `develop` branch (changes are collected there before making a release).
* Please follow [PEP8](https://www.python.org/dev/peps/pep-0008/) and [PEP257](https://www.python.org/dev/peps/pep-0257/) style rules.
* Please don't repeat code.
* Please cover the code with tests.
## Local conventions
* Tests: Use `from tests.testcase_with_tmpdir import TestCase` in place of `unittest.TestCase` to get a test object with `self.tmpdir` as temporary location to write to.

2
MANIFEST.in Normal file
View File

@ -0,0 +1,2 @@
include *.txt
recursive-include docs *.txt

95
README Normal file
View File

@ -0,0 +1,95 @@
======
resync
======
**resync** is a ResourceSync client and library in python. `ResourceSync
<http://www.openarchives.org/rs/>`_ is a synchronization framework for
the web consisting of various capabilities that allow third party
systems to remain synchronized with a server's evolving resources.
The capabilities can be combined in a modular manner to meet local
or community requirements.
Client usage
------------
Typical client usage to synchronize from a source at
``http://source.example.com/`` to a set of local files would be::
resync http://source.example.com/
which will create or update a local directory ``./source.example.com``.
Alternatively, the destination directory may be specified explicitly::
resync http://source.example.com/ /tmp/my_copy
Option details and a number of different modes are described with::
resync -h
Library usage
-------------
Typical library usage in a source::
from resync.resource_list import ResourceList
from resync.resource import Resource
rl = ResourceList()
rl.add( Resource('http://example.com/res1', lastmod='2013-01-01') )
rl.add( Resource('http://example.com/res2', lastmod='2013-01-02') )
print rl.as_xml(rl)
Installation
------------
The client and library are designed to work with Python 2.6 or 2.7.
**Automatic installation**::
easy_install resync
rsync is listed in `PyPI
<http://pypi.python.org/pypi/resync>`_ and can be installed with
``pip`` or ``easy_install``.
**Manual installation from github**::
cd /tmp
git clone git://github.com/resync/resync.git
cd resync/
python setup.py build
sudo python setup.py install
This will install the library code in the appropriate place within
your python setup, and the client ``resync`` in an appropriate system
path (perhaps ``/usr/local/bin`` or ``/usr/bin`` depending on your system).
The source code is maintained on `Github
<https://github.com/resync/resync>`_
and there may be branches/versions available there that are not
yet packaged for `PyPI
<http://pypi.python.org/pypi/resync>`_.
See also
--------
http://github.com/resync/simulator
Copyright and License
---------------------
Copyright 2012,2013 Simeon Warner
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
See LICENSE.txt

124
README.md
View File

@ -1,124 +0,0 @@
# resync
[![Build Status](https://travis-ci.org/resync/resync.svg?branch=main)](https://travis-ci.org/resync/resync)
[![Test Coverage](https://coveralls.io/repos/github/resync/resync/badge.svg?branch=main)](https://coveralls.io/github/resync/resync)
**resync** is a ResourceSync library with supporting client scipts,
written in python.
[ResourceSync](http://www.openarchives.org/rs/) is a synchronization
framework for the web consisting of various capabilities that allow
third party systems to remain synchronized with a server's evolving
resources. The capabilities can be combined in a modular manner to
meet local or community requirements.
## Client usage
Typical client usage to synchronize from a source at
`http://source.example.com/` to a set of local files would be
```
resync-sync http://source.example.com/
```
which will create or update a local directory `./source.example.com`.
Alternatively, the destination directory may be specified explicitly::
```
resync-sync http://source.example.com/ /tmp/my_copy
```
Option details and a number of different modes are described with::
```
resync-sync -h
```
## Python library usage
Typical library use in a source (create and output a Resource List)::
```
from resync import Resource, ResourceList
rl = ResourceList()
rl.add( Resource('http://example.com/res1', lastmod='2013-01-01') )
rl.add( Resource('http://example.com/res2', lastmod='2013-01-02') )
print rl.as_xml()
```
Typical library use in a destination (get and examine a Capability List)::
```
from resync import CapabilityList
# Read Capability List and show supported capabilities
cl = CapabilityList()
cl.read("https://raw.github.com/resync/resync/1.0/resync/test/testdata/examples_from_spec/resourcesync_ex_2_6.xml")
for resource in cl:
print "supports %s (at %s)" % (resource.capability,resource.uri)
```
## Installation
The client and library are designed to work with Python 3.5 and up. (The
last version supporting Python 2.7 was
[1.0.9, also on PyPI](https://pypi.org/project/resync/1.0.9/))
### Automatic installation
```
pip install resync
```
rsync is listed in [PyPI](http://pypi.python.org/pypi/resync) and can be
installed with `pip` or `easy_install`.
### Manual installation from github
```
cd /tmp
git clone git://github.com/resync/resync.git
cd resync/
python setup.py build
python setup.py install
```
This will install the library code in the appropriate place within
a user-space python setup, including the clients `resync-sync`,
`resync-build` and `resync-explorer`. Use of `sudo` to install in
system spaces is generally discouraged.
The source code is maintained on [Github](https://github.com/resync/resync)
and there may be branches/versions available there that are not
yet packaged for PyPI.
## See also
* [ResourceSync simulator](http://github.com/resync/simulator)
## Contributors
Thanks to: [Bernhard Haslhofer](https://github.com/behas),
[Robert Sanderson](<https://github.com/azaroth42)
and [other contributors](https://github.com/resync/resync/graphs/contributors).
See [CONTRIBUTING.md](https://github.com/resync/resync/blob/main/CONTRIBUTING.md)
for guidelines for contributing.
## Copyright and License
Copyright 2012--2020 Simeon Warner and contributors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
See `LICENSE.txt`

1
README.rst Symbolic link
View File

@ -0,0 +1 @@
README

14
TODO.md
View File

@ -1,14 +0,0 @@
To do list for resync client and library
========================================
See also issues on github tracker. Other big threads that need attention are:
* work through and test creation of Resource and Change Dump ZIPs
* add use of a Resource and Change Dump ZIPs
* add archives (ResoourceListArchive, ChangeListArchive, ResourceDumpArchive, ChangeDumpArchive)
* consider breaking out handling of links that is somewhat duplicated in resource.py and resource_container.py
Planned changes and deprecation
-------------------------------
* Remove support for ListBase.parse(str=...) which was deprecated in favor of ListBase.parse(str_data=...) 2016-03-22 in 1.0.1

320
bin/resync Executable file
View File

@ -0,0 +1,320 @@
#!/usr/bin/python
"""
resync: The ResourceSync command line client
Created by Simeon Warner on 2012-04...
"""
import logging
import logging.config
import optparse
import sys
from resync.client import Client, ClientFatalError
from resync.utils import UTCFormatter
DEFAULT_CLIENT_LOGFILE = 'resync-client.log'
def init_logging(to_file=False, logfile=None, human=True,
verbose=False, eval_mode=False):
"""Initialize logging
Use of log levels:
DEBUG - very verbose, for evaluation of output (-e)
INFO - verbose, only seen by users if they ask for it (-v)
WARNING - messages output messages to console
"""
fmt = '%(asctime)s | %(name)s | %(levelname)s | %(message)s'
formatter = UTCFormatter(fmt)
if human:
# Create a special handler designed just for human readable output
hh = logging.StreamHandler()
hh.setLevel( logging.INFO if (verbose) else logging.WARNING )
hh.setFormatter(logging.Formatter(fmt='%(message)s'))
if to_file:
if (logfile is None):
logfile = DEFAULT_CLIENT_LOGFILE
fh = logging.FileHandler(filename=logfile, mode='a')
fh.setFormatter(formatter)
fh.setLevel( logging.DEBUG if (eval_mode) else logging.INFO )
loggers = ['client','resourcelist_builder','sitemap']
for logger in loggers:
log = logging.getLogger(logger)
log.setLevel(logging.DEBUG) #control at handler instead
if human:
log.addHandler(hh)
if to_file:
log.addHandler(fh)
log=logging.getLogger('client')
if (to_file):
log.info("Writing detailed log to %s" % (logfile))
def count_true_args(*args):
"""Count number of list of arguments that evaluate True"""
count=0
for arg in args:
if (arg):
count+=1
return(count)
def parse_links(args_link):
links=[]
if (args_link is not None):
for link_str in args_link:
try:
links.append(parse_link(link_str))
except ValueError as e:
raise ClientFatalError("Bad --link option '%s' (%s)"%(link_str,str(e)))
return(links)
def parse_link(link_str):
"""Parse --link option to add to <rs:ln> links
Input string of the form: rel,href,att1=val1,att2=val2
"""
atts={}
try:
segs = link_str.split(',')
# First segments are relation and subject
atts['rel'] = segs.pop(0)
atts['href'] = segs.pop(0)
# Remaining segments are attributes
for term in segs:
(k,v)=term.split('=')
atts[k]=v
except ValueError as e:
raise ClientFatalError("Bad component of --link option '%s' (%s)"%(link_str,str(e)))
return(atts)
def parse_capabilities(caps_str):
"""Parse list of capabilities in --capabilitylist option
Input string of the form: cap_name=uri,cap_name=uri
"""
capabilities={}
try:
segs = caps_str.split(',')
for term in segs:
(k,v)=term.split('=')
capabilities[k]=v
except ValueError as e:
raise ClientFatalError("Bad component of --capabilitylist option '%s' (%s)"%(link_str,str(e)))
return(capabilities)
def parse_capability_lists(cls_str):
"""Parse list of capability lists in --capabilitylistindex option
Input string of the form: uri,uri
"""
return(cls_str.split(','))
def main():
if (sys.version_info < (2,6)):
sys.exit("This program requires python version 2.6 or later")
# Options and arguments
p = optparse.OptionParser(description='ResourceSync sync script',
usage='usage: %prog [options] uri_path local_path (-h for help)',
add_help_option=False)
# Modes
# a. modes using remote sitemap/resources
rem = p.add_option_group('REMOTE MODES',
'These modes use a remote source that is specified in a set of uri=path mappings '
'and potentially also using an explicit --sitemap location. The default mode is '
'--baseline')
rem.add_option('--baseline', '-b', action='store_true',
help='baseline sync of resources from remote source (src) to local filesystem (dst)')
rem.add_option('--inc', '-i', action='store_true',
help='incremental sync of resources from remote source (src) to local filesystem (dst). Uses either timestamp recorded from last baseline or incremental sync for this source, or explicit --from parameter, to determine the earlier update timestamp to act on.')
rem.add_option('--audit', '-a', action='store_true',
help="audit sync state of destination wrt source")
rem.add_option('--parse', '-p', action='store_true',
help="parse a remote sitemap/sitemapindex (from mapping or explicit --sitemap) and show summary information including document type and number of entries")
rem.add_option('--explore','-x', action='store_true',
help="read discovery information and show capabilities. Will use either an explicit --sitemap or the first mapping URI specified to determine which server to look for .well-known discovery information or which URI to try")
# b. modes based solely on files on local disk
loc = p.add_option_group('LOCAL MODES',
'These modes act on files on the local disk')
loc.add_option('--resourcelist', '--resource-list', '-r', action='store_true',
help="write a resource list based on files on disk using uri=path mappings "
"in reverse to calculate URIs from the local paths. To STDOUT by "
"default, override with --outfile")
loc.add_option('--changelist', '--change-list', '-c', action='store_true',
help="write a change list based on comparison of a reference sitemap "
"(specify file with --reference) and either files on disk (using "
"the mapping provided) or a second sitemap (specify file with "
"--newreference). Otherwise follows --resourcelist options. Also accepts "
"the --empty option (with no mapping) to write and empty changelist.")
loc.add_option('--capabilitylist', '--capability-list', type=str, action='store',
help="write a capability list based on the set of capabilities and "
"URIs supplied in cap_name=URI,cap_name=URI format. Otherwise "
"follows --resourcelist options.")
loc.add_option('--capabilitylistindex', '--capability-list-index', type=str, action='store',
help="write a capability list index based on the set of capabilitie "
"list URIs supplied as a comma separated list. Otherwise "
"follows --resourcelist options.")
# Specification of map between remote URI and local file paths, and remote sitemap
nam = p.add_option_group('FILE/URI NAMING OPTIONS')
nam.add_option('--outfile', type=str, action='store',
help='write sitemap to specified file rather than STDOUT')
nam.add_option('--sitemap', type=str, action='store',
help='explicitly set sitemap name, overriding default sitemap.xml '
'appended to first source URI specified in the mappings')
nam.add_option('--reference', type=str, action='store',
help='reference sitemap name for --changelist calculation')
nam.add_option('--newreference', type=str, action='store',
help='updated reference sitemap name for --changelist calculation')
nam.add_option('--dump', metavar="DUMPFILE", type=str, action='store',
help='write dump to specified file for --resourcelist or --changelist')
nam.add_option('--changelist-uri','--change-list-uri', type=str, action='store',
help='explicitly set the changelist URI that will be use in --inc mode, '
'overrides process of getting this from the sitemap')
# Options that apply to multiple modes
opt = p.add_option_group('MISCELANEOUS OPTIONS')
opt.add_option('--checksum', action='store_true',
help="use checksum (md5) in addition to last modification time and size")
opt.add_option('--delete', action='store_true',
help="allow files on destination to be deleted")
opt.add_option('--from', type=str, action='store', dest='from_datetime', metavar="DATETIME",
help="explicit datetime value used to filter updates in change list for "
"--incremental sync")
opt.add_option('--exclude', type=str, action='append',
help="exclude resources with URI or filename matching pattern "
"(repeat option for multiple excludes)")
opt.add_option('--empty', action='store_true',
help="combine with --changelist to write and empty changelist, perhaps with links")
opt.add_option('--link', type=str, action='append',
help="add discovery links to the output sitemap, "
"format: rel,href[,att1=val1,att2=val2] "
"(repeat option for multiple links)")
opt.add_option('--describedby', type=str, action='store',
help="add an <rs:md rel=\"describedby\" link to the "
"URI given")
opt.add_option('--multifile', '-m', action='store_true',
help="disable reading and output of sitemapindex for multifile sitemap")
opt.add_option('--noauth', action='store_true',
help="disable checking of URL paths to ensure that the sitemaps refer "
"only to resources on the same server/sub-path etc. Use with care.")
opt.add_option('--warc', action='store_true',
help="write dumps in WARC format (instead of ZIP+Sitemap default)")
opt.add_option('--dryrun', '-n', action='store_true',
help="don't update local resources, say what would be done")
opt.add_option('--ignore-failures', action='store_true',
help="continue past download failures")
# These likely only useful for experimentation
opt.add_option('--max-sitemap-entries', type=int, action='store',
help="override default size limits")
# Want these to show at the end
opt.add_option('--verbose', '-v', action='store_true',
help="verbose")
opt.add_option('--logger', '-l', action='store_true',
help="create detailed log of client actions (will write "
"to %s unless specified with --logfile" %
DEFAULT_CLIENT_LOGFILE)
opt.add_option('--logfile', type='str', action='store',
help="create detailed log of client actions")
opt.add_option('--eval', '-e', action='store_true',
help="output evaluation of source/client synchronization performance... "
"be warned, this is very verbose")
opt.add_option('--help', '-h', action='help',
help="this help")
(args, map) = p.parse_args()
# Implement exclusive arguments and default --baseline (support for exclusive
# groups in argparse is incomplete is python2.6)
if (not args.baseline and not args.inc and not args.audit and
not args.parse and not args.explore and
not args.resourcelist and not args.changelist and
not args.capabilitylist and not args.capabilitylistindex):
args.baseline=True
elif (count_true_args(args.baseline,args.inc,args.audit,args.parse,args.explore,
args.resourcelist,args.changelist,args.capabilitylist,
args.capabilitylistindex)>1):
p.error("Only one of --baseline, --inc, --audit, --parse, --explore, --resourcelist, --changelist, --capabilitylist, --capabilitylistindex modes allowed")
# Configure logging module and create logger instance
init_logging( to_file=args.logger, logfile=args.logfile,
verbose=args.verbose, eval_mode=args.eval )
c = Client( checksum=args.checksum,
verbose=args.verbose,
dryrun=args.dryrun )
try:
if (map):
# Mappings apply to (almost) everything
c.set_mappings(map)
if (args.sitemap):
c.sitemap_name=args.sitemap
if (args.warc):
c.dump_format='warc'
if (args.exclude):
c.exclude_patterns=args.exclude
if (args.multifile):
c.allow_multifile=not args.multifile
if (args.noauth):
c.noauth=args.noauth
if (args.max_sitemap_entries):
c.max_sitemap_entries=args.max_sitemap_entries
if (args.ignore_failures):
c.ignore_failures=args.ignore_failures
# Links apply to anything that writes sitemaps
links = parse_links(args.link)
# expand --describedby shorthand for a link, prepend
if (args.describedby):
links.insert(0,{'rel':'describedby','href':args.describedby})
# Finally, do something...
if (args.baseline or args.audit):
c.baseline_or_audit(allow_deletion=args.delete,
audit_only=args.audit)
elif (args.inc):
c.incremental(allow_deletion=args.delete,
change_list_uri=args.changelist_uri,
from_datetime=args.from_datetime)
elif (args.parse):
c.parse_document()
elif (args.explore):
c.explore()
elif (args.resourcelist):
c.write_resource_list(outfile=args.outfile,
links=links,
dump=args.dump)
elif (args.changelist):
if (not args.reference and not args.empty):
p.error("Must supply --reference sitemap for --changelist, or --empty")
c.write_change_list(ref_sitemap=args.reference,
newref_sitemap=( args.newreference if (args.newreference) else None ),
empty=args.empty,
outfile=args.outfile,
links=links,
dump=args.dump)
elif (args.capabilitylist):
c.write_capability_list(capabilities=parse_capabilities(args.capabilitylist),
outfile=args.outfile,
links=links)
elif (args.capabilitylistindex):
c.write_capability_list_index(capability_lists=parse_capability_lists(args.capabilitylistindex),
outfile=args.outfile,
links=links)
else:
p.error("Unknown mode requested")
# Any problem we expect will come as a ClientFatalError, anything else
# is... an exception ;-)
except ClientFatalError as e:
sys.stderr.write("\nFatalError: " + str(e) + "\n")
if __name__ == '__main__':
main()

0
docs/documentation.txt Normal file
View File

7
examples/README_client_library_in_source.py → docs/examples/01_make_resourcelist.py
View File

@ -1,7 +1,8 @@
#!/usr/bin/env python
from resync import Resource, ResourceList
from resync.resource_list import ResourceList
from resync.resource import Resource
from resync.sitemap import Sitemap
rl = ResourceList()
rl.add( Resource('http://example.com/res1', lastmod='2013-01-01') )
rl.add( Resource('http://example.com/res2', lastmod='2013-01-02') )
print rl.as_xml()
print rl.as_xml(pretty_xml=True)

190
docs/examples/99_make_spec_examples.pl Normal file
View File

@ -0,0 +1,190 @@
from resync.resource_list import ResourceList
from resync.change_list import ChangeList
from resync.resource_dump import ResourceDump
from resync.resource_dump_manifest import ResourceDumpManifest
from resync.capability_list import CapabilityList
from resync.resource import Resource
from resync.sitemap import Sitemap
def head(txt):
print "\n=== "+txt+" ===\n"
print "\nExample script to build objects to write the examples in the spec\n"
head("Example 2.1: A Resource List")
rl = ResourceList()
rl.modified='2013-01-03T09:00:00Z'
rl.add( Resource('http://example.com/res1') )
rl.add( Resource('http://example.com/res2') )
print rl.as_xml(pretty_xml=True)
head("Example 2.2: A Resource List with additional information")
rl = ResourceList()
rl.modified='2013-01-03T09:00:00Z'
rl.add( Resource('http://example.com/res1',
lastmod='2013-01-02T13:00:00Z',
md5='1584abdf8ebdc9802ac0c6a7402c03b') )
rl.add( Resource('http://example.com/res2',
lastmod='2013-01-02T14:00:00Z',
md5='1e0d5cb8ef6ba40c99b14c0237be735e') )
print rl.as_xml(pretty_xml=True)
head("Example 2.3: A Change List")
cl = ChangeList()
cl.modified="2013-01-03T11:00:00Z"
cl.add( Resource('http://example.com/res2.pdf',
lastmod='2013-01-02T13:00:00Z',
change='updated') )
cl.add( Resource('http://example.com/res3.tiff',
lastmod='2013-01-02T18:00:00Z',
change='deleted') )
print cl.as_xml(pretty_xml=True)
head("Example 2.4: A Resource Dump")
rd = ResourceDump()
rd.modified="2013-01-03T09:00:00Z"
rd.add( Resource('http://example.com/resourcedump.zip',
lastmod='2013-01-03T09:00:00Z') )
print rd.as_xml(pretty_xml=True)
head("Example 2.5: A Resource Dump Manifest detailing the content of a ZIP file")
rdm = ResourceDumpManifest()
rdm.modified="2013-01-03T09:00:00Z"
rdm.add( Resource('http://example.com/res1',
lastmod='2013-01-02T13:00:00Z',
md5='1584abdf8ebdc9802ac0c6a7402c03b6',
path='/resources/res1') )
rdm.add( Resource('http://example.com/res2',
lastmod='2013-01-02T14:00:00Z',
md5='1e0d5cb8ef6ba40c99b14c0237be735e',
path='/resources/res2') )
print rdm.as_xml(pretty_xml=True)
head("Example 2.6: A Capability List with the description of the ResourceSync capabilities of a Source")
caplist=CapabilityList()
caplist.modified="2013-01-02T14:00:00Z"
caplist.ln.append({'href':"http://example.com/info-about-source.xml",
'rel':"describedby",
'type':"application/xml"})
caplist.add( Resource('http://example.com/dataset1/resourcelist.xml',
capability="resourcelist") )
caplist.add( Resource('http://example.com/dataset1/resourcedump.xml',
capability="resourcedump") )
caplist.add( Resource('http://example.com/dataset1/changelist.xml',
capability="changelist") )
print caplist.as_xml(pretty_xml=True)
head("Example 2.7: A Resource List Index expressed using the <sitemapindex> document format")
rl = ResourceList()
rl.modified="2013-01-03T09:00:00Z"
rl.add( Resource("http://example.com/resourcelist-part2.xml",
lastmod="2013-01-03T09:00:00Z") )
rl.add( Resource("http://example.com/resourcelist-part1.xml",
lastmod="2013-01-03T09:00:00Z") )
print rl.index_as_xml(pretty_xml=True)
head("Example 4.1: A Resource List")
rl = ResourceList()
rl.modified="2013-01-03T09:00:00Z"
rl.ln.append({"rel":"resourcesync",
"href":"http://example.com/dataset1/capabilitylist.xml"})
rl.add( Resource("http://example.com/res1",
lastmod="2013-01-02T13:00:00Z",
md5="1584abdf8ebdc9802ac0c6a7402c03b6",
length="8876",
type="text/html") )
rl.add( Resource("http://example.com/res2",
lastmod="2013-01-02T14:00:00Z",
md5="1e0d5cb8ef6ba40c99b14c0237be735e",
sha256="854f61290e2e197a11bc91063afce22e43f8ccc655237050ace766adc68dc784",
length="14599",
type="application/pdf") )
print rl.as_xml(pretty_xml=True)
head("Example 4.2: A Resource List Index")
rl = ResourceList()
rl.modified="2013-01-03T09:00:00Z"
rl.ln.append({"rel":"resourcesync",
"href":"http://example.com/dataset1/capabilitylist.xml"})
rl.add( Resource("http://example.com/resourcelist3.xml",
lastmod="2013-01-03T09:00:00Z") )
rl.add( Resource("http://example.com/resourcelist2.xml",
lastmod="2013-01-03T09:00:00Z") )
rl.add( Resource("http://example.com/resourcelist1.xml",
lastmod="2013-01-03T09:00:00Z") )
print rl.index_as_xml(pretty_xml=True)
head("Example 4.3: A Resource List with a navigational link to its parent Resource List Index")
rl = ResourceList()
rl.modified="2013-01-03T09:00:00Z"
rl.ln.append({"rel":"resourcesync",
"href":"http://example.com/dataset1/capabilitylist.xml"})
rl.ln.append({"rel":"up",
"href":"http://example.com/dataset1/resourcelist-index.xml"})
rl.add( Resource("http://example.com/res3",
lastmod="2013-01-03T09:00:00Z",
md5="1584abdf8ebdc9802ac0c6a7402c8753",
length="4385",
type="application/pdf") )
rl.add( Resource("http://example.com/res2",
lastmod="2013-01-03T09:00:00Z",
md5="4556abdf8ebdc9802ac0c6a7402c9881",
length="883",
type="image/png") )
print rl.as_xml(pretty_xml=True)
head("Example 5.1: A Resource Dump document")
rd = ResourceDump()
rd.modified = "2013-01-03T09:00:00Z"
rd.ln.append({"rel":"resourcesync",
"href":"http://example.com/dataset1/capabilitylist.xml"})
rd.add( Resource("http://example.com/resourcedump-part3.zip",
lastmod="2013-01-03T09:00:00Z",
type="application/zip",
length="4765") )
rd.add( Resource("http://example.com/resourcedump-part2.zip",
lastmod="2013-01-03T09:00:00Z",
type="application/zip",
length="9875") )
rd.add( Resource("http://example.com/resourcedump-part1.zip",
lastmod="2013-01-03T09:00:00Z",
type="application/zip",
length="2298") )
print rd.as_xml(pretty_xml=True)
head("Example 5.2: A Resource Dump Manifest")
rdm = ResourceDumpManifest()
rdm.modified="2013-01-03T09:00:00Z"
rdm.ln.append({"rel":"resourcesync",
"href":"http://example.com/dataset1/capabilitylist.xml"})
rdm.add( Resource("http://example.com/res1",
lastmod="2013-01-02T13:00:00Z",
md5="1584abdf8ebdc9802ac0c6a7402c03b6",
length="8876",
type="text/html",
path="/resources/res1") )
rdm.add( Resource("http://example.com/res2",
lastmod="2013-01-02T14:00:00Z",
md5="1e0d5cb8ef6ba40c99b14c0237be735e",
sha256="854f61290e2e197a11bc91063afce22e43f8ccc655237050ace766adc68dc784",
length="14599",
type="application/pdf",
path="/resources/res2") )
print rdm.as_xml(pretty_xml=True)
head("Example 5.3: A Resource Dump Manifest Index")
rdm = ResourceDumpManifest()
rdm.modified="2013-01-03T09:00:00Z"
rdm.ln.append({"rel":"resourcesync",
"href":"http://example.com/dataset1/capabilitylist.xml"})
rdm.add( Resource("/manifests/part3.xml",
lastmod="2013-01-03T09:00:00Z",
path="/manifests/part3.xml") )
rdm.add( Resource("/manifests/part2.xml",
lastmod="2013-01-03T09:00:00Z",
path="/manifests/part2.xml") )
rdm.add( Resource("/manifests/part1.xml",
lastmod="2013-01-03T09:00:00Z",
path="/manifests/part1.xml") )
print rdm.index_as_xml(pretty_xml=True)

79
docs/pypi_upload.md
View File

@ -1,79 +0,0 @@
# Updating resync on pypi
Notes to remind @zimeon...
resync is at <https://pypi.python.org/pypi/resync> on pypi
## Putting up a new version
0. In `develop` branch: bump version number in `resync/__init__.py` and check `CHANGES.md` is up to date
1. Check all tests good with appropriate Python 3.x (`python setup.py test` and CI)
2. Check code is up-to-date with github version
3. Check out `main` and merge in `develop`
4. Check all tests still good (`python setup.py test` and CI)
5. Make sure main `README.md` has correct travis-ci icon link
6. Check branches as expected (`git branch -a`)
7. Check local build and version reported OK (`python setup.py build; python setup.py install; resync-sync -h`)
8. Check client works with simulator:
```
# Run simulator in one window
resync-simulator> ./resync-simulator
# Run client in another
resync> rm -rf localhost_8888
resync> ./resync-sync --baseline --delete http://localhost:8888/
Using URI mapping: http://localhost:8888/ -> localhost_8888
Status: NOT IN SYNC (same=0, to create=1003, to update=0, to delete=0)
Will GET 1003 resources, and delete 0 resources
Status: SYNCED (same=0, created=1003, updated=0, deleted=0)
resync> ./resync-sync --baseline --delete http://localhost:8888/
Using URI mapping: http://localhost:8888/ -> localhost_8888
Status: NOT IN SYNC (same=1000, to create=2, to update=1, to delete=2)
Will GET 3 resources, and delete 2 resources
Status: SYNCED (same=1000, created=2, updated=1, deleted=2)
resync> ./resync-sync --incremental --delete http://localhost:8888/
Using URI mapping: http://localhost:8888/ -> localhost_8888
Status: CHANGES APPLIED (created=4, updated=5, deleted=4)
Will apply 13 changes, and delete 4 resources
Status: NO CHANGES (created=4, updated=5, deleted=4)
esync> ./resync-sync --incremental --delete http://localhost:8888/
Using URI mapping: http://localhost:8888/ -> localhost_8888
Status: CHANGES APPLIED (created=0, updated=3, deleted=0)
Will apply 3 changes, and delete 0 resources
Status: NO CHANGES (created=0, updated=3, deleted=0)
resync> ./resync-sync --incremental --delete http://localhost:8888/; ./resync-sync --audit http://localhost:8888/
Using URI mapping: http://localhost:8888/ -> localhost_8888
Status: CHANGES APPLIED (created=1, updated=2, deleted=0)
Will apply 3 changes, and delete 0 resources
Status: NO CHANGES (created=1, updated=2, deleted=0)
Using URI mapping: http://localhost:8888/ -> localhost_8888
Status: IN SYNC (same=1001, to create=0, to update=0, to delete=0)
```
9. If all checks out OK, tag and push the new version to github:
```
git tag -n1
#...current tags
git tag -a -m "ResourceSync library and client" v2.0.0
git push --tags
```
10. Upload to PyPI
```
rm -r dist
python setup.py sdist bdist_wheel; ls dist
# Should be source and wheel files for just this version
twine upload dist/*
```
10. Check on PyPI at <https://pypi.python.org/pypi/resync>
11. Finally, back on `develop` branch start new version number by editing `resync/__init__.py` and `CHANGES.md`

47
docs/tutorial.md
View File

@ -1,47 +0,0 @@
# ResourceSync tutorial exercises using resync
## Writing a Resource Dump
Relative to the base directory of the github distribution there is a directory
`tests/testdata/dir1` with two files, `file_a` and `file_b`. We can create
a Resource Dump of this directory, as if it were mapped to `http://example.com/dir1`,
with the following:
```
resync> ./resync-build -v --write-resourcedump --outfile /tmp/rd http://example.com/dir1 tests/testdata/dir1
Scanning disk from tests/testdata/dir1
Writing resource dump to True...
Total size of files to include in dump 65 bytes
Writing sitemap /tmp/rd00000.xml...
Wrote sitemap /tmp/rd00000.xml
Wrote ZIP file dump /tmp/rd00000.zip with size 623 bytes
Wrote 1 dump files
resync> zipinfo /tmp/rd00000.zip
Archive: /tmp/rd00000.zip
Zip file size: 623 bytes, number of entries: 3
?rw------- 2.0 unx 453 b- defN 20-Dec-16 10:58 manifest.xml
-rw-r--r-- 2.0 unx 20 b- defN 12-Jul-25 13:13 file_a
-rw-r--r-- 2.0 unx 45 b- defN 01-Sep-08 21:46 file_b
3 files, 518 bytes uncompressed, 325 bytes compressed: 37.3%
```
This can then be unpacked and the files checked:
```
resync> mkdir /tmp/rd
mkdir: /tmp/rd: File exists
resync> cd /tmp/rd
rd> unzip ../rd00000.zip
Archive: ../rd00000.zip
inflating: manifest.xml
inflating: file_a
inflating: file_b
rd> more manifest.xml
<?xml version='1.0' encoding='UTF-8'?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"><rs:md capability="resourcedump-manifest" /><url><loc>http://example.com/dir1/file_a</loc><lastmod>2012-07-25T17:13:46Z</lastmod><rs:md length="20" path="file_a" /></url><url><loc>http://example.com/dir1/file_b</loc><lastmod>2001-09-09T01:46:40Z</lastmod><rs:md length="45" path="file_b" /></url></urlset>
rd> more file_a
I am file a in dir1
rd> more file_b
I am file b in dir1, I am bigger than file_a
```

8
examples/README_client_library_in_destination.py
View File

@ -1,8 +0,0 @@
#!/usr/bin/env python
from resync import CapabilityList
# Read Capability List and show supported capabilities
cl = CapabilityList()
cl.read("https://raw.github.com/resync/resync/0.6/resync/test/testdata/examples_from_spec/resourcesync_ex_2_6.xml")
for resource in cl:
print "supports %s (at %s)" % (resource.capability,resource.uri)

41
examples/sitemapindex_and_generator.py
View File

@ -1,41 +0,0 @@
#!/usr/bin/env python
from resync.resource import Resource
from resync.resource_list import ResourceList
from resync.list_base_with_index import ListBaseIndexError
import sys
try:
max_sitemap_entries = int(sys.argv[1])
except:
max_sitemap_entries = 2
print "### max_sitemap_entries=%d" % max_sitemap_entries
basename = 'http://example.com/resourcelist.xml'
my_resources = [ Resource('a'), Resource('b'), Resource('c') ]
def my_resource_list():
"""Simulate the generator used by simulator"""
rl = ResourceList( resources=iter(my_resources), count=len(my_resources) )
rl.max_sitemap_entries = max_sitemap_entries
return(rl)
print "\n### ResourceListHandler"
rl = my_resource_list()
if (rl.requires_multifile):
print "Single sitemap:"
else:
print "Sitemapindex:"
print rl.as_xml( allow_multifile=True, basename=basename )
print "\n### ResourceListPartHandler"
rl = my_resource_list()
num_parts=rl.requires_multifile()
if (num_parts):
for part_number in range(0,num_parts):
rl = my_resource_list()
print "Part %d:" % (part_number)
print rl.as_xml_part( basename=basename, part_number=0 )
else:
print "404 - no parts"

201
resync-build
View File

@ -1,201 +0,0 @@
#!/usr/bin/env python
"""resync-build: The ResourceSync command line list builder.
Copyright 2012-2020 Simeon Warner
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License
"""
import argparse
import sys
from resync import __version__
from resync.client import Client, ClientFatalError
from resync.client_utils import init_logging, count_true_args, parse_links, parse_capabilities, parse_capability_lists, add_shared_misc_options, process_shared_misc_options
DEFAULT_LOGFILE = 'resync-client.log'
def main():
"""Main function to implement command line script."""
if (sys.version_info < (3, 5)):
sys.exit("This program requires python version 3.5 or later")
# Options and arguments
parser = argparse.ArgumentParser(
description="ResourceSync build script (v" + __version__ + ")",
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser._optionals = parser.add_argument_group(
'MODES OF OPERATION (must specify one only). This script operates only '
'to create ResourceSync descriptions on the local filesystem based on '
'local content')
rem = parser.add_mutually_exclusive_group(required=True)
rem.add_argument('--write-resourcelist', '--write-resource-list', action='store_true',
help="write a resource list based on files on disk using uri=path mappings "
"in reverse to calculate URIs from the local paths. Scans local disk "
"based either on explicit --paths setting, else starting from all local "
"paths specified in the mappings. Writes to STDOUT by default, override "
"with --outfile")
rem.add_argument('--write-changelist', '--write-change-list', action='store_true',
help="write a change list based on comparison of a reference sitemap "
"(specify file with --reference) and either files on disk (using "
"the mapping provided) or a second sitemap (specify file with "
"--newreference). Otherwise follows --write-resourcelist options. Also accepts "
"the --empty option (with no mapping) to write and empty changelist.")
rem.add_argument('--write-capabilitylist', '--write-capability-list', type=str, action='store',
help="write a capability list based on the set of capabilities and "
"URIs supplied in cap_name=URI,cap_name=URI format. Otherwise "
"follows --write-resourcelist options.")
rem.add_argument('--write-sourcedescription', '--write-source-description', type=str, action='store',
help="write a Source Description document based on the set of capability "
"list URIs supplied as a comma separated list. Otherwise "
"follows --write-resourcelist options.")
rem.add_argument('--write-resourcedump', '--write-resource-dump', '-d', action='store_true',
help="write a Resource Dump. Specify output file with --outfile and use other "
"options as for --write-resourcelist")
rem.add_argument('--write-changedump', '--write-change-dump', action='store_true',
help="write a Resource Dump. Specify output file with --outfile and use other "
"options as for --write-changelist")
# Positional arguments
map = parser.add_argument_group('URI MAPPING TO FILESYSTEM for REMOTE modes')
map.add_argument(metavar='uri=path | uri path', dest='map', type=str, nargs='*',
help="remote URI of source for remote synchronization operations (may "
"also combine uri=local path)")
nam = parser.add_argument_group('FILE/URI NAMING OPTIONS')
nam.add_argument('--outfile', type=str, action='store',
help="write output to specified file rather than STDOUT or default")
nam.add_argument('--paths', type=str, action='store',
help="explicit set of paths for disk scan --resourceslist or --changelist "
"generation")
nam.add_argument('--reference', type=str, action='store',
help="reference sitemap name for --write-changelist calculation")
nam.add_argument('--newreference', type=str, action='store',
help="updated reference sitemap name for --write-changelist calculation")
lks = parser.add_argument_group("LINK GENERATION")
lks.add_argument('--link', type=str, action='append',
help="add discovery links to the output sitemap, "
"format: rel,href[,att1=val1,att2=val2] "
"(repeat option for multiple links)")
lks.add_argument('--describedby-link', type=str, action='store',
help="add an <rs:md rel=\"describedby\" link to "
"a description of the feed at the URI given")
lks.add_argument('--sourcedescription-link', '--source-description-link',
type=str, action='store',
help="for a Capability List add a <rs:md rel=\"up\" link to the"
"Source Description document at the URI given, else ignored")
lks.add_argument('--capabilitylist-link', '--capability-list-link',
type=str, action='store',
help="for all documents except a Capability List or a "
"Source Description, add an <rs:md rel=\"up\" link "
"to the Capability List at the URI given")
# Options that apply to multiple modes
opt = parser.add_argument_group('MISCELANEOUS OPTIONS')
add_shared_misc_options(opt, default_logfile=DEFAULT_LOGFILE)
opt.add_argument('--empty', action='store_true',
help="combine with --changelist to write and empty changelist, perhaps with links")
opt.add_argument('--warc', action='store_true',
help="write dumps in WARC format (instead of ZIP+Sitemap default)")
opt.add_argument('--dryrun', '-n', action='store_true',
help="don't update local resources, say what would be done")
# These likely only useful for experimentation
opt.add_argument('--max-sitemap-entries', type=int, action='store',
help="override default size limits")
opt.add_argument('--eval', '-e', action='store_true',
help="output evaluation of source/client synchronization performance... "
"be warned, this is very verbose")
args = parser.parse_args()
# Configure logging module and create logger instance
init_logging(to_file=args.logger, logfile=args.logfile, default_logfile=DEFAULT_LOGFILE,
verbose=args.verbose, eval_mode=args.eval)
process_shared_misc_options(args)
c = Client(spec_version=args.spec_version,
hashes=args.hash,
verbose=args.verbose,
dryrun=args.dryrun)
try:
if (args.map):
# Mappings apply to (almost) everything
c.set_mappings(args.map)
if (args.warc):
c.dump_format = 'warc'
if (args.exclude):
c.exclude_patterns = args.exclude
if (args.multifile):
c.allow_multifile = not args.multifile
if (args.max_sitemap_entries):
c.max_sitemap_entries = args.max_sitemap_entries
# Links apply to anything that writes sitemaps
links = parse_links(args.link)
# Add specific links is appropriate cases
if (args.capabilitylist_link
and not args.write_capabilitylist
and not args.write_sourcedescription):
# rel="up" to Capability List in all but Capability List
# and Source Description
links.insert(0, {'rel': 'up', 'href': args.capabilitylist_link})
if (args.sourcedescription_link and args.write_capabilitylist):
# rel="up" to Source Description from Capability List
links.insert(0, {'rel': 'up', 'href': args.sourcedescription_link})
if (args.describedby_link):
links.insert(0, {'rel': 'describedby',
'href': args.describedby_link})
# Finally, do something...
if (args.write_resourcelist or args.write_resourcedump):
c.write_resource_list(paths=args.paths,
outfile=args.outfile,
links=links,
dump=args.write_resourcedump)
elif (args.write_changelist or args.write_changedump):
if (not args.reference and not args.empty):
parser.error("Must supply --reference sitemap for --changelist, or --empty")
c.write_change_list(ref_sitemap=args.reference,
newref_sitemap=(args.newreference if (
args.newreference) else None),
empty=args.empty,
paths=args.paths,
outfile=args.outfile,
links=links,
dump=args.write_changedump)
elif (args.write_capabilitylist):
c.write_capability_list(
capabilities=parse_capabilities(args.write_capabilitylist),
outfile=args.outfile,
links=links)
elif (args.write_sourcedescription):
c.write_source_description(
capability_lists=parse_capability_lists(
args.write_sourcedescription),
outfile=args.outfile,
links=links)
else:
parser.error("Unknown mode requested")
# Any problem we expect will come as a ClientFatalError, anything else
# is... an exception ;-)
except ClientFatalError as e:
sys.stderr.write("\nFatalError: " + str(e) + "\n")
if __name__ == '__main__':
main()

99
resync-explorer
View File

@ -1,99 +0,0 @@
#!/usr/bin/env python
"""resync-explorer: The ResourceSync explorer.
Copyright 2012-2020 Simeon Warner
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License
"""
import argparse
import sys
from resync import __version__
from resync.client import ClientFatalError
from resync.client_utils import init_logging, add_shared_misc_options, process_shared_misc_options
from resync.explorer import Explorer
DEFAULT_LOGFILE = 'resync-explorer.log'
def main():
"""Main function to implement command line script."""
if (sys.version_info < (3, 5)):
sys.exit("This program requires python version 3.5 or later")
# Options and arguments
parser = argparse.ArgumentParser(
description='ResourceSync explorer (v' + __version__ + ')',
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
# Specification of map between remote URI and local file paths, and remote
# sitemap
nam = parser.add_argument_group('FILE/URI NAMING OPTIONS')
nam.add_argument('--outfile', type=str, action='store',
help="write sitemap to specified file rather than STDOUT")
nam.add_argument('--paths', type=str, action='store',
help="explicit set of paths for disk scan --resourceslist or --changelist "
"generation")
nam.add_argument('--sitemap', type=str, action='store',
help="explicitly set sitemap name, overriding default sitemap.xml "
"appended to first source URI specified in the mappings")
nam.add_argument('--reference', type=str, action='store',
help="reference sitemap name for --changelist calculation")
nam.add_argument('--newreference', type=str, action='store',
help="updated reference sitemap name for --changelist calculation")
nam.add_argument('--dump', metavar='DUMPFILE', type=str, action='store',
help="write dump to specified file for --resourcelist or --changelist")
nam.add_argument('--changelist-uri', '--change-list-uri', type=str, action='store',
help="explicitly set the changelist URI that will be use in --inc mode, "
"overrides process of getting this from the sitemap")
# Options that apply to multiple modes
opt = parser.add_argument_group('MISCELANEOUS OPTIONS')
opt.add_argument('--max-sitemap-entries', type=int, action='store',
help="override default size limits")
add_shared_misc_options(opt, default_logfile=DEFAULT_LOGFILE, include_remote=True)
args = parser.parse_args()
init_logging(to_file=args.logger, logfile=args.logfile, default_logfile=DEFAULT_LOGFILE,
verbose=args.verbose)
print("----- ResourceSync Explorer -----")
process_shared_misc_options(args, include_remote=True)
c = Explorer(hashes=args.hash,
verbose=args.verbose)
try:
if (args.sitemap):
c.sitemap_name = args.sitemap
if (args.exclude):
c.exclude_patterns = args.exclude
if (args.multifile):
c.allow_multifile = not args.multifile
if (args.noauth):
c.noauth = args.noauth
if (args.max_sitemap_entries):
c.max_sitemap_entries = args.max_sitemap_entries
c.explore()
# Any problem we expect will come as a ClientFatalError, anything else
# is... an exception ;-)
except ClientFatalError as e:
sys.stderr.write("\nFatalError: " + str(e) + "\n")
if __name__ == '__main__':
main()

163
resync-sync
View File

@ -1,163 +0,0 @@
#!/usr/bin/env python
"""resync-sync: The ResourceSync command line synchronization client.
Copyright 2012-2020 Simeon Warner
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License
"""
import argparse
import sys
from resync import __version__
from resync.client import Client, ClientFatalError
from resync.client_utils import init_logging, count_true_args, parse_links, parse_capabilities, parse_capability_lists, add_shared_misc_options, process_shared_misc_options
DEFAULT_LOGFILE = 'resync-client.log'
def main():
"""Main function to implement command line script."""
if (sys.version_info < (3, 5)):
sys.exit("This program requires python version 3.5 or later")
# Options and arguments
parser = argparse.ArgumentParser(
description="ResourceSync command line client (v" + __version__ + ")",
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser._optionals = parser.add_argument_group(
'MODES OF OPERATION (must specify one only). A source that is specified '
'either in a set of uri=path mappings or else using an explicit --sitemap '
'location')
rem = parser.add_mutually_exclusive_group(required=True)
rem.add_argument('--baseline', '-b', action='store_true',
help='baseline sync of resources from remote source (src) to local filesystem (dst)')
rem.add_argument('--incremental', '--inc', '-i', action='store_true',
help='incremental sync of resources from remote source (src) to local filesystem (dst). Uses either timestamp recorded from last baseline or incremental sync for this source, or explicit --from parameter, to determine the earlier update timestamp to act on.')
rem.add_argument('--audit', '-a', action='store_true',
help="audit sync state of destination wrt source")
rem.add_argument('--parse', '-p', action='store_true',
help="parse a remote sitemap/sitemapindex (from mapping or explicit --sitemap) and show summary information including document type and number of entries")
# Positional arguments
map = parser.add_argument_group('URI MAPPING TO FILESYSTEM for REMOTE modes')
map.add_argument(metavar='uri=path | uri path', dest='map', type=str, nargs='*',
help="remote URI of source for remote synchronization operations (may "
"also combine uri=local path)")
# Specification of map between remote URI and local file paths, and remote
# sitemap
nam = parser.add_argument_group('FILE/URI NAMING OPTIONS')
nam.add_argument('--sitemap', type=str, action='store',
help="explicitly set sitemap name, overriding default sitemap.xml "
"appended to first source URI specified in the mappings")
nam.add_argument('--capabilitylist', '--capability-list', type=str, action='store',
help="explicitly set capability list URI to search for instead of "
"looking for the source description")
nam.add_argument('--reference', type=str, action='store',
help="reference sitemap name for --write-changelist calculation")
nam.add_argument('--newreference', type=str, action='store',
help="updated reference sitemap name for --write-changelist calculation")
nam.add_argument('--changelist-uri', '--change-list-uri', type=str, action='store',
help="explicitly set the changelist URI that will be use in --inc mode, "
"overrides process of getting this from the sitemap")
# Options that apply to multiple modes
opt = parser.add_argument_group('MISCELANEOUS OPTIONS')
add_shared_misc_options(opt, default_logfile=DEFAULT_LOGFILE, include_remote=True)
opt.add_argument('--delete', action='store_true',
help="allow files on destination to be deleted")
opt.add_argument('--empty', action='store_true',
help="combine with --changelist to write and empty changelist, perhaps with links")
opt.add_argument('--strictauth', action='store_true',
help="use more strict checking of URLs to ensure that the ResourceSync "
"documents refer only to resources on the same server or sub-domains, "
"and on the same server to sub-paths. This is the authority model "
"of Sitemaps but there are legitimate uses where these rules would "
"not be followed.")
opt.add_argument('--dryrun', '-n', action='store_true',
help="don't update local resources, say what would be done")
opt.add_argument('--ignore-failures', action='store_true',
help="continue past download failures")
# These likely only useful for experimentation
opt.add_argument('--max-sitemap-entries', type=int, action='store',
help="override default size limits")
opt.add_argument('--eval', '-e', action='store_true',
help="output evaluation of source/client synchronization performance... "
"be warned, this is very verbose")
opt.add_argument('--tries', '-t', type=int, action='store', metavar='TRIES',
help="set number of tries to TRIES. The default is to retry 20 times, "
"with the exception of fatal errors like \"connection refused\" "
"or \"not found\" (404), which are not retried.")
opt.add_argument('--timeout', '-T', type=int, action='store', metavar='SECONDS',
help="set the request timeout for resource downloads to SECONDS seconds")
args = parser.parse_args()
# Configure logging module and create logger instance
init_logging(to_file=args.logger, logfile=args.logfile, default_logfile=DEFAULT_LOGFILE,
verbose=args.verbose, eval_mode=args.eval)
process_shared_misc_options(args, include_remote=True)
c = Client(spec_version=args.spec_version,
hashes=args.hash,
verbose=args.verbose,
dryrun=args.dryrun)
try:
if (args.map):
# Mappings apply to (almost) everything
c.set_mappings(args.map)
if (args.sitemap):
c.sitemap_name = args.sitemap
if (args.capabilitylist):
c.capability_list_uri = args.capabilitylist
if (args.exclude):
c.exclude_patterns = args.exclude
if (args.multifile):
c.allow_multifile = not args.multifile
if (args.noauth):
c.noauth = args.noauth
if (args.strictauth):
c.strictauth = args.strictauth
if (args.max_sitemap_entries):
c.max_sitemap_entries = args.max_sitemap_entries
if (args.ignore_failures):
c.ignore_failures = args.ignore_failures
if (args.tries):
c.tries = args.tries
if (args.timeout):
c.timeout = args.timeout
# Finally, do something...
if (args.baseline or args.audit):
c.baseline_or_audit(allow_deletion=args.delete,
audit_only=args.audit)
elif (args.incremental):
c.incremental(allow_deletion=args.delete,
change_list_uri=args.changelist_uri,
from_datetime=args.from_datetime)
elif (args.parse):
c.parse_document()
else:
parser.error("Unknown mode requested")
# Any problem we expect will come as a ClientFatalError, anything else
# is... an exception ;-)
except ClientFatalError as e:
sys.stderr.write("\nFatalError: " + str(e) + "\n")
if __name__ == '__main__':
main()

17
resync/__init__.py
View File

@ -1,17 +0,0 @@
"""Module config for resync.
This is the one place the version number for resync is stored.
"""
__version__ = '2.0.1'
# Enable easy import for core classes, e.g.
# from resync import Resource
from .source_description import SourceDescription
from .capability_list import CapabilityList
from .resource_list import ResourceList
from .change_list import ChangeList
from .resource_dump import ResourceDump
from .resource_dump_manifest import ResourceDumpManifest
from .change_dump import ChangeDump
from .archives import ResourceListArchive, ResourceDumpArchive, ChangeListArchive, ChangeDumpArchive
from .resource import Resource

72
resync/archives.py
View File

@ -1,72 +0,0 @@
"""ResourceSync Archive objects.
The ResourceSync Archives specification
http://www.openarchives.org/rs/archives specifies capabilities
that provide archives of the 4 core capabilities. While some
optional attributes are different the basic structure is the
same for the Resource List Archive, Change List Archive,
Resource Dump Archive, and Change Dump Archive.
"""
import collections
from .list_base_with_index import ListBaseWithIndex
from .resource import Resource
from .sitemap import Sitemap
class ResourceListArchive(ListBaseWithIndex):
"""Class representing an Resource List Archive."""
def __init__(self, resources=None, md=None, ln=None, uri=None,
spec_version='1.1', add_lastmod=False,
resources_class=None):
"""Initialize ResourceListArchive."""
self.resources_class = list if resources_class is None else resources_class
if (resources is None):
resources = self.resources_class()
super(ResourceListArchive, self).__init__(
resources=resources, md=md, ln=ln, uri=uri,
capability_name='resourcelist-archive',
spec_version=spec_version, add_lastmod=add_lastmod)
class ChangeListArchive(ListBaseWithIndex):
"""Class representing an Change List Archive."""
def __init__(self, resources=None, md=None, ln=None, uri=None,
spec_version='1.1', add_lastmod=False,
resources_class=None):
"""Initialize ChangeListArchive."""
super(ChangeListArchive, self).__init__(
resources=resources, md=md, ln=ln, uri=uri,
capability_name='changelist-archive',
spec_version=spec_version, add_lastmod=add_lastmod)
class ResourceDumpArchive(ListBaseWithIndex):
"""Class representing an Resource Dump Archive."""
def __init__(self, resources=None, md=None, ln=None, uri=None,
spec_version='1.1', add_lastmod=False,
resources_class=None):
"""Initialize ResourceDumpArchive."""
super(ResourceDumpArchive, self).__init__(
resources=resources, md=md, ln=ln, uri=uri,
capability_name='resourcedump-archive',
spec_version=spec_version, add_lastmod=add_lastmod,
resources_class=resources_class)
class ChangeDumpArchive(ListBaseWithIndex):
"""Class representing an Change Dump Archive."""
def __init__(self, resources=None, md=None, ln=None, uri=None,
spec_version='1.1', add_lastmod=False,
resources_class=None):
"""Initialize ChangeDumpArchive."""
super(ChangeDumpArchive, self).__init__(
resources=resources, md=md, ln=ln, uri=uri,
capability_name='changedump-archive',
spec_version=spec_version, add_lastmod=add_lastmod,
resources_class=resources_class)

138
resync/capability_list.py
View File

@ -1,127 +1,73 @@
"""ResourceSync Capability List object.
"""ResourceSync capability_list object
An Capability List is a set of capabilitys with some metadata for
each capability. The Capability List object may also contain metadata
and links like other lists.
An capability_list is a set of capabilitys with some metadata for each
capability.
The capability_list object may also contain metadata and links.
"""
import collections.abc
from .resource import Resource
from .resource_set import ResourceSet
from .list_base import ListBase
from .sitemap import Sitemap
class CapabilitySet(ResourceSet):
"""Class for storage of resources in a Capability List.
Extends the ResourceSet to add checks to ensure that there are
never two entries for the same resource, and that values are
returned in the canonical order.
"""
def __init__(self):
"""Initialize CpabilitySet."""
self.order = ['resourcelist', 'resourcedump',
'changelist', 'changedump',
'resourcelist-archive', 'resourcedump-archive',
'changelist-archive', 'changedump-archive']
def __iter__(self):
"""Iterator over all the resources in capability order.
Deals with the case of unknown capabilities or duplicate entries
by using uri order for duplicates and adding any unknown ones
at the end
"""
self._iter_next_list = []
# look through all resources and build capability to uri index
uris = {}
for uri in self.keys():
cap = self[uri].capability
if (cap not in uris):
uris[cap] = []
uris[cap].append(uri)
# build list or uris in defined order for iterator
for cap in self.order:
if (cap in uris):
for uri in sorted(uris[cap]):
self._iter_next_list.append(uri)
del uris[cap]
# add any left over capabilities we don't know about in alphabetical
# order
for cap in uris:
for uri in sorted(uris[cap]):
self._iter_next_list.append(uri)
self._iter_next_list.reverse()
return(iter(self._iter_next, None))
def _iter_next(self):
if (len(self._iter_next_list) > 0):
return(self[self._iter_next_list.pop()])
else:
return(None)
import collections
from resource import Resource
from resource_set import ResourceSet
from list_base import ListBase
from sitemap import Sitemap
class CapabilityList(ListBase):
"""Class representing a Capability List.
"""Class representing an capability_list of resources
An Capability List will admit only one resource with any given
URI. The iterator over resources is expected to return them in
canonical order of capability names as defined in main specification
section 7 and archives specification section 6.
This same class is used for both the source and the destination
and is the central point of comparison the decide whether they
are in sync or what needs to be copied to bring the destinaton
into sync.
An capability_list will admit only one resource with any given URI.
Storage is unordered but the iterator imposes a canonical order
which is currently alphabetical by URI.
"""
def __init__(self, resources=None, md=None, ln=None, uri=None,
spec_version='1.1', add_lastmod=False):
"""Initialize CapabilityList."""
def __init__(self, resources=None, md=None, ln=None):
if (resources is None):
resources = CapabilitySet()
super(CapabilityList, self).__init__(
resources=resources, md=md, ln=ln, uri=uri,
capability_name='capabilitylist',
spec_version=spec_version,
add_lastmod=add_lastmod)
resources = ResourceSet()
super(CapabilityList, self).__init__(resources=resources, md=md, ln=ln)
self.capability_name='capabilitylist'
self.capability_md='capabilitylist'
def add(self, resource, replace=False):
"""Add a resource or an iterable collection of resources.
"""Add a resource or an iterable collection of resources
Will throw a ValueError if the resource (ie. same uri) already
exists in the capability_list, unless replace=True.
See add_capability() for normal method of adding capabilities.
"""
if isinstance(resource, collections.abc.Iterable):
if isinstance(resource, collections.Iterable):
for r in resource:
self.resources.add(r, replace)
self.resources.add(r,replace)
else:
self.resources.add(resource, replace)
def add_capability(self, capability=None, uri=None, name=None):
"""Specific add function for capabilities.
self.resources.add(resource,replace)
def add_capability(self,capability=None,uri=None,name=None):
"""Specific add function for capabilities
Takes either:
- a capability object (derived from ListBase) as the first argument
from which the capability name is extracted, and the URI if given
- a capability object (derived from ListBase) as the first argument
from which the capability name is extracted
- or a plain name string
and
- the URI of the capability
"""
if (capability is not None):
name = capability.capability_name
if (capability.uri is not None):
uri = capability.uri
self.add(Resource(uri=uri, capability=name))
name = capability.capability_md
self.add( Resource(uri=uri,capability=name) )
def has_capability(self, name=None):
"""True if the Capability List includes the named capability."""
return(self.capability_info(name) is not None)
def has_capability(self,name=None):
"""True if the Capability List includes the named capability"""
return( self.capability(name) is not None )
def capability_info(self, name=None):
"""Return information about the requested capability from this list.
def capability(self,name=None):
"""Return information about the requested capability from this list
Will return None if there is no information about the requested capability.
Will return None if there is no information about the requested capability
"""
for r in self.resources:
if (r.capability == name):

48
resync/capability_list_index.py Normal file
View File

@ -0,0 +1,48 @@
"""ResourceSync capability list index object
A capability list index is a set of capability lists which
may also contain metadata and links.
"""
import collections
from resource import Resource
from resource_set import ResourceSet
from list_base import ListBase
class CapabilityListIndex(ListBase):
"""Class representing an capability list index
An capability list index will admit only one resource with any
given URI.
Storage is unordered but the iterator imposes a canonical order
which is currently alphabetical by URI.
"""
def __init__(self, resources=None, md=None, ln=None):
if (resources is None):
resources = ResourceSet()
super(CapabilityListIndex, self).__init__(resources=resources, md=md, ln=ln)
self.capability_name='capabilitylistindex'
self.capability_md='capabilitylist'
self.sitemapindex=True
def add(self, resource, replace=False):
"""Add a resource or an iterable collection of resources
Will throw a ValueError if the resource (ie. same uri) already
exists in the capability_list, unless replace=True.
"""
if isinstance(resource, collections.Iterable):
for r in resource:
self.resources.add(r,replace)
else:
self.resources.add(resource,replace)
def add_capability_list(self,uri):
"""Specific add function for capability_lists
Takes uri and adds Resource with that URI
"""
self.add( Resource(uri=uri) )

35
resync/change_dump.py
View File

@ -1,35 +0,0 @@
"""ResourceSync Change Dump object.
A Change Dump is a set of content dump package resources
with some metadata for each resource.
The Change Dump may also contain metadata and links like
other ResourceSync documents.
Described in specification at:
http://www.openarchives.org/rs/resourcesync#ChangeDump
"""
from .resource_list import ResourceList
class ChangeDump(ResourceList):
"""Class representing an Change Dump.
A ChangeDump comprises a set of content dump packages that
are the resources listed. Properties similar to a ResourceList
and implemented as a sub-class of ResourceList.
"""
def __init__(self, resources=None, md=None, ln=None,
uri=None, allow_multifile=None, mapper=None,
spec_version='1.1', add_lastmod=False):
"""Initialize ChangeDump.
Simply sets capability_name to 'changedump' when
subclassing ResourceList.
"""
super(ChangeDump, self).__init__(
resources=resources, md=md, ln=ln, uri=uri, mapper=mapper,
spec_version=spec_version, add_lastmod=add_lastmod)
self.capability_name = 'changedump'

36
resync/change_dump_manifest.py
View File

@ -1,36 +0,0 @@
"""ResourceSync ChangeDumpManifest object.
A ChangeDumpManifest lists the set of files/resources included
within a content package that is included in a ChangeDump.
The ChangeeDumpManifest object will include the change type,
a path for each item (except in the case of "deleted"), and may
also contain metadata and links.
Described in specification at:
http://www.openarchives.org/rs/resourcesync#ChangeDumpManifest
"""
from .change_list import ChangeList
class ChangeDumpManifest(ChangeList):
"""Class representing a Change Dump Manifest.
A ChangeDumpManifest comprises a set of files/resources
in a content package. Properties much Like a ChangeList
and implemented as a sub-class of ChangeeList.
"""
def __init__(self, resources=None, md=None, ln=None,
uri=None, allow_multifile=None, mapper=None,
spec_version='1.1', add_lastmod=False):
"""Initialize ChangeDumpManifest.
Simply sets capability_name to 'changedump-manifest' when
subclassing ChangeList.
"""
super(ChangeDumpManifest, self).__init__(
resources=resources, md=md, ln=ln, uri=uri, mapper=mapper,
spec_version=spec_version, add_lastmod=add_lastmod)
self.capability_name = 'changedump-manifest'

85
resync/change_list.py
View File

@ -1,4 +1,4 @@
"""ResourceSync ChangeList object.
"""ResourceSync ChangeList object
A ChangeList is a list of resource descriptions which includes
both metadata associated with the resource at some point in
@ -8,79 +8,48 @@ are Resource objects.
Different from an resource_list, a change_list may include multiple
descriptions for the same resource. The change_list is ordered
from first entry to last entry.
from first entry to last entry.
Different from an resource_list, dereference by a URI yields a
ChangeList containing descriptions pertaining to that
Different from an resource_list, dereference by a URI yields a
ChangeList containing descriptions pertaining to that
particular resource.
"""
import collections.abc
import collections
from urllib import URLopener
from .list_base_with_index import ListBaseWithIndex
from .resource import Resource, ChangeTypeError
from .sitemap import Sitemap
from list_base import ListBase
from resource import Resource
from sitemap import Sitemap
class ChangeList(ListBase):
"""Class representing an Change List"""
class ChangeList(ListBaseWithIndex):
"""Class representing an Change List."""
def __init__(self, resources=None, md=None, ln=None, uri=None,
mapper=None, spec_version='1.1', add_lastmod=False,
resources_class=list):
"""Initialize ChangeList."""
super(ChangeList, self).__init__(
resources=resources, md=md, ln=ln, uri=uri,
capability_name='changelist', mapper=mapper,
spec_version=spec_version, add_lastmod=add_lastmod,
resources_class=resources_class)
def add_if_changed(self, resource):
"""Add resource if change is not None else ChangeTypeError."""
if (resource.change is not None):
self.resources.append(resource)
else:
raise ChangeTypeError(resource.change)
def __init__(self, resources=None, md=None, ln=None):
if (resources is None):
resources = list()
super(ChangeList, self).__init__(resources=resources, md=md, ln=ln)
self.capability_name='changelist'
self.capability_md='changelist'
def add(self, resource):
"""Add a resource change or an iterable collection of them.
Allows multiple resource_change objects for the same
resource (ie. URI) and preserves the order of addition.
"""Add a resource_change or an iterable collection to this ChangeList
Allows multiple resourec_change objects for the same resource (ie. URI) and
preserves the order of addition.
"""
if isinstance(resource, collections.abc.Iterable):
if isinstance(resource, collections.Iterable):
for r in resource:
self.add_if_changed(r)
self.resources.append(r)
else:
self.add_if_changed(resource)
self.resources.append(resource)
def add_changed_resources(self, resources, change=None):
"""Add items from a ResourceContainer resources.
"""Add items from a ResourceContainer resources to this ChangeList
If change is specified then the attribute is set in the Resource
If change is specified then the attribute is set in the Resource
objects created.
"""
for resource in resources:
rc = Resource(resource=resource, change=change)
rc = Resource( resource=resource, change=change )
self.add(rc)
def prune_updates_before(self, timestamp, spec_version='1.1'):
"""Remove all resource updates earlier than the given timestamp.
Returns the number of entries removed. Will raise an excpetion
if there are any entries without a datetime (1.1) or
timestamp (1.0).
"""
n = 0
pruned = []
use_timestamp = (spec_version == '1.0') # Else use datetime
for r in self.resources:
ts = r.timestamp if use_timestamp else r.ts_datetime
if (ts is None):
raise Exception("Entry %s has no update datetime/timestamp" % (r.uri))
elif (ts >= timestamp):
pruned.append(r)
else:
n += 1
self.resources = pruned
return(n)

975
resync/client.py
View File

File diff suppressed because it is too large Load Diff

57
resync/client_state.py
View File

@ -1,4 +1,4 @@
"""ResourceSync client state class.
"""ResourceSync client state class
The client requires memory of state to support incremental
synchronization. At minimum it must store the source timestamp
@ -6,70 +6,55 @@ of the last change seen.
"""
import sys
import urllib
import urlparse
import os.path
import datetime
import distutils.dir_util
import distutils.dir_util
import re
import time
import logging
try: # python3
from configparser import ConfigParser, NoSectionError, NoOptionError
except ImportError: # pragma: no cover python2
from ConfigParser import SafeConfigParser as ConfigParser, NoSectionError, NoOptionError # pragma: no cover
import ConfigParser
class ClientState(object):
"""Read and store client state on disk."""
"""
"""
def __init__(self):
"""Initialize ClientState object with default status file name."""
self.status_file = '.resync-client-status.cfg'
def set_state(self, site, timestamp=None):
"""Write status dict to client status file.
def set_state(self,site,timestamp=None):
"""Write status dict to client status file
FIXME - should have some file lock to avoid race
"""
parser = ConfigParser()
parser = ConfigParser.SafeConfigParser()
parser.read(self.status_file)
status_section = 'incremental'
if (not parser.has_section(status_section)):
parser.add_section(status_section)
if (timestamp is None):
parser.remove_option(
status_section,
self.config_site_to_name(site))
parser.remove_option(status_section, self.config_site_to_name(site))
else:
parser.set(
status_section,
self.config_site_to_name(site),
str(timestamp))
with open(self.status_file, 'w') as configfile:
parser.set(status_section, self.config_site_to_name(site), str(timestamp))
with open(self.status_file, 'wb') as configfile:
parser.write(configfile)
configfile.close()
def get_state(self, site):
"""Read client status file and return dict."""
parser = ConfigParser()
def get_state(self,site):
"""Read client status file and return dict"""
parser = ConfigParser.SafeConfigParser()
status_section = 'incremental'
parser.read(self.status_file)
timestamp = None
try:
timestamp = float(
parser.get(
status_section,
self.config_site_to_name(site)))
except NoSectionError as e:
timestamp = float(parser.get(status_section,self.config_site_to_name(site)))
except ConfigParser.NoSectionError as e:
pass
except NoOptionError as e:
except ConfigParser.NoOptionError as e:
pass
return(timestamp)
def config_site_to_name(self, name):
"""Convert site name to safe string for config.
Simply replaces any non-word chars with underscore. This could
lead to multiple site names mapping to one config name but
probably not too likely.
"""
return(re.sub(r"[^\w]", '_', name))
return( re.sub(r"[^\w]",'_',name) )

222
resync/client_utils.py
View File

@ -1,222 +0,0 @@
"""ResourceSync Client Utilities.
Code shared by client scripts.
"""
import argparse
from datetime import datetime
import logging
import logging.config
from .url_or_file_open import set_url_or_file_open_config
class ClientFatalError(Exception):
"""Non-recoverable error in client, should include message to user."""
pass
class ClientError(Exception):
"""Possibly recoverable error in client, should include message to user."""
pass
class UTCFormatter(logging.Formatter):
"""Format datetime values as ISO8601 UTC Z form.
Based on http://bit.ly/T2n3Xk
"""
def formatTime(self, record, datefmt=None):
"""Format datetime of record.created as ISO8601 UTC Z form."""
timestamp = record.created
return datetime.utcfromtimestamp(timestamp).isoformat() + 'Z'
def init_logging(to_file=False, logfile=None, default_logfile='/tmp/resync.log',
human=True, verbose=False, eval_mode=False,
default_logger='client', extra_loggers=None):
"""Initialize logging.
Use of log levels:
DEBUG - very verbose, for evaluation of output (-e)
INFO - verbose, only seen by users if they ask for it (-v)
WARNING - messages output messages to console
Logging to a file: If to_file is True then output will be written to
a file. This will be logfile if set, else default_logfile (which may
also be overridden).
"""
fmt = '%(asctime)s | %(name)s | %(levelname)s | %(message)s'
formatter = UTCFormatter(fmt)
if human:
# Create a special handler designed just for human readable output
hh = logging.StreamHandler()
hh.setLevel(logging.INFO if (verbose) else logging.WARNING)
hh.setFormatter(logging.Formatter(fmt='%(message)s'))
if to_file:
if (logfile is None):
logfile = default_logfile
fh = logging.FileHandler(filename=logfile, mode='a')
fh.setFormatter(formatter)
fh.setLevel(logging.DEBUG if (eval_mode) else logging.INFO)
loggers = [default_logger, 'resync']
if (extra_loggers is not None):
for logger in extra_loggers:
loggers.append(logger)
for logger in loggers:
log = logging.getLogger(logger)
log.setLevel(logging.DEBUG) # control at handler instead
if human:
log.addHandler(hh)
if to_file:
log.addHandler(fh)
log = logging.getLogger(default_logger)
if (to_file):
log.info("Writing detailed log to %s" % (logfile))
def count_true_args(*args):
"""Count number of list of arguments that evaluate True."""
count = 0
for arg in args:
if (arg):
count += 1
return(count)
def parse_links(args_link):
"""Parse --link options.
Uses parse_link() to parse each option.
"""
links = []
if (args_link is not None):
for link_str in args_link:
try:
links.append(parse_link(link_str))
except ClientFatalError as e:
raise ClientFatalError(
"Bad --link option '%s' (%s)" %
(link_str, str(e)))
return(links)
def parse_link(link_str):
"""Parse one --link option to add to <rs:ln> links.
Input string of the form: rel,href,att1=val1,att2=val2
"""
atts = {}
help_str = "--link option '%s' (format rel,href,att1=val1...)" % (link_str)
try:
segs = link_str.split(',')
# First segments are relation and subject
atts['rel'] = segs.pop(0)
atts['href'] = segs.pop(0)
if (atts['href'] == ''):
raise ClientFatalError("Missing uri in " + help_str)
# Remaining segments are attributes
for term in segs:
(k, v) = term.split('=')
if (k == '' or v == ''):
raise ClientFatalError(
"Bad attribute (%s) in " %
(term) + help_str)
atts[k] = v
except ValueError as e:
raise ClientFatalError("Bad component of " + help_str)
except IndexError as e:
raise ClientFatalError("Incomplete component of " + help_str)
return(atts)
def parse_capabilities(caps_str):
"""Parse list of capabilities in --capabilitylist option.
Input string of the form: cap_name=uri,cap_name=uri
"""
capabilities = {}
try:
segs = caps_str.split(',')
for term in segs:
(k, v) = term.split('=')
capabilities[k] = v
except ValueError as e:
raise ClientFatalError(
"Bad component of --capabilitylist option '%s' (%s)" %
(caps_str, str(e)))
return(capabilities)
def parse_capability_lists(cls_str):
"""Parse list of capability lists in --capabilitylistindex option.
Input string of the form: uri,uri
"""
return(cls_str.split(','))
def add_shared_misc_options(opt, default_logfile, include_remote=False):
"""Add shared miscellaneous options to the argument_group opt.
Options that the resync-sync, resync-build and resync-explorer scripts use.
"""
opt.add_argument('--hash', type=str, action='append',
help="use specified hash types in addition to last modification time "
"and size (repeatable, may include `md5`, `sha-1` and `sha-256`)")
opt.add_argument('--checksum', action='store_true',
help="use md5 checksum in addition to last modification time and size "
"(same as --hash=md5)")
opt.add_argument('--from', type=str, action='store', dest='from_datetime', metavar="DATETIME",
help="explicit datetime value used to filter updates in change list for "
"--incremental sync")
opt.add_argument('--exclude', type=str, action='append',
help="exclude resources with URI or filename matching the python regex "
"supplied (see: <https://docs.python.org/3/howto/regex.html> for regex "
"information, repeat option for multiple excludes)")
opt.add_argument('--multifile', '-m', action='store_true',
help="disable reading and output of sitemapindex for multifile sitemap")
if include_remote:
opt.add_argument('--noauth', action='store_true',
help="disable checking of URL paths to ensure that the sitemaps refer "
"only to resources on the same server/sub-path etc. Use with care.")
opt.add_argument('--access-token', type=str, default=None,
help="include this access token (a bearer token) in web requests")
opt.add_argument('--delay', type=float, default=None,
help="add a delay between web requests (default is None)")
# Want these to show at the end
opt.add_argument('--logger', '-l', action='store_true',
help="create detailed log of client actions (will write "
"to %s unless specified with --logfile" % (default_logfile))
opt.add_argument('--logfile', type=str, action='store',
help="create detailed log of client actions")
opt.add_argument('--spec-version', default='1.1', choices=('1.0', '1.1'),
help="follow given ResourceSync specification version. The key difference is that v1.0 "
"used lastmod for the time of a change (often also the resource Last-Modification "
"time but not always). In v1.1 the rs:md datetime attribute in a ChangeList "
"indicates the time of the change, and use of lastmod is entirely optional")
opt.add_argument('--verbose', '-v', action='store_true',
help="verbose, show additional informational messages")
def process_shared_misc_options(args, include_remote=False):
"""Process shared miscellaneous options in args.
Parse options that the resync-sync, resync-build and resync-explorer scripts use.
"""
if args.checksum:
args.hash.append('md5')
if include_remote:
if args.access_token:
set_url_or_file_open_config('bearer_token', args.access_token)
if args.delay:
if args.delay < 0.0:
raise argparse.ArgumentTypeError("--delay must be non-negative!")
set_url_or_file_open_config('delay', args.delay)

223
resync/dump.py
View File

@ -1,106 +1,61 @@
"""Dump handler for ResourceSync."""
"""Dump handler for ResourceSync"""
import logging
import os.path
from zipfile import ZipFile, ZIP_STORED, ZIP_DEFLATED
from resync.resource_dump_manifest import ResourceDumpManifest
from sitemap import Sitemap
class DumpError(Exception):
pass
class Dump(object):
"""Dump of content for a Resource Dump or Change Dump.
"""Dump of resource content associated with an resource_list or change set
The resources itearable must be comprised of Resource objects
The resource_list must be comprised of Resource objects
which have the path attributes set to indicate the local
location of the copies of the resources.
rl = ResourceList()
# ... add items by whatever means, may have >50k items and/or
# >100MB total size of files ...
d = Dump(resources=rl)
d.write(basename="/tmp/rd_")
# will create dump files /tmp/rd_00000.zip etc.
"""
def __init__(self, resources=None, format=None, compress=True):
"""Initialize Dump object and set defaults."""
self.resources = resources
def __init__(self, format=None, compress=True):
self.format = ('zip' if (format is None) else format)
self.compress = compress
self.manifest_class = ResourceDumpManifest # FIXME
self.max_size = 100 * 1024 * 1024 # 100MB
self.max_size = 100*1024*1024 #100MB
self.max_files = 50000
self.path_prefix = None
self.logger = logging.getLogger('resync.dump')
def write(self, resource_list=None, dumpfile=None):
"""Write a dump file"""
self.check_files(resource_list)
if (self.format == 'zip'):
self.write_zip(resource_list,dumpfile)
elif (self.format == 'warc'):
self.write_warc(resource_list,dumpfile)
else:
raise DumpError("Unknown dump format '%s'" % (self.format))
def write(self, basename=None, write_separate_manifests=True):
"""Write one or more dump files to complete this dump.
Returns the number of dump/archive files written.
"""
self.check_files()
n = 0
for manifest in self.partition_dumps():
dumpbase = "%s%05d" % (basename, n)
dumpfile = "%s.%s" % (dumpbase, self.format)
if (write_separate_manifests):
manifest.write(basename=dumpbase + '.xml')
if (self.format == 'zip'):
self.write_zip(manifest.resources, dumpfile)
elif (self.format == 'warc'):
self.write_warc(manifest.resources, dumpfile)
else:
raise DumpError(
"Unknown dump format requested (%s)" %
(self.format))
n += 1
self.logger.info("Wrote %d dump files" % (n))
return(n)
def write_zip(self, resources=None, dumpfile=None):
"""Write a ZIP format dump file.
Writes a ZIP file containing the resources in the iterable resources along with
a manifest file manifest.xml (written first). No checks on the size of files
or total size are performed, this is expected to have been done beforehand.
"""
compression = (ZIP_DEFLATED if self.compress else ZIP_STORED)
zf = ZipFile(
dumpfile,
mode="w",
compression=compression,
allowZip64=True)
# Write resources first
rdm = ResourceDumpManifest(resources=resources)
real_path = {}
for resource in resources:
archive_path = self.archive_path(resource.path)
real_path[archive_path] = resource.path
resource.path = archive_path
zf.writestr('manifest.xml', rdm.as_xml())
# Add all files in the resources
for resource in resources:
zf.write(real_path[resource.path], arcname=resource.path)
def write_zip(self, resource_list=None, dumpfile=None):
"""Write a ZIP dump file"""
compression = ( ZIP_DEFLATED if self.compress else ZIP_STORED )
zf = ZipFile(dumpfile, mode="w", compression=compression, allowZip64=True)
# Write resource_list first
s = Sitemap(pretty_xml=True, allow_multifile=False)
zf.writestr('manifest.xml',s.resources_as_xml(resource_list))
# Add all files in the resource_list
for resource in resource_list:
zf.write(resource.uri)
zf.close()
zipsize = os.path.getsize(dumpfile)
self.logger.info(
"Wrote ZIP file dump %s with size %d bytes" %
(dumpfile, zipsize))
def write_warc(self, resources=None, dumpfile=None):
"""Write a WARC dump file.
WARC support is not part of ResourceSync v1.0 (Z39.99 2014) but is left
in this library for experimentation.
"""
# Load library late as we want to be able to run rest of code
print "Wrote ZIP file dump %s with size %d bytes" % (dumpfile,zipsize)
def write_warc(self, resource_list=None, dumpfile=None):
"""Write a WARC dump file"""
# Load library late as we want to be able to run rest of code
# without this installed
try:
from warc import WARCFile, WARCHeader, WARCRecord
except ImportError:
from warc import WARCFile,WARCHeader,WARCRecord
except:
raise DumpError("Failed to load WARC library")
wf = WARCFile(dumpfile, mode="w", compress=self.compress)
# Add all files in the resources
for resource in resources:
# Add all files in the resource_list
for resource in resource_list:
wh = WARCHeader({})
wh.url = resource.uri
wh.ip_address = None
@ -108,95 +63,23 @@ class Dump(object):
wh.content_type = 'text/plain'
wh.result_code = 200
wh.checksum = 'aabbcc'
wh.location = self.archive_path(resource.path)
wf.write_record(WARCRecord(header=wh, payload=resource.path))
wh.location = 'loc'
wf.write_record( WARCRecord( header=wh, payload=resource.path ) )
wf.close()
warcsize = os.path.getsize(dumpfile)
self.logging.info(
"Wrote WARC file dump %s with size %d bytes" %
(dumpfile, warcsize))
def check_files(self, set_length=True, check_length=True):
"""Check all files in self.resources, find longest common prefix.
Go though and check all files in self.resources, add up size, and find
longest common path that can be used when writing the dump file. Saved in
self.path_prefix.
Parameters set_length and check_length control control whether then set_length
attribute should be set from the file size if not specified, and whether any
length specified should be checked. By default both are True. In any event, the
total size calculated is the size of files on disk.
"""
total_size = 0 # total size of all files in bytes
path_prefix = None
for resource in self.resources:
print "Wrote WARC file dump %s with size %d bytes" % (dumpfile,warcsize)
def check_files(self,resource_list):
"""Go though and check all files in resource_list, add up size"""
if (len(resource_list) > self.max_files):
raise DumpError("Number of files to dump (%d) exceeds maximum (%d)" % (len(resource_list),self.max_files))
total_size = 0 #total size of all files in bytes
for resource in resource_list:
if (resource.path is None):
# explicit test because exception raised by getsize otherwise
# confusing
raise DumpError(
"No file path defined for resource %s" %
resource.uri)
if (path_prefix is None):
path_prefix = os.path.dirname(resource.path)
else:
path_prefix = os.path.commonprefix(
[path_prefix, os.path.dirname(resource.path)])
size = os.path.getsize(resource.path)
if (resource.length is not None):
if (check_length and resource.length != size):
raise DumpError("Size of resource %s is %d on disk, not %d as specified" %
(resource.uri, size, resource.length))
elif (set_length):
resource.length = size
if (size > self.max_size):
raise DumpError(
"Size of file (%s, %d) exceeds maximum (%d) dump size" %
(resource.path, size, self.max_size))
total_size += size
self.path_prefix = path_prefix
#explicit test because exception raised by getsize otherwise confusing
raise DumpError("No file path defined for resource %s" % resource.uri)
total_size += os.path.getsize(resource.path)
self.total_size = total_size
self.logger.info(
"Total size of files to include in dump %d bytes" %
(total_size))
return True
def partition_dumps(self):
"""Yeild a set of manifest object that parition the dumps.
Simply adds resources/files to a manifest until their are either the
the correct number of files or the size limit is exceeded, then yields
that manifest.
"""
manifest = self.manifest_class()
manifest_size = 0
manifest_files = 0
for resource in self.resources:
manifest.add(resource)
manifest_size += resource.length
manifest_files += 1
if (manifest_size >= self.max_size or manifest_files >= self.max_files):
yield(manifest)
# Need to start a new manifest
manifest = self.manifest_class()
manifest_size = 0
manifest_files = 0
if (manifest_files > 0):
yield(manifest)
def archive_path(self, real_path):
"""Return the archive path for file with real_path.
Mapping is based on removal of self.path_prefix which is determined
by self.check_files().
"""
if (not self.path_prefix):
return(real_path)
else:
return(os.path.relpath(real_path, self.path_prefix))
class DumpError(Exception):
"""Error class used by Dump() objects."""
pass
print "Total size of files to include in dump %d bytes" % (total_size)
if (total_size > self.max_size):
raise DumpError("Size of files to dump (%d) exceeds maximum (%d)" % (total_size,self.max_size))

370
resync/explorer.py
View File

@ -1,370 +0,0 @@
"""ResourceSync explorer.
This is the guts of a client designed to 'explore' the ResourceSync
facilities offered by a source. Will use standard practices to
look for and interpret capabilities.
"""
import sys
try: # python3
from urllib.parse import urlparse, urlunparse, urljoin
except ImportError: # python2
from urlparse import urlparse, urlunparse, urljoin
input = raw_input
import os.path
import datetime
import distutils.dir_util
import re
import time
import logging
from .mapper import Mapper
from .sitemap import Sitemap
from .client import Client, ClientFatalError
from .client_state import ClientState
from .client_utils import ClientFatalError
from .resource import Resource
from .url_or_file_open import url_or_file_open
from .w3c_datetime import str_to_datetime, datetime_to_str
class Explorer(Client):
"""Extension of the client code to explore a ResourceSync source.
Designed to support a text-based command-line client that starts from
a given URI, server, or local file, and then allows interactive
exploration of ResourceSync capabilities offered by the source.
"""
def explore(self):
"""INTERACTIVE exploration source capabilities.
Will use sitemap URI taken either from explicit self.sitemap_name
or derived from the mappings supplied.
"""
# Where do we start? Build options in starts which has entries
# that are a pair comprised of the uri and a list of acceptable
# capabilities
starts = []
if (self.sitemap_name is not None):
print("Starting from explicit --sitemap %s" % (self.sitemap_name))
starts.append(XResource(self.sitemap_name))
elif (len(self.mapper) > 0):
uri = self.mapper.default_src_uri()
(scheme, netloc, path, params, query, fragment) = urlparse(uri)
if (not scheme and not netloc):
if (os.path.isdir(path)):
# have a dir, look for 'likely' file names
print(
"Looking for capability documents in local directory %s" %
(path))
for name in ['resourcesync', 'capabilities.xml',
'resourcelist.xml', 'changelist.xml']:
file = os.path.join(path, name)
if (os.path.isfile(file)):
starts.append(XResource(file))
if (len(starts) == 0):
raise ClientFatalError("No likely capability files found in local directory %s" %
(path))
else:
# local file, might be anything (or not exist)
print("Starting from local file %s" % (path))
starts.append(XResource(path))
else:
# remote, can't tell whether we have a sitemap or a server name or something
# else, build list of options depending on whether there is a path and whether
# there is an extension/name
well_known = urlunparse(
[scheme, netloc, '/.well-known/resourcesync', '', '', ''])
if (not path):
# root, just look for .well-known
starts.append(
XResource(
well_known, [
'capabilitylist', 'capabilitylistindex']))
else:
starts.append(XResource(uri))
starts.append(
XResource(
well_known, [
'capabilitylist', 'capabilitylistindex']))
print("Looking for discovery information based on mappings")
else:
raise ClientFatalError(
"No source information (server base uri or capability uri) specified, use -h for help")
#
# Have list of one or more possible starting point, try them in turn
try:
for start in starts:
# For each starting point we create a fresh history
history = [start]
input = None
while (len(history) > 0):
print()
xr = history.pop()
new_xr = self.explore_uri(xr, len(history) > 0)
if (new_xr):
# Add current and new to history
history.append(xr)
history.append(new_xr)
except ExplorerQuit:
pass # expected way to exit
print("\nresync-explorer done, bye...\n")
def explore_uri(self, explorer_resource, show_back=True):
"""INTERACTIVE exploration of capabilities document(s) starting at a given URI.
Will flag warnings if the document is not of type listed in caps.
"""
uri = explorer_resource.uri
caps = explorer_resource.acceptable_capabilities
checks = explorer_resource.checks
print("Reading %s" % (uri))
options = {}
capability = None
try:
if (caps == 'resource'):
# Not expecting a capability document
self.explore_show_head(uri, check_headers=checks)
else:
s = Sitemap()
list = s.parse_xml(url_or_file_open(uri))
(options, capability) = self.explore_show_summary(
list, s.parsed_index, caps, context=uri)
except IOError as e:
print("Cannot read %s (%s)" % (uri, str(e)))
except Exception as e:
print("Cannot parse %s (%s)" % (uri, str(e)))
#
# Loop until we have some valid input
#
while (True):
# don't offer number option for no resources/capabilities
num_prompt = '' if (len(options) == 0) else 'number, '
up_prompt = 'b(ack), ' if (show_back) else ''
if (self.fake_input):
inp = self.fake_input
else:
inp = input("Follow [%s%sq(uit)]?" % (num_prompt, up_prompt))
if (inp in options.keys()):
break
if (inp == 'q'):
raise ExplorerQuit()
if (inp == 'b'):
return(None)
#
# Got input that is one of the options
#
checks = {}
r = options[inp]
if (r.capability is None):
if (capability in ['resourcelist', 'changelist',
'resourcedump', 'changedump']):
caps = 'resource'
else:
caps = self.allowed_entries(capability)
elif (r.capability == 'resource'):
caps = r.capability
else:
caps = [r.capability]
# Record anything we know about the resource to check
if (r.length is not None):
checks['content-length'] = r.length
if (r.lastmod is not None):
checks['last-modified'] = r.lastmod
if (r.mime_type is not None):
checks['content-type'] = r.mime_type
# FIXME - could add fixity checks here too
return(XResource(r.uri, caps, checks))
def explore_show_summary(self, list, index=False,
expected=None, context=None):
"""Show summary of one capability document.
Given a capability document or index (in list, index True if it is an
index), write out a simply textual summary of the document with all
related documents shown as numbered options (of the form
[#] ...description...) which will then form a menu for the next
exploration.
If expected is not None then it should be a list of expected document
types. If this is set then a warning will be printed if list is not
one of these document types.
Look for certain top-level link types including rel="up".
"""
num_entries = len(list.resources)
capability = '(unknown capability)'
if ('capability' in list.md):
capability = list.md['capability']
if (index):
capability += 'index'
print(
"Parsed %s document with %d entries:" %
(capability, num_entries))
if (expected is not None and capability not in expected):
print("WARNING - expected a %s document" % (','.join(expected)))
if (capability not in ['description', 'descriptionoindex', 'capabilitylist',
'resourcelist', 'resourcelistindex', 'changelist', 'changelistindex',
'resourcedump', 'resourcedumpindex', 'changedump', 'changedumpindex',
'resourcelist-archive', 'resourcedump-archive',
'changelist-archive', 'changedump-archive']):
print("WARNING - unknown %s document type" % (capability))
to_show = num_entries
if (num_entries > 21):
to_show = 20
# What capability entries are allowed/expected?
entry_caps = self.allowed_entries(capability)
options = {}
n = 0
# Look for <rs:ln> elements in this document
ln_describedby = list.link('describedby')
if (ln_describedby):
if ('href' in ln_describedby):
uri = ln_describedby['href']
print("[%s] rel='describedby' link to %s" % ('d', uri))
uri = self.expand_relative_uri(context, uri)
options['d'] = Resource(uri, capability='resource')
else:
print("WARNING - describedby link with no href, ignoring")
ln_up = list.link('up')
if (ln_up):
if ('href' in ln_up):
uri = ln_up['href']
print("[%s] rel='up' link to %s" % ('u', uri))
uri = self.expand_relative_uri(context, uri)
options['u'] = Resource(uri)
else:
print("WARNING - up link with no href, ignoring")
ln_index = list.link('index')
if (ln_index):
if ('href' in ln_index):
uri = ln_index['href']
print("[%s] rel='index' link to %s" % ('i', uri))
uri = self.expand_relative_uri(context, uri)
options['i'] = Resource(uri)
else:
print("WARNING - index link with no href, ignoring")
# Show listed resources as numbered options
for r in list.resources:
if (n >= to_show):
print("(not showing remaining %d entries)" % (num_entries - n))
break
n += 1
options[str(n)] = r
print("[%d] %s" % (n, r.uri))
if (self.verbose):
print(" " + str(r))
r.uri = self.expand_relative_uri(context, r.uri)
if (r.capability is not None):
warning = ''
if (r.capability not in entry_caps):
warning = " (EXPECTED %s)" % (' or '.join(entry_caps))
print(" %s%s" % (r.capability, warning))
elif (len(entry_caps) == 1):
r.capability = entry_caps[0]
print(
" capability not specified, should be %s" %
(r.capability))
return(options, capability)
def explore_show_head(self, uri, check_headers=None):
"""Do HEAD on uri and show infomation.
Will also check headers against any values specified in
check_headers.
"""
print("HEAD %s" % (uri))
if (re.match(r'^\w+:', uri)):
# Looks like a URI
with url_or_file_open(uri, method='HEAD') as response:
status_code = response.code()
headers = response.headers()
else:
# Mock up response if we have a local file
(status_code, headers) = self.head_on_file(uri)
print(" status: %s" % (status_code))
if (status_code == '200'):
# print some of the headers
for header in ['content-length', 'last-modified',
'lastmod', 'content-type', 'etag']:
if header in headers:
check_str = ''
if (check_headers is not None and header in check_headers):
if (headers[header] == check_headers[header]):
check_str = ' MATCHES EXPECTED VALUE'
else:
check_str = ' EXPECTED %s' % (
check_headers[header])
print(" %s: %s%s" % (header, headers[header], check_str))
def head_on_file(self, file):
"""Get fake status code and headers from local file."""
status_code = '404'
headers = {}
if os.path.isfile(file):
status_code = '200'
headers['last-modified'] = datetime_to_str(os.path.getmtime(file))
headers['content-length'] = os.path.getsize(file)
return(status_code, headers)
def allowed_entries(self, capability):
"""Return list of allowed entries for given capability document.
Includes handling of capability = *index where the only acceptable
entries are *.
"""
index = re.match(r'(.+)index$', capability)
archive = re.match(r'(.+)\-archive$', capability)
if (capability == 'capabilitylistindex'):
return([]) # not allowed so no valid references
elif (index):
return([index.group(1)]) # name without index ending
elif (archive):
return([archive.group(1)]) # name without -archive ending
elif (capability == 'description'):
return(['capabilitylist'])
elif (capability == 'capabilitylist'):
return(['resourcelist', 'resourcedump',
'changelist', 'changedump',
'resourcelist-archive', 'resourcedump-archive',
'changelist-archive', 'changedump-archive'])
return([])
def expand_relative_uri(self, context, uri):
"""If uri is relative then expand in context.
Prints warning if expansion happens.
"""
full_uri = urljoin(context, uri)
if (full_uri != uri):
print(" WARNING - expanded relative URI to %s" % (full_uri))
uri = full_uri
return(uri)
class XResource(object):
"""Information about a resource for the explorer.
Must have a uri but may also store:
acceptable_capabilities - None for any acceptable, 'resource' if a
resource rather than a capability document is expected, else
a list of capability names
checks - a set of information to check when then XResource is inspected
base_uri - on creation interpret any relative URI specified in the
context of base_uri, store the resulting full URI
"""
def __init__(self, uri, acceptable_capabilities=None,
checks=None, context=None):
"""Initialize XResource object."""
self.uri = urljoin(context, uri)
self.acceptable_capabilities = acceptable_capabilities
self.checks = checks
class ExplorerQuit(Exception):
"""Exception raised when user quits normally, no error."""
pass

116
resync/hashes.py
View File

@ -1,116 +0,0 @@
"""util.py: A collection of utility functions for source and/or client."""
import base64
import hashlib
class Hashes(object):
"""Compute hash digests for ResourceSync.
These are all base64 encoded according to the rules of
http://www.ietf.org/rfc/rfc4648.txt
MD5
ResourceSync defined to be the same as for Content-MD5 in HTTP,
http://www.ietf.org/rfc/rfc2616.txt which, in turn, defined the
digest string as the "base64 of 128 bit MD5 digest as per RFC 1864"
http://www.ietf.org/rfc/rfc1864.txt
Unfortunately, RFC1864 is rather vague and contains only and example
which doesn't use encoding characters for 62 or 63. It points to
RFC1521 to describe base64 which is explicit that the encoding alphabet
is [A-Za-z0-9+/] with = to pad.
The above corresponds with the alphabet of "3. Base 64 Encoding" in RFC3548
http://www.ietf.org/rfc/rfc3548.txt
and not the url safe version, "Base 64 Encoding with URL and Filename Safe
Alphabet" which replaces + and / with - and _ respectively.
This is the same as the alphabet of "4. Base 64 Encoding" in RFC4648
http://www.ietf.org/rfc/rfc4648.txt.
This algorithm is implemented by base64.standard_b64encode() or
base64.b64encode() with no altchars specified. Available in python2.4 and
up [http://docs.python.org/library/base64.html]
"""
NAME_TO_ATTRIBUTE = {'md5': 'md5', 'sha-1': 'sha1', 'sha-256': 'sha256'}
def __init__(self, hashes=None, file=None):
"""Initialize Hashes object with types of hash to caluclate.
If file is supplied then compute for that file.
"""
self.hashes = set()
for hash in hashes:
if (hash not in self.NAME_TO_ATTRIBUTE.keys()):
raise Exception("Hash type %s not supported" % (hash))
self.hashes.add(hash)
#
self.md5_calc = None
self.sha1_calc = None
self.sha256_calc = None
#
if (file is not None):
self.compute_for_file(file)
def initialize_hashes(self):
"""Create new hashlib objects for each hash we are going to calculate."""
if ('md5' in self.hashes):
self.md5_calc = hashlib.md5()
if ('sha-1' in self.hashes):
self.sha1_calc = hashlib.sha1()
if ('sha-256' in self.hashes):
self.sha256_calc = hashlib.sha256()
def compute_for_file(self, file, block_size=2**14):
"""Compute hash digests for a file.
Calculate the hashes based on one read through the file.
Optional block_size parameter controls memory used to do
calculations. This should be a multiple of 128 bytes.
"""
self.initialize_hashes()
f = open(file, 'rb')
while True:
data = f.read(block_size)
if not data:
break
if self.md5_calc is not None:
self.md5_calc.update(data)
if self.sha1_calc is not None:
self.sha1_calc.update(data)
if self.sha256_calc is not None:
self.sha256_calc.update(data)
f.close()
def set(self, resource):
"""Set hash values for resource from current file.
Assumes that resource has appropriate attributes or setters
with names md5, sha1, etc. and that hashes have been calculated.
"""
for hash in self.hashes:
att = self.NAME_TO_ATTRIBUTE[hash]
setattr(resource, att, getattr(self, att))
@property
def md5(self):
"""Return MD5 hash calculated."""
if (self.md5_calc is None):
return None
return self.md5_calc.hexdigest()
@property
def sha1(self):
"""Return SHA-1 hash calculated."""
if (self.sha1_calc is None):
return None
return self.sha1_calc.hexdigest()
@property
def sha256(self):
"""Return SHA-256 hash calculated."""
if (self.sha256_calc is None):
return None
return self.sha256_calc.hexdigest()

134
resync/list_base.py
View File

@ -1,154 +1,96 @@
"""Base class for ResourceSync capabilities with list of resources.
"""Base class for ResourceSync capabilities with list of resources
Adds Sitemap based IO to the ResourceContainer class and is
intended as the base class for ResourceList, ChangeList,
CapabilityList etc.. Adds common read() and write() methods.
"""
from datetime import datetime
import io
import collections
import os
from datetime import datetime
import re
import sys
import StringIO
import logging
from urllib import URLopener
from .resource_container import ResourceContainer
from .sitemap import Sitemap
from .url_or_file_open import url_or_file_open
from resource_container import ResourceContainer
from sitemap import Sitemap
class ListBase(ResourceContainer):
"""Class that adds Sitemap based IO to ResourceContainer.
resources - an iterable of resources
count - add optional explicit setting of the number of items in
resources which is useful when this is an iterator/generator.
Is used instead of trying len(resources)
md - metadata information for the list (<rs:md>)
ln - link information for the list (<rs:ln>)
uri - the URL of this list
capability_name -
spec_version - default to None for latest version supported, else explicit
version such as '1.0'
Internal variables:
sitemapindex - defaults to False, set True if this is an index object
pretty_xml - defaults to False, set True for more human readable output
"""Class that adds Sitemap based IO to ResourceContainer
"""
def __init__(self, resources=None, count=None, md=None, ln=None, uri=None,
capability_name='unknown', spec_version='1.1', add_lastmod=False):
"""Initialize ListBase."""
super(ListBase, self).__init__(resources=resources, md=md, ln=ln, uri=uri,
capability_name=capability_name)
self.count = count
self.spec_version = spec_version
self.add_lastmod = add_lastmod # Optional in v1.1
def __init__(self, resources=None, md=None, ln=None):
super(ListBase, self).__init__(resources=resources, md=md, ln=ln)
self.capability_name = 'unknown'
self.capability_md = 'unknown'
self.sitemapindex = False
self.pretty_xml = False
#
self.logger = logging.getLogger('resync.list_base')
self.logger = logging.getLogger('list_base')
self.bytes_read = 0
self.parsed_index = None
def __iter__(self):
"""Default to iterator provided by resources object."""
"""Default to iterator provided by resources object"""
return(iter(self.resources))
def __len__(self):
"""Number of entries in this list.
To handle the case where self.resources is a generator or an iterator and
thus cannot provide len(...) we first check to see whether an explicit
self.count is set and return that, otherwise just do len(...).
Typical usage would be to set this on instantiation:
list = ListBase( resources=my_generator, count=count)
print list.as_xml()
"""
if (self.count is not None):
return(self.count)
"""Number of entries in this list"""
return(len(self.resources))
# INPUT
##### INPUT #####
def read(self, uri=None):
"""Default case is just to parse document at this URI.
def read(self,uri=None):
"""Default case is just to parse document at this URI
Intention is that the read() method may be overridden to support reading
of compound documents in more then one sitemapindex/sitemap.
"""
self.parse(uri=uri)
def parse(self, uri=None, fh=None, str_data=None, **kwargs):
"""Parse a single XML document for this list.
def parse(self,uri=None,fh=None):
"""Parse a single XML document for this list
Accepts either a uri (uri or default if parameter not specified),
or a filehandle (fh) or a string (str_data). Note that this method
does not handle the case of a sitemapindex+sitemaps.
LEGACY SUPPORT - the parameter str may be used in place of str_data
but is deprecated and will be removed in a later version.
Does not handle the case of sitemapindex+sitemaps ResourceList
"""
if (uri is not None):
try:
fh = url_or_file_open(uri)
fh = URLopener().open(uri)
except IOError as e:
raise Exception(
"Failed to load sitemap/sitemapindex from %s (%s)" %
(uri, str(e)))
elif (str_data is not None):
fh = io.StringIO(str_data)
elif ('str' in kwargs):
# Legacy support for str argument, see
# https://github.com/resync/resync/pull/21
# One test for this in tests/test_list_base.py
self.logger.warning(
"Legacy parse(str=...), use parse(str_data=...) instead")
fh = io.StringIO(kwargs['str'])
raise Exception("Failed to load sitemap/sitemapindex from %s (%s)" % (uri,str(e)))
if (fh is None):
raise Exception("Nothing to parse")
s = self.new_sitemap()
s.parse_xml(
fh=fh,
resources=self,
capability=self.capability_name,
sitemapindex=False)
s.parse_xml(fh=fh,resources=self,capability=self.capability_md,sitemapindex=False)
self.parsed_index = s.parsed_index
# OUTPUT
##### OUTPUT #####
def as_xml(self):
"""Return XML serialization of this list.
"""Return XML serialization of this list
This code does not support the case where the list is too big for
This code does not support the case where the list is too big for
a single XML document.
"""
self.default_capability()
self.default_capability_and_modified()
s = self.new_sitemap()
return s.resources_as_xml(self, sitemapindex=self.sitemapindex)
return s.resources_as_xml(self,sitemapindex=self.sitemapindex)
def write(self, basename="/tmp/resynclist.xml"):
"""Write a single sitemap or sitemapindex XML document.
def write(self,basename="/tmp/resynclist.xml"):
"""Write a single sitemap or sitemapindex XML document
Must be overridden to support multi-file lists.
"""
self.default_capability()
fh = open(basename, 'w')
self.default_capability_and_modified()
fh = open(basename,'w')
s = self.new_sitemap()
s.resources_as_xml(self, fh=fh, sitemapindex=self.sitemapindex)
s.resources_as_xml(self,fh=fh,sitemapindex=self.sitemapindex)
fh.close()
# UTILITY
##### UTILITY #####
def new_sitemap(self):
"""Create new Sitemap object with default settings."""
return Sitemap(pretty_xml=self.pretty_xml,
spec_version=self.spec_version,
add_lastmod=self.add_lastmod)
"""Create new Sitemap object with default settings"""
return Sitemap(pretty_xml=self.pretty_xml)

376
resync/list_base_with_index.py
View File

@ -1,349 +1,209 @@
"""Base class for ResourceSync capabilities with large lists of resources.
"""Base class for ResourceSync capabilities with lists of resources including
support for both sitemaps and sitemapindexes.
Handles possibly large lists by including support for both sitemaps and
sitemapindexes. Extends ListBase to add the support for sitemapindexes.
Extends ListBase to add support for sitemapindexes.
"""
import collections
import math
import os
from datetime import datetime
import re
import sys
import itertools
from urllib import URLopener
from .hashes import Hashes
from .list_base import ListBase
from .mapper import Mapper, MapperError
from .resource import Resource
from .sitemap import Sitemap
from .url_authority import UrlAuthority
from .url_or_file_open import url_or_file_open
from list_base import ListBase
from resource import Resource
from sitemap import Sitemap
from mapper import Mapper, MapperError
from url_authority import UrlAuthority
from utils import compute_md5_for_file
class ListBaseIndexError(Exception):
"""Exception for problems with sitemapindexes"""
pass
class ListBaseWithIndex(ListBase):
"""Class that add handling of sitemapindexes to ListBase.
Splitting of a list into multiple sitemaps with a sitemapindex is currently
handled based solely on the number of entries in the list. The configurable
self.max_sitemap_entries controls the number of entries that will be written
in a single sitemap or a component sitemap that has a sitemapindex. Support
for sitemapindexes can be disabled by setting allow_multifile False.
resources - an iterable of resources
count - add optional explicit setting of the number of items in
resources which is useful when this is an iterator/generator.
Is used instead of trying len(resources)
md - metadata information for the list (<rs:md>)
ln - link information for the list (<rs:ln>)
allow_multifile - set False to disable support for indexes. Defaults to True
mapper - Mapper instance used to map between file names and URIs so that
the correct URIs can be written into a sitemapindex which correspond
to those that the component sitemap files will be exposed as
"""
def __init__(self, resources=None, count=None, md=None, ln=None, uri=None,
capability_name='unknown', allow_multifile=None, mapper=None,
spec_version='1.1', add_lastmod=False, resources_class=None):
"""Initialize ListBaseWithIndex."""
self.resources_class = list if resources_class is None else resources_class
if (resources is None):
resources = self.resources_class()
super(ListBaseWithIndex, self).__init__(
resources=resources,
count=count,
md=md,
ln=ln,
uri=uri,
capability_name=capability_name,
spec_version=spec_version,
add_lastmod=add_lastmod)
def __init__(self, resources=None, md=None, ln=None, allow_multifile=None, mapper=None):
super(ListBaseWithIndex, self).__init__(resources=resources, md=md, ln=ln)
# specific to lists with indexes
self.max_sitemap_entries = 50000
self.resources_class=dict
self.max_sitemap_entries=50000
self.mapper = mapper
self.allow_multifile = (
True if (
allow_multifile is None) else allow_multifile)
self.allow_multifile = (True if (allow_multifile is None) else allow_multifile)
self.check_url_authority = False
self.content_length = 0
self.num_files = 0 # Number of files read
self.bytes_read = 0 # Aggregate of content_length values
# INPUT
##### INPUT #####
def read(self, uri=None, resources=None, index_only=False):
"""Read sitemap from a URI including handling sitemapindexes.
def read(self, uri=None, resources=None, capability=None, index_only=False):
"""Read sitemap from a URI including handling sitemapindexes
If index_only is True then individual sitemaps references in a sitemapindex
will not be read. This will result in no resources being returned and is
useful only to read the metadata and links listed in the sitemapindex.
Includes the subtlety that if the input URI is a local file and is a
Includes the subtlety that if the input URI is a local file and is a
sitemapindex which contains URIs for the individual sitemaps, then these
are mapped to the filesystem also.
"""
try:
fh = url_or_file_open(uri)
fh = URLopener().open(uri)
self.num_files += 1
except IOError as e:
raise IOError(
"Failed to load sitemap/sitemapindex from %s (%s)" %
(uri, str(e)))
raise IOError("Failed to load sitemap/sitemapindex from %s (%s)" % (uri,str(e)))
# Get the Content-Length if we can (works fine for local files)
try:
self.content_length = int(fh.info()['Content-Length'])
self.bytes_read += self.content_length
self.logger.debug(
"Read %d bytes from %s" %
(self.content_length, uri))
except (KeyError, TypeError):
self.logger.debug( "Read %d bytes from %s" % (self.content_length,uri) )
except KeyError:
# If we don't get a length then c'est la vie
self.logger.debug("Read ????? bytes from %s" % (uri))
self.logger.debug( "Read ????? bytes from %s" % (uri) )
pass
self.logger.info("Read sitemap/sitemapindex from %s" % (uri))
self.logger.info( "Read sitemap/sitemapindex from %s" % (uri) )
s = self.new_sitemap()
s.parse_xml(fh=fh, resources=self, capability=self.capability_name)
s.parse_xml(fh=fh,resources=self,capability='resourcelist')
# what did we read? sitemap or sitemapindex?
if (s.parsed_index):
# sitemapindex
if (not self.allow_multifile):
raise ListBaseIndexError(
"Got sitemapindex from %s but support for sitemapindex disabled" %
(uri))
self.logger.info(
"Parsed as sitemapindex, %d sitemaps" %
(len(
self.resources)))
raise ListBaseIndexError("Got sitemapindex from %s but support for sitemapindex disabled" % (uri))
self.logger.info( "Parsed as sitemapindex, %d sitemaps" % (len(self.resources)) )
sitemapindex_is_file = self.is_file_uri(uri)
if (index_only):
# don't read the component sitemaps
self.sitemapindex = True
return
# now loop over all entries to read each sitemap and add to
# resources
# now loop over all entries to read each sitemap and add to resources
sitemaps = self.resources
self.resources = self.resources_class()
self.logger.info("Now reading %d sitemaps" % len(sitemaps.uris()))
self.logger.info( "Now reading %d sitemaps" % len(sitemaps.uris()) )
for sitemap_uri in sorted(sitemaps.uris()):
self.read_component_sitemap(
uri, sitemap_uri, s, sitemapindex_is_file)
self.read_component_sitemap(uri,sitemap_uri,s,sitemapindex_is_file)
else:
# sitemap
self.logger.info("Parsed as sitemap, %d resources" %
(len(self.resources)))
self.logger.info( "Parsed as sitemap, %d resources" % (len(self.resources)) )
def read_component_sitemap(
self, sitemapindex_uri, sitemap_uri, sitemap, sitemapindex_is_file):
"""Read a component sitemap of a Resource List with index.
Each component must be a sitemap with the
def read_component_sitemap(self, sitemapindex_uri, sitemap_uri, sitemap, sitemapindex_is_file):
"""Read a component sitemap of a Resource List with index
Each component must be a sitemap with the
"""
if (sitemapindex_is_file):
if (not self.is_file_uri(sitemap_uri)):
# Attempt to map URI to local file
remote_uri = sitemap_uri
sitemap_uri = self.mapper.src_to_dst(remote_uri)
self.logger.info(
"Mapped %s to local file %s" %
(remote_uri, sitemap_uri))
self.logger.info("Mapped %s to local file %s" % (remote_uri, sitemap_uri))
else:
# The individual sitemaps should be at a URL (scheme/server/path)
# that the sitemapindex URL can speak authoritatively about
if (self.check_url_authority
and not UrlAuthority(sitemapindex_uri).has_authority_over(sitemap_uri)):
raise ListBaseIndexError(
"The sitemapindex (%s) refers to sitemap at a location it does not have authority over (%s)" %
(sitemapindex_uri, sitemap_uri))
if (self.check_url_authority and
not UrlAuthority(sitemapindex_uri).has_authority_over(sitemap_uri)):
raise ListBaseIndexError("The sitemapindex (%s) refers to sitemap at a location it does not have authority over (%s)" % (sitemapindex_uri,sitemap_uri))
try:
fh = url_or_file_open(sitemap_uri)
fh = URLopener().open(sitemap_uri)
self.num_files += 1
except IOError as e:
raise ListBaseIndexError(
"Failed to load sitemap from %s listed in sitemap index %s (%s)" %
(sitemap_uri, sitemapindex_uri, str(e)))
raise ListBaseIndexError("Failed to load sitemap from %s listed in sitemap index %s (%s)" % (sitemap_uri,sitemapindex_uri,str(e)))
# Get the Content-Length if we can (works fine for local files)
try:
self.content_length = int(fh.info()['Content-Length'])
self.bytes_read += self.content_length
except (KeyError, TypeError):
except KeyError:
# If we don't get a length then c'est la vie
pass
self.logger.info(
"Reading sitemap from %s (%d bytes)" %
(sitemap_uri, self.content_length))
component = sitemap.parse_xml(fh=fh, sitemapindex=False)
self.logger.info( "Reading sitemap from %s (%d bytes)" % (sitemap_uri,self.content_length) )
component = sitemap.parse_xml( fh=fh, sitemapindex=False )
# Copy resources into self, check any metadata
for r in component:
self.resources.add(r)
# FIXME - if rel="up" check it goes to correct place
# FIXME - check capability
# OUTPUT
##### OUTPUT #####
def requires_multifile(self):
"""Return False or the number of component sitemaps required.
def as_xml(self):
"""Return XML serialization of this list
In the case that no len() is available for self.resources then
then self.count must be set beforehand to avoid an exception.
A single XML serailization does not make sense in the case that the list
resources is more than is allowed in a single sitemap so will raise an
exception if that is the case. Otherwise passes to superclass method.
"""
if (self.max_sitemap_entries is None
or len(self) <= self.max_sitemap_entries):
return(False)
return(int(math.ceil(len(self) / float(self.max_sitemap_entries))))
def as_xml(self, allow_multifile=False, basename="/tmp/sitemap.xml"):
"""Return XML serialization of this list.
If this list can be serialized as a single sitemap then the
superclass method is used.
If there is no single XML serialization (in the case that the
number of list resources is more than is allowed in a single sitemap)
then raise an exception unless allow_multifile is set True.
If allow_multifile is set True then will return the sitemapindex
for the set of component sitemaps.
"""
if (not self.requires_multifile()):
return super(ListBaseWithIndex, self).as_xml()
elif (allow_multifile):
return self.as_xml_index(basename)
else:
raise ListBaseIndexError(
"Attempt to write single XML string for list with %d entries when max_sitemap_entries is set to %d" %
(len(self), self.max_sitemap_entries))
def as_xml_index(self, basename="/tmp/sitemap.xml"):
"""Return a string of the index for a large list that is split.
All we need to do is determine the number of component sitemaps will
be is and generate their URIs based on a pattern.
Q - should there be a flag to select generation of each component sitemap
in order to calculate the md5sum?
Q - what timestamp should be used?
"""
num_parts = self.requires_multifile()
if (not num_parts):
raise ListBaseIndexError(
"Request for sitemapindex for list with only %d entries when max_sitemap_entries is set to %s" %
(len(self), str(
self.max_sitemap_entries)))
index = ListBase()
index.sitemapindex = True
index.capability_name = self.capability_name
index.default_capability()
for n in range(num_parts):
r = Resource(uri=self.part_name(basename, n))
index.add(r)
return(index.as_xml())
def as_xml_part(self, basename="/tmp/sitemap.xml", part_number=0):
"""Return a string of component sitemap number part_number.
Used in the case of a large list that is split into component
sitemaps.
basename is used to create "index" links to the sitemapindex
Q - what timestamp should be used?
"""
if (not self.requires_multifile()):
raise ListBaseIndexError(
"Request for component sitemap for list with only %d entries when max_sitemap_entries is set to %s" %
(len(self), str(
self.max_sitemap_entries)))
start = part_number * self.max_sitemap_entries
if (start > len(self)):
raise ListBaseIndexError(
"Request for component sitemap with part_number too high, would start at entry %d yet the list has only %d entries" %
(start, len(self)))
stop = start + self.max_sitemap_entries
if (stop > len(self)):
stop = len(self)
part = ListBase(itertools.islice(self.resources, start, stop))
part.capability_name = self.capability_name
part.default_capability()
part.index = basename
s = self.new_sitemap()
return(s.resources_as_xml(part))
if (self.max_sitemap_entries is not None):
if (len(self)>self.max_sitemap_entries):
raise ListBaseIndexError("Attempt to write single XLM string for list with %d entries when max_sitemap_entries is set to %d" % (len(self),self.max_sitemap_entries))
return super(ListBaseWithIndex, self).as_xml()
def write(self, basename='/tmp/sitemap.xml'):
"""Write one or a set of sitemap files to disk.
"""Write one or a set of sitemap files to disk
resources is a ResourceContainer that may be an ResourceList or
a ChangeList. This may be a generator so data is read as needed
and length is determined at the end.
basename is used as the name of the single sitemap file or the
basename is used as the name of the single sitemap file or the
sitemapindex for a set of sitemap files.
Uses self.max_sitemap_entries to determine whether the resource_list can
be written as one sitemap. If there are more entries and
self.allow_multifile is set True then a set of sitemap files,
Uses self.max_sitemap_entries to determine whether the resource_list can
be written as one sitemap. If there are more entries and
self.allow_multifile is set true then a set of sitemap files,
with an sitemapindex, will be written.
"""
# Access resources through iterator only
resources_iter = iter(self.resources)
(chunk, nxt) = self.get_resources_chunk(resources_iter)
( chunk, next ) = self.get_resources_chunk(resources_iter)
s = self.new_sitemap()
if (nxt is not None):
if (next is not None):
# Have more than self.max_sitemap_entries => sitemapindex
if (not self.allow_multifile):
raise ListBaseIndexError(
"Too many entries for a single sitemap but multifile disabled")
raise ListBaseIndexError("Too many entries for a single sitemap but multifile disabled")
# Work out how to name the sitemaps, attempt to add %05d before ".xml$", else append
sitemap_prefix = basename
sitemap_suffix = '.xml'
if (basename[-4:] == '.xml'):
sitemap_prefix = basename[:-4]
# Work out URI of sitemapindex so that we can link up to
# it from the individual sitemap files
try:
index_uri = self.mapper.dst_to_src(basename)
except MapperError as e:
raise ListBaseIndexError(
"Cannot map sitemapindex filename to URI (%s)" %
str(e))
raise ListBaseIndexError("Cannot map sitemapindex filename to URI (%s)" % str(e))
# Use iterator over all resources and count off sets of
# max_sitemap_entries to go into each sitemap, store the
# names of the sitemaps as we go. Copy md from self into
# the index and use this for all chunks also
index = ListBase(md=self.md.copy(), ln=list(self.ln))
# names of the sitemaps as we go
index=ListBase()
index.capability_name = self.capability_name
index.default_capability()
while (len(chunk) > 0):
file = self.part_name(basename, len(index))
index.capability_md = self.capability_md
index.default_capability_and_modified()
while (len(chunk)>0):
file = sitemap_prefix + ( "%05d" % (len(index)) ) + sitemap_suffix
# Check that we can map the filename of this sitemap into
# URI space for the sitemapindex
try:
uri = self.mapper.dst_to_src(file)
except MapperError as e:
raise ListBaseIndexError(
"Cannot map sitemap filename to URI (%s)" % str(e))
raise ListBaseIndexError("Cannot map sitemap filename to URI (%s)" % str(e))
self.logger.info("Writing sitemap %s..." % (file))
f = open(file, 'w')
chunk.index = index_uri
chunk.md = index.md
chunk.ln.append({'rel': 'up', 'href': index_uri})
s.resources_as_xml(chunk, fh=f)
f.close()
# Record information about this sitemap for index
r = Resource(uri=uri,
timestamp=os.stat(file).st_mtime,
md5=Hashes(['md5'], file).md5)
r = Resource( uri = uri, path = file,
timestamp = os.stat(file).st_mtime,
md5 = compute_md5_for_file(file) )
index.add(r)
# Get next chunk
(chunk, nxt) = self.get_resources_chunk(resources_iter, nxt)
( chunk, next ) = self.get_resources_chunk(resources_iter,next)
self.logger.info("Wrote %d sitemaps" % (len(index)))
f = open(basename, 'w')
self.logger.info("Writing sitemapindex %s..." % (basename))
s.resources_as_xml(index, sitemapindex=True, fh=f)
f.close()
self.logger.info("Wrote sitemapindex %s" % (basename))
elif self.sitemapindex:
f = open(basename, 'w')
self.logger.info("Writing sitemapindex %s..." % (basename))
s.resources_as_xml(chunk, sitemapindex=True, fh=f)
s.resources_as_xml(index,sitemapindex=True,fh=f)
f.close()
self.logger.info("Wrote sitemapindex %s" % (basename))
else:
@ -354,69 +214,47 @@ class ListBaseWithIndex(ListBase):
self.logger.info("Wrote sitemap %s" % (basename))
def index_as_xml(self):
"""XML serialization of this list taken to be sitemapindex entries."""
self.default_capability()
s = self.new_sitemap()
return s.resources_as_xml(self, sitemapindex=True)
"""Return XML serialization of this list taken to be sitemapindex entries
# Utility
"""
self.default_capability_and_modified()
s = self.new_sitemap()
return s.resources_as_xml(self,sitemapindex=True)
##### Utility #####
def get_resources_chunk(self, resource_iter, first=None):
"""Return next chunk of resources from resource_iter, and next item.
"""Return next chunk of resources from resource_iter, and next item
If first parameter is specified then this will be prepended to
the list.
The chunk will contain self.max_sitemap_entries if the iterator
The chunk will contain self.max_sitemap_entries if the iterator
returns that many. next will have the value of the next value from
the iterator, providing indication of whether more is available.
the iterator, providing indication of whether more is available.
Use this as first when asking for the following chunk.
"""
chunk = ListBase(md=self.md.copy(), ln=list(self.ln))
chunk = ListBase()
chunk.capability_name = self.capability_name
chunk.default_capability()
chunk.capability_md = self.capability_md
chunk.default_capability_and_modified()
if (first is not None):
chunk.add(first)
for r in resource_iter:
chunk.add(r)
if (len(chunk) >= self.max_sitemap_entries):
if (len(chunk)>=self.max_sitemap_entries):
break
# Get next to see whether there are more resources
try:
nxt = next(resource_iter)
next = resource_iter.next()
except StopIteration:
nxt = None
return(chunk, nxt)
def part_name(self, basename='/tmp/sitemap.xml', part_number=0):
"""Name (file or URI) for one component sitemap.
Works for both filenames and URIs because manipulates only the end
of the string.
Abstracting this into a function that starts from the basename to get
prefix and suffix each time seems a bit wasteful but perhaps not worth
worrying about. Allows same code to be used for the write() and
as_xml_index() cases.
"""
# Work out how to name the sitemaps, attempt to add %05d before
# ".xml$", else append
sitemap_prefix = basename
sitemap_suffix = '.xml'
if (basename[-4:] == '.xml'):
sitemap_prefix = basename[:-4]
return(sitemap_prefix + ("%05d" % (part_number)) + sitemap_suffix)
next = None
return(chunk,next)
def is_file_uri(self, uri):
"""Return true if uri looks like a local file URI, false otherwise.
"""Return true if uri looks like a local file URI, false otherwise
Test is to see whether have either an explicit file: URI or whether
there is no scheme name.
"""
return(re.match(r'file:', uri) or not re.match(r'\w{3,4}:', uri))
class ListBaseIndexError(Exception):
"""Exception for problems with sitemapindexes in ListBaseIndex."""
pass
return(re.match('file:',uri) or not re.match('\w{3,4}:',uri))

208
resync/mapper.py
View File

@ -1,95 +1,78 @@
"""Map between source URIs and destination paths."""
"""Map between source URIs and destination paths"""
import os
import os.path
import re
try: # python3
from urllib.parse import urlparse
except ImportError: # pragma: no cover python2
from urlparse import urlparse # pragma: no cover
import urlparse
import logging
class MapperError(Exception):
pass
class Mapper():
"""Mapper object to map between source URIs and destination paths.
Implemented as a list of Map objects.
"""
def __init__(self, mappings=None, use_default_path=False):
"""Initialize Mapper."""
self.logger = logging.getLogger('resync.mapper')
self.mappings = []
self.logger = logging.getLogger('mapper')
self.mappings=[]
if (mappings):
self.parse(mappings, use_default_path)
def __len__(self):
"""Length is number of mappings."""
"""Length is number of mappings"""
return(len(self.mappings))
def parse(self, mappings, use_default_path=False):
"""Parse a list of map strings (mappings).
"""Parse a list of map strings (mappings)
Accepts two distinct formats:
1. If there are exactly two entries then these may be the source
base URI and the destination base path. Neither may contain an
equals (=) sign.
2. For any number of mapping stings interpret each as a mapping
URI=path. These are in the order they will be tried.
1. If there are exactly two entries then these may be the source base URI
and the destination base path. Neither may contain an equals (=) sign.
2. For any number of mapping stings interpret each as a mapping URI=path.
These are in the order they will be tried.
And if use_default_path is True then a third:
3. If there is exactly one entry and it does not contain an
equals (=) sign then a safe local path is created based on the
source base URI. In the case that the source base URI is a local
path already then an indentity mapping is used.
3. If there is exactly one entry and it does not contain an equals (=)
sign then a safe local path is created based on the source base URI. In
the case that the source base URI is a local path already then an indentity
mapping is used.
"""
if (use_default_path
and len(mappings) == 1
and re.search(r"=", mappings[0]) is None):
if (use_default_path and
len(mappings)==1 and
re.search(r"=",mappings[0])==None):
path = self.path_from_uri(mappings[0])
self.logger.warning("Using URI mapping: %s -> %s" %
(mappings[0], path))
self.mappings.append(Map(mappings[0], path))
elif (len(mappings) == 2
and re.search(r"=", mappings[0]) is None
and re.search(r"=", mappings[1]) is None):
self.mappings.append(Map(mappings[0], mappings[1]))
self.logger.info("Assuming mapping: %s -> %s" % (mappings[0],path))
self.mappings.append(Map(mappings[0],path))
elif (len(mappings)==2 and
re.search(r"=",mappings[0])==None and
re.search(r"=",mappings[1])==None):
self.mappings.append(Map(mappings[0],mappings[1]))
else:
for mapping in mappings:
entry = mapping.split('=')
if (len(entry) != 2):
raise MapperError(
"Bad mapping argument (%s), got %s" %
(mapping, str(entry)))
(src_uri, dst_path) = entry
l=mapping.split('=')
if (len(l)!=2):
raise MapperError("Bad mapping argument (%s), got %s"%(mapping,str(l)))
(src_uri,dst_path)=l
# Check for dupes
for map in self.mappings:
if (src_uri == map.src_uri):
raise MapperError(
"Attempt to set duplicate mapping for "
"source URI %s (with destination path %s)" %
(src_uri, dst_path))
raise MapperError("Attempt to set duplicate mapping for source URI %s (with destination path %s)" % (src_uri,dst_path))
if (dst_path == map.dst_path):
raise MapperError(
"Attempt to set duplicate mapping for "
"destination path %s (with source URI %s)" %
(dst_path, src_uri))
raise MapperError("Attempt to set duplicate mapping for destination path %s (with source URI %s)" % (dst_path,src_uri))
self.mappings.append(Map(src_uri, dst_path))
def default_src_uri(self):
"""Default src_uri from mapping.
"""Default src_uri from mapping
This is take just to be the src_uri of the first entry
"""
if (len(self.mappings) > 0):
if (len(self.mappings)>0):
return(self.mappings[0].src_uri)
raise MapperError("Can't get default source URI from empty mapping")
def unsafe(self):
"""True if one or more mapping is unsafe.
"""True is one or more mapping is unsafe
See Map.unsafe() for logic. Provide this as a test rather than
building into object creation/parse because it is useful to
See Map.unsafe() for logic. Provide this as a test rather than
building into object creation/parse because it is useful to
allow unsafe mappings in situations where it doesn't matter.
"""
for map in self.mappings:
@ -97,46 +80,41 @@ class Mapper():
return(True)
return(False)
def dst_to_src(self, dst_file):
"""Map destination path to source URI."""
def dst_to_src(self,dst_file):
"""Map destination path to source URI"""
for map in self.mappings:
src_uri = map.dst_to_src(dst_file)
if (src_uri is not None):
return(src_uri)
# Must have failed if loop exited
raise MapperError(
"Unable to translate destination path (%s) "
"into a source URI." % (dst_file))
raise MapperError("Unable to translate destination path (%s) into a source URI." % (dst_file))
def src_to_dst(self, src_uri):
"""Map source URI to destination path."""
def src_to_dst(self,src_uri):
"""Map source URI to destination path"""
for map in self.mappings:
dst_path = map.src_to_dst(src_uri)
if (dst_path is not None):
return(dst_path)
# Must have failed if loop exited
raise MapperError(
"Unable to translate source URI (%s) into "
"a destination path." % (src_uri))
raise MapperError("Unable to translate source URI (%s) into a destination path." % (src_uri))
def path_from_uri(self, uri):
"""Make a safe path name from uri.
def path_from_uri(self,uri):
"""Make a safe path name from uri
In the case that uri is already a local path then the
same path is returned.
In the case that uri is already a local path then the same path
is returned.
"""
(scheme, netloc, path, params, query, frag) = urlparse(uri)
(scheme, netloc, path, params, query, fragment) = urlparse.urlparse( uri )
if (netloc == ''):
return(uri)
path = '/'.join([netloc, path])
path = re.sub(r'[^\w\-\.]', '_', path)
path = re.sub(r'__+', '_', path)
path = re.sub(r'[_\.]+$', '', path)
path = re.sub(r'^[_\.]+', '', path)
path = '/'.join([netloc,path])
path = re.sub('[^\w\-\.]', '_', path)
path = re.sub('__+', '_', path)
path = re.sub('[_\.]+$', '', path)
path = re.sub('^[_\.]+', '', path)
return(path)
def __repr__(self):
"""Human readable dump of all Maps in the Mapper."""
s = 'Mapper with %d maps:\n' % (len(self.mappings))
for map in self.mappings:
s += str(map) + '\n'
@ -144,85 +122,63 @@ class Mapper():
class Map:
"""A single map from source URI to destination path.
"""A single map from source URI to destination path
Both URI and destination paths are assumed to use / as the path
separator. No account is take for other path separators used
for paths on non-unix systems. This translation must be done
for paths on non-unix systems. This translation must be done
elsewhere by consideration of os.sep.
"""
def __init__(self, src_uri=None, dst_path=None):
"""Initialize Map object.
Will strip trailing slashes from src_uri and dst_path.
"""
self.src_uri = self.strip_trailing_slashes(src_uri)
def __init__(self,src_uri=None,dst_path=None):
self.src_uri = self.strip_trailing_slashes(src_uri)
self.dst_path = self.strip_trailing_slashes(dst_path)
def strip_trailing_slashes(self, path):
"""Return input path minus any trailing slashes."""
m = re.match(r"(.*)/+$", path)
def strip_trailing_slashes(self,path):
"""Return input path minus any trailing slashes"""
m=re.match(r"(.*)/+$",path)
if (m is None):
return(path)
return(m.group(1))
def dst_to_src(self, dst_file):
"""Return the src URI from the dst filepath.
def dst_to_src(self,dst_file):
"""Return the src URI from the dst filepath
This does not rely on the destination filepath actually
existing on the local filesystem, just on pattern matching.
Return source URI on success, None on failure.
Relies upon self.dst_path and self.src_path not including trailing
slashes. However, a match of just self.dst_path withouth a trailing
slash will return self.src_path with a trailing slash.
This does not rely on the destination filepath actually existing on the local
filesystem, just on pattern matching. Return source URI on success, None on
failure.
"""
m = re.match(self.dst_path + "(/.*)?$", dst_file)
m=re.match(self.dst_path+"/(.*)$",dst_file)
if (m is None):
return(None)
rel_path = m.group(1)
if rel_path is None:
rel_path = '/'
return self.src_uri + rel_path
def src_to_dst(self, src_uri):
"""Return the dst filepath from the src URI.
Returns None on failure, local destination path on success.
Relies upon self.dst_path and self.src_path not including trailing
slashes. However, a match of just self.src_path withouth a trailing
slash will return self.dst_path with a trailing slash.
rel_path=m.group(1)
return(self.src_uri+'/'+rel_path)
def src_to_dst(self,src_uri):
"""Return the dst filepath from the src URI
FIXME -- look at whether urlparse can be used here?
Returns None on failure, destination path on success.
"""
m = re.match(self.src_uri + "(/.*)?$", src_uri)
m=re.match(self.src_uri+"/(.*)$",src_uri)
if (m is None):
return(None)
rel_path = m.group(1)
if rel_path is None:
rel_path = '/'
return self.dst_path + rel_path
rel_path=m.group(1)
return(self.dst_path+'/'+rel_path)
def unsafe(self):
"""True if the mapping is unsafe for an update.
"""True is the mapping is unsafe for an update
Applies only to local source. Returns True if the paths for source and
Applies only to local source. Returns True if the paths for source and
destination are the same, or if one is a component of the other path.
"""
(scheme, netloc, path, params, query, frag) = urlparse(self.src_uri)
(scheme, netloc, path, params, query, fragment) = urlparse.urlparse( self.src_uri )
if (scheme != ''):
return(False)
s = os.path.normpath(self.src_uri)
d = os.path.normpath(self.dst_path)
lcp = os.path.commonprefix([s, d])
lcp = os.path.commonprefix([s,d])
return(s == lcp or d == lcp)
def __repr__(self):
"""Human readable string for one map."""
return("Map( %s -> %s )" % (self.src_uri, self.dst_path))
class MapperError(Exception):
"""Exception for errors in Mapper class."""
pass

565
resync/resource.py
View File

@ -1,104 +1,63 @@
"""ResourceSync Resources - information about a web resource and changes."""
"""Information about a web resource
Each web resource is identified by a URI and may optionally have
other metadata such as timestamp, length, md5. The lastmod property
provides ISO8601 format string access to the timestamp.
The timestamp is assumed to be stored in UTC.
This object is optimized for size in the case whether there is not large
data in the attributes. This is done using __slots__ for the attributes
so that there is no __dict__ defined. The 'ln' attribute is used if
necessary to add links or other information in a hash which is convenient
but will significantly increase the size of each object with such
information.
"""
import re
from urlparse import urlparse
from posixpath import basename
from urllib.parse import urlparse
from .w3c_datetime import str_to_datetime, datetime_to_str
from w3c_datetime import str_to_datetime, datetime_to_str
class Resource(object):
"""ResourceSync Resource object.
Each web resource is identified by a URI and may optionally
have other metadata such as timestamp, length, md5, etc..
The lastmod property provides ISO8601 format string access
to the timestamp. The timestamp is stored internally as a
unix timestamp value in UTC. This limits the range of
possible lastmod values but covers all web-era values for
a good way into the future.
This object is optimized for size in the case whether there
is not large a large amount of data in the attributes. This
is done using __slots__ for the core attributes so that there
is no __dict__ defined for a Resource object. Core attributes
are:
uri - Resource URI
timestamp - Last-Modification time, has lastmod accessor
length - size in bytes
mime_type - MIME type
md5, sha1, sha256 - digests, have hash accessor
change - change type
ts_datetime - time of the change, has datetime accessor
path - path in dump
If non-core attributes are needed then the '_extra' attribute
has a dict of values. The ones explicitly used here are:
capability - Capability name
ts_at - at time, has md_at accessor
ts_completed - completed time, has md_completed accessor
ts_from - from time, has md_from accessor
ts_until - until time, has md_until accessor
The accessor names mime_type, md_from etc. are used to avoid conflict
with Python built-in functions and keywords type(), from etc...
The 'ln' attribute is used when it is necessary to add links
or other information to the object. Use of non-core attributes
or links results in dicts being created which is convenient
but will significantly increase the size of each Resource
object that contains such information.
"""
__slots__ = ('uri', 'timestamp', 'length', 'mime_type',
'md5', 'sha1', 'sha256', 'change', 'ts_datetime',
'path', 'ln', '_extra')
CHANGE_TYPES = ['created', 'updated', 'deleted']
def __init__(self, uri=None, timestamp=None, length=None,
mime_type=None, md5=None, sha1=None, sha256=None,
change=None, ts_datetime=None, path=None, ln=None,
# the following in _extra
capability=None,
ts_at=None, ts_completed=None, ts_from=None, ts_until=None,
# and the following via setters
lastmod=None, datetime=None,
md_at=None, md_completed=None, md_from=None, md_until=None,
# and finally another Resource to copy from
resource=None):
"""Initialize Resource object.
Initialize either from parameters specified or from an existing
Resource object. If explicit parameters are specified then they
will override values copied from a Resource object supplied.
__slots__=('uri', 'timestamp', 'length',
'md5', 'sha1', 'sha256', 'type',
'change', 'path',
'capability', 'ln' )
def __init__(self, uri = None, timestamp = None, length = None,
md5 = None, sha1 = None, sha256 = None, type = None,
change = None, path = None,
lastmod = None, resource = None,
capability = None, ln = None):
""" Initialize object either from parameters specified or
from an existing Resource object. If explicit parameters
are specified then they will override values copied from
a resource object supplied.
"""
# Initialize core attributes
# Create from a Resource?
self.uri = None
self.timestamp = None
self.length = None
self.mime_type = None
self.md5 = None
self.sha1 = None
self.sha256 = None
self.type = None
self.change = None
self.ts_datetime = None # Added in ResourceSync v1.1
self.path = None
self.capability = None
self.ln = None
self._extra = None
# Create from a Resource-like object? Copy any attributes, both ones
# that have slots and ones that live in _extra
if (resource is not None):
for att in ['uri', 'timestamp', 'length', 'mime_type',
'md5', 'sha1', 'sha256', 'change', 'ts_datetime', 'path', 'ln',
# the following in _extra
'capability', 'ts_at', 'ts_completed', 'ts_from', 'ts_until']:
if hasattr(resource, att):
setattr(self, att, getattr(resource, att))
# Any arguments will then override
self.uri = resource.uri
self.timestamp = resource.timestamp
self.length = resource.length
self.md5 = resource.md5
self.sha1 = resource.sha1
self.sha256 = resource.sha256
self.change = resource.change
self.path = resource.path
self.capability = resource.capability
self.ln = resource.ln
if (uri is not None):
self.uri = uri
if (timestamp is not None):
@ -111,418 +70,138 @@ class Resource(object):
self.sha1 = sha1
if (sha256 is not None):
self.sha256 = sha256
if (mime_type is not None):
self.mime_type = mime_type
if (type is not None):
self.type = type
if (change is not None):
self.change = change
if (ts_datetime is not None):
self.ts_datetime = ts_datetime
if (path is not None):
self.path = path
if (ts_at is not None):
self.ts_at = ts_at
if (ts_completed is not None):
self.ts_completed = ts_completed
if (ts_from is not None):
self.ts_from = ts_from
if (ts_until is not None):
self.ts_until = ts_until
if (capability is not None):
self.capability = capability
if (ln is not None):
self.ln = ln
# Timestamp setters
if (lastmod is not None):
self.lastmod = lastmod
if (datetime is not None):
self.datetime = datetime
if (md_at is not None):
self.md_at = md_at
if (md_completed is not None):
self.md_completed = md_completed
if (md_from is not None):
self.md_from = md_from
if (md_until is not None):
self.md_until = md_until
self.lastmod=lastmod
# Sanity check
if (self.uri is None):
raise ValueError("Cannot create Resource without a URI")
def __setattr__(self, prop, value):
"""Attribute setter with check and support for extra attributes.
Because this class is optimized to use __slots__ it cannot
ordinarily be dynamically extended. We implement extension with
the idea of extra properties.
"""
# Add validity check for self.change
if (prop == 'change' and Resource.CHANGE_TYPES
and value is not None and value not in Resource.CHANGE_TYPES):
raise ChangeTypeError(value)
else:
try:
object.__setattr__(self, prop, value)
except AttributeError:
# assume an extra one...
self._set_extra(prop, value)
def _set_extra(self, prop, value):
# Use self._extra dict to hold non-core attributes to
# save space
if (self._extra is None):
self._extra = dict()
self._extra[prop] = value
def _get_extra(self, prop):
# Returns None for a property not set
if (self._extra is None):
return None
return self._extra.get(prop)
raise ValueError("Cannot create resoure without a URI")
@property
def lastmod(self):
"""The Last-Modified date for the resource in W3C Datetime syntax, Z notation."""
"""The Last-Modified data in W3C Datetime syntax, Z notation"""
if (self.timestamp is None):
return None
return datetime_to_str(self.timestamp)
@lastmod.setter
def lastmod(self, lastmod):
"""Set last modified timestamp from a W3C Datetime Last-Modified value."""
self.timestamp = str_to_datetime(lastmod, context='lastmod')
@property
def datetime(self):
"""The datetime of the resource change in W3C Datetime syntax, Z notation."""
return datetime_to_str(self.ts_datetime)
@datetime.setter
def datetime(self, datetime):
"""Set timestamp from a W3C Datetime Last-Modified value."""
self.ts_datetime = str_to_datetime(datetime, context='ts_datetime')
@property
def ts_at(self):
"""ts_at is the timestamp for md_at."""
return self._get_extra('ts_at')
@property
def md_at(self):
"""md_at values in W3C Datetime syntax, Z notation."""
return datetime_to_str(self._get_extra('ts_at'))
@md_at.setter
def md_at(self, md_at):
"""Set at value from a W3C Datetime value."""
self._set_extra(
'ts_at',
str_to_datetime(md_at, context='md_at datetime'))
@property
def ts_completed(self):
"""ts_completed is the timestamp for md_completed."""
return self._get_extra('ts_completed')
@property
def md_completed(self):
"""md_completed value in W3C Datetime syntax, Z notation."""
return datetime_to_str(self._get_extra('ts_completed'))
@md_completed.setter
def md_completed(self, md_completed):
"""Set md_completed value from a W3C Datetime value."""
self._set_extra(
'ts_completed',
str_to_datetime(md_completed, context='md_completed datetime'))
@property
def ts_from(self):
"""ts_from is the timestamp for md_from."""
return self._get_extra('ts_from')
@property
def md_from(self):
"""md_from value in W3C Datetime syntax, Z notation."""
return datetime_to_str(self._get_extra('ts_from'))
@md_from.setter
def md_from(self, md_from):
"""Set md_from value from a W3C Datetime value."""
self._set_extra(
'ts_from',
str_to_datetime(md_from, context='md_from datetime'))
@property
def ts_until(self):
"""ts_until is the timestamp for md_until."""
return self._get_extra('ts_until')
@property
def md_until(self):
"""md_until value in W3C Datetime syntax, Z notation."""
return datetime_to_str(self._get_extra('ts_until'))
@md_until.setter
def md_until(self, md_until):
"""Set md_until value from a W3C Datetime value."""
self._set_extra(
'ts_until',
str_to_datetime(md_until, context='md_until datetime'))
@property
def capability(self):
"""Get Capability name string."""
return self._get_extra('capability')
@capability.setter
def capability(self, capability):
"""Set Capability name string."""
self._set_extra('capability', capability)
"""Set timestamp from an W3C Datetime Last-Modified value"""
if (lastmod is None):
self.timestamp = None
return
if (lastmod == ''):
raise ValueError('Attempt to set empty lastmod')
self.timestamp = str_to_datetime(lastmod)
@property
def hash(self):
"""Provide access to the complete hash string.
The hash string may have zero or more hash values with
appropriate prefixes. All hash values are assumed to be
strings
"""
"""Provide access to the total hash string"""
hashvals = []
if (self.md5 is not None):
hashvals.append('md5:' + self.md5)
hashvals.append('md5:'+self.md5)
if (self.sha1 is not None):
hashvals.append('sha-1:' + self.sha1)
hashvals.append('sha-1:'+self.sha1)
if (self.sha256 is not None):
hashvals.append('sha-256:' + self.sha256)
if (len(hashvals) > 0):
hashvals.append('sha-256:'+self.sha256)
if (len(hashvals)>0):
return(' '.join(hashvals))
return(None)
@hash.setter
def hash(self, hashes_str):
"""Parse space separated set of values to set md5, sha1 and sha256.
def hash(self, hash):
"""Parse space separated set of values
Any existing values for md5, sha1 and sha256 will be removed first.
See specification at:
http://tools.ietf.org/html/draft-snell-atompub-link-extensions-09
which defines many types. We implement md5, sha-1, sha-256
"""
defines many types, we implement md5, sha-1, sha-256"""
self.md5 = None
self.sha1 = None
self.sha256 = None
if (hash is None):
return
hash_seen = set()
errors = []
for entry in hashes_str.split():
(hash_type, value) = entry.split(':', 1)
if hash_type in hash_seen:
errors.append("duplicate hash type %s" % (hash_type))
for entry in hash.split():
( hash_type, value ) = entry.split(':',1)
if (hash_type in hash_seen):
errors.append("Ignored duplicate hash type %s" % (hash_type))
else:
hash_seen.add(hash_type)
if hash_type == 'md5':
self.md5 = value
elif hash_type == 'sha-1':
self.sha1 = value
elif hash_type == 'sha-256':
self.sha256 = value
if (hash_type == 'md5'):
self.md5=value
elif (hash_type == 'sha-1'):
self.sha1=value
elif (hash_type == 'sha-256'):
self.sha256=value
else:
errors.append("unsupported hash type %s" % (hash_type))
if len(errors) > 0:
raise ValueError("Ignored " + ", ".join(errors))
def link(self, rel):
"""Look for link with specified rel, return else None.
Searches through dicts in self.ln looking for one with the
specified rel value. If there are multiple links with the
same rel then just the first will be returned
"""
if self.ln is not None:
for link in self.ln:
if 'rel' in link and link['rel'] == rel:
return link
return None
def link_href(self, rel):
"""Look for link with specified rel, return href from it or None."""
link = self.link(rel)
return None if link is None else link['href']
def link_set(self, rel, href, allow_duplicates=False, **atts):
"""Set/create link with specified rel, set href and any other attributes.
Any link element must have both rel and href values, the specification
also defines the type attributes and others are permitted also. See
description of allowed formats in
http://www.openarchives.org/rs/resourcesync.html#DocumentFormats
Be aware that adding links to a Resource object will
significantly increase the size of the object.
"""
if self.ln is None:
# automagically create a self.ln list
self.ln = []
link = None
else:
link = self.link(rel)
if link is not None and not allow_duplicates:
# overwrite current value
link['href'] = href
else:
# create new link
link = {'rel': rel, 'href': href}
self.ln.append(link)
for k in atts:
link[k] = atts[k]
def link_add(self, rel, href, **atts):
"""Create an link with specified rel.
Will add a link even if one with that rel already exists.
"""
self.link_set(rel, href, allow_duplicates=True, **atts)
@property
def describedby(self):
"""Convenient access to <rs:ln rel="describedby" href="uri">."""
return(self.link_href('describedby'))
@describedby.setter
def describedby(self, uri):
"""Set ResourceSync Description link to given URI."""
self.link_set('describedby', uri)
@property
def up(self):
"""Get the URI of any ResourceSync rel="up" link."""
return(self.link_href('up'))
@up.setter
def up(self, uri):
"""Set rel="up" link to given URI."""
self.link_set('up', uri)
@property
def index(self):
"""Get the URI of and ResourceSync rel="index" link."""
return(self.link_href('index'))
@index.setter
def index(self, uri):
"""Set rel="index" link to given URI."""
self.link_set('index', uri)
@property
def contents(self):
"""Get the URI of and ResourceSync rel="contents" link."""
return(self.link_href('contents'))
@contents.setter
def contents(self, uri, type='application/xml'):
"""Set rel="contents" link to given URI.
Will also set the type="application/xml" unless overridden
"""
self.link_set('contents', uri, type=type)
errors.append("Ignored unsupported hash type (%s)" % (hash_type))
if (len(errors)>0):
raise ValueError(". ".join(errors))
@property
def basename(self):
"""The resource basename.
For example from http://example.com/resource/1 returns 1
"""
"""The resource basename (http://example.com/resource/1 -> 1)"""
parse_object = urlparse(self.uri)
return(basename(parse_object.path))
return basename(parse_object.path)
def __eq__(self, other):
"""Equality test for resources allowing <1s difference in timestamp.
def __eq__(self,other):
"""Equality test for resources allowing <1s difference in timestamp"""
return( self.equal(other,delta=1.0) )
See equal(...) for more details of equality test.
"""
return(self.equal(other, delta=1.0))
def equal(self, other, delta=0.0):
"""Equality or near equality test for resources.
Equality means that any parameters that exist for both resources match,
essentially "no proof of inequality":
def equal(self,other,delta=0.0):
"""Equality or near equality test for resources
Equality means:
1. same uri, AND
2. same timestamp WITHIN delta if specified for either, AND
3. same md5|sha1|sha256 if specified for both, AND
3. same md5 if specified for both, AND
4. same length if specified for both
"""
if other is None:
return False
if self.uri != other.uri:
return False
if (self.timestamp is not None or other.timestamp is not None):
if (other is None): return False
if (self.uri != other.uri):
return(False)
if ( self.timestamp is not None or other.timestamp is not None ):
# not equal if only one timestamp specified
if (self.timestamp is None
or other.timestamp is None
or abs(self.timestamp - other.timestamp) >= delta):
return False
if ((self.md5 is not None and other.md5 is not None)
and self.md5 != other.md5):
return False
if ((self.sha1 is not None and other.sha1 is not None)
and self.sha1 != other.sha1):
return False
if ((self.sha256 is not None and other.sha256 is not None)
and self.sha256 != other.sha256):
return False
if ((self.length is not None and other.length is not None)
and self.length != other.length):
return False
return True
if ( self.timestamp is None or
other.timestamp is None or
abs(self.timestamp-other.timestamp)>=delta ):
return(False)
if ( ( self.md5 is not None and other.md5 is not None ) and
self.md5 != other.md5 ):
return(False)
if ( ( self.length is not None and other.length is not None ) and
self.length != other.length ):
return(False)
return(True)
def __str__(self):
"""Return a human readable string for this resource.
"""Return a human readable string for this resource
Includes only the parts necessary for synchronizaion and
designed to support logging.
Includes only part necessary for synchronizaion
"""
s = [str(self.uri), str(self.lastmod), str(self.length),
str(self.md5 if self.md5 else (self.sha1 if self.sha1 else self.sha256))]
s = [ str(self.uri), str(self.lastmod), str(self.length),
str(self.md5 if self.md5 else self.sha1) ]
if (self.change is not None):
ch = str(self.change)
if self.datetime is not None:
ch += " @ " + str(self.datetime)
s.append(ch)
s.append(str(self.change))
if (self.path is not None):
s.append(str(self.path))
return "[ " + " | ".join(s) + " ]"
def __repr__(self):
"""Return an unambigous representation of this resource.
Uses format like Python's representation of a dict() for
attributes. Includes only those attributes with values that
are not None. Order defined by __slots__.
"""
s = []
for attr in self.__slots__:
val = getattr(self, attr, None)
if (val is not None):
s.append(repr(attr) + ': ' + repr(val))
return "{" + ", ".join(s) + "}"
class ChangeTypeError(Exception):
"""Exception class raised by Resource for bad change attribute.
The change attribute of a Resource object may be either None
or one of the values in Resource.CHANGE_TYPES. Checking is
disabled by setting Resource.CHANGE_TYPES False.
"""
def __init__(self, val):
"""Initialize ChangeTypeError, store val."""
self.supplied = val
def __repr__(self):
"""Helpful error message."""
return "<ChangeTypeError: got %s, expected one of %s>" % \
(self.supplied, str(Resource.CHANGE_TYPES))
def __str__(self):
"""Helpful error message."""
return repr(self)
"""Return an unambigous representation"""
dict_repr = dict((name, getattr(self, name, None))
for name in dir(self) if not (name.startswith('__')
or name == 'equal'
or name == 'basename'
or name == 'lastmod'))
return str(dict_repr)

234
resync/resource_container.py
View File

@ -1,37 +1,19 @@
"""ResourceSync Resource Container object.
"""ResourceSync Resource Container object
All documents in ResourceSync have the same types of information:
they have some top-level metadata and links, and then a list of
resources, each of which may also have metadata and links. This
resources, each of which may also have metadata and links. This
class implements this model.
This is a base class for the ListBase class which is in turn the
base class for ResourceList, ChangeList, etc. This class provides
only the data storage and manipulation, the ListBase class
adds IO.
only the data storage and manipulation, ListBase adds IO.
"""
import collections.abc
from .w3c_datetime import datetime_to_str
def _str_datetime_now(x='now'):
"""Return datetime string for use with time attributes.
Handling depends on input:
'now' - returns datetime for now
other string - no change, return same value
otherwise - assume datetime value, generate string
"""
if (x == 'now' or not isinstance(x, str)):
# Now, this is wht datetime_to_str() with no arg gives
return datetime_to_str(x)
else:
# Assume valid datetime string
return x
import collections
from w3c_datetime import datetime_to_str
class ResourceContainer(object):
"""Class containing resource-like objects.
"""Class containing resource-like objects
Core functionality::
- resources property that is the set/list of resources
@ -39,179 +21,102 @@ class ResourceContainer(object):
-- iter() to get iterator over self.resource in appropriate order
- md property that is a dict of metadata
- ln property that is a list of links
- uri is optional identifier of this container object
- capability_name - name of this capability
The properties md_from, md_at, md_until and md_completed are datetime
strings. The setters acceopt either a string value (not checked), the value
'now' to generate the current datetime, or otherwise an integer timestamp
value that is converted to a string.
Derived classes may add extra functionality such as len() etc..
However, any code designed to work with any ResourceContainer
should use only the core functionality.
"""
def __init__(self, resources=None, md=None, ln=None,
uri=None, capability_name=None):
"""Initialize ResourceContainer."""
self.resources = (resources if (resources is not None) else [])
self.md = (md if (md is not None) else {})
self.ln = (ln if (ln is not None) else [])
self.uri = uri
self.capability_name = capability_name
def __init__(self, resources=None, md=None, ln=None):
self.resources=(resources if (resources is not None) else [])
self.md=(md if (md is not None) else {})
self.ln=(ln if (ln is not None) else [])
self.capability_name=None
self.capability_md=None
def __iter__(self):
"""Iterator over all the resources in this resource list.
"""Iterator over all the resources in this resource list
Baseline implementation use iterator given by resources property.
Baseline implementation use iterator given by resources property
"""
return iter(self.resources)
def __getitem__(self, index):
"""Feed through for __getitem__ of resources property."""
return self.resources[index]
return(iter(self.resources))
@property
def capability(self):
"""Get/set the <rs:md capability="" .../> attribute, None if not set."""
return self.md.get('capability')
if ('capability' in self.md):
return(self.md['capability'])
else:
return(None)
@capability.setter
def capability(self, capability):
self.md['capability'] = capability
def capability(self,capability):
"""Get/set the capability attribute of this resource container"""
self.md['capability']=capability
@property
def md_from(self):
"""Get/set the <rs:md from="" .../> attribute, None if not set."""
return self.md.get('md_from')
@md_from.setter
def md_from(self, md_from):
self.md['md_from'] = _str_datetime_now(md_from)
@property
def md_until(self):
"""Get/set the <rs:md until="" .../> attribute, None if not set."""
return self.md.get('md_until')
@md_until.setter
def md_until(self, md_until):
self.md['md_until'] = _str_datetime_now(md_until)
@property
def md_at(self):
"""Get/set the <rs:md at="" attribute."""
return self.md.get('md_at')
@md_at.setter
def md_at(self, md_at):
self.md['md_at'] = _str_datetime_now(md_at)
@property
def md_completed(self):
"""Get/set the <rs:md completed="" .../> attribute, None if not set."""
return self.md.get('md_completed')
@md_completed.setter
def md_completed(self, md_completed):
self.md['md_completed'] = _str_datetime_now(md_completed)
def link(self, rel):
"""Look for link with specified rel, return else None."""
for link in self.ln:
if ('rel' in link and link['rel'] == rel):
return link
return None
def link_href(self, rel):
"""Look for link with specified rel, return href from it or None."""
link = self.link(rel)
if (link is not None):
link = link['href']
return link
def link_set(self, rel, href, **atts):
"""Set/create link with specified rel, set href and any other attributes.
Any link element must have both rel and href values, the specification
also defines the type attributes and others are permitted also. See
description of allowed formats in
http://www.openarchives.org/rs/resourcesync#DocumentFormats
"""
link = self.link(rel)
if link is None:
# create new link
link = {'rel': rel, 'href': href}
self.ln.append(link)
def modified(self):
if ('modified' in self.md):
return(self.md['modified'])
else:
# overwrite current value
link['href'] = href
for k in atts:
link[k] = atts[k]
return(None)
@property
def describedby(self):
"""Convenient access to <rs:ln rel="describedby" href="uri">."""
return(self.link_href('describedby'))
@modified.setter
def modified(self,modified):
"""Get/set the modified attribute of this resource container"""
self.md['modified']=modified
@describedby.setter
def describedby(self, uri):
"""Set ResourceSync Description link to given URI."""
self.link_set('describedby', uri)
@property
def up(self):
"""Get the URI of any ResourceSync rel="up" link."""
return(self.link_href('up'))
def default_capability_and_modified(self):
"""Set capability name and modified time in md
@up.setter
def up(self, uri):
"""Set ResourceSync rel="up" link to given URI."""
self.link_set('up', uri)
@property
def index(self):
"""Get the URI of and ResourceSync rel="index" link."""
return(self.link_href('index'))
@index.setter
def index(self, uri):
"""Set index link to given URI."""
self.link_set('index', uri)
def default_capability(self):
"""Set capability name in md.
Every ResourceSync document should have the top-level
capability attributes.
Every ResourceSync document should have these two top-level
metadata attributes.
"""
if ('capability' not in self.md and self.capability_name is not None):
self.md['capability'] = self.capability_name
if ('capability' not in self.md and self.capability_md is not None):
self.md['capability']=self.capability_md
if ('modified' not in self.md):
self.md['modified']=datetime_to_str(no_fractions=True)
def add(self, resource):
"""Add a resource or an iterable collection of resources to this container.
"""Add a resource or an iterable collection of resources to this container
Must be implemented in derived class.
Must be implemented in derived class
"""
if isinstance(resource, collections.abc.Iterable):
if isinstance(resource, collections.Iterable):
for r in resource:
self.resources.append(r)
else:
self.resources.append(resource)
def uris(self):
"""Return list of all URIs, possibly including dupes."""
"""Return list of all URIs, possibly including dupes"""
uris = []
for r in self.resources:
uris.append(r.uri)
return(uris)
def prune_dupes(self):
"""Remove all but the last entry for a given resource URI.
def prune_before(self, timestamp):
"""Remove all resources with timestamp earlier than that given
Returns the number of entries removed. Will raise an excpetion
if there are any entries without a timestamp.
"""
n = 0
pruned = []
for r in self.resources:
if (r.timestamp is None):
raise Exception("Entry %s has no timestamp" % (r.uri))
elif (r.timestamp >= timestamp):
pruned.append(r)
else:
n += 1
self.resources = pruned
return(n)
Returns the number of entries removed. Also removes all entries for a
def prune_dupes(self):
"""Remove all but the last entry for a given resource URI
Returns the number of entries removed. Also removes all entries for a
given URI where the first entry is a create and the last entry is a
delete.
"""
@ -223,12 +128,12 @@ class ResourceContainer(object):
if (r.uri in seen):
n += 1
if (r.uri in deletes):
deletes[r.uri] = r.change
deletes[r.uri]=r.change
else:
pruned1.append(r)
seen.add(r.uri)
if (r.change == 'deleted'):
deletes[r.uri] = r.change
deletes[r.uri]=r.change
# go through all deletes and prune if first was create
pruned2 = []
for r in reversed(pruned1):
@ -236,5 +141,12 @@ class ResourceContainer(object):
n += 1
else:
pruned2.append(r)
self.resources = pruned2
self.resources=pruned2
return(n)
def __str__(self):
"""Return string of all resources in order given by interator"""
s = ''
for resource in self:
s += str(resource) + "\n"
return(s)

39
resync/resource_dump.py
View File

@ -1,42 +1,25 @@
"""ResourceSync Resource Dump object.
"""ResourceSync ResourceDump object
A Resource Dump is a descriptions of a set of content packages
that contain copies of resources at a point in time.
A ResourceDump is a set of content package resources with
some metadata for each resource.
The Resource Dump object may also contain metadata and links.
The ResourceDump object may also contain metadata and links.
Described in specification at:
http://www.openarchives.org/rs/resourcesync#ResourceDump
"""
from .resource_list import ResourceList
from resource_list import ResourceList
class ResourceDump(ResourceList):
"""Class representing an Resource Dump.
"""Class representing an Resource Dump
A Resource Dump comprises a set of content packages that are
the resources listed. Properties similar to a ResourceList
A ResourceDump comprises a set of content packages that are
the resources listed. Properties much Like a ResourceList
and implemented as a sub-class of ResourceList.
See the Dump class for details of how to create a Resource
Dump from a ResourceList.
"""
def __init__(self, resources=None, md=None, ln=None, uri=None,
allow_multifile=None, mapper=None,
spec_version='1.1', add_lastmod=False):
"""Initialize ResourceDump."""
super(ResourceDump, self).__init__(
resources=resources, md=md, ln=ln, uri=uri, mapper=mapper,
spec_version=spec_version, add_lastmod=add_lastmod)
def __init__(self, resources=None, md=None, ln=None, allow_multifile=None, mapper=None):
super(ResourceDump, self).__init__(resources=resources, md=md, ln=ln, mapper=mapper)
self.capability_name = 'resourcedump'
def write(self, basename="/tmp/resource_dump.xml"):
"""Write out a ResourceDump document.
FIXME - should there be support for a tie to writing out
the Dump files themselves, complete with Resource Dump
Manifests?
"""
super(ResourceDump, self).write(basename)
self.capability_md = 'resourcedump'

19
resync/resource_dump_manifest.py
View File

@ -1,6 +1,6 @@
"""ResourceSync ResourceDumpManifest object.
"""ResourceSync ResourceDumpManifest object
A ResourceDumpManifest lists the set of files/resources included
A ResourceDumpManifest lists the set of files/resources included
within a content package that is included in a ResourceDump.
The ResourceDumpManifest object may also contain metadata and links.
@ -9,22 +9,17 @@ Described in specification at:
http://www.openarchives.org/rs/resourcesync#ResourceDumpManifest
"""
from .resource_list import ResourceList
from resource_list import ResourceList
class ResourceDumpManifest(ResourceList):
"""Class representing a Resource Dump Manifest.
"""Class representing a Resource Dump Manifest
A ResourceDumpManifest comprises a set of files/resources
in a content package. Properties much Like a ResourceList
and implemented as a sub-class of ResourceList.
"""
def __init__(self, resources=None, md=None, ln=None,
uri=None, allow_multifile=None, mapper=None,
spec_version='1.1', add_lastmod=False):
"""Initialize ResourceDumpManifest."""
super(ResourceDumpManifest, self).__init__(
resources=resources, md=md, ln=ln, uri=uri, mapper=mapper,
spec_version=spec_version, add_lastmod=add_lastmod)
def __init__(self, resources=None, md=None, ln=None, allow_multifile=None, mapper=None):
super(ResourceDumpManifest, self).__init__(resources=resources, md=md, ln=ln, mapper=mapper)
self.capability_name = 'resourcedump-manifest'
self.capability_md = 'resourcedump-manifest'

255
resync/resource_list.py
View File

@ -1,226 +1,155 @@
"""ResourceSync Resource List object.
"""ResourceSync resource_list object
A Resource List is a set of resources with some metadata for
each resource. Every resource must have a uri attribute.
Comparison of resource lists from a source and a destination
allows understanding of whether the two are in sync or whether
some resources need to be updated at the destination.
An resource_list is a set of resources with some metadata for
each resource. Comparison of resource lists from a source and
a destination allows understanding of whether the two are in
sync or whether some resources need to be updated at the
destination.
There may also be metadata about the Resource List, and links
to other ResourceSync documents. Metadata include the
timestamp of the Resource List (md_at) and, optionally, the
timestamp when creation of the Resource List was completed
(md_completed) at the top level. These include a creation timestamp
(from) and links to the Capability List.
The resource_list object may also contain metadata and links.
Described in specification at:
http://www.openarchives.org/rs/resourcesync#DescResources
- mandatory <rs:md> element:
-- must have capability="resourcelist"
-- should have modified=".." attribute also, which should match the
mtime of a file written.
- one <url> element for each resource
"""
import collections.abc
from collections import OrderedDict
import collections
import os
from datetime import datetime
import re
import sys
from urllib import URLopener
from .list_base_with_index import ListBaseWithIndex
from .sitemap import Sitemap
from .mapper import Mapper, MapperError
from .url_authority import UrlAuthority
from list_base_with_index import ListBaseWithIndex
from sitemap import Sitemap
from mapper import Mapper, MapperError
from url_authority import UrlAuthority
class SortedIterMixin(object):
"""Mixin to provide sorted_iter() method."""
class ResourceListDict(dict):
"""Default implementation of class to store resources in ResourceList
def sorted_iter(self):
"""Iterator over all the resources in this dict by sorted key order."""
self._sorted_iter_next_list = sorted(self.keys(), reverse=True)
return iter(self._sorted_iter_next, None)
Key properties of this class are:
- has add(resource) method
- is iterable and results given in alphanumeric order by resource.uri
"""
def _sorted_iter_next(self):
if (len(self._sorted_iter_next_list) > 0):
return self[self._sorted_iter_next_list.pop()]
def __iter__(self):
"""Iterator over all the resources in this resource_list"""
self._iter_next_list = sorted(self.keys())
self._iter_next_list.reverse()
return(iter(self._iter_next, None))
def _iter_next(self):
if (len(self._iter_next_list)>0):
return(self[self._iter_next_list.pop()])
else:
return None
class ResourceListDict(dict, SortedIterMixin):
"""Default implementation of class to store resources in ResourceList.
Key properties of this class are:
- has add(resource) method
- is iterable and gives resources (not keys) in alphanumeric order by
resource.uri
"""
def __iter__(self):
"""Iterator over all the resources in this ResourceListDict."""
return self.sorted_iter()
return(None)
def uris(self):
"""Extract sorted list of URIs for resources in this ResourceListDict."""
return sorted(self.keys())
return(sorted(self.keys()))
def add(self, resource, replace=False):
"""Add just a single resource."""
uri = resource.uri
if (uri in self.keys() and not replace):
raise ResourceListDupeError(
"Attempt to add resource already in resource_list")
self[uri] = resource
class ResourceListOrdered(OrderedDict, SortedIterMixin):
"""Alternative implementation of class to store resources in ResourceList.
Key properties of this class are:
- has add(resource) method
- is iterable and gives resources (not keys) in the order added
"""
def __iter__(self):
"""Iterator over the resources in the list in the order added."""
return iter(self.values())
def uris(self):
"""Extract list of all resource URIs in the order added."""
return self.keys()
def add(self, resource, replace=False):
"""Add just a single resource."""
"""Add just a single resource"""
uri = resource.uri
if (uri in self and not replace):
raise ResourceListDupeError(
"Attempt to add resource already in resource_list")
self[uri] = resource
raise ResourceListDupeError("Attempt to add resource already in resource_list")
self[uri]=resource
class ResourceListDupeError(Exception):
"""Exception in case of duplicate resource."""
pass
class ResourceList(ListBaseWithIndex):
"""Class representing a ResourceList.
"""Class representing an resource_list of resources
This same class is used for both the source and the destination
and is the central point of comparison the decide whether they
are in sync or what needs to be copied to bring the destinaton
into sync.
A ResourceList will admit only one resource with any given URI.
An resource_list will admit only one resource with any given URI.
Typical usage for a small ResourceList is:
rl = ResourceList()
rl.add( Resource(...) )
rl.add( Resource(...) )
print rl.as_xml()
The default storage is unordered but the iterator imposes a canonical
order which is alphabetical by URI. If it is desired to have
resources listed in the order they are added then the ResourceListOrdered
class may be specified on creation:
rl = ResourceList( resources_class=ResourceDictOrdered )
Use of ResourceListOrdered will be faster than the default.
In normal use it is expected that any Resource List Index will be
created automatically when writing out a large Resource List in
multiple sitemap files. However, should it be necessary to
explicitly create an index then this may be specified with:
rli = ResourceList( resources_class=ResourceDictOrdered )
rli.sitemapindex=True
See additional descriptions in ListBaseWithIndex and ListBase.
Storage is unordered but the iterator imposes a canonical order
which is currently alphabetical by URI.
"""
def __init__(self, resources=None, count=None, md=None, ln=None, uri=None,
allow_multifile=None, mapper=None,
spec_version='1.1', add_lastmod=False,
resources_class=ResourceListDict):
"""Initialize ResourceList."""
super(ResourceList, self).__init__(
resources=resources,
count=count,
md=md,
ln=ln,
uri=uri,
capability_name='resourcelist',
allow_multifile=allow_multifile,
mapper=mapper, spec_version=spec_version,
add_lastmod=add_lastmod,
resources_class=resources_class)
def __init__(self, resources=None, md=None, ln=None, allow_multifile=None, mapper=None):
if (resources is None):
resources = ResourceListDict()
super(ResourceList, self).__init__(resources=resources, md=md, ln=ln, mapper=mapper)
self.capability_name = 'resourcelist'
self.capability_md = 'resourcelist'
self.resources_class = ResourceListDict
def add(self, resource, replace=False):
"""Add a resource or an iterable collection of resources.
"""Add a resource or an iterable collection of resources
Will throw a ValueError if the resource (ie. same uri) already
exists in the ResourceList, unless replace=True.
exists in the resource_list, unless replace=True.
"""
if isinstance(resource, collections.abc.Iterable):
if isinstance(resource, collections.Iterable):
for r in resource:
self.resources.add(r, replace)
self.resources.add(r,replace)
else:
self.resources.add(resource, replace)
self.resources.add(resource,replace)
def compare(self, src):
"""Compare this ResourceList object with that specified as src.
def compare(self,src):
"""Compare the current resource_list object with the specified resource_list
The parameter src must also be a ResourceList object, it is assumed
to be the source, and the current object is the destination. This
The parameter src must also be an resource_list object, it is assumed
to be the source, and the current object is the destination. This
written to work for any objects in self and sc, provided that the
== operator can be used to compare them.
The functioning of this method depends on the sorted_iter() iterators
for self and src providing access to the resource objects in URI order.
The functioning of this method depends on the iterators for self and
src providing access to the resource objects in URI order.
"""
dst_iter = self.resources.sorted_iter()
src_iter = src.resources.sorted_iter()
same = ResourceList()
updated = ResourceList()
deleted = ResourceList()
created = ResourceList()
dst_cur = next(dst_iter, None)
src_cur = next(src_iter, None)
dst_iter = iter(self.resources)
src_iter = iter(src.resources)
same=ResourceList()
updated=ResourceList()
deleted=ResourceList()
created=ResourceList()
dst_cur=next(dst_iter,None)
src_cur=next(src_iter,None)
while ((dst_cur is not None) and (src_cur is not None)):
# print 'dst='+dst_cur+' src='+src_cur
if dst_cur.uri == src_cur.uri:
if (dst_cur == src_cur):
#print 'dst='+dst_cur+' src='+src_cur
if (dst_cur.uri == src_cur.uri):
if (dst_cur==src_cur):
same.add(dst_cur)
else:
updated.add(src_cur)
dst_cur = next(dst_iter, None)
src_cur = next(src_iter, None)
elif dst_cur.uri < src_cur.uri:
dst_cur=next(dst_iter,None)
src_cur=next(src_iter,None)
elif (not src_cur or dst_cur.uri < src_cur.uri):
deleted.add(dst_cur)
dst_cur = next(dst_iter, None)
else: # dst_cur.uri > src_cur.uri:
dst_cur=next(dst_iter,None)
elif (not dst_cur or dst_cur.uri > src_cur.uri):
created.add(src_cur)
src_cur = next(src_iter, None)
src_cur=next(src_iter,None)
else:
raise InternalError("this should not be possible")
# what do we have leftover in src or dst lists?
while (dst_cur is not None):
deleted.add(dst_cur)
dst_cur = next(dst_iter, None)
dst_cur=next(dst_iter,None)
while (src_cur is not None):
created.add(src_cur)
src_cur = next(src_iter, None)
src_cur=next(src_iter,None)
# have now gone through both lists
return(same, updated, deleted, created)
return(same,updated,deleted,created)
def hashes(self):
"""Return set of hashes uses in this resource_list."""
hashes = set()
if (self.resources is not None):
for resource in self:
if (resource.md5 is not None):
hashes.add('md5')
if (resource.sha1 is not None):
hashes.add('sha-1')
if (resource.sha256 is not None):
hashes.add('sha-256')
return(hashes)
def has_md5(self):
"""Return true if at least one contained resource-like object has md5 data"""
if (self.resources is None):
return(False)
for resource in self:
if (resource.md5 is not None):
return(True)
return(False)

252
resync/resource_list_builder.py
View File

@ -1,188 +1,138 @@
"""ResourceListBuilder to create ResourceList objects."""
"""ResourceListBuilder to create ResourceList objects
Currently implements build from files on disk only.
Attributes:
- do_md5 set true to calculate MD5 sums for all files
- do_length set true to include file length in resource_list
- exclude_dirs is a list of directory names to exclude
(defaults to ['CVS','.git'))
"""
import os
import os.path
import re
import time
import logging
from defusedxml.ElementTree import parse
from .hashes import Hashes
from .resource import Resource
from .resource_list import ResourceList
from .sitemap import Sitemap
from .w3c_datetime import datetime_to_str
from urllib import URLopener
from xml.etree.ElementTree import parse
from resource import Resource
from resource_list import ResourceList
from sitemap import Sitemap
from utils import compute_md5_for_file
class ResourceListBuilder():
"""ResourceListBuilder to create ResourceList objects.
Currently implements build from files on disk only.
def __init__(self, do_md5=False, do_length=True, mapper=None):
"""Create ResourceListBuilder object, optionally set options
Attributes:
- set_hashes to indicate which hashes should be calculated for each resource
- set_path set true to add path attribute for each resource
- set_length set true to include file length in resource_list (defaults true)
- exclude_patterns is a list of files and directory names patterns to exclude
using re.match(..). These patterns are left anchored so thus need to be
preceded with .* if there may be arbitrary leading characters (defaults to
empty)
"""
def __init__(self, mapper=None, set_hashes=None,
set_length=True, set_path=False):
"""Create ResourceListBuilder object, optionally set options.
The mapper attribute must be set before a call to from_disk() in order to
map between local filenames and URIs.
The following attributes may be set to determine information added to
each Resource object based on the disk scan:
- set_hashes - Set of digests to compute for each resource. This may add
significant time to the scan process as each file has to be read to
compute the hash. Empty set or None means no hashes calculated, names
defined in Hashes object
- set_length - False to not add length for each resources
- set_path - True to add local path information for each file/resource
Optionaly sets the following attributes:
- do_md5 - True to add md5 digests for each resource
- do_length - False to not add length for each resources
"""
self.do_md5 = do_md5
self.do_length = do_length
self.mapper = mapper
self.set_path = set_path
self.set_hashes = set_hashes if (set_hashes and len(set_hashes) > 0) else None
self.set_length = set_length
self.exclude_patterns = []
self.exclude_files = ['sitemap\d{0,5}.xml']
self.exclude_dirs = ['CVS','.git']
self.include_symlinks = False
self.log_count_increment = 50000 # Write log message after 50000 files
# Used internally only:
self.logger = logging.getLogger('resync.resource_list_builder')
self.compiled_exclude_patterns = []
self.logger = logging.getLogger('resource_list_builder')
self.compiled_exclude_files = []
def add_exclude_patterns(self, exclude_patterns):
"""Add more patterns of files or directories to exclude while building resource_list.
After patterns are added the set of compiled patterns is updated.
"""
def add_exclude_files(self, exclude_patterns):
"""Add more patterns of files to exclude while building resource_list"""
for pattern in exclude_patterns:
self.exclude_patterns.append(pattern)
self._compile_excludes()
self.exclude_files.append(pattern)
def _compile_excludes(self):
# Compile a set of regexps for files and directories to be exlcuded from scans
self.compiled_exclude_patterns = []
for pattern in self.exclude_patterns:
try:
self.compiled_exclude_patterns.append(re.compile(pattern))
except re.error as e:
raise ValueError(
"Bad python regex in exclude pattern '%s': %s" % (pattern, str(e)))
def compile_excludes(self):
self.compiled_exclude_files = []
for pattern in self.exclude_files:
self.compiled_exclude_files.append(re.compile(pattern))
def _exclude(self, file):
# True if file should be exclude based on name pattern.
for pattern in self.compiled_exclude_patterns:
if pattern.match(file):
return True
return False
def exclude_file(self, file):
"""True if file should be exclude based on name pattern"""
for pattern in self.compiled_exclude_files:
if (pattern.match(file)):
return(True)
return(False)
def from_disk(self, resource_list=None, paths=None):
"""Create or extend resource_list with resources from disk scan.
def from_disk(self, resource_list=None, set_path=False):
"""Create or extend resource_list with resources from disk scan
Assumes very simple disk path to URL mapping (in self.mapping): chop
path and replace with url_path. Returns the new or extended ResourceList
Assumes very simple disk path to URL mapping: chop path and
replace with url_path. Returns the new or extended ResourceList
object.
If a resource_list is specified then items are added to that rather
than creating a new one.
If paths is specified then these are used instead of the set
of local paths in self.mapping.
Example usage with mapping start paths:
If set_path is True then the path attribue will be set with the
local path for each Resource.
mapper=Mapper('http://example.org/path','/path/to/files')
rlb = ResourceListBuilder(mapper=mapper)
m = rlb.from_disk()
Example usage with explicit paths:
mapper=Mapper('http://example.org/path','/path/to/files')
rlb = ResourceListBuilder(mapper=mapper)
m = rlb.from_disk(paths=['/path/to/files/a','/path/to/files/b'])
mb = ResourceListBuilder(mapper=mapper)
m = resource_list_from_disk()
"""
num = 0
num=0
# Either use resource_list passed in or make a new one
if (resource_list is None):
resource_list = ResourceList()
# Work out start paths from map if not explicitly specified
if (paths is None):
paths = []
for map in self.mapper.mappings:
paths.append(map.dst_path)
# Set start time unless already set (perhaps because building in
# chunks)
if (resource_list.md_at is None):
resource_list.md_at = datetime_to_str()
# Compile exclude pattern matches
self.compile_excludes()
# Run for each map in the mappings
for path in paths:
self.logger.info("Scanning disk from %s" % (path))
self.from_disk_add_path(path=path, resource_list=resource_list)
# Set end time
resource_list.md_completed = datetime_to_str()
for map in self.mapper.mappings:
self.logger.info("Scanning disk for %s" % (str(map)))
self.from_disk_add_map(resource_list=resource_list, map=map, set_path=set_path)
return(resource_list)
def from_disk_add_path(self, path=None, resource_list=None):
"""Add to resource_list with resources from disk scan starting at path."""
# sanity
if (path is None or resource_list is None or self.mapper is None):
raise ValueError("Must specify path, resource_list and mapper")
# is path a directory or a file? for each file: create Resource object,
# add, increment counter
if os.path.isdir(path):
num_files = 0
for dirpath, dirs, files in os.walk(path, topdown=True):
for file_in_dirpath in files:
num_files += 1
if (num_files % self.log_count_increment == 0):
self.logger.info(
"ResourceListBuilder.from_disk_add_path: %d files..." % (num_files))
self.add_file(resource_list=resource_list,
dir=dirpath, file=file_in_dirpath)
# prune list of dirs based on self.exclude_dirs
prune = []
for dir in dirs:
if self._exclude(dir):
self.logger.debug("Excluding dir '%s'" % (dir))
prune.append(dir)
for dir in prune:
dirs.remove(dir)
else:
# single file
self.add_file(resource_list=resource_list, file=path)
def from_disk_add_map(self, resource_list=None, map=None, set_path=False):
"""Add to resource_list with resources from disk scan based one map
def add_file(self, resource_list=None, dir=None, file=None):
"""Add a single file to resource_list.
Follows object settings of set_path, set_hashes and set_length.
If set_path is True then the path attribue will be set with the
local path for each Resource.
"""
if self._exclude(file):
self.logger.debug("Excluding file '%s'" % (file))
return
# get abs filename and also URL
if (dir is not None):
file = os.path.join(dir, file)
if os.path.islink(file) and not self.include_symlinks:
self.logger.warning("Ignoring symlink '%s'" % (file))
return
try:
uri = self.mapper.dst_to_src(file) # might throw MapperError
file_stat = os.stat(file)
except OSError as e:
self.logger.warning("Ignoring file '%s' (error: %s)" % (file, str(e)))
return
timestamp = file_stat.st_mtime # UTC
r = Resource(uri=uri, timestamp=timestamp)
if self.set_path: # add full local path
r.path = file
if self.set_hashes: # add any hashes requested
Hashes(self.set_hashes, file).set(r)
if self.set_length: # add length
r.length = file_stat.st_size
resource_list.add(r)
# sanity
if (resource_list is None or map is None):
raise ValueError("Must specify resource_list and map")
path=map.dst_path
#print "walking: %s" % (path)
# for each file: create Resource object, add, increment counter
num_files=0
for dirpath, dirs, files in os.walk(path,topdown=True):
for file_in_dirpath in files:
num_files+=1
if (num_files%50000 == 0):
self.logger.info("ResourceListBuilder.from_disk_add_map: %d files..." % (num_files))
try:
if self.exclude_file(file_in_dirpath):
self.logger.debug("Excluding file %s" % (file_in_dirpath))
continue
# get abs filename and also URL
file = os.path.join(dirpath,file_in_dirpath)
if (not os.path.isfile(file) or not (self.include_symlinks or not os.path.islink(file))):
continue
uri = map.dst_to_src(file)
if (uri is None):
raise Exception("Internal error, mapping failed")
file_stat=os.stat(file)
except OSError as e:
sys.stderr.write("Ignoring file %s (error: %s)" % (file,str(e)))
continue
timestamp = file_stat.st_mtime #UTC
r = Resource(uri=uri,timestamp=timestamp)
if (set_path):
r.path=file
if (self.do_md5):
# add md5
r.md5=compute_md5_for_file(file)
if (self.do_length):
# add length
r.length=file_stat.st_size
resource_list.add(r)
# prune list of dirs based on self.exclude_dirs
for exclude in self.exclude_dirs:
if exclude in dirs:
self.logger.debug("Excluding dir %s" % (exclude))
dirs.remove(exclude)

29
resync/resource_set.py
View File

@ -1,14 +1,9 @@
"""Base class representing a set of Resource objects."""
"""A set of Resource objects
"""
class ResourceSet(dict):
"""Base class representing a set of Resource objects.
The ResourceSet class is used for Capability List Indexes and
ResourceSync Description documents.
Ordering of Resources is currently alphanumeric (using sorted(..))
on the uri which is the key.
"""Default implementation of class to store resources in ResourceList
Key properties of this class are:
- has add(resource) method
@ -16,31 +11,23 @@ class ResourceSet(dict):
"""
def __iter__(self):
"""Iterator over all the resources in this ResourceSet."""
"""Iterator over all the resources in this resource_list"""
self._iter_next_list = sorted(self.keys())
self._iter_next_list.reverse()
return(iter(self._iter_next, None))
def _iter_next(self):
if (len(self._iter_next_list) > 0):
if (len(self._iter_next_list)>0):
return(self[self._iter_next_list.pop()])
else:
return(None)
def add(self, resource, replace=False):
"""Add just a single resource."""
"""Add just a single resource"""
uri = resource.uri
if (uri in self and not replace):
raise ResourceSetDupeError(
"Attempt to add resource already in this set")
self[uri] = resource
def first(self):
"""First item obtained from iterator."""
return(next(iter(self)))
raise ResourceSetDupeError("Attempt to add resource already in this set")
self[uri]=resource
class ResourceSetDupeError(Exception):
"""Exception for case of attempt to add duplicate resource."""
pass

488
resync/sitemap.py
View File

@ -1,112 +1,67 @@
"""Read and write ResourceSync documents as sitemaps."""
"""Read and write ResourceSync documents as sitemaps"""
import io
import logging
import re
import os
import sys
from defusedxml.ElementTree import parse
from xml.etree.ElementTree import ElementTree, Element, tostring
import logging
from xml.etree.ElementTree import ElementTree, Element, parse, tostring
import StringIO
from .resource import Resource
from .resource_container import ResourceContainer
from resource import Resource
from resource_container import ResourceContainer
SITEMAP_NS = 'http://www.sitemaps.org/schemas/sitemap/0.9'
RS_NS = 'http://www.openarchives.org/rs/terms/'
# Mapping of Resource object atts to XML atts, see Sitemap._xml_att_name()
XML_ATT_NAME = {
'mime_type': 'type',
'md_at': 'at',
'md_completed': 'completed',
'md_from': 'from',
'md_until': 'until'
}
class SitemapIndexError(Exception):
"""Exception if sitemapindex read instead of sitemap or vice-versa.
"""Exception on attempt to read a sitemapindex instead of sitemap
or vice-versa
Provides both a message and a place to store the etree so that
the parse tree may be reused as a sitemapindex.
the parse tree may be reused as a sitemapingex.
"""
def __init__(self, message=None, etree=None):
"""Initialize SitemapIndexError with supplied parameters."""
self.message = message
self.etree = etree
def __str__(self):
"""Return just the message attribute."""
def __repr__(self):
return(self.message)
class SitemapParseError(Exception):
"""Exception for sitemap parsing structural error."""
pass
class SitemapDupeError(Exception):
"""Exception for case of duplicate resources in sitemap."""
pass
class Sitemap(object):
"""Read and write sitemaps.
"""Read and write sitemaps
Implemented as a separate class that uses ResourceContainer
(ResourceList or ChangeList) and Resource classes as data objects.
Implemented as a separate class that uses ResourceContainer
(ResourceList or ChangeList) and Resource classes as data objects.
Reads and write sitemaps, including multiple file sitemaps.
This class does not automatically handle the reading or writing
of a sitemapindex and multiple sitemap documents. Instead it will
take a default and throw and exception if the other case is found
so that the calling code can handle it.
Default is to follow ResourceSync v1.1 which distinguishes the meanings
of the <lastmod> element (resource modified time as used for Last-Modified
header) and the newly introduced <rs:md datetime="..."> attribute (the
change time in ChangeList and similar resources). There are two parameters
to control these behaviors:
spec_version - set to either '1.0' or '1.1'. If set to '1.0' then no
datetime attribute will be written or parsed
add_lastmod - set True to add a lastmod with the same value of datetime
if no specific lastmod is specified. Applies only when spec_version
'1.1' is selected, provide compatibility with systems that do not
understand datetime but instead rely on lastmod
"""
def __init__(self, pretty_xml=False, spec_version='1.1', add_lastmod=False):
"""Initialize Sitemap object."""
self.logger = logging.getLogger('resync.sitemap')
self.pretty_xml = pretty_xml
self.spec_1_0 = (spec_version == '1.0') # v1.0 else assume v1.1
self.add_lastmod = add_lastmod # Optional in v1.1
def __init__(self, pretty_xml=False):
self.logger = logging.getLogger('sitemap')
self.pretty_xml=pretty_xml
# Classes used when parsing
self.resource_class = Resource
self.resource_class=Resource
# Information recorded for logging
self.resources_created = 0 # Set during parsing sitemap
self.parsed_index = None # Set True for sitemapindex, False for sitemap
# rs:md attributes
self.md_att_keys = ['md_at', 'capability', 'change', 'datetime',
'md_completed', 'md_from', 'hash', 'length',
'path', 'mime_type', 'md_until']
# capabilities
self.capabilities = ['resourcelist', 'changelist', 'resourcedump',
'changedump', 'resourcedump-manifest',
'changedump-manifest', 'capabilitylist',
'description',
'resourcelist-archive', 'resourcedump-archive',
'changelist-archive', 'changedump-archive']
if self.spec_1_0:
self.md_att_keys.remove('datetime')
self.resources_created=0 # Set during parsing sitemap
self.parsed_index=None # Set True for sitemapindex, False for sitemap
# Write the XML for a sitemap or sitemapindex
##### Write the XML for a sitemap or sitemapindex #####
def resources_as_xml(self, resources, sitemapindex=False, fh=None):
"""Write or return XML for a set of resources in sitemap format.
"""Write or return XML for a set of resources in sitemap format
Arguments:
- resources - either an iterable or iterator of Resource objects;
if there an md attribute this will go to <rs:md>
@ -115,50 +70,48 @@ class Sitemap(object):
- fh - write to filehandle fh instead of returning string
"""
# element names depending on sitemapindex or not
root_element = ('sitemapindex' if (sitemapindex) else 'urlset')
item_element = ('sitemap' if (sitemapindex) else 'url')
root_element = ( 'sitemapindex' if (sitemapindex) else 'urlset' )
item_element = ( 'sitemap' if (sitemapindex) else 'url')
# namespaces and other settings
namespaces = {'xmlns': SITEMAP_NS, 'xmlns:rs': RS_NS}
namespaces = { 'xmlns': SITEMAP_NS, 'xmlns:rs': RS_NS }
root = Element(root_element, namespaces)
if (self.pretty_xml):
root.text = "\n"
# <rs:ln>
if (hasattr(resources, 'ln')):
for ln in resources.ln:
self.add_element_with_atts_to_etree(root, 'rs:ln', ln)
root.text="\n"
# <rs:md>
if (hasattr(resources, 'md')):
self.add_element_with_atts_to_etree(root, 'rs:md', resources.md)
if (hasattr(resources,'md')):
self.add_md_to_etree(root,resources.md)
# <rs:ln>
if (hasattr(resources,'ln')):
for ln in resources.ln:
self.add_ln_to_etree(root,ln)
# <url> entries from either an iterable or an iterator
for r in resources:
e = self.resource_etree_element(r, element_name=item_element)
e=self.resource_etree_element(r, element_name=item_element)
root.append(e)
# have tree, now serialize
tree = ElementTree(root)
xml_buf = None
tree = ElementTree(root);
xml_buf=None
if (fh is None):
xml_buf = io.StringIO()
fh = xml_buf
tree.write(
fh,
encoding='unicode',
xml_declaration=True,
method='xml')
xml_buf=StringIO.StringIO()
fh=xml_buf
if (sys.version_info < (2,7)):
tree.write(fh,encoding='UTF-8')
else:
tree.write(fh,encoding='UTF-8',xml_declaration=True,method='xml')
if (xml_buf is not None):
return(xml_buf.getvalue())
# Read/parse an XML sitemap or sitemapindex
def parse_xml(self, fh=None, etree=None, resources=None,
capability=None, sitemapindex=None):
"""Parse XML Sitemap and add to resources object.
##### Read/parse an XML sitemap or sitemapindex #####
Reads from fh or etree and adds resources to a resorces object
(which must support the add method). Returns the resources object.
def parse_xml(self, fh=None, etree=None, resources=None, capability=None, sitemapindex=None):
"""Parse XML Sitemap from fh or etree and add resources to a
resorces object (which must support the add method). Returns
the resources object.
Also sets self.resources_created to be the number of resources created.
We adopt a very lax approach here. The parsing is properly namespace
aware but we search just for the elements wanted and leave everything
Also sets self.resources_created to be the number of resources created.
We adopt a very lax approach here. The parsing is properly namespace
aware but we search just for the elements wanted and leave everything
else alone.
This method will read either sitemap or sitemapindex documents. Behavior
@ -167,308 +120,281 @@ class Sitemap(object):
- False - SitemapIndexError exception if sitemapindex detected
- True - SitemapIndexError exception if sitemap detected
Will set self.parsed_index based on whether a sitemap or sitemapindex
Will set self.parsed_index based on whether a sitemap or sitemapindex
document was read:
- False - sitemap
- True - sitemapindex
"""
if (resources is None):
resources = ResourceContainer()
resources=ResourceContainer()
if (fh is not None):
etree = parse(fh)
etree=parse(fh)
elif (etree is None):
raise ValueError("Neither fh or etree set")
# check root element: urlset (for sitemap), sitemapindex or bad
root_tag = etree.getroot().tag
resource_tag = None # will be <url> or <sitemap> depending on type
resource_tag = None # will be <url> or <sitemap> depending on type
self.parsed_index = None
if (root_tag == '{' + SITEMAP_NS + "}urlset"):
if (root_tag == '{'+SITEMAP_NS+"}urlset"):
self.parsed_index = False
if (sitemapindex is not None and sitemapindex):
raise SitemapIndexError(
"Got sitemap when expecting sitemapindex", etree)
resource_tag = '{' + SITEMAP_NS + "}url"
elif (root_tag == '{' + SITEMAP_NS + "}sitemapindex"):
raise SitemapIndexError("Got sitemap when expecting sitemapindex",etree)
resource_tag = '{'+SITEMAP_NS+"}url"
elif (root_tag == '{'+SITEMAP_NS+"}sitemapindex"):
self.parsed_index = True
if (sitemapindex is not None and not sitemapindex):
raise SitemapIndexError(
"Got sitemapindex when expecting sitemap", etree)
resource_tag = '{' + SITEMAP_NS + "}sitemap"
raise SitemapIndexError("Got sitemapindex when expecting sitemap",etree)
resource_tag = '{'+SITEMAP_NS+"}sitemap"
else:
raise SitemapParseError(
"XML is not sitemap or sitemapindex (root element is <%s>)" %
root_tag)
raise SitemapParseError("XML is not sitemap or sitemapindex")
# have what we expect, read it
in_preamble = True
self.resources_created = 0
seen_top_level_md = False
for e in list(etree.getroot()):
# look for <rs:md> and <rs:ln>, first <url>/<sitemap> ends
# then look for resources in <url>/<sitemap> blocks.
# ignore any elements we don't recognize
self.resources_created=0
for e in etree.getroot().getchildren():
# look for <rs:md> and <rs:ln>, first <url> ends
# then look for resources in <url> blocks
if (e.tag == resource_tag):
in_preamble = False # any later rs:md or rs:ln is error
in_preamble = False #any later rs:md or rs:ln is error
r = self.resource_from_etree(e, self.resource_class)
resources.add(r)
self.resources_created += 1
elif (e.tag == "{" + RS_NS + "}md"):
try:
resources.add( r )
except SitemapDupeError:
self.logger.warning("dupe of: %s (lastmod=%s)" % (r.uri,r.lastmod))
self.resources_created+=1
elif (e.tag == "{"+RS_NS+"}md"):
if (in_preamble):
if (seen_top_level_md):
raise SitemapParseError(
"Multiple <rs:md> at top level of sitemap")
else:
resources.md = self.md_from_etree(e, 'preamble')
seen_top_level_md = True
# FIXME - more then one <rs:md> should be error
resources.md = self.md_from_etree(e,'preamble')
else:
raise SitemapParseError(
"Found <rs:md> after first <url> in sitemap")
elif (e.tag == "{" + RS_NS + "}ln"):
raise SitemapParseError("Found <rs:md> after first <url> in sitemap")
elif (e.tag == "{"+RS_NS+"}ln"):
if (in_preamble):
resources.ln.append(self.ln_from_etree(e, 'preamble'))
resources.ln.append(self.ln_from_etree(e,'preamble'))
else:
raise SitemapParseError(
"Found <rs:ln> after first <url> in sitemap")
raise SitemapParseError("Found <rs:md> after first <url> in sitemap")
else:
# element we don't recognize, ignore
# FIXME - might add check for debug?
pass
# check that we read to right capability document
if (capability is not None):
if ('capability' not in resources.md):
if (capability == 'resourcelist'):
self.logger.warning(
'No capability specified in sitemap, assuming resourcelist')
self.logger.warning('No capability specified in sitemap, assuming resourcelist')
resources.md['capability'] = 'resourcelist'
else:
raise SitemapParseError("Expected to read a %s document, but no capability specified in sitemap" %
(capability))
(capability))
if (resources.md['capability'] != capability):
raise SitemapParseError("Expected to read a %s document, got %s" %
(capability, resources.md['capability']))
(capability,resources.md['capability']))
# return the resource container object
return(resources)
# Resource methods
##### Resource methods #####
def resource_etree_element(self, resource, element_name='url'):
"""Return xml.etree.ElementTree.Element representing the resource.
"""Return xml.etree.ElementTree.Element representing the resource
Returns and element for the specified resource, of the form <url>
Returns and element for the specified resource, of the form <url>
with enclosed properties that are based on the sitemap with extensions
for ResourceSync.
"""
e = Element(element_name)
sub = Element('loc')
sub.text = resource.uri
sub.text=resource.uri
e.append(sub)
lm = resource.lastmod # W3C Datetime in UTC
if lm is not None or self.spec_1_0 or self.add_lastmod:
# In 1.0 we either use the lastmod specified or else use the
# datetime value because there should always be a lastmod
if lm is None and (self.spec_1_0 or self.add_lastmod):
lm = resource.datetime # W3C Datetime in UTC
if lm is not None:
sub = Element('lastmod')
sub.text = lm
e.append(sub)
if (resource.timestamp is not None):
# Create appriate element for timestamp
sub = Element('lastmod')
sub.text = str(resource.lastmod) #W3C Datetime in UTC
e.append(sub)
md_atts = {}
for att in self.md_att_keys:
val = getattr(resource, att, None)
for att in ('capability','change','hash','length','path','type'):
val = getattr(resource,att,None)
if (val is not None):
md_atts[att] = str(val)
if (len(md_atts) > 0):
self.add_element_with_atts_to_etree(e, 'rs:md', md_atts, add_return=False)
if (len(md_atts)>0):
md = Element('rs:md',md_atts)
e.append(md)
# add any <rs:ln>
if (hasattr(resource, 'ln') and resource.ln is not None):
if (hasattr(resource,'ln') and
resource.ln is not None):
for ln in resource.ln:
self.add_element_with_atts_to_etree(e, 'rs:ln', ln)
self.add_ln_to_etree(e,ln)
if (self.pretty_xml):
e.tail = "\n"
e.tail="\n"
return(e)
def resource_as_xml(self, resource):
"""Return string for the resource as part of an XML sitemap.
def resource_as_xml(self,resource,indent=' '):
"""Return string for the the resource as part of an XML sitemap
Returns a string with the XML snippet representing the resource,
without any XML declaration. (So much simpler now only Python 3.x
supported, see earlier versions for 2.6, 2.7 etc.)
"""
e = self.resource_etree_element(resource)
return(tostring(e, encoding='unicode', method='xml'))
if (sys.version_info < (2,7)):
#must not specify method='xml' in python2.6
return(tostring(e, encoding='UTF-8'))
else:
return(tostring(e, encoding='UTF-8', method='xml'))
def resource_from_etree(self, etree, resource_class):
"""Construct a Resource from an etree.
"""Construct a Resource from an etree
Parameters:
etree - the etree to parse
resource_class - class of Resource object to create
The parsing is properly namespace aware but we search just
for the elements wanted and leave everything else alone. Will
raise an error if there are multiple <loc> or multiple <lastmod>
elements. Otherwise, provided there is a <loc> element then will
go ahead and extract as much as possible.
All errors raised are SitemapParseError with messages intended
to help debug problematic sitemap XML.
The parsing is properly namespace aware but we search just for
the elements wanted and leave everything else alone. Provided
there is a <loc> element then we'll go ahead and extract as much
as possible.
"""
loc_elements = etree.findall('{' + SITEMAP_NS + "}loc")
if (len(loc_elements) > 1):
raise SitemapParseError(
"Multiple <loc> elements while parsing <url> in sitemap")
elif (len(loc_elements) == 0):
raise SitemapParseError(
"Missing <loc> element while parsing <url> in sitemap")
else:
loc = loc_elements[0].text
if (loc is None or loc == ''):
raise SitemapParseError(
"Bad <loc> element with no content while parsing <url> in sitemap")
loc = etree.findtext('{'+SITEMAP_NS+"}loc")
if (loc is None):
raise SitemapParseError("Missing <loc> element while parsing <url> in sitemap")
###FIXME - test for mutliple <loc> -> error
# must at least have a URI, make this object
resource = resource_class(uri=loc)
# and hopefully a lastmod datetime (but none is OK)
lastmod_elements = etree.findall('{' + SITEMAP_NS + "}lastmod")
if (len(lastmod_elements) > 1):
raise SitemapParseError(
"Multiple <lastmod> elements while parsing <url> in sitemap")
elif (len(lastmod_elements) == 1):
resource.lastmod = lastmod_elements[0].text
# proceed to look for other resource attributes in an rs:md element
md_elements = etree.findall('{' + RS_NS + "}md")
if (len(md_elements) > 1):
raise SitemapParseError(
"Found multiple (%d) <rs:md> elements for %s", (len(md_elements), loc))
elif (len(md_elements) == 1):
resource=resource_class(uri=loc)
# and hopefully a lastmod datetime
lastmod = etree.findtext('{'+SITEMAP_NS+"}lastmod")
###FIXME - test for multple <lastmod> -> error
if (lastmod is not None):
resource.lastmod=lastmod
# then proceed to look for other resource attributes in an rs:md element
md_elements = etree.findall('{'+RS_NS+"}md")
if (len(md_elements)>1):
raise SitemapParseError("Found multiple (%d) <rs:md> elements for %s", (len(md_elements),loc))
elif (len(md_elements)==1):
# have on element, look at attributes
md = self.md_from_etree(md_elements[0], context=loc)
md = self.md_from_etree(md_elements[0],context=loc)
# simple attributes that map directly to Resource object attributes
for att in ('capability', 'change', 'datetime', 'length', 'path', 'mime_type'):
for att in ('capability','change','length','path','type'):
if (att in md):
setattr(resource, att, md[att])
setattr(resource,att,md[att])
# The ResourceSync beta spec lists md5, sha-1 and sha-256 fixity
# digest types. Parse and warn of errors ignored.
if ('hash' in md):
try:
resource.hash = md['hash']
except ValueError as e:
self.logger.warning("%s in <rs:md> for %s" % (str(e), loc))
self.logger.warning("%s in <rs:md> for %s" % (str(e),loc))
# look for rs:ln elements (optional)
ln_elements = etree.findall('{' + RS_NS + "}ln")
if (len(ln_elements) > 0):
ln_elements = etree.findall('{'+RS_NS+"}ln")
if (len(ln_elements)>0):
resource.ln = []
for ln_element in ln_elements:
resource.ln.append(self.ln_from_etree(ln_element, loc))
resource.ln.append(self.ln_from_etree(ln_element,loc))
return(resource)
def md_from_etree(self, md_element, context=''):
"""Parse rs:md attributes returning a dict of the data.
"""Parse rs:md attributes returning a dict of the data
Parameters:
md_element - etree element <rs:md>
context - context for error reporting
ResourceSync v1.1 adds the datetime attribute
md_element - etree element <rs:md>
"""
md = {}
# grab all understood attributes into md dict
for att in self.md_att_keys:
xml_att = self._xml_att_name(att)
val = md_element.attrib.get(xml_att, None)
for att in ('capability','change','hash','length','modified','path','type'):
val = md_element.attrib.get(att,None)
if (val is not None):
md[att] = val
# capability. Allow this to be missing or a new value but warn if it
# isn't recognized
if ('capability' in md and md['capability'] not in self.capabilities):
self.logger.warning("Unknown capability name '%s' in %s" % (md['capability'], context))
# change must be one of defined values
if ('change' in md and md['change'] not in ['created', 'updated', 'deleted']):
raise SitemapParseError("Bad change attribute in <rs:md> for %s" % (context))
# length must be an integer
# capability. Allow this to be missing but do a very simple syntax
# check on plausible values if present
if ('capability' in md):
if (re.match(r"^[\w\-]+$", md['capability']) is None):
raise SitemapParseError("Bad capability name '%s' in %s" % (capability,context))
# change should be one of defined values
if ('change' in md):
if (md['change'] not in ['created','updated','deleted'] ):
self.logger.warning("Bad change attribute in <rs:md> for %s" % (context))
# length should be an integer
if ('length' in md):
try:
md['length'] = int(md['length'])
md['length']=int(md['length'])
except ValueError as e:
raise SitemapParseError("Invalid length element in <rs:md> for %s" %
(context))
raise SitemapParseError("Invalid length element in <rs:md> for %s" % (context))
return(md)
def ln_from_etree(self, ln_element, context=''):
"""Parse rs:ln element from an etree, returning a dict of the data.
def ln_from_etree(self,ln_element,context=''):
"""Parse rs:ln element from an etree, returning a dict of the data
Parameters:
md_element - etree element <rs:md>
context - context string for error reporting
md_element - etree element <rs:md>
"""
ln = {}
# grab all understood attributes into ln dict
for att in ('hash', 'href', 'length', 'modified',
'path', 'rel', 'pri', 'mime_type'):
xml_att = self._xml_att_name(att)
val = ln_element.attrib.get(xml_att, None)
for att in ('hash','href','length','modified','path','rel','pri','type'):
val = ln_element.attrib.get(att,None)
if (val is not None):
ln[att] = val
# now do some checks and conversions...
# href (MANDATORY)
if ('href' not in ln):
raise SitemapParseError("Missing href in <rs:ln> in %s" %
(context))
raise SitemapParseError("Missing href in <rs:ln> in %s" % (context))
# rel (MANDATORY)
if ('rel' not in ln):
raise SitemapParseError("Missing rel in <rs:ln> in %s" % (context))
# length in bytes
if ('length' in ln):
try:
ln['length'] = int(ln['length'])
ln['length']=int(ln['length'])
except ValueError as e:
raise SitemapParseError("Invalid length attribute value in <rs:ln> for %s" %
(context))
raise SitemapParseError("Invalid length attribute value in <rs:ln> for %s" % (context))
# pri - priority, must be a number between 1 and 999999
if ('pri' in ln):
try:
ln['pri'] = int(ln['pri'])
ln['pri']=int(ln['pri'])
except ValueError as e:
raise SitemapParseError(
"Invalid pri attribute in <rs:ln> for %s" %
(context))
if (ln['pri'] < 1 or ln['pri'] > 999999):
raise SitemapParseError(
"Bad pri attribute value in <rs:ln> for %s" %
(context))
raise SitemapParseError("Invalid pri attribute in <rs:ln> for %s" % (context))
if (ln['pri']<1 or ln['pri']>999999):
raise SitemapParseError("Bad pri attribute value in <rs:ln> for %s" % (context))
return(ln)
# Metadata and link elements
def add_element_with_atts_to_etree(self, etree, name, atts, add_return=True):
"""Add empty element with name and atts to etree iff there are any atts.
##### Metadata and links #####
Parameters:
etree - an etree object
name - XML element name
atts - dicts of attribute values. Attribute names are transformed
def add_md_to_etree(self, etree, md):
""" Add <rs:md> element to the etree supplied
"""
xml_atts = {}
for att in atts.keys():
val = atts[att]
if (val is not None):
xml_atts[self._xml_att_name(att)] = str(val)
if (len(xml_atts) > 0):
e = Element(name, xml_atts)
if (add_return and self.pretty_xml):
e.tail = "\n"
# There is no real reason why the attribute keys should be sorted
# but ElementTree serialization up to Python 3.7 always wrote XML
# with attributes in sort order. This was changed in Python 3.8
# to be the order of addition. Adding sorting here keeps consistent
# behavior for all versions. See:
# https://docs.python.org/3/library/xml.etree.elementtree.html#element-objects
attribs = sorted(e.attrib.items())
e.attrib.clear()
e.attrib.update(attribs)
atts = {}
for a in md.keys():
# make attributes by space concatenating any capability dict values
# that are arrays
value = md[a]
if (value is None):
# skip None values
pass
elif (isinstance(value, str)):
atts[a]=value
else:
atts[a]=' '.join(value)
if (len(atts)>0):
e = Element('rs:md', atts)
if (self.pretty_xml):
e.tail="\n"
etree.append(e)
def _xml_att_name(self, att):
"""Get XML attribute name corresponding to supplied Resource object attribute.
We cannot use the XML attribute names in Python because 'from' and others
conflict with Python reserved words. To be consistent all the extra timestamps
are prefixed with md_. Assumption is that attibute name is same unless
specified in XML_ATT_NAME.
def add_ln_to_etree(self, etree, ln):
""" Add <rs:ln> element to the etree supplied
"""
return XML_ATT_NAME.get(att, att)
atts = {}
for a in ln.keys():
# make attributes by space concatenating any capability dict values
# that are arrays
value = ln[a]
if (value is None):
# skip None values
pass
elif (isinstance(value, str)):
atts[a]=value
else:
atts[a]=' '.join(value)
if (len(atts)>0):
e = Element('rs:ln', atts)
if (self.pretty_xml):
e.tail="\n"
etree.append(e)

79
resync/source_description.py
View File

@ -1,79 +0,0 @@
"""ResourceSync Source Description object.
A ResourceSync Source Description enumerates the Capability
Lists offered by a Source. Since a Source has one Capability
List per set of resources that it distinguishes, the
ResourceSync Source Description will enumerate as many
Capability Lists as the Source has distinct sets of resources.
The ResourceSync Source Description is based on the <urlset>
format. The specification allows for a very large Source Descriptions
to use and index based on a <sitemapindex>, though this is probably
rarely necessary!
There is no meaning in the order of description of sets of resources
in a Source Description so the default is to store these descriptions
as a set (ResourceSet).
May also contain metadata and links like other ResourceSync
documents.
See: http://www.openarchives.org/rs/resourcesync#SourceDesc
"""
import collections.abc
from resync.resource import Resource
from resync.resource_set import ResourceSet
from resync.list_base_with_index import ListBaseWithIndex
class SourceDescription(ListBaseWithIndex):
"""Class representing the set of Capability Lists supported.
Will admit only one resource with any given URI.
Storage is unordered but the iterator imposes a canonical order
which is currently alphabetical by URI. Pass in a resources object
to override.
"""
def __init__(self, resources=None, md=None, ln=None,
spec_version='1.1', add_lastmod=False):
"""Initialize based on superclass ListBaseWithIndex."""
super(SourceDescription, self).__init__(
resources=resources, md=md, ln=ln,
capability_name='description',
resources_class=ResourceSet,
spec_version=spec_version,
add_lastmod=add_lastmod)
self.md['from'] = None # usually don't want a from date
def add(self, resource, replace=False):
"""Add a resource or an iterable collection of resources.
Will throw a ValueError if the resource (ie. same uri) already
exists in the capability_list, unless replace=True.
"""
if isinstance(resource, collections.abc.Iterable):
for r in resource:
self.resources.add(r, replace)
else:
self.resources.add(resource, replace)
def add_capability_list(self, capability_list=None):
"""Add a capability list.
Adds either a CapabiltyList object specified in capability_list
or else creates a Resource with the URI given in capability_list
and adds that to the Source Description
"""
if (hasattr(capability_list, 'uri')):
r = Resource(uri=capability_list.uri,
capability=capability_list.capability_name)
if (capability_list.describedby is not None):
r.link_set(rel='describedby', href=capability_list.describedby)
else:
r = Resource(uri=capability_list,
capability='capabilitylist')
self.add(r)

0
resync/test/__init__.py Normal file
View File

31
resync/test/test_capability_list.py Normal file
View File

@ -0,0 +1,31 @@
import unittest
import re
from resync.resource import Resource
from resync.resource_list import ResourceList
from resync.change_list import ChangeList
from resync.capability_list import CapabilityList
class TestCapabilityList(unittest.TestCase):
def test01_resourcelist(self):
rl = ResourceList()
caps = CapabilityList()
caps.add_capability( rl, "http://example.org/resourcelist.xml" )
caps.md['modified'] = "2013-02-07T22:39:00"
self.assertEqual( len(caps), 1 )
self.assertEqual( caps.as_xml(), '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"><rs:md capability="capabilitylist" modified="2013-02-07T22:39:00" /><url><loc>http://example.org/resourcelist.xml</loc><rs:md capability="resourcelist" /></url></urlset>' )
def test02_multiple(self):
caps = CapabilityList()
rl = ResourceList()
caps.add_capability( rl, "rl.xml" )
cl = ChangeList()
caps.add_capability( cl, "cl.xml" )
self.assertEqual( len(caps), 2 )
xml = caps.as_xml()
self.assertTrue( re.search( r'<loc>rl.xml</loc><rs:md capability="resourcelist" />', xml ) )
self.assertTrue( re.search( r'<loc>cl.xml</loc><rs:md capability="changelist" />', xml) )
if __name__ == '__main__':
suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestCapabilityList)
unittest.TextTestRunner().run(suite)

132
resync/test/test_change_list.py Normal file
View File

@ -0,0 +1,132 @@
import unittest
import StringIO
import re
from resync.resource import Resource
from resync.change_list import ChangeList
from resync.resource_list import ResourceList
from resync.sitemap import SitemapParseError
class TestChangeList(unittest.TestCase):
def test1_set_with_repeats(self):
src = ChangeList()
src.add( Resource('a',timestamp=1) )
src.add( Resource('b',timestamp=1) )
src.add( Resource('c',timestamp=1) )
src.add( Resource('a',timestamp=2) )
src.add( Resource('b',timestamp=2) )
self.assertEqual(len(src), 5, "5 changes in change_list")
def test2_with_repeats_again(self):
r1 = Resource(uri='a',length=1)
r2 = Resource(uri='b',length=2)
i = ChangeList()
i.add(r1)
i.add(r2)
self.assertEqual( len(i), 2 )
# Can add another Resource with same URI
r1d = Resource(uri='a',length=10)
i.add(r1d)
self.assertEqual( len(i), 3 )
def test3_change_list(self):
src = ChangeList()
src.add( Resource('a',timestamp=1) )
src.add( Resource('b',timestamp=2) )
src.add( Resource('c',timestamp=3) )
src.add( Resource('d',timestamp=4))
src.add( Resource('e',timestamp=5) )
self.assertEqual(len(src), 5, "5 things in src")
def test4_iter(self):
i = ChangeList()
i.add( Resource('a',timestamp=1) )
i.add( Resource('b',timestamp=2) )
i.add( Resource('c',timestamp=3) )
i.add( Resource('d',timestamp=4) )
resources=[]
for r in i:
resources.append(r)
self.assertEqual(len(resources), 4)
self.assertEqual( resources[0].uri, 'a')
self.assertEqual( resources[3].uri, 'd')
def test5_add_changed_resources(self):
added = ResourceList()
added.add( Resource('a',timestamp=1) )
added.add( Resource('d',timestamp=4))
self.assertEqual(len(added), 2, "2 things in added resource_list")
changes = ChangeList()
changes.add_changed_resources( added, change='created' )
self.assertEqual(len(changes), 2, "2 things added")
i = iter(changes)
first = i.next()
self.assertEqual(first.uri, 'a', "changes[0].uri=a")
self.assertEqual(first.timestamp, 1, "changes[0].timestamp=1")
self.assertEqual(first.change, 'created') #, "changes[0].change=created")
second = i.next()
self.assertEqual(second.timestamp, 4, "changes[1].timestamp=4")
self.assertEqual(second.change, 'created', "changes[1].change=created")
# Now add some with updated (one same, one diff)
updated = ResourceList()
updated.add( Resource('a',timestamp=5) )
updated.add( Resource('b',timestamp=6))
self.assertEqual(len(updated), 2, "2 things in updated resource_list")
changes.add_changed_resources( updated, change='updated' )
self.assertEqual(len(changes), 4, "4 = 2 old + 2 things updated")
# Make new resource_list from the changes which should not have dupes
dst = ResourceList()
dst.add( changes, replace=True )
self.assertEqual(len(dst), 3, "3 unique resources")
self.assertEqual(dst.resources['a'].timestamp, 5 ) # 5 was later in last the 1
self.assertEqual(dst.resources['a'].change, 'updated')
self.assertEqual(dst.resources['b'].timestamp, 6)
self.assertEqual(dst.resources['b'].change, 'updated')
self.assertEqual(dst.resources['d'].timestamp, 4)
self.assertEqual(dst.resources['d'].change, 'created')
def test20_as_xml(self):
cl = ChangeList()
cl.add( Resource('a',timestamp=1,change='updated') )
cl.add( Resource('b',timestamp=2,change='updated') )
xml = cl.as_xml()
print xml
self.assertTrue( re.search(r'<rs:md .*capability="changelist"', xml), 'XML has capability' )
self.assertTrue( re.search(r'<rs:md .*modified="\d\d\d\d\-\d\d\-\d\dT\d\d:\d\d:\d\dZ"', xml), 'XML has modified to seconds precision (and not more)' )
self.assertTrue( re.search(r'<url><loc>a</loc><lastmod>1970-01-01T00:00:01Z</lastmod>', xml), 'XML has resource a' )
def test30_parse(self):
xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
<rs:md capability="changelist" modified="2013-01-01"/>\
<url><loc>/tmp/rs_test/src/file_a</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md change="updated" length="12" /></url>\
<url><loc>/tmp/rs_test/src/file_b</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="32" /></url>\
</urlset>'
cl=ChangeList()
cl.parse(fh=StringIO.StringIO(xml))
self.assertEqual( len(cl.resources), 2, 'got 2 resources')
self.assertEqual( cl.md['capability'], 'changelist', 'capability set' )
self.assertEqual( cl.md['modified'], '2013-01-01' )
def test31_parse_no_capability(self):
# missing capability is an error for changelist
xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">\
<url><loc>http://example.com/res1</loc><lastmod>2012-03-14T18:37:36Z</lastmod></url>\
</urlset>'
cl=ChangeList()
self.assertRaises( SitemapParseError, cl.parse, fh=StringIO.StringIO(xml) )
def test32_parse_bad_capability(self):
# the <rs:md capability="bad_capability".. should give error
xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
<rs:md capability="bad_capability" modified="2013-01-01"/>\
<url><loc>http://example.com/bad_res_1</loc><lastmod>2012-03-14T18:37:36Z</lastmod></url>\
</urlset>'
cl=ChangeList()
self.assertRaises( SitemapParseError, cl.parse, fh=StringIO.StringIO(xml) )
if __name__ == '__main__':
suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestChangeList)
unittest.TextTestRunner().run(suite)

86
resync/test/test_client.py Normal file
View File

@ -0,0 +1,86 @@
import unittest
import re
import logging
import sys, StringIO, contextlib
from resync.client import Client, ClientFatalError
# From http://stackoverflow.com/questions/2654834/capturing-stdout-within-the-same-process-in-python
class Data(object):
pass
@contextlib.contextmanager
def capture_stdout():
old = sys.stdout
capturer = StringIO.StringIO()
sys.stdout = capturer
data = Data()
yield data
sys.stdout = old
data.result = capturer.getvalue()
class TestClient(unittest.TestCase):
@classmethod
def setUpClass(cls):
logging.basicConfig(level=logging.INFO)
def test01_make_resource_list_empty(self):
c = Client()
# No mapping is error
#
def wrap_resource_list_property_call(c):
# do this because assertRaises( ClientFatalError, c.resource_list ) doesn't work
return(c.resource_list)
self.assertRaises( ClientFatalError, wrap_resource_list_property_call, c )
def test02_bad_source_uri(self):
c = Client()
self.assertRaises( ClientFatalError, c.baseline_or_audit )
c.set_mappings( ['http://example.org/bbb','/tmp/this_does_not_exist'] )
self.assertRaises( ClientFatalError, c.baseline_or_audit )
def test06_write_capability_list(self):
c = Client()
with capture_stdout() as capturer:
c.write_capability_list( { 'a':'uri_a', 'b':'uri_b' } )
self.assertTrue( re.search(r'<urlset ',capturer.result) )
self.assertTrue( re.search(r'<rs:md capability="capabilitylist" modified=',capturer.result) )
self.assertTrue( re.search(r'<url><loc>uri_a</loc><rs:md capability="a"',capturer.result) )
self.assertTrue( re.search(r'<url><loc>uri_b</loc><rs:md capability="b"',capturer.result) )
def test07_write_capability_list_index(self):
c = Client()
with capture_stdout() as capturer:
c.write_capability_list_index( [ 'a','b','c' ] )
self.assertTrue( re.search(r'<sitemapindex ',capturer.result) )
self.assertTrue( re.search(r'<rs:md capability="capabilitylist" modified=',capturer.result) )
#print capturer.result
self.assertTrue( re.search(r'<sitemap><loc>a</loc></sitemap>',capturer.result) )
self.assertTrue( re.search(r'<sitemap><loc>b</loc></sitemap>',capturer.result) )
def test20_parse_document(self):
# Key property of the parse_document() method is that it parses the
# document and identifies its type
c = Client()
with capture_stdout() as capturer:
c.sitemap_name='resync/test/testdata/examples_from_spec/resourcesync_ex_2_1.xml'
c.parse_document()
self.assertTrue( re.search(r'Parsed resourcelist document with 2 entries',capturer.result) )
with capture_stdout() as capturer:
c.sitemap_name='resync/test/testdata/examples_from_spec/resourcesync_ex_5_1.xml'
c.parse_document()
self.assertTrue( re.search(r'Parsed resourcedump document with 3 entries',capturer.result) )
with capture_stdout() as capturer:
c.sitemap_name='resync/test/testdata/examples_from_spec/resourcesync_ex_6_1.xml'
c.parse_document()
self.assertTrue( re.search(r'Parsed changelist document with 4 entries',capturer.result) )
with capture_stdout() as capturer:
c.sitemap_name='resync/test/testdata/examples_from_spec/resourcesync_ex_7_1.xml'
c.parse_document()
self.assertTrue( re.search(r'Parsed changedump document with 3 entries',capturer.result) )
if __name__ == '__main__':
suite = unittest.TestLoader().loadTestsFromTestCase(TestClient)
unittest.TextTestRunner(verbosity=2).run(suite)

20
resync/test/test_dump.py Normal file
View File

@ -0,0 +1,20 @@
import unittest
from resync.dump import Dump, DumpError
from resync.resource_list import ResourceList
from resync.resource import Resource
class TestDump(unittest.TestCase):
def test00_dump_creation(self):
i=ResourceList()
i.add( Resource('http://ex.org/a', length=1, path='resync/test/testdata/a') )
i.add( Resource('http://ex.org/b', length=2, path='resync/test/testdata/b') )
d=Dump()
d.check_files(resource_list=i)
self.assertEqual(d.total_size, 28)
#FIXME -- need some code to actually write and read dump
if __name__ == '__main__':
suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestDump)
unittest.TextTestRunner(verbosity=2).run(suite)

130
resync/test/test_examples_from_spec.py Normal file
View File

@ -0,0 +1,130 @@
import sys
import unittest
import StringIO
from resync.resource import Resource
from resync.resource_list import ResourceList
from resync.change_list import ChangeList
from resync.resource_dump import ResourceDump
from resync.resource_dump_manifest import ResourceDumpManifest
from resync.capability_list import CapabilityList
from resync.sitemap import Sitemap
class TestSitemap(unittest.TestCase):
def test_all_simple(self):
"""Just try to read each one"""
for ex in ("archives_ex_2_1.xml",
"archives_ex_3_1.xml",
"archives_ex_4_1.xml",
"archives_ex_5_1.xml",
"archives_ex_6_1.xml",
"resourcesync_ex_2_1.xml", "resourcesync_ex_2_2.xml",
"resourcesync_ex_2_3.xml", "resourcesync_ex_2_4.xml",
"resourcesync_ex_2_5.xml", "resourcesync_ex_2_6.xml",
"resourcesync_ex_2_7.xml",
"resourcesync_ex_4_1.xml", "resourcesync_ex_4_2.xml",
"resourcesync_ex_4_3.xml",
"resourcesync_ex_5_1.xml", "resourcesync_ex_5_2.xml",
"resourcesync_ex_6_1.xml",
"resourcesync_ex_7_1.xml", "resourcesync_ex_7_2.xml",
"resourcesync_ex_8_1.xml", "resourcesync_ex_8_2.xml",
"resourcesync_ex_8_3.xml", "resourcesync_ex_8_4.xml",
"resourcesync_ex_8_5.xml", "resourcesync_ex_8_6.xml",
"resourcesync_ex_8_7.xml", "resourcesync_ex_8_8.xml",
"resourcesync_ex_8_9.xml", "resourcesync_ex_8_10.xml",
"resourcesync_ex_9_1.xml" ):
s=Sitemap()
fh = open( 'resync/test/testdata/examples_from_spec/%s' % (ex), 'r')
si = s.parse_xml( fh=fh )
def test_ex_2_1(self):
"""resourcesync_ex2_1 is a simple resource_list with 2 resources, no metadata"""
rl=ResourceList()
rl.parse(uri='resync/test/testdata/examples_from_spec/resourcesync_ex_2_1.xml')
self.assertEqual( rl.capability, 'resourcelist' )
self.assertEqual( len(rl.resources), 2, '2 resources')
sms = sorted(rl.uris())
self.assertEqual( sms, ['http://example.com/res1','http://example.com/res2'] )
self.assertEqual( rl.resources['http://example.com/res1'].lastmod, None )
def test_ex_2_2(self):
"""resourcesync_ex_2_2 is a simple resource_list with 2 resources, some metadata"""
rl=ResourceList()
rl.parse(uri='resync/test/testdata/examples_from_spec/resourcesync_ex_2_2.xml')
self.assertEqual( len(rl.resources), 2, '2 resources')
sms = sorted(rl.uris())
self.assertEqual( sms, ['http://example.com/res1','http://example.com/res2'] )
self.assertEqual( rl.resources['http://example.com/res1'].lastmod, '2013-01-02T13:00:00Z' )
self.assertEqual( rl.resources['http://example.com/res2'].lastmod, '2013-01-02T14:00:00Z' )
self.assertEqual( rl.resources['http://example.com/res1'].md5, '1584abdf8ebdc9802ac0c6a7402c03b6' )
self.assertEqual( rl.resources['http://example.com/res2'].md5, '1e0d5cb8ef6ba40c99b14c0237be735e' )
def test_ex_2_3(self):
"""resourcesync_ex_2_3 is a simple change_list with 2 resources"""
cl=ChangeList()
cl.parse('resync/test/testdata/examples_from_spec/resourcesync_ex_2_3.xml')
self.assertEqual( len(cl.resources), 2, '2 resources')
sms = sorted(cl.uris())
self.assertEqual( sms, ['http://example.com/res2.pdf','http://example.com/res3.tiff'] )
self.assertEqual( cl.resources[0].lastmod, '2013-01-02T13:00:00Z' )
self.assertEqual( cl.resources[1].lastmod, '2013-01-02T18:00:00Z' )
self.assertEqual( cl.resources[0].change, 'updated' )
self.assertEqual( cl.resources[1].change, 'deleted' )
def test_ex_2_4(self):
"""resourcesync_ex_2_4 is a simple resource dump with one ZIP listed"""
rd=ResourceDump()
rd.parse('resync/test/testdata/examples_from_spec/resourcesync_ex_2_4.xml')
self.assertEqual( len(rd.resources), 1, '1 resources')
self.assertTrue( 'http://example.com/resourcedump.zip' in rd.resources )
self.assertEqual( rd.resources['http://example.com/resourcedump.zip'].lastmod, '2013-01-03T09:00:00Z' )
def test_ex_2_5(self):
"""resourcesync_ex_2_5 is a simple resource dump manifest with two files listed"""
rdm=ResourceDumpManifest()
rdm.parse('resync/test/testdata/examples_from_spec/resourcesync_ex_2_5.xml')
self.assertEqual( len(rdm.resources), 2, '2 resources')
sms = sorted(rdm.uris())
self.assertEqual( sms, ['http://example.com/res1','http://example.com/res2'] )
self.assertEqual( rdm.resources['http://example.com/res1'].lastmod, '2013-01-03T03:00:00Z' )
self.assertEqual( rdm.resources['http://example.com/res1'].md5, '1584abdf8ebdc9802ac0c6a7402c03b6' )
self.assertEqual( rdm.resources['http://example.com/res1'].path, '/resources/res1' )
self.assertEqual( rdm.resources['http://example.com/res2'].lastmod, '2013-01-03T04:00:00Z' )
self.assertEqual( rdm.resources['http://example.com/res2'].md5, '1e0d5cb8ef6ba40c99b14c0237be735e' )
self.assertEqual( rdm.resources['http://example.com/res2'].path, '/resources/res2' )
def test_ex_2_6(self):
"""resourcesync_ex_2_6 is a simple capability list with capabilities"""
capl=CapabilityList()
capl.parse('resync/test/testdata/examples_from_spec/resourcesync_ex_2_6.xml')
self.assertEqual( len(capl.resources), 3, '3 capabilities')
# What capabilities are present?
self.assertTrue( capl.has_capability('resourcelist') )
self.assertEqual( capl.capability('resourcelist').uri, 'http://example.com/dataset1/resourcelist.xml')
self.assertTrue( capl.has_capability('resourcedump') )
self.assertEqual( capl.capability('resourcedump').uri, 'http://example.com/dataset1/resourcedump.xml')
self.assertTrue( capl.has_capability('changelist') )
self.assertEqual( capl.capability('changelist').uri, 'http://example.com/dataset1/changelist.xml')
# Check some that aren't
self.assertFalse( capl.has_capability() )
self.assertFalse( capl.has_capability('bogus') )
self.assertFalse( capl.has_capability('capabilitylist') )
def test_ex_2_7(self):
"""resourcesync_ex2_7 is a simple Resource List Index with 2 Resource Lists"""
rl=ResourceList()
rl.read(uri='resync/test/testdata/examples_from_spec/resourcesync_ex_2_7.xml',index_only=True)
self.assertEqual( rl.capability, 'resourcelist' )
self.assertEqual( len(rl.resources), 2, '2 resources')
sms = sorted(rl.uris())
self.assertEqual( sms, ['http://example.com/resourcelist-part1.xml',
'http://example.com/resourcelist-part2.xml'] )
self.assertEqual( rl.resources['http://example.com/resourcelist-part1.xml'].lastmod,
'2013-01-03T09:00:00Z' )
self.assertEqual( rl.resources['http://example.com/resourcelist-part2.xml'].lastmod,
'2013-01-03T09:00:00Z' )
if __name__ == '__main__':
suite = unittest.TestLoader().loadTestsFromTestCase(TestSitemap)
unittest.TextTestRunner(verbosity=2).run(suite)

41
resync/test/test_list_base.py Normal file
View File

@ -0,0 +1,41 @@
import sys
import unittest
import StringIO
from resync.resource import Resource
from resync.list_base import ListBase
from resync.sitemap import Sitemap, SitemapIndexError
# etree gives ParseError in 2.7, ExpatError in 2.6
etree_error_class = None
if (sys.version_info < (2,7)):
from xml.parsers.expat import ExpatError
etree_error_class = ExpatError
else:
from xml.etree.ElementTree import ParseError
etree_error_class = ParseError
class TestSitemap(unittest.TestCase):
def test_08_print(self):
lb = ListBase()
lb.add( Resource(uri='a',lastmod='2001-01-01',length=1234) )
lb.add( Resource(uri='b',lastmod='2002-02-02',length=56789) )
lb.add( Resource(uri='c',lastmod='2003-03-03',length=0) )
lb.md['modified']=None #avoid now being added
#print lb
self.assertEqual( lb.as_xml(), '<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"><rs:md capability="unknown" /><url><loc>a</loc><lastmod>2001-01-01T00:00:00Z</lastmod><rs:md length="1234" /></url><url><loc>b</loc><lastmod>2002-02-02T00:00:00Z</lastmod><rs:md length="56789" /></url><url><loc>c</loc><lastmod>2003-03-03T00:00:00Z</lastmod><rs:md length="0" /></url></urlset>' )
def test_11_parse_2(self):
xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
<rs:md capability="unknown" modified="2013-02-12T14:09:00Z" />\
<url><loc>/tmp/rs_test/src/file_a</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="12" /></url>\
<url><loc>/tmp/rs_test/src/file_b</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="32" /></url>\
</urlset>'
lb=ListBase()
lb.parse(fh=StringIO.StringIO(xml))
self.assertEqual( len(lb.resources), 2, 'got 2 resources')
if __name__ == '__main__':
suite = unittest.TestLoader().loadTestsFromTestCase(TestSitemap)
unittest.TextTestRunner(verbosity=2).run(suite)

96
resync/test/test_mapper.py Normal file
View File

@ -0,0 +1,96 @@
import unittest
from resync.mapper import Mapper, MapperError, Map
class TestMapper(unittest.TestCase):
def test00_mapper_creation(self):
m1=Mapper( ['http://e.org/p/','/tmp/q/'] )
self.assertEqual( len(m1), 1 )
m2=Mapper( mappings=['http://e.org/p','/tmp/q'] )
self.assertEqual( len(m2), 1 )
self.assertEqual( str(m1), str(m2) )
m3=Mapper( ['http://e.org/p/=/tmp/q/'] )
self.assertEqual( len(m3), 1 )
self.assertEqual( str(m1), str(m3) )
m4=Mapper( ['http://e.org/p/=/tmp/q/','http://e.org/r/=/tmp/s/'] )
m5=Mapper( ['http://e.org/r/=/tmp/s/','http://e.org/p/=/tmp/q/'] )
self.assertEqual( len(m4), 2 )
self.assertEqual( len(m5), 2 )
self.assertNotEqual( str(m4), str(m5) )
def test01_mapper_src_to_dst(self):
m=Mapper( ['http://e.org/p/','/tmp/q/'] )
self.assertEqual( m.src_to_dst('http://e.org/p/'), '/tmp/q/')
self.assertEqual( m.src_to_dst('http://e.org/p/aa'), '/tmp/q/aa')
self.assertEqual( m.src_to_dst('http://e.org/p/aa/bb'), '/tmp/q/aa/bb')
self.assertEqual( m.src_to_dst('http://e.org/p/aa/bb/'), '/tmp/q/aa/bb/')
self.assertRaises( MapperError, m.src_to_dst, 'http://e.org/p' )
self.assertRaises( MapperError, m.src_to_dst, 'http://e.org/pa' )
self.assertRaises( MapperError, m.src_to_dst, 'nomatch' )
def test02_mapper_dst_to_src(self):
m=Mapper( ['http://e.org/p/','/tmp/q/'] )
self.assertEqual( m.dst_to_src('/tmp/q/'), 'http://e.org/p/')
self.assertEqual( m.dst_to_src('/tmp/q/bb'), 'http://e.org/p/bb')
self.assertEqual( m.dst_to_src('/tmp/q/bb/cc'), 'http://e.org/p/bb/cc')
self.assertRaises( MapperError, m.dst_to_src, '/tmp/q' )
self.assertRaises( MapperError, m.dst_to_src, '/tmp/qa')
self.assertRaises( MapperError, m.dst_to_src, 'nomatch' )
def test03_mapper2_src_to_dst(self):
m=Mapper( ['http://e.org/p=/tmp/q','http://e.org/r=/tmp/s'] )
self.assertEqual( m.src_to_dst('http://e.org/p/'), '/tmp/q/')
self.assertEqual( m.src_to_dst('http://e.org/p/aa'), '/tmp/q/aa')
self.assertEqual( m.src_to_dst('http://e.org/r/'), '/tmp/s/')
self.assertEqual( m.src_to_dst('http://e.org/r/aa'), '/tmp/s/aa')
def test04_mapper2_dst_to_src(self):
m=Mapper( ['http://e.org/p=/tmp/q','http://e.org/r=/tmp/s'] )
self.assertEqual( m.dst_to_src('/tmp/q/'), 'http://e.org/p/')
self.assertEqual( m.dst_to_src('/tmp/q/bb'), 'http://e.org/p/bb')
self.assertEqual( m.dst_to_src('/tmp/s/'), 'http://e.org/r/')
self.assertEqual( m.dst_to_src('/tmp/s/bb'), 'http://e.org/r/bb')
def test05_path_from_uri(self):
m=Mapper()
self.assertEqual( m.path_from_uri('a_file'), 'a_file' )
self.assertEqual( m.path_from_uri('some_path/a_file'), 'some_path/a_file' )
self.assertEqual( m.path_from_uri('http://localhost/p'), 'localhost_p' )
self.assertEqual( m.path_from_uri('http://localhost:8888/p'), 'localhost_8888_p' )
self.assertEqual( m.path_from_uri('https://localhost:8888/p'), 'localhost_8888_p' )
self.assertEqual( m.path_from_uri('http://example.com'), 'example.com' )
self.assertEqual( m.path_from_uri('http://example.com/'), 'example.com' )
self.assertEqual( m.path_from_uri('http://example.com/ex1'), 'example.com_ex1' )
self.assertEqual( m.path_from_uri('http://example.com/ex1/'), 'example.com_ex1' )
def test06_mapper_unsafe(self):
self.assertFalse( Mapper( ['http://example.com/=/tmp/a'] ).unsafe() )
self.assertFalse( Mapper( ['http://example.com/=http://example.com/'] ).unsafe() )
self.assertFalse( Mapper( ['http://example.com/'], use_default_path=True ).unsafe() )
# Following hits case of single local arg supplied
self.assertTrue( Mapper( ['/tmp/a'], use_default_path=True ).unsafe() )
# One good, one bad -> bad
self.assertTrue( Mapper( ['http://example.com/=/tmp/a','/tmp/a=/tmp'] ).unsafe() )
def test07_default_src_uri(self):
self.assertEqual( Mapper(['a=b']).default_src_uri(), 'a')
self.assertEqual( Mapper(['a=b','b=c']).default_src_uri(), 'a')
self.assertRaises( MapperError, Mapper().default_src_uri )
##### Map #####
def test10_map_unsafe(self):
self.assertFalse( Map('http://example.com/','path').unsafe() )
# Note the first is a URI and the second is a (silly) path in the following
self.assertFalse( Map('http://example.com/','http://example.com/').unsafe() )
self.assertFalse( Map('a','b').unsafe() )
self.assertFalse( Map('path/a','path/b').unsafe() )
# The following are unsafe
self.assertTrue( Map('path','path').unsafe() )
self.assertTrue( Map('path/a','path').unsafe() )
self.assertTrue( Map('path','path/b').unsafe() )
if __name__ == '__main__':
suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestMapper)
unittest.TextTestRunner(verbosity=2).run(suite)

120
resync/test/test_resource.py Normal file
View File

@ -0,0 +1,120 @@
import unittest
import re
from resync.resource import Resource
class TestResource(unittest.TestCase):
def test1a_same(self):
r1 = Resource('a')
r2 = Resource('a')
self.assertEqual( r1, r1 )
self.assertEqual( r1, r2 )
def test1b_same(self):
r1 = Resource(uri='a',timestamp=1234.0)
r2 = Resource(uri='a',timestamp=1234.0)
self.assertEqual( r1, r1 )
self.assertEqual( r1, r2 )
def test1c_same(self):
"""Same with lastmod instead of direct timestamp"""
r1 = Resource('a')
r1lm = '2012-01-01T00:00:00Z'
r1.lastmod = r1lm
r2 = Resource('a')
for r2lm in ('2012',
'2012-01',
'2012-01-01',
'2012-01-01T00:00Z',
'2012-01-01T00:00:00Z',
'2012-01-01T00:00:00.000000Z',
'2012-01-01T00:00:00.000000000000Z',
'2012-01-01T00:00:00.000000000001Z', #below resolution
'2012-01-01T00:00:00.00+00:00',
'2012-01-01T00:00:00.00-00:00',
'2012-01-01T02:00:00.00-02:00',
'2011-12-31T23:00:00.00+01:00'
):
r2.lastmod = r2lm
self.assertEqual( r1.timestamp, r2.timestamp, ('%s (%f) == %s (%f)' % (r1lm,r1.timestamp,r2lm,r2.timestamp)) )
self.assertEqual( r1, r2 )
def test1d_same(self):
"""Same with slight timestamp diff"""
r1 = Resource('a')
r1.lastmod='2012-01-02T01:02:03Z'
r2 = Resource('a')
r2.lastmod='2012-01-02T01:02:03.99Z'
self.assertNotEqual( r1.timestamp, r2.timestamp )
self.assertEqual( r1, r2 )
def test2a_diff(self):
r1 = Resource('a')
r2 = Resource('b')
self.assertNotEqual(r1,r2)
def test2b_diff(self):
r1 = Resource('a',lastmod='2012-01-11')
r2 = Resource('a',lastmod='2012-01-22')
#print 'r1 == r2 : '+str(r1==r2)
self.assertNotEqual( r1, r2 )
def test4_bad_lastmod(self):
def setlastmod(r,v):
r.lastmod=v
r = Resource('4')
# Bad formats
self.assertRaises( ValueError, setlastmod, r, "bad_lastmod" )
self.assertRaises( ValueError, setlastmod, r, "" )
self.assertRaises( ValueError, setlastmod, r, "2012-13-01" )
self.assertRaises( ValueError, setlastmod, r, "2012-12-32" )
self.assertRaises( ValueError, setlastmod, r, "2012-11-01T10:10:60" )
self.assertRaises( ValueError, setlastmod, r, "2012-11-01T10:10:59.9x" )
# Valid ISO8601 but not alloed in W3C Datetime
self.assertRaises( ValueError, setlastmod, r, "2012-11-01T01:01:01" )
self.assertRaises( ValueError, setlastmod, r, "2012-11-01 01:01:01Z" )
self.assertRaises( ValueError, setlastmod, r, "2012-11-01T01:01:01+0000" )
self.assertRaises( ValueError, setlastmod, r, "2012-11-01T01:01:01-1000" )
def test5_lastmod_roundtrips(self):
r = Resource('a')
r.lastmod='2012-03-14'
self.assertEqual( r.lastmod, '2012-03-14T00:00:00Z' )
r.lastmod='2012-03-14T00:00:00+00:00'
print r.timestamp
self.assertEqual( r.lastmod, '2012-03-14T00:00:00Z' )
r.lastmod='2012-03-14T00:00:00-00:00'
print r.timestamp
self.assertEqual( r.lastmod, '2012-03-14T00:00:00Z' )
r.lastmod='2012-03-14T18:37:36Z'
print r.timestamp
self.assertEqual( r.lastmod, '2012-03-14T18:37:36Z' )
def test6_str(self):
r1 = Resource('abc',lastmod='2012-01-01')
self.assertTrue( re.match( r"\[ abc \| 2012-01-01T", str(r1) ) )
def test7_multiple_hashes(self):
r1 = Resource('abcd')
r1.md5 = "some_md5"
r1.sha1 = "some_sha1"
r1.sha256 = "some_sha256"
self.assertEqual( r1.md5, "some_md5" )
self.assertEqual( r1.sha1, "some_sha1" )
self.assertEqual( r1.sha256, "some_sha256" )
self.assertEqual( r1.hash, "md5:some_md5 sha-1:some_sha1 sha-256:some_sha256" )
r2 = Resource('def')
r2.hash = "md5:ddd"
self.assertEqual( r2.md5, 'ddd' )
self.assertEqual( r2.sha1, None )
r2.hash = "sha-1:eee"
self.assertEqual( r2.md5, None )
self.assertEqual( r2.sha1, 'eee' )
r2.hash = "md5:fff sha-1:eee sha-256:ggg"
self.assertEqual( r2.md5, 'fff' )
self.assertEqual( r2.sha1, 'eee' )
self.assertEqual( r2.sha256, 'ggg' )
if __name__ == '__main__':
suite = unittest.TestLoader().loadTestsFromTestCase(TestResource)
unittest.TextTestRunner(verbosity=2).run(suite)

75
resync/test/test_resource_container.py Normal file
View File

@ -0,0 +1,75 @@
import unittest
from resync.resource import Resource
from resync.resource_container import ResourceContainer
class TestResourceContainer(unittest.TestCase):
def test1_create_and_add(self):
rc = ResourceContainer( resources=[] )
self.assertEqual( len(rc.resources), 0, "empty" )
rc.resources.append( Resource('a',timestamp=1) )
rc.resources.append( Resource('b',timestamp=2) )
self.assertEqual( len(rc.resources), 2, "two resources" )
def test2_iter(self):
rc = ResourceContainer( resources=[] )
rc.resources.append( Resource('a',timestamp=1) )
rc.resources.append( Resource('b',timestamp=2) )
rc.resources.append( Resource('c',timestamp=3) )
rc.resources.append( Resource('d',timestamp=4) )
resources=[]
for r in rc:
resources.append(r)
self.assertEqual(len(resources), 4)
self.assertEqual( resources[0].uri, 'a')
self.assertEqual( resources[3].uri, 'd')
def test3_prune_before(self):
rc = ResourceContainer()
rc.resources.append( Resource('a',timestamp=1) )
rc.resources.append( Resource('b',timestamp=2) )
rc.resources.append( Resource('c',timestamp=3) )
rc.resources.append( Resource('d',timestamp=4) )
rc.prune_before(3)
self.assertEqual( len(rc.resources), 2 )
i = iter(rc)
self.assertEqual( i.next().uri, 'c' )
self.assertEqual( i.next().uri, 'd' )
# put some more back out of order
rc.resources.append( Resource('a',timestamp=1) )
rc.resources.append( Resource('b',timestamp=2) )
rc.resources.append( Resource('e',timestamp=1000) )
rc.prune_before(3.5)
self.assertEqual( len(rc.resources), 2 )
i = iter(rc)
self.assertEqual( i.next().uri, 'd' )
self.assertEqual( i.next().uri, 'e' )
def test4_prune_dupes(self):
rc = ResourceContainer()
rc.resources.append( Resource('a',timestamp=1, change='created') )
rc.resources.append( Resource('b',timestamp=2, change='created') )
rc.resources.append( Resource('c',timestamp=3, change='updated') )
rc.resources.append( Resource('d',timestamp=4, change='deleted') )
rc.resources.append( Resource('a',timestamp=4, change='updated') )
rc.resources.append( Resource('b',timestamp=4, change='deleted') )
rc.resources.append( Resource('b',timestamp=5, change='created') )
rc.resources.append( Resource('b',timestamp=6, change='deleted') )
rc.resources.append( Resource('c',timestamp=7, change='deleted') )
rc.resources.append( Resource('d',timestamp=8, change='created') )
rc.prune_dupes()
self.assertEqual( len(rc.resources), 3 )
i = iter(rc)
i1 = i.next()
self.assertEqual( i1.uri, 'a' )
self.assertEqual( i1.change, 'updated' )
i2 = i.next()
self.assertEqual( i2.uri, 'c' )
self.assertEqual( i2.change, 'deleted' )
i3 = i.next()
self.assertEqual( i3.uri, 'd' )
self.assertEqual( i3.change, 'created' )
if __name__ == '__main__':
suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestResourceContainer)
unittest.TextTestRunner().run(suite)

59
resync/test/test_resource_dump.py Normal file
View File

@ -0,0 +1,59 @@
import unittest
import StringIO
import re
from resync.resource import Resource
from resync.resource_dump import ResourceDump
from resync.sitemap import SitemapParseError
class TestResourceDump(unittest.TestCase):
def test01_as_xml(self):
rd = ResourceDump()
rd.add( Resource('a.zip',timestamp=1) )
rd.add( Resource('b.zip',timestamp=2) )
xml = rd.as_xml()
print xml
self.assertTrue( re.search(r'<rs:md .*capability="resourcedump"', xml), 'XML has capability' )
self.assertTrue( re.search(r'<rs:md .*modified="\d\d\d\d\-\d\d\-\d\dT\d\d:\d\d:\d\dZ"', xml), 'XML has modified to seconds precision (and not more)' )
self.assertTrue( re.search(r'<url><loc>a.zip</loc><lastmod>1970-01-01T00:00:01Z</lastmod></url>', xml), 'XML has resource a' )
def test10_parse(self):
xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
<rs:md capability="resourcedump" modified="2013-01-01"/>\
<url><loc>http://example.com/a.zip</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="12345" /></url>\
<url><loc>http://example.com/b.zip</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="56789" /></url>\
</urlset>'
rd=ResourceDump()
rd.parse(fh=StringIO.StringIO(xml))
self.assertEqual( len(rd.resources), 2, 'got 2 resource dumps')
self.assertEqual( rd.md['capability'], 'resourcedump', 'capability set' )
self.assertEqual( rd.md['modified'], '2013-01-01' )
self.assertTrue( 'http://example.com/a.zip' in rd.resources )
self.assertTrue( rd.resources['http://example.com/a.zip'].length, 12345 )
self.assertTrue( 'http://example.com/b.zip' in rd.resources )
self.assertTrue( rd.resources['http://example.com/b.zip'].length, 56789 )
def test11_parse_no_capability(self):
# For a resource dump this should be an error
xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
<rs:md modified="2013-01-01"/>\
<url><loc>http://example.com/a.zip</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="12" /></url>\
</urlset>'
rd=ResourceDump()
self.assertRaises( SitemapParseError, rd.parse, fh=StringIO.StringIO(xml) )
def test12_parse_bad_capability(self):
# the <rs:md capability="bad_capability".. should give error
xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
<rs:md capability="bad_capability" modified="2013-01-01"/>\
<url><loc>http://example.com/bad_res_1</loc><lastmod>2012-03-14T18:37:36Z</lastmod></url>\
</urlset>'
rd=ResourceDump()
self.assertRaises( SitemapParseError, rd.parse, fh=StringIO.StringIO(xml) )
if __name__ == '__main__':
suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestResourceDump)
unittest.TextTestRunner().run(suite)

61
resync/test/test_resource_dump_manifest.py Normal file
View File

@ -0,0 +1,61 @@
import unittest
import StringIO
import re
from resync.resource import Resource
from resync.resource_dump_manifest import ResourceDumpManifest
from resync.sitemap import SitemapParseError
class TestResourceDumpManifest(unittest.TestCase):
def test01_as_xml(self):
rdm = ResourceDumpManifest()
rdm.add( Resource('a.zip',timestamp=1) )
rdm.add( Resource('b.zip',timestamp=2) )
xml = rdm.as_xml()
print xml
self.assertTrue( re.search(r'<rs:md .*capability="resourcedump-manifest"', xml), 'XML has capability' )
self.assertTrue( re.search(r'<rs:md .*modified="\d\d\d\d\-\d\d\-\d\dT\d\d:\d\d:\d\dZ"', xml), 'XML has modified to seconds precision (and not more)' )
self.assertTrue( re.search(r'<url><loc>a.zip</loc><lastmod>1970-01-01T00:00:01Z</lastmod></url>', xml), 'XML has resource a' )
def test10_parse(self):
xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
<rs:md capability="resourcedump-manifest" modified="2013-01-01"/>\
<url><loc>http://example.com/res1</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="12" path="/res1" /></url>\
<url><loc>http://example.com/res2</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="32" path="/res2"/></url>\
</urlset>'
rdm=ResourceDumpManifest()
rdm.parse(fh=StringIO.StringIO(xml))
self.assertEqual( len(rdm.resources), 2, 'got 2 resource dumps')
self.assertEqual( rdm.md['capability'], 'resourcedump-manifest', 'capability set' )
self.assertEqual( rdm.md['modified'], '2013-01-01' )
self.assertTrue( 'http://example.com/res1' in rdm.resources )
self.assertTrue( rdm.resources['http://example.com/res1'].length, 12 )
self.assertTrue( rdm.resources['http://example.com/res1'].path, '/res1' )
self.assertTrue( 'http://example.com/res2' in rdm.resources )
self.assertTrue( rdm.resources['http://example.com/res2'].path, '/res2' )
def test11_parse_no_capability(self):
# For a resource dump this should be an error
xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
<rs:md modified="2013-01-01"/>\
<url><loc>http://example.com/res1</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="12" path="/res1" /></url>\
<url><loc>http://example.com/res2</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="32" path="/res2"/></url>\
</urlset>'
rdm=ResourceDumpManifest()
self.assertRaises( SitemapParseError, rdm.parse, fh=StringIO.StringIO(xml) )
def test12_parse_bad_capability(self):
# the <rs:md capability="bad_capability".. should give error
xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
<rs:md capability="bad_capability" modified="2013-01-01"/>\
<url><loc>http://example.com/a.zip</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="12" /></url>\
</urlset>'
rdm=ResourceDumpManifest()
self.assertRaises( SitemapParseError, rdm.parse, fh=StringIO.StringIO(xml) )
if __name__ == '__main__':
suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestResourceDumpManifest)
unittest.TextTestRunner().run(suite)

175
resync/test/test_resource_list.py Normal file
View File

@ -0,0 +1,175 @@
import unittest
import StringIO
import re
from resync.resource import Resource
from resync.resource_list import ResourceList, ResourceListDupeError
from resync.sitemap import SitemapParseError
class TestResourceList(unittest.TestCase):
def test01_same(self):
src = ResourceList()
src.add( Resource('a',timestamp=1) )
src.add( Resource('b',timestamp=2) )
dst = ResourceList()
dst.add( Resource('a',timestamp=1) )
dst.add( Resource('b',timestamp=2) )
( same, changed, deleted, added ) = dst.compare(src)
self.assertEqual( len(same), 2, "2 things unchanged" )
i = iter(same)
self.assertEqual( i.next().uri, 'a', "first was a" )
self.assertEqual( i.next().uri, 'b', "second was b" )
self.assertEqual( len(changed), 0, "nothing changed" )
self.assertEqual( len(deleted), 0, "nothing deleted" )
self.assertEqual( len(added), 0, "nothing added" )
def test02_changed(self):
src = ResourceList()
src.add( Resource('a',timestamp=1) )
src.add( Resource('b',timestamp=2) )
dst = ResourceList()
dst.add( Resource('a',timestamp=3) )
dst.add( Resource('b',timestamp=4) )
( same, changed, deleted, added ) = dst.compare(src)
self.assertEqual( len(same), 0, "0 things unchanged" )
self.assertEqual( len(changed), 2, "2 things changed" )
i = iter(changed)
self.assertEqual( i.next().uri, 'a', "first was a" )
self.assertEqual( i.next().uri, 'b', "second was b" )
self.assertEqual( len(deleted), 0, "nothing deleted" )
self.assertEqual( len(added), 0, "nothing added" )
def test03_deleted(self):
src = ResourceList()
src.add( Resource('a',timestamp=1) )
src.add( Resource('b',timestamp=2) )
dst = ResourceList()
dst.add( Resource('a',timestamp=1) )
dst.add( Resource('b',timestamp=2) )
dst.add( Resource('c',timestamp=3) )
dst.add( Resource('d',timestamp=4) )
( same, changed, deleted, added ) = dst.compare(src)
self.assertEqual( len(same), 2, "2 things unchanged" )
self.assertEqual( len(changed), 0, "nothing changed" )
self.assertEqual( len(deleted), 2, "c and d deleted" )
i = iter(deleted)
self.assertEqual( i.next().uri, 'c', "first was c" )
self.assertEqual( i.next().uri, 'd', "second was d" )
self.assertEqual( len(added), 0, "nothing added" )
def test04_added(self):
src = ResourceList()
src.add( Resource('a',timestamp=1) )
src.add( Resource('b',timestamp=2) )
src.add( Resource('c',timestamp=3) )
src.add( Resource('d',timestamp=4) )
dst = ResourceList()
dst.add( Resource('a',timestamp=1) )
dst.add( Resource('c',timestamp=3) )
( same, changed, deleted, added ) = dst.compare(src)
self.assertEqual( len(same), 2, "2 things unchanged" )
self.assertEqual( len(changed), 0, "nothing changed" )
self.assertEqual( len(deleted), 0, "nothing deleted" )
self.assertEqual( len(added), 2, "b and d added" )
i = iter(added)
self.assertEqual( i.next().uri, 'b', "first was b" )
self.assertEqual( i.next().uri, 'd', "second was d" )
def test05_add(self):
r1 = Resource(uri='a',length=1)
r2 = Resource(uri='b',length=2)
i = ResourceList()
i.add(r1)
self.assertRaises( ResourceListDupeError, i.add, r1)
i.add(r2)
self.assertRaises( ResourceListDupeError, i.add, r2)
# allow dupes
r1d = Resource(uri='a',length=10)
i.add(r1d,replace=True)
self.assertEqual( len(i), 2 )
self.assertEqual( i.resources['a'].length, 10 )
def test06_add_iterable(self):
r1 = Resource(uri='a',length=1)
r2 = Resource(uri='b',length=2)
i = ResourceList()
i.add( [r1,r2] )
self.assertRaises( ResourceListDupeError, i.add, r1)
self.assertRaises( ResourceListDupeError, i.add, r2)
# allow dupes
r1d = Resource(uri='a',length=10)
i.add( [r1d] ,replace=True)
self.assertEqual( len(i), 2 )
self.assertEqual( i.resources['a'].length, 10 )
def test07_has_md5(self):
r1 = Resource(uri='a')
r2 = Resource(uri='b')
i = ResourceList()
self.assertFalse( i.has_md5() )
i.add(r1)
i.add(r2)
self.assertFalse( i.has_md5() )
r1.md5="aabbcc"
self.assertTrue( i.has_md5() )
def test08_iter(self):
i = ResourceList()
i.add( Resource('a',timestamp=1) )
i.add( Resource('b',timestamp=2) )
i.add( Resource('c',timestamp=3) )
i.add( Resource('d',timestamp=4) )
resources=[]
for r in i:
resources.append(r)
self.assertEqual(len(resources), 4)
self.assertEqual( resources[0].uri, 'a')
self.assertEqual( resources[3].uri, 'd')
def test20_as_xml(self):
rl = ResourceList()
rl.add( Resource('a',timestamp=1) )
rl.add( Resource('b',timestamp=2) )
xml = rl.as_xml()
print xml
self.assertTrue( re.search(r'<rs:md .*capability="resourcelist"', xml), 'XML has capability' )
self.assertTrue( re.search(r'<rs:md .*modified="\d\d\d\d\-\d\d\-\d\dT\d\d:\d\d:\d\dZ"', xml), 'XML has modified to seconds precision (and not more)' )
self.assertTrue( re.search(r'<url><loc>a</loc><lastmod>1970-01-01T00:00:01Z</lastmod></url>', xml), 'XML has resource a' )
def test30_parse(self):
xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
<rs:md capability="resourcelist" modified="2013-01-01"/>\
<url><loc>/tmp/rs_test/src/file_a</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md change="updated" length="12" /></url>\
<url><loc>/tmp/rs_test/src/file_b</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="32" /></url>\
</urlset>'
rl=ResourceList()
rl.parse(fh=StringIO.StringIO(xml))
self.assertEqual( len(rl.resources), 2, 'got 2 resources')
self.assertEqual( rl.md['capability'], 'resourcelist', 'capability set' )
self.assertEqual( rl.md['modified'], '2013-01-01' )
def test31_parse_no_capability(self):
xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">\
<url><loc>http://example.com/res1</loc><lastmod>2012-03-14T18:37:36Z</lastmod></url>\
</urlset>'
rl=ResourceList()
rl.parse(fh=StringIO.StringIO(xml))
self.assertEqual( len(rl.resources), 1, 'got 1 resource')
self.assertEqual( rl.md['capability'], 'resourcelist', 'capability set by reading routine' )
self.assertFalse( 'modified' in rl.md )
def test32_parse_bad_capability(self):
# the <rs:md capability="bad_capability".. should give error
xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
<rs:md capability="bad_capability" modified="2013-01-01"/>\
<url><loc>http://example.com/bad_res_1</loc><lastmod>2012-03-14T18:37:36Z</lastmod></url>\
</urlset>'
rl=ResourceList()
self.assertRaises( SitemapParseError, rl.parse, fh=StringIO.StringIO(xml) )
if __name__ == '__main__':
suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestResourceList)
unittest.TextTestRunner().run(suite)

55
resync/test/test_resource_list_builder.py Normal file
View File

@ -0,0 +1,55 @@
import unittest
import re
import os
import time
from resync.resource_list_builder import ResourceListBuilder
from resync.mapper import Mapper
class TestResourceListBuilder(unittest.TestCase):
def setUp(self):
# Set timestamps (mtime) for test data. Timestamps on disk are
# in UTC so no conversion issues.
# Test case file_a: 1343236426 = 2012-07-25T17:13:46Z
# Test case file_b: 1000000000 = 2001-09-09T01:46:40Z
os.utime( "resync/test/testdata/dir1/file_a", (0, 1343236426 ) )
os.utime( "resync/test/testdata/dir1/file_b", (0, 1000000000 ) )
def test1_simple_output(self):
rlb = ResourceListBuilder()
rlb.mapper = Mapper(['http://example.org/t','resync/test/testdata/dir1'])
rl = rlb.from_disk()
rl.md['modified']=None #don't write so we can test output easily
self.assertEqual(rl.as_xml(),'<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/"><rs:md capability="resourcelist" /><url><loc>http://example.org/t/file_a</loc><lastmod>2012-07-25T17:13:46Z</lastmod><rs:md length=\"20\" /></url><url><loc>http://example.org/t/file_b</loc><lastmod>2001-09-09T01:46:40Z</lastmod><rs:md length=\"45\" /></url></urlset>' )
def test2_pretty_output(self):
rlb = ResourceListBuilder()
rlb.mapper = Mapper(['http://example.org/t','resync/test/testdata/dir1'])
rl = rlb.from_disk()
rl.md['modified']=None #don't write so we can test output easily
rl.pretty_xml=True
self.assertEqual(rl.as_xml(),'<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\n<rs:md capability="resourcelist" />\n<url><loc>http://example.org/t/file_a</loc><lastmod>2012-07-25T17:13:46Z</lastmod><rs:md length=\"20\" /></url>\n<url><loc>http://example.org/t/file_b</loc><lastmod>2001-09-09T01:46:40Z</lastmod><rs:md length=\"45\" /></url>\n</urlset>' )
def test3_with_md5(self):
rlb = ResourceListBuilder(do_md5=True)
rlb.mapper = Mapper(['http://example.org/t','resync/test/testdata/dir1'])
rl = rlb.from_disk()
xml = rl.as_xml()
self.assertNotEqual( None, re.search('<loc>http://example.org/t/file_a</loc><lastmod>[\w\:\-]+Z</lastmod><rs:md hash=\"md5:a/Jv1mYBtSjS4LR\+qoft/Q==\" length=\"20\" />',xml) ) #must escape + in md5
self.assertNotEqual( None, re.search('<loc>http://example.org/t/file_b</loc><lastmod>[\w\:\-]+Z</lastmod><rs:md hash=\"md5:RS5Uva4WJqxdbnvoGzneIQ==\" length=\"45\" />',xml) )
def test4_data(self):
rlb = ResourceListBuilder(do_md5=True)
rlb.mapper = Mapper(['http://example.org/t','resync/test/testdata/dir1'])
rl = rlb.from_disk(set_path=True)
self.assertEqual( len(rl), 2)
r1 = rl.resources.get('http://example.org/t/file_a')
self.assertTrue( r1 is not None )
self.assertEqual( r1.uri, 'http://example.org/t/file_a' )
self.assertEqual( r1.lastmod, '2012-07-25T17:13:46Z' )
self.assertEqual( r1.md5, 'a/Jv1mYBtSjS4LR+qoft/Q==' )
self.assertEqual( r1.path, 'resync/test/testdata/dir1/file_a' )
if __name__ == '__main__':
suite = unittest.defaultTestLoader.loadTestsFromTestCase(TestResourceListBuilder)
unittest.TextTestRunner(verbosity=2).run(suite)

102
resync/test/test_resource_list_multifile.py Normal file
View File

@ -0,0 +1,102 @@
import sys
import unittest
import StringIO
import tempfile
import os.path
import shutil
from resync.list_base_with_index import ListBaseIndexError
from resync.resource import Resource
from resync.resource_list import ResourceList
from resync.mapper import Mapper
# etree gives ParseError in 2.7, ExpatError in 2.6
etree_error_class = None
if (sys.version_info < (2,7)):
from xml.parsers.expat import ExpatError
etree_error_class = ExpatError
else:
from xml.etree.ElementTree import ParseError
etree_error_class = ParseError
class TestResourceListMultifile(unittest.TestCase):
def test_01_read_local_filenames(self):
rl = ResourceList()
rl.read('resync/test/testdata/sitemapindex2/sitemap.xml')
self.assertEqual( len(rl.resources), 17, '17 resources from 3 sitemaps listed')
sr = sorted(rl.uris())
self.assertEqual( sr[0], 'http://localhost:8888/resources/1' )
self.assertEqual( sr[1], 'http://localhost:8888/resources/10' )
self.assertEqual( sr[2], 'http://localhost:8888/resources/100' )
self.assertEqual( sr[3], 'http://localhost:8888/resources/1000' )
self.assertEqual( sr[16], 'http://localhost:8888/resources/826' )
def test_02_read_with_mapper(self):
rl = ResourceList()
rl.mapper = Mapper(['http://localhost/=resync/test/testdata/sitemapindex2/'])
rl.read('resync/test/testdata/sitemapindex2/sitemap_mapper.xml')
self.assertEqual( len(rl.resources), 17, '17 resources from 3 sitemaps listed')
sr = sorted(rl.uris())
self.assertEqual( sr[0], 'http://localhost:8888/resources/1' )
self.assertEqual( sr[1], 'http://localhost:8888/resources/10' )
self.assertEqual( sr[2], 'http://localhost:8888/resources/100' )
self.assertEqual( sr[3], 'http://localhost:8888/resources/1000' )
self.assertEqual( sr[16], 'http://localhost:8888/resources/826' )
def test_11_write_multifile(self):
tempdir = tempfile.mkdtemp(prefix='test_resource_list_multifile')
rl = ResourceList()
rl.mapper = Mapper(['http://localhost/=%s/' % (tempdir)])
rl.add( Resource( uri='http://localhost/a' ) )
rl.add( Resource( uri='http://localhost/b' ) )
rl.add( Resource( uri='http://localhost/c' ) )
rl.add( Resource( uri='http://localhost/d' ) )
rl.max_sitemap_entries = 2
# first try writing without mutlifile allowed
rl.allow_multifile = False
self.assertRaises( ListBaseIndexError, rl.write, basename=os.path.join(tempdir,'sitemap.xml') )
# second actually do it
rl.allow_multifile = True
rl.write( basename = os.path.join(tempdir,'sitemap.xml') )
# check the two component sitemaps
rl1 = ResourceList()
rl1.read( os.path.join(tempdir,'sitemap00000.xml') )
self.assertEquals( len(rl1), 2 )
self.assertEquals( rl1.capability, 'resourcelist' )
self.assertFalse( rl1.sitemapindex )
i = iter(rl1)
self.assertEquals( i.next().uri, 'http://localhost/a' )
self.assertEquals( i.next().uri, 'http://localhost/b' )
rl2 = ResourceList()
rl2.read( os.path.join(tempdir,'sitemap00001.xml') )
self.assertEquals( len(rl2), 2 )
i = iter(rl2)
self.assertEquals( i.next().uri, 'http://localhost/c' )
self.assertEquals( i.next().uri, 'http://localhost/d' )
# check the sitemapindex (read just as index)
rli = ResourceList()
rli.read( os.path.join(tempdir,'sitemap.xml'), index_only=True )
self.assertEquals( len(rli), 2 )
i = iter(rli)
self.assertEquals( rli.capability, 'resourcelist' )
self.assertTrue( rli.sitemapindex )
self.assertEquals( i.next().uri, 'http://localhost/sitemap00000.xml' )
self.assertEquals( i.next().uri, 'http://localhost/sitemap00001.xml' )
# check the sitemapindex and components
rli = ResourceList( mapper=rl.mapper )
rli.read( os.path.join(tempdir,'sitemap.xml') )
self.assertEquals( len(rli), 4 )
self.assertEquals( rli.capability, 'resourcelist' )
self.assertFalse( rli.sitemapindex )
i = iter(rli)
self.assertEquals( i.next().uri, 'http://localhost/a' )
self.assertEquals( i.next().uri, 'http://localhost/b' )
self.assertEquals( i.next().uri, 'http://localhost/c' )
self.assertEquals( i.next().uri, 'http://localhost/d' )
# cleanup tempdir
shutil.rmtree(tempdir)
if __name__ == '__main__':
suite = unittest.TestLoader().loadTestsFromTestCase(TestResourceListMultifile)
unittest.TextTestRunner(verbosity=2).run(suite)

39
resync/test/test_resource_set.py Normal file
View File

@ -0,0 +1,39 @@
import unittest
import re
from resync.resource import Resource
from resync.resource_set import ResourceSet,ResourceSetDupeError
class TestResourceSet(unittest.TestCase):
def test01_add(self):
rs = ResourceSet()
self.assertEqual( len(rs), 0 )
rs.add( Resource('a') )
self.assertEqual( len(rs), 1 )
rs.add( Resource('b') )
self.assertEqual( len(rs), 2 )
rs.add( Resource('c') )
self.assertEqual( len(rs), 3 )
def test02_order(self):
rs = ResourceSet()
rs.add( Resource('a2') )
rs.add( Resource('a3') )
rs.add( Resource('a1') )
i = iter(rs)
self.assertEqual( i.next().uri, 'a1' )
self.assertEqual( i.next().uri, 'a2' )
self.assertEqual( i.next().uri, 'a3' )
self.assertRaises( StopIteration, i.next )
def test03_dupe(self):
rs = ResourceSet()
self.assertEqual( len(rs), 0 )
rs.add( Resource('a') )
self.assertEqual( len(rs), 1 )
self.assertRaises( ResourceSetDupeError, rs.add, Resource('a') )
if __name__ == '__main__':
suite = unittest.TestLoader().loadTestsFromTestCase(TestResourceSet)
unittest.TextTestRunner(verbosity=2).run(suite)

227
resync/test/test_sitemap.py Normal file
View File

@ -0,0 +1,227 @@
import sys
import unittest
import StringIO
from resync.resource import Resource
from resync.resource_list import ResourceList
from resync.sitemap import Sitemap, SitemapIndexError, SitemapParseError
# etree gives ParseError in 2.7, ExpatError in 2.6
etree_error_class = None
if (sys.version_info < (2,7)):
from xml.parsers.expat import ExpatError
etree_error_class = ExpatError
else:
from xml.etree.ElementTree import ParseError
etree_error_class = ParseError
class TestSitemap(unittest.TestCase):
def test_01_resource_str(self):
r1 = Resource('a3')
r1.lastmod='2012-01-11T01:02:03Z'
self.assertEqual( Sitemap().resource_as_xml(r1), "<?xml version='1.0' encoding='UTF-8'?>\n<url><loc>a3</loc><lastmod>2012-01-11T01:02:03Z</lastmod></url>" )
def test_02_resource_str(self):
r1 = Resource('3b',1234.1,9999,'ab54de')
self.assertEqual( Sitemap().resource_as_xml(r1), "<?xml version='1.0' encoding='UTF-8'?>\n<url><loc>3b</loc><lastmod>1970-01-01T00:20:34.100000Z</lastmod><rs:md hash=\"md5:ab54de\" length=\"9999\" /></url>" )
def test_03_resource_str_hashes(self):
r1 = Resource('03hashes',1234.1)
r1.md5 = 'aaa'
r1.sha1 = 'bbb'
self.assertEqual( Sitemap().resource_as_xml(r1), "<?xml version='1.0' encoding='UTF-8'?>\n<url><loc>03hashes</loc><lastmod>1970-01-01T00:20:34.100000Z</lastmod><rs:md hash=\"md5:aaa sha-1:bbb\" /></url>" )
r1.sha256 = 'ccc'
self.assertEqual( Sitemap().resource_as_xml(r1), "<?xml version='1.0' encoding='UTF-8'?>\n<url><loc>03hashes</loc><lastmod>1970-01-01T00:20:34.100000Z</lastmod><rs:md hash=\"md5:aaa sha-1:bbb sha-256:ccc\" /></url>" )
r1.sha1 = None
self.assertEqual( Sitemap().resource_as_xml(r1), "<?xml version='1.0' encoding='UTF-8'?>\n<url><loc>03hashes</loc><lastmod>1970-01-01T00:20:34.100000Z</lastmod><rs:md hash=\"md5:aaa sha-256:ccc\" /></url>" )
def test_04_resource_str(self):
r1 = Resource(uri='4a',lastmod="2013-01-02",length=9999,md5='ab54de')
r1.ln = [{ 'rel':'duplicate',
'pri':'1',
'href':'http://mirror1.example.com/res1',
'modified':'2013-01-02T18:00:00Z' }]
self.assertEqual( Sitemap().resource_as_xml(r1), "<?xml version='1.0' encoding='UTF-8'?>\n<url><loc>4a</loc><lastmod>2013-01-02T00:00:00Z</lastmod><rs:md hash=\"md5:ab54de\" length=\"9999\" /><rs:ln href=\"http://mirror1.example.com/res1\" modified=\"2013-01-02T18:00:00Z\" pri=\"1\" rel=\"duplicate\" /></url>" )
# add another two rs:ln's
r1.ln.append( { 'rel':'num2' } )
r1.ln.append( { 'rel':'num3' } )
self.assertEqual( Sitemap().resource_as_xml(r1), "<?xml version='1.0' encoding='UTF-8'?>\n<url><loc>4a</loc><lastmod>2013-01-02T00:00:00Z</lastmod><rs:md hash=\"md5:ab54de\" length=\"9999\" /><rs:ln href=\"http://mirror1.example.com/res1\" modified=\"2013-01-02T18:00:00Z\" pri=\"1\" rel=\"duplicate\" /><rs:ln rel=\"num2\" /><rs:ln rel=\"num3\" /></url>" )
def test_08_print(self):
r1 = Resource(uri='a',lastmod='2001-01-01',length=1234)
r2 = Resource(uri='b',lastmod='2002-02-02',length=56789)
r3 = Resource(uri='c',lastmod='2003-03-03',length=0)
m = ResourceList(md={'capability':'resourcelist','modified':None})
m.add(r1)
m.add(r2)
m.add(r3)
#print m
self.assertEqual( Sitemap().resources_as_xml(m), "<?xml version='1.0' encoding='UTF-8'?>\n<urlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\" xmlns:rs=\"http://www.openarchives.org/rs/terms/\"><rs:md capability=\"resourcelist\" /><url><loc>a</loc><lastmod>2001-01-01T00:00:00Z</lastmod><rs:md length=\"1234\" /></url><url><loc>b</loc><lastmod>2002-02-02T00:00:00Z</lastmod><rs:md length=\"56789\" /></url><url><loc>c</loc><lastmod>2003-03-03T00:00:00Z</lastmod><rs:md length=\"0\" /></url></urlset>")
def test_09_print_from_iter(self):
r1 = Resource(uri='a',lastmod='2001-01-01',length=1234)
r2 = Resource(uri='b',lastmod='2002-02-02',length=56789)
m = ResourceList()
m.add(r1)
m.add(r2)
i = iter(m)
self.assertEqual( Sitemap().resources_as_xml(i), "<?xml version='1.0' encoding='UTF-8'?>\n<urlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\" xmlns:rs=\"http://www.openarchives.org/rs/terms/\"><url><loc>a</loc><lastmod>2001-01-01T00:00:00Z</lastmod><rs:md length=\"1234\" /></url><url><loc>b</loc><lastmod>2002-02-02T00:00:00Z</lastmod><rs:md length=\"56789\" /></url></urlset>")
def test_10_sitemap(self):
xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
<url><loc>http://e.com/a</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md hash="md5:Q2hlY2sgSW50ZWdyaXR5IQ==" length=\"12\" /></url>\
</urlset>'
s=Sitemap()
i=s.parse_xml(fh=StringIO.StringIO(xml))
self.assertFalse( s.parsed_index, 'was a sitemap')
self.assertEqual( s.resources_created, 1, 'got 1 resources')
for r in i.resources:
self.assertTrue( r is not None, 'got the uri expected')
self.assertEqual( r.uri, 'http://e.com/a' )
self.assertEqual( r.lastmod, '2012-03-14T18:37:36Z' )
self.assertEqual( r.length, 12 )
self.assertEqual( r.md5, 'Q2hlY2sgSW50ZWdyaXR5IQ==' )
def test_11_parse_2(self):
xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
<url><loc>/tmp/rs_test/src/file_a</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length=\"12\" /></url>\
<url><loc>/tmp/rs_test/src/file_b</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length=\"32\" /></url>\
</urlset>'
s=Sitemap()
i=s.parse_xml(fh=StringIO.StringIO(xml))
self.assertFalse( s.parsed_index, 'was a sitemap')
self.assertEqual( s.resources_created, 2, 'got 2 resources')
def test_13_parse_illformed(self):
s=Sitemap()
# ExpatError in python2.6, ParserError in 2.7
self.assertRaises( etree_error_class, s.parse_xml, StringIO.StringIO('not xml') )
self.assertRaises( etree_error_class, s.parse_xml, StringIO.StringIO('<urlset><url>something</urlset>') )
def test_13_parse_valid_xml_but_other(self):
s=Sitemap()
self.assertRaises( SitemapParseError, s.parse_xml, StringIO.StringIO('<urlset xmlns="http://example.org/other_namespace"> </urlset>') )
self.assertRaises( SitemapParseError, s.parse_xml, StringIO.StringIO('<other xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> </other>') )
def test_14_parse_sitemapindex_as_sitemap(self):
s=Sitemap()
self.assertRaises( SitemapIndexError, s.parse_xml, StringIO.StringIO('<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> </sitemapindex>'), sitemapindex=False )
def test_15_parse_with_rs_ln_on_resource(self):
xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
<rs:md capability=\"resourcelist\"/>\
<url>\
<loc>http://example.com/file_a</loc><lastmod>2012-03-14T18:37:36Z</lastmod>\
<rs:md hash=\"md5:r2d2\" length=\"12345\" />\
<rs:ln rel=\"duplicate\" href=\"http://mirror1.example.com/res1\" modified=\"2013-01-02\" pri=\"1\" />\
<rs:ln rel=\"num2\" href=\"http://m2.example.com/res1\"/>\
<rs:ln rel=\"num3\" href=\"http://m3.example.com/res1\"/>\
</url>\
<url>\
<loc>http://example.com/file_b</loc><lastmod>2012-03-14T18:37:36Z</lastmod>\
<rs:md length=\"32\" />\
</url>\
</urlset>'
s=Sitemap()
rc=s.parse_xml(fh=StringIO.StringIO(xml))
self.assertFalse( s.parsed_index, 'was a sitemap')
self.assertEqual( s.resources_created, 2, 'got 2 resources')
i = iter(rc)
r1 = i.next()
r2 = i.next()
self.assertEqual( r1.uri, 'http://example.com/file_a' )
self.assertEqual( r1.ln[0]['rel'], 'duplicate' )
self.assertEqual( r1.ln[0]['href'], 'http://mirror1.example.com/res1' )
self.assertEqual( r1.ln[0]['modified'], '2013-01-02' )
self.assertEqual( r1.ln[0]['pri'], 1 )
self.assertEqual( r2.uri, 'http://example.com/file_b' )
def test_16_parse_with_bad_rs_ln(self):
xmlstart='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
<rs:md capability="resourcelist"/>\
<url><loc>http://example.com/file_a</loc>'
xmlend='</url></urlset>'
s=Sitemap()
#
# missing href
xml=xmlstart+'<rs:ln rel="duplicate"/>'+xmlend
self.assertRaises( SitemapParseError, s.parse_xml, fh=StringIO.StringIO(xml))
# missing rel
xml=xmlstart+'<rs:ln href="http://example.com/"/>'+xmlend
self.assertRaises( SitemapParseError, s.parse_xml, fh=StringIO.StringIO(xml))
# bad length
xml=xmlstart+'<rs:ln rel="duplicate" href="http://example.com/" length="a"/>'+xmlend
self.assertRaises( SitemapParseError, s.parse_xml, fh=StringIO.StringIO(xml))
# bad pri
xml=xmlstart+'<rs:ln rel="duplicate" href="http://example.com/" pri="fff"/>'+xmlend
self.assertRaises( SitemapParseError, s.parse_xml, fh=StringIO.StringIO(xml))
xml=xmlstart+'<rs:ln rel="duplicate" href="http://example.com/" pri="0"/>'+xmlend
self.assertRaises( SitemapParseError, s.parse_xml, fh=StringIO.StringIO(xml))
xml=xmlstart+'<rs:ln rel="duplicate" href="http://example.com/" pri="1000000"/>'+xmlend
self.assertRaises( SitemapParseError, s.parse_xml, fh=StringIO.StringIO(xml))
# and finally OK with errors fixes
xml=xmlstart+'<rs:ln rel="duplicate" href="http://example.com/" length="12345" pri="1" other="whatever"/>'+xmlend
rc = s.parse_xml(fh=StringIO.StringIO(xml))
self.assertEqual( len(rc.resources), 1, 'good at last, extra attribute ignored' )
def test_20_parse_sitemapindex_empty(self):
s=Sitemap()
si = s.parse_xml( fh=StringIO.StringIO('<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> </sitemapindex>'), sitemapindex=True )
self.assertTrue( s.parsed_index, 'was a sitemapindex')
self.assertEqual( len(si.resources), 0, '0 sitemaps')
def test_21_parse_sitemapindex(self):
s=Sitemap()
si = s.parse_xml( fh=StringIO.StringIO('<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><sitemap><loc>aaa</loc></sitemap><sitemap><loc>bbb</loc></sitemap></sitemapindex>'), sitemapindex=True )
self.assertEqual( len(si.resources), 2, '2 sitemaps')
sms = sorted(si.uris())
self.assertEqual( sms, ['aaa','bbb'] )
# add a couple more
s.parse_xml( fh=StringIO.StringIO('<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><sitemap><loc>cc</loc></sitemap><sitemap><loc>dd</loc></sitemap></sitemapindex>'), resources=si )
self.assertTrue( s.parsed_index, 'was a sitemapindex')
self.assertEqual( len(si.resources), 4, '4 sitemaps total')
sms = sorted(si.uris())
self.assertEqual( sms, ['aaa','bbb', 'cc', 'dd'] )
def test_22_parse_sitemapindex_file(self):
s=Sitemap()
fh=open('resync/test/testdata/sitemapindex1/sitemap.xml','r')
si = s.parse_xml( fh=fh, sitemapindex=True )
self.assertTrue( s.parsed_index, 'was a sitemapindex')
self.assertEqual( len(si.resources), 3, '3 sitemaps')
sms = sorted(si.uris())
self.assertEqual( sms, ['http://localhost:8888/sitemap00000.xml','http://localhost:8888/sitemap00001.xml','http://localhost:8888/sitemap00002.xml'] )
#self.assertEqual( si.resources['http://localhost:8888/sitemap00000.xml'].lastmod, '2012-06-13T18:09:13Z' )
def test_21_parse_multi_sitemapindex(self):
s = Sitemap()
fh=open('resync/test/testdata/sitemapindex2/sitemap.xml','r')
si = s.parse_xml( fh=fh, sitemapindex=True )
self.assertTrue( s.parsed_index, 'was a sitemapindex')
self.assertEqual( len(si.resources), 3, '3 sitemaps listed')
def test_30_parse_change_list(self):
xml='<?xml version=\'1.0\' encoding=\'UTF-8\'?>\n\
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">\
<url><loc>/tmp/rs_test/src/file_a</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md change="updated" length="12" /></url>\
<url><loc>/tmp/rs_test/src/file_b</loc><lastmod>2012-03-14T18:37:36Z</lastmod><rs:md length="32" /></url>\
</urlset>'
s=Sitemap()
s.resource_class=Resource
c=s.parse_xml(fh=StringIO.StringIO(xml))
self.assertEqual( s.resources_created, 2, 'got 2 resources')
i = iter(c)
r1 = i.next()
self.assertEqual( r1.uri, '/tmp/rs_test/src/file_a' )
self.assertEqual( r1.change, 'updated' )
r2 = i.next()
self.assertEqual( r2.uri, '/tmp/rs_test/src/file_b' )
self.assertEqual( r2.change, None )
if __name__ == '__main__':
suite = unittest.TestLoader().loadTestsFromTestCase(TestSitemap)
unittest.TextTestRunner(verbosity=2).run(suite)

35
resync/test/test_url_authority.py Normal file
View File

@ -0,0 +1,35 @@
import unittest
from resync.url_authority import UrlAuthority
class TestUrlAuthority(unittest.TestCase):
def test1(self):
uauth = UrlAuthority( 'http://example.org/sitemap.xml' )
self.assertTrue( uauth.has_authority_over( 'http://example.org/sitemap.xml' ) )
self.assertTrue( uauth.has_authority_over( 'http://example.org/sitemap.xml?anything' ) )
self.assertTrue( uauth.has_authority_over( 'http://example.org/sitemap.xml#frag' ) )
self.assertTrue( uauth.has_authority_over( 'http://example.org/same_level' ) )
self.assertTrue( uauth.has_authority_over( 'http://example.org/one/deeper' ) )
self.assertTrue( uauth.has_authority_over( 'http://example.org/one/two/deeper' ) )
self.assertTrue( uauth.has_authority_over( 'http://example.org/' ) )
self.assertTrue( uauth.has_authority_over( 'http://sub.example.org/subdomain' ) )
self.assertTrue( uauth.has_authority_over( 'http://sub.sub.example.org/subsubdomain' ) )
def test2_no_authority(self):
uauth = UrlAuthority( 'http://example.org/dir/sitemap.xml' )
self.assertFalse( uauth.has_authority_over( 'http://example.org/sitemap.xml' ) )
self.assertFalse( uauth.has_authority_over( 'http://sub.example.org/sitemap.xml' ) )
self.assertFalse( uauth.has_authority_over( 'https://example.org/dir/sitemap.xml' ) )
self.assertFalse( uauth.has_authority_over( 'unknown://example.org/dir/sitemap.xml' ) )
def test3_domains(self):
uauth = UrlAuthority( 'http://a.example.org/sitemap.xml' )
self.assertTrue( uauth.has_authority_over( 'http://a.example.org/sitemap.xml' ) )
self.assertTrue( uauth.has_authority_over( 'http://sub.a.example.org/sitemap.xml' ) )
self.assertFalse( uauth.has_authority_over( 'http://b.example.org/sitemap.xml' ) )
self.assertFalse( uauth.has_authority_over( 'http://sub.b.example.org/sitemap.xml' ) )
if __name__ == '__main__':
suite = unittest.TestLoader().loadTestsFromTestCase(TestUrlAuthority)
unittest.TextTestRunner(verbosity=2).run(suite)

16
resync/test/test_utils.py Normal file
View File

@ -0,0 +1,16 @@
import unittest
import resync.utils
class TestUtill(unittest.TestCase):
def test1_string(self):
self.assertEqual( resync.utils.compute_md5_for_string('A file\n'),
'j912liHgA/48DCHpkptJHg==')
def test2_file(self):
# Should be same as the string above
self.assertEqual( resync.utils.compute_md5_for_file('resync/test/testdata/a'),
'j912liHgA/48DCHpkptJHg==')
if __name__ == '__main__':
unittest.main()

83
resync/test/test_w3c_datetime.py Normal file
View File

@ -0,0 +1,83 @@
import unittest
import re
from resync.w3c_datetime import str_to_datetime, datetime_to_str
def rt(dts):
""" Do simple round-trip """
return( datetime_to_str(str_to_datetime( dts )) )
class TestW3cDatetime(unittest.TestCase):
def test1_datetime_to_str(self):
"""Writing..."""
self.assertEqual( datetime_to_str(0), "1970-01-01T00:00:00Z" )
self.assertEqual( datetime_to_str(0.000001), "1970-01-01T00:00:00.000001Z" )
self.assertEqual( datetime_to_str(0.1), "1970-01-01T00:00:00.100000Z" )
self.assertEqual( datetime_to_str(1), "1970-01-01T00:00:01Z" )
self.assertEqual( datetime_to_str(60), "1970-01-01T00:01:00Z" )
self.assertEqual( datetime_to_str(60*60), "1970-01-01T01:00:00Z" )
self.assertEqual( datetime_to_str(60*60*24), "1970-01-02T00:00:00Z" )
self.assertEqual( datetime_to_str(60*60*24*31), "1970-02-01T00:00:00Z" )
self.assertEqual( datetime_to_str(60*60*24*365), "1971-01-01T00:00:00Z" )
#
self.assertEqual( datetime_to_str(1234567890), "2009-02-13T23:31:30Z" )
def test2_str_to_datetime(self):
"""Reading..."""
self.assertEqual( str_to_datetime("1970-01-01T00:00:00Z"), 0 )
self.assertEqual( str_to_datetime("1970-01-01T00:00:00.000Z"), 0 )
self.assertEqual( str_to_datetime("1970-01-01T00:00:00+00:00"), 0 )
self.assertEqual( str_to_datetime("1970-01-01T00:00:00-00:00"), 0 )
self.assertEqual( str_to_datetime("1970-01-01T00:00:00.000001Z"), 0.000001 )
self.assertEqual( str_to_datetime("1970-01-01T00:00:00.1Z"), 0.1 )
self.assertEqual( str_to_datetime("1970-01-01T00:00:00.100000Z"), 0.1 )
#
self.assertEqual( str_to_datetime("2009-02-13T23:31:30Z"), 1234567890 )
def test2_same(self):
"""Datetime values that are the same"""
astr = '2012-01-01T00:00:00Z'
a = str_to_datetime( astr )
for bstr in ('2012',
'2012-01',
'2012-01-01',
'2012-01-01T00:00Z',
'2012-01-01T00:00:00Z',
'2012-01-01T00:00:00.000000Z',
'2012-01-01T00:00:00.000000000000Z',
'2012-01-01T00:00:00.000000000001Z', #below resolution
'2012-01-01T00:00:00.00+00:00',
'2012-01-01T00:00:00.00-00:00',
'2012-01-01T02:00:00.00-02:00',
'2011-12-31T23:00:00.00+01:00'
):
b = str_to_datetime( bstr )
self.assertEqual( a, b, ('%s (%f) == %s (%f)' % (astr,a,bstr,b)) )
def test4_bad_str(self):
# Bad formats
self.assertRaises( ValueError, str_to_datetime, "bad_lastmod" )
self.assertRaises( ValueError, str_to_datetime, "" )
self.assertRaises( ValueError, str_to_datetime, "2012-13-01" )
self.assertRaises( ValueError, str_to_datetime, "2012-12-32" )
self.assertRaises( ValueError, str_to_datetime, "2012-11-01T10:10:60" )
self.assertRaises( ValueError, str_to_datetime, "2012-11-01T10:10:59.9x" )
# Valid ISO8601 but not allowed in W3C Datetime
self.assertRaises( ValueError, str_to_datetime, "2012-11-01T01:01:01" )
self.assertRaises( ValueError, str_to_datetime, "2012-11-01 01:01:01Z" )
self.assertRaises( ValueError, str_to_datetime, "2012-11-01T01:01:01+0000" )
self.assertRaises( ValueError, str_to_datetime, "2012-11-01T01:01:01-1000" )
def test5_roundtrips(self):
self.assertEqual( rt('2012-03-14T00:00:00+00:00'),
'2012-03-14T00:00:00Z')
self.assertEqual( rt('2012-03-14T00:00:00-00:00'),
'2012-03-14T00:00:00Z')
self.assertEqual( rt('2012-03-14T11:00:00-11:00' ),
'2012-03-14T00:00:00Z')
self.assertEqual( rt('2012-03-14T18:37:36Z' ),
'2012-03-14T18:37:36Z' )
if __name__ == '__main__':
suite = unittest.TestLoader().loadTestsFromTestCase(TestW3cDatetime)
unittest.TextTestRunner(verbosity=2).run(suite)

0
tests/testdata/a → resync/test/testdata/a vendored
View File

0
tests/testdata/b → resync/test/testdata/b vendored
View File

0
tests/testdata/c → resync/test/testdata/c vendored
View File

0
tests/testdata/dir1/file_a → resync/test/testdata/dir1/file_a vendored
View File

0
tests/testdata/dir1/file_b → resync/test/testdata/dir1/file_b vendored
View File

0
tests/testdata/dir2/file_x → resync/test/testdata/dir2/file_x vendored
View File

14
tests/testdata/examples_from_spec_v1_0/archives_ex_5_1.xml → resync/test/testdata/examples_from_spec/archives_ex_2_1.xml vendored
View File

@ -1,22 +1,20 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="up"
<rs:ln rel="resourcesync"
href="http://example.com/dataset1/capabilitylist.xml"/>
<rs:md capability="changelist-archive"/>
<rs:md capability="changelist-archive"
modified="2013-01-03T09:00:00Z"/>
<url>
<loc>http://example.com/changelist1.xml</loc>
<rs:md from="2013-01-01T09:00:00Z"
until="2013-01-02T09:00:00Z"/>
<lastmod>2013-01-01T09:00:00Z</lastmod>
</url>
<url>
<loc>http://example.com/changelist2.xml</loc>
<rs:md from="2013-01-02T09:00:00Z"
until="2013-01-03T09:00:00Z"/>
<lastmod>2013-01-02T09:00:00Z</lastmod>
</url>
<url>
<loc>http://example.com/changelist3.xml</loc>
<rs:md from="2013-01-03T09:00:00Z"
until="2013-01-04T09:00:00Z"/>
<lastmod>2013-01-03T09:00:00Z</lastmod>
</url>
</urlset>

17
tests/testdata/examples_from_spec_v1_0/archives_ex_6_1.xml → resync/test/testdata/examples_from_spec/archives_ex_3_1.xml vendored
View File

@ -1,19 +1,20 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="up"
<rs:ln rel="resourcesync"
href="http://example.com/dataset1/capabilitylist.xml"/>
<rs:md capability="changedump-archive"/>
<rs:md capability="changedump-archive"
modified="2013-01-03T09:00:00Z"/>
<url>
<loc>http://example.com/changedump-w1.xml</loc>
<lastmod>2012-12-20T09:02:43Z</lastmod>
<rs:md from="2012-01-13T09:00:00Z"
until="2013-01-20T09:00:00Z"/>
<lastmod>2012-12-20T09:00:00Z</lastmod>
</url>
<url>
<loc>http://example.com/changedump-w2.xml</loc>
<lastmod>2012-12-27T09:01:57Z</lastmod>
<rs:md from="2012-01-20T09:00:00Z"
until="2013-01-27T09:00:00Z"/>
<lastmod>2012-12-27T09:00:00Z</lastmod>
</url>
<url>
<loc>http://example.com/changedump-w3.xml</loc>
<lastmod>2013-01-03T09:00:00Z</lastmod>
</url>
</urlset>

11
tests/testdata/examples_from_spec_v1_0/archives_ex_3_1.xml → resync/test/testdata/examples_from_spec/archives_ex_4_1.xml vendored
View File

@ -1,19 +1,20 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="up"
<rs:ln rel="resourcesync"
href="http://example.com/dataset1/capabilitylist.xml"/>
<rs:md capability="resourcelist-archive"/>
<rs:md capability="resourcelist-archive"
modified="2013-01-03T09:00:00Z"/>
<url>
<loc>http://example.com/resourcelist1.xml</loc>
<rs:md at="2012-11-03T09:00:00Z"/>
<lastmod>2012-11-03T09:00:00Z</lastmod>
</url>
<url>
<loc>http://example.com/resourcelist2.xml</loc>
<rs:md at="2012-12-03T09:00:00Z"/>
<lastmod>2012-12-03T09:00:00Z</lastmod>
</url>
<url>
<loc>http://example.com/resourcelist3.xml</loc>
<rs:md at="2013-01-03T09:00:00Z"/>
<lastmod>2013-01-03T09:00:00Z</lastmod>
</url>
</urlset>

17
tests/testdata/examples_from_spec_v1_0/archives_ex_4_1.xml → resync/test/testdata/examples_from_spec/archives_ex_5_1.xml vendored
View File

@ -1,19 +1,20 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="up"
<rs:ln rel="resourcesync"
href="http://example.com/dataset1/capabilitylist.xml"/>
<rs:md capability="resourcedump-archive"/>
<rs:md capability="resourcedump-archive"
modified="2013-01-03T09:00:00Z"/>
<url>
<loc>http://example.com/resourcedump1.xml</loc>
<lastmod>2012-11-03T09:05:42Z</lastmod>
<rs:md at="2012-11-03T09:00:00Z"
completed="2012-11-03T09:05:01Z"/>
<lastmod>2012-11-03T09:00:00Z</lastmod>
</url>
<url>
<loc>http://example.com/resourcedump2.xml</loc>
<lastmod>2012-12-03T09:06:12Z</lastmod>
<rs:md at="2012-12-03T09:00:00Z"
completed="2012-12-03T09:05:17Z"/>
<lastmod>2012-12-03T09:00:00Z</lastmod>
</url>
<url>
<loc>http://example.com/resourcedump3.xml</loc>
<lastmod>2013-01-03T09:00:00Z</lastmod>
</url>
</urlset>

8
tests/testdata/examples_from_spec_v1_0/archives_ex_2_1.xml → resync/test/testdata/examples_from_spec/archives_ex_6_1.xml vendored
View File

@ -2,11 +2,9 @@
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="describedby"
href="http://example.com/info_about_set1_of_resources.xml"
type="application/xml"/>
<rs:ln rel="up"
href="http://example.com/resourcesync_description.xml"/>
<rs:md capability="capabilitylist"/>
href="http://example.com/dataset1/info_about_source.xml"/>
<rs:md capability="capabilitylist"
modified="2013-01-02T14:00:00Z"/>
<url>
<loc>http://example.com/dataset1/resourcelist.xml</loc>
<rs:md capability="resourcelist"/>

2
tests/testdata/examples_from_spec_v1_0/resourcesync_ex_1.xml → resync/test/testdata/examples_from_spec/resourcesync_ex_2_1.xml vendored
View File

@ -2,7 +2,7 @@
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:md capability="resourcelist"
at="2013-01-03T09:00:00Z"/>
modified="2013-01-03T09:00:00Z"/>
<url>
<loc>http://example.com/res1</loc>
</url>

4
tests/testdata/examples_from_spec_v1_1/resourcesync_ex_2.xml → resync/test/testdata/examples_from_spec/resourcesync_ex_2_2.xml vendored
View File

@ -2,7 +2,7 @@
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:md capability="resourcelist"
at="2013-01-03T09:00:00Z"/>
modified="2013-01-03T09:00:00Z"/>
<url>
<loc>http://example.com/res1</loc>
<lastmod>2013-01-02T13:00:00Z</lastmod>
@ -12,7 +12,5 @@
<loc>http://example.com/res2</loc>
<lastmod>2013-01-02T14:00:00Z</lastmod>
<rs:md hash="md5:1e0d5cb8ef6ba40c99b14c0237be735e"/>
<rs:ln rel="duplicate"
href="http://mirror.example.com/res2"/>
</url>
</urlset>

3
tests/testdata/examples_from_spec_v1_0/resourcesync_ex_3.xml → resync/test/testdata/examples_from_spec/resourcesync_ex_2_3.xml vendored
View File

@ -2,8 +2,7 @@
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:md capability="changelist"
from="2013-01-02T00:00:00Z"
until="2013-01-03T00:00:00Z"/>
modified="2013-01-03T11:00:00Z"/>
<url>
<loc>http://example.com/res2.pdf</loc>
<lastmod>2013-01-02T13:00:00Z</lastmod>

2
tests/testdata/examples_from_spec_v1_0/resourcesync_ex_4.xml → resync/test/testdata/examples_from_spec/resourcesync_ex_2_4.xml vendored
View File

@ -2,7 +2,7 @@
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:md capability="resourcedump"
at="2013-01-03T09:00:00Z"/>
modified="2013-01-03T09:00:00Z"/>
<url>
<loc>http://example.com/resourcedump.zip</loc>
<lastmod>2013-01-03T09:00:00Z</lastmod>

2
tests/testdata/examples_from_spec_v1_0/resourcesync_ex_5.xml → resync/test/testdata/examples_from_spec/resourcesync_ex_2_5.xml vendored
View File

@ -2,7 +2,7 @@
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:md capability="resourcedump-manifest"
at="2013-01-03T09:00:00Z"/>
modified="2013-01-03T09:00:00Z"/>
<url>
<loc>http://example.com/res1</loc>
<lastmod>2013-01-03T03:00:00Z</lastmod>

10
tests/testdata/examples_from_spec_v1_1/resourcesync_ex_6.xml → resync/test/testdata/examples_from_spec/resourcesync_ex_2_6.xml vendored
View File

@ -1,11 +1,11 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="describedby"
href="http://example.com/info_about_set1_of_resources.xml"/>
<rs:ln rel="up"
href="http://example.com/resourcesync_description.xml"/>
<rs:md capability="capabilitylist"/>
<rs:ln href="http://example.com/info-about-source.xml"
rel="describedby"
type="application/xml"/>
<rs:md capability="capabilitylist"
modified="2013-01-02T14:00:00Z"/>
<url>
<loc>http://example.com/dataset1/resourcelist.xml</loc>
<rs:md capability="resourcelist"/>

10
tests/testdata/examples_from_spec_v1_1/resourcesync_ex_8.xml → resync/test/testdata/examples_from_spec/resourcesync_ex_2_7.xml vendored
View File

@ -2,11 +2,13 @@
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:md capability="resourcelist"
at="2013-01-03T09:00:00Z"/>
<sitemap>
<loc>http://example.com/resourcelist-part1.xml</loc>
</sitemap>
modified="2013-01-03T09:00:00Z"/>
<sitemap>
<loc>http://example.com/resourcelist-part2.xml</loc>
<lastmod>2013-01-03T09:00:00Z</lastmod>
</sitemap>
<sitemap>
<loc>http://example.com/resourcelist-part1.xml</loc>
<lastmod>2013-01-03T09:00:00Z</lastmod>
</sitemap>
</sitemapindex>

5
tests/testdata/examples_from_spec_v1_0/resourcesync_ex_14.xml → resync/test/testdata/examples_from_spec/resourcesync_ex_4_1.xml vendored
View File

@ -1,11 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="up"
<rs:ln rel="resourcesync"
href="http://example.com/dataset1/capabilitylist.xml"/>
<rs:md capability="resourcelist"
at="2013-01-03T09:00:00Z"
completed="2013-01-03T09:01:00Z"/>
modified="2013-01-03T09:00:00Z"/>
<url>
<loc>http://example.com/res1</loc>
<lastmod>2013-01-02T13:00:00Z</lastmod>

15
tests/testdata/examples_from_spec_v1_1/resourcesync_ex_15.xml → resync/test/testdata/examples_from_spec/resourcesync_ex_4_2.xml vendored
View File

@ -1,21 +1,20 @@
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="up"
<rs:ln rel="resourcesync"
href="http://example.com/dataset1/capabilitylist.xml"/>
<rs:md capability="resourcelist"
at="2013-01-03T09:00:00Z"
completed="2013-01-03T09:10:00Z"/>
modified="2013-01-03T09:00:00Z"/>
<sitemap>
<loc>http://example.com/resourcelist1.xml</loc>
<rs:md at="2013-01-03T09:00:00Z"/>
<loc>http://example.com/resourcelist3.xml</loc>
<lastmod>2013-01-03T09:00:00Z</lastmod>
</sitemap>
<sitemap>
<loc>http://example.com/resourcelist2.xml</loc>
<rs:md at="2013-01-03T09:03:00Z"/>
<lastmod>2013-01-03T09:00:00Z</lastmod>
</sitemap>
<sitemap>
<loc>http://example.com/resourcelist3.xml</loc>
<rs:md at="2013-01-03T09:07:00Z"/>
<loc>http://example.com/resourcelist1.xml</loc>
<lastmod>2013-01-03T09:00:00Z</lastmod>
</sitemap>
</sitemapindex>

10
tests/testdata/examples_from_spec_v1_0/resourcesync_ex_16.xml → resync/test/testdata/examples_from_spec/resourcesync_ex_4_3.xml vendored
View File

@ -1,22 +1,22 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="up"
<rs:ln rel="resourcesync"
href="http://example.com/dataset1/capabilitylist.xml"/>
<rs:ln rel="index"
<rs:ln rel="up"
href="http://example.com/dataset1/resourcelist-index.xml"/>
<rs:md capability="resourcelist"
at="2013-01-03T09:00:00Z"/>
modified="2013-01-03T09:00:00Z"/>
<url>
<loc>http://example.com/res3</loc>
<lastmod>2013-01-02T13:00:00Z</lastmod>
<lastmod>2013-01-03T09:00:00Z</lastmod>
<rs:md hash="md5:1584abdf8ebdc9802ac0c6a7402c8753"
length="4385"
type="application/pdf"/>
</url>
<url>
<loc>http://example.com/res4</loc>
<lastmod>2013-01-02T14:00:00Z</lastmod>
<lastmod>2013-01-03T09:00:00Z</lastmod>
<rs:md hash="md5:4556abdf8ebdc9802ac0c6a7402c9881"
length="883"
type="image/png"/>

26
resync/test/testdata/examples_from_spec/resourcesync_ex_5_1.xml vendored Normal file
View File

@ -0,0 +1,26 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="resourcesync"
href="http://example.com/dataset1/capabilitylist.xml"/>
<rs:md capability="resourcedump"
modified="2013-01-03T09:00:00Z"/>
<url>
<loc>http://example.com/resourcedump-part3.zip</loc>
<lastmod>2013-01-03T09:00:00Z</lastmod>
<rs:md type="application/zip"
length="4765"/>
</url>
<url>
<loc>http://example.com/resourcedump-part2.zip</loc>
<lastmod>2013-01-03T09:00:00Z</lastmod>
<rs:md type="application/zip"
length="9875"/>
</url>
<url>
<loc>http://example.com/resourcedump-part1.zip</loc>
<lastmod>2013-01-03T09:00:00Z</lastmod>
<rs:md type="application/zip"
length="2298"/>
</url>
</urlset>

5
tests/testdata/examples_from_spec_v1_1/resourcesync_ex_18.xml → resync/test/testdata/examples_from_spec/resourcesync_ex_5_2.xml vendored
View File

@ -1,11 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="up"
<rs:ln rel="resourcesync"
href="http://example.com/dataset1/capabilitylist.xml"/>
<rs:md capability="resourcedump-manifest"
at="2013-01-03T09:00:00Z"
completed="2013-01-03T09:02:00Z"/>
modified="2013-01-03T09:00:00Z"/>
<url>
<loc>http://example.com/res1</loc>
<lastmod>2013-01-02T13:00:00Z</lastmod>

12
tests/testdata/examples_from_spec_v1_0/resourcesync_ex_19.xml → resync/test/testdata/examples_from_spec/resourcesync_ex_6_1.xml vendored
View File

@ -1,28 +1,28 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="up"
<rs:ln rel="resourcesync"
href="http://example.com/dataset1/capabilitylist.xml"/>
<rs:md capability="changelist"
from="2013-01-03T00:00:00Z"/>
modified="2013-01-03T11:00:00Z"/>
<url>
<loc>http://example.com/res1.html</loc>
<lastmod>2013-01-03T11:00:00Z</lastmod>
<lastmod>2013-01-02T11:00:00Z</lastmod>
<rs:md change="created"/>
</url>
<url>
<loc>http://example.com/res2.pdf</loc>
<lastmod>2013-01-03T13:00:00Z</lastmod>
<lastmod>2013-01-02T13:00:00Z</lastmod>
<rs:md change="updated"/>
</url>
<url>
<loc>http://example.com/res3.tiff</loc>
<lastmod>2013-01-03T18:00:00Z</lastmod>
<lastmod>2013-01-02T18:00:00Z</lastmod>
<rs:md change="deleted"/>
</url>
<url>
<loc>http://example.com/res2.pdf</loc>
<lastmod>2013-01-03T21:00:00Z</lastmod>
<lastmod>2013-01-02T21:00:00Z</lastmod>
<rs:md change="updated"/>
</url>
</urlset>

26
resync/test/testdata/examples_from_spec/resourcesync_ex_7_1.xml vendored Normal file
View File

@ -0,0 +1,26 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="resourcesync"
href="http://example.com/dataset1/capabilitylist.xml"/>
<rs:md capability="changedump"
modified="2013-01-03T09:00:00Z"/>
<url>
<loc>http://example.com/changedump-part3.zip</loc>
<lastmod>2013-01-03T09:00:00Z</lastmod>
<rs:md type="application/zip"
length="3109"/>
</url>
<url>
<loc>http://example.com/changedump-part2.zip</loc>
<lastmod>2013-01-03T09:00:00Z</lastmod>
<rs:md type="application/zip"
length="6629"/>
</url>
<url>
<loc>http://example.com/changedump-part1.zip</loc>
<lastmod>2013-01-03T09:00:00Z</lastmod>
<rs:md type="application/zip"
length="8124"/>
</url>
</urlset>

41
resync/test/testdata/examples_from_spec/resourcesync_ex_7_2.xml vendored Normal file
View File

@ -0,0 +1,41 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="resourcesync"
href="http://example.com/dataset1/capabilitylist.xml"/>
<rs:md capability="changedump-manifest"
modified="2013-01-03T21:00:00Z"/>
<url>
<loc>http://example.com/res1.html</loc>
<lastmod>2013-01-01T05:00:00Z</lastmod>
<rs:md change="created"
hash="md5:1c1b0e264fa9b7e1e9aa6f9db8d6362b"
length="4339"
type="text/html"
path="/changes/res1.html"/>
</url>
<url>
<loc>http://example.com/res2.pdf</loc>
<lastmod>2013-01-01T09:00:00Z</lastmod>
<rs:md change="updated"
hash="md5:f906610c3d4aa745cb2b986f25b37c5a
sha-256:f138185cddef488264a0323aee56e7647e89cd7a4d6e45ba28b3be26234a6d09"
length="38297"
type="application/pdf"
path="/changes/res2.pdf"/>
</url>
<url>
<loc>http://example.com/res3.tiff</loc>
<lastmod>2013-01-02T11:00:00Z</lastmod>
<rs:md change="deleted"/>
</url>
<url>
<loc>http://example.com/res1.html</loc>
<lastmod>2013-01-02T14:00:00Z</lastmod>
<rs:md change="updated"
hash="md5:0988647082c8bc51778894a48ec3b576"
length="5426"
type="text/html"
path="/changes/res1-v2.html"/>
</url>
</urlset>

12
tests/testdata/examples_from_spec_v1_0/resourcesync_ex_24.xml → resync/test/testdata/examples_from_spec/resourcesync_ex_8_1.xml vendored
View File

@ -1,13 +1,13 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="up"
<rs:ln rel="resourcesync"
href="http://example.com/dataset1/capabilitylist.xml"/>
<rs:md capability="changelist"
from="2013-01-03T00:00:00Z"/>
modified="2013-01-03T11:00:00Z"/>
<url>
<loc>http://example.com/res1</loc>
<lastmod>2013-01-03T18:00:00Z</lastmod>
<lastmod>2013-01-02T17:00:00Z</lastmod>
<rs:md change="updated"
hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6"
length="8876"
@ -15,14 +15,14 @@
<rs:ln rel="duplicate"
pri="1"
href="http://mirror1.example.com/res1"
modified="2013-01-03T18:00:00Z"/>
modified="2013-01-02T18:00:00Z"/>
<rs:ln rel="duplicate"
pri="2"
href="http://mirror2.example.com/res1"
modified="2013-01-03T18:00:00Z"/>
modified="2013-01-02T18:30:00Z"/>
<rs:ln rel="duplicate"
pri="3"
href="gsiftp://gridftp.example.com/res1"
modified="2013-01-03T18:00:00Z"/>
modified="2013-01-02T18:30:00Z"/>
</url>
</urlset>

10
tests/testdata/examples_from_spec_v1_0/resourcesync_ex_33.xml → resync/test/testdata/examples_from_spec/resourcesync_ex_8_10.xml vendored
View File

@ -1,10 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="up"
<rs:ln rel="resourcesync"
href="http://aggregator2.example.com/dataset1/capabilitylist.xml"/>
<rs:md capability="changelist"
from="2013-01-03T12:00:00Z"/>
modified="2013-01-04T15:00:00Z"/>
<url>
<loc>http://aggregator2.example.com/res1.html</loc>
<lastmod>2013-01-04T09:00:00Z</lastmod>
@ -18,5 +18,11 @@
hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6"
length="8876"
type="text/html"/>
<rs:ln rel="via"
href="http://aggregator1.example.com/res1.html"
modified="2013-01-03T20:00:00Z"
hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6"
length="8876"
type="text/html"/>
</url>
</urlset>

10
tests/testdata/examples_from_spec_v1_0/resourcesync_ex_25.xml → resync/test/testdata/examples_from_spec/resourcesync_ex_8_2.xml vendored
View File

@ -1,21 +1,21 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="up"
<rs:ln rel="resourcesync"
href="http://example.com/dataset1/capabilitylist.xml"/>
<rs:md capability="changelist"
from="2013-01-03T11:00:00Z"/>
modified="2013-01-03T11:00:00Z"/>
<url>
<loc>http://example.com/res1</loc>
<lastmod>2013-01-03T18:00:00Z</lastmod>
<lastmod>2013-01-02T18:00:00Z</lastmod>
<rs:md change="updated"/>
<rs:ln rel="alternate"
href="http://example.com/res1.html"
modified="2013-01-03T18:00:00Z"
modified="2013-01-02T18:00:00Z"
type="text/html"/>
<rs:ln rel="alternate"
href="http://example.com/res1.pdf"
modified="2013-01-03T18:00:00Z"
modified="2013-01-02T18:00:00Z"
type="application/pdf"/>
</url>
</urlset>

8
tests/testdata/examples_from_spec_v1_0/resourcesync_ex_26.xml → resync/test/testdata/examples_from_spec/resourcesync_ex_8_3.xml vendored
View File

@ -1,18 +1,18 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="up"
<rs:ln rel="resourcesync"
href="http://example.com/dataset1/capabilitylist.xml"/>
<rs:md capability="changelist"
from="2013-01-03T00:00:00Z"/>
modified="2013-01-03T11:00:00Z"/>
<url>
<loc>http://example.com/res1.html</loc>
<lastmod>2013-01-03T18:00:00Z</lastmod>
<lastmod>2013-01-02T18:00:00Z</lastmod>
<rs:md change="updated"
hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6"
length="8876"/>
<rs:ln rel="canonical"
href="http://example.com/res1"
modified="2013-01-03T18:00:00Z"/>
modified="2013-01-02T18:00:00Z"/>
</url>
</urlset>

12
tests/testdata/examples_from_spec_v1_0/resourcesync_ex_27.xml → resync/test/testdata/examples_from_spec/resourcesync_ex_8_4.xml vendored
View File

@ -1,34 +1,34 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="up"
<rs:ln rel="resourcesync"
href="http://example.com/dataset1/capabilitylist.xml"/>
<rs:md capability="changelist"
from="2013-01-03T00:00:00Z"/>
modified="2013-01-03T11:00:00Z"/>
<url>
<loc>http://example.com/res4</loc>
<lastmod>2013-01-03T17:00:00Z</lastmod>
<lastmod>2013-01-02T17:00:00Z</lastmod>
<rs:md change="updated"
hash="sha-256:f4OxZX_x_DFGFDgghgdfb6rtSx-iosjf6735432nklj"
length="56778"
type="application/json"/>
<rs:ln rel="http://www.openarchives.org/rs/terms/patch"
href="http://example.com/res4-json-patch"
modified="2013-01-03T17:00:00Z"
modified="2013-01-02T17:00:00Z"
hash="sha-256:y66dER_t_HWEIKpesdkeb7rtSc-ippjf9823742opld"
length="73"
type="application/json-patch"/>
</url>
<url>
<loc>http://example.com/res5-full.tiff</loc>
<lastmod>2013-01-03T18:00:00Z</lastmod>
<lastmod>2013-01-02T18:00:00Z</lastmod>
<rs:md change="updated"
hash="sha-256:f4OxZX_x_FO5LcGBSKHWXfwtSx-j1ncoSt3SABJtkGk"
length="9788456778"
type="image/tiff"/>
<rs:ln rel="http://www.openarchives.org/rs/terms/patch"
href="http://example.com/res5-diff"
modified="2013-01-03T18:00:00Z"
modified="2013-01-02T18:00:00Z"
hash="sha-256:h986gT_t_87HTkjHYE76G558hY-jdfgy76t55sadJUYT"
length="4533"
type="application/x-tiff-diff"/>

16
tests/testdata/examples_from_spec_v1_1/resourcesync_ex_28.xml → resync/test/testdata/examples_from_spec/resourcesync_ex_8_5.xml vendored
View File

@ -1,32 +1,32 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="up"
<rs:ln rel="resourcesync"
href="http://example.com/dataset1/capabilitylist.xml"/>
<rs:md capability="changelist"
from="2013-01-03T00:00:00Z"/>
modified="2013-01-03T11:00:00Z"/>
<url>
<loc>http://example.com/res2.pdf</loc>
<lastmod>2013-01-02T18:00:00Z</lastmod>
<rs:md change="updated"
datetime="2013-01-03T18:00:00Z"
hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6"
length="8876"
type="application/pdf"/>
<rs:ln rel="describedby"
href="http://example.com/res2_dublin-core_metadata.xml"
modified="2013-01-02T18:00:00Z"
type="application/xml"/>
</url>
<url>
<loc>http://example.com/res2_dublin-core_metadata.xml</loc>
<lastmod>2013-01-02T19:00:00Z</lastmod>
<rs:md change="updated"
datetime="2013-01-03T19:00:00Z"
type="application/xml"/>
<rs:ln rel="describes"
href="http://example.com/res2.pdf"
hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6"
length="8876"
modified="2013-01-02T19:00:00Z"
hash="md5:1e0d5cb8ef6ba40c99b14c0237be735e"
length="14599"
type="application/pdf"/>
<rs:ln rel="profile"
href="http://purl.org/dc/elements/1.1/"/>
</url>
</urlset>

11
tests/testdata/examples_from_spec_v1_1/resourcesync_ex_29.xml → resync/test/testdata/examples_from_spec/resourcesync_ex_8_6.xml vendored
View File

@ -1,27 +1,24 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="up"
<rs:ln rel="resourcesync"
href="http://example.com/dataset1/capabilitylist.xml"/>
<rs:md capability="changelist"
from="2013-01-03T00:00:00Z"/>
modified="2013-01-03T11:00:00Z"/>
<url>
<loc>http://example.com/res1</loc>
<lastmod>2013-01-03T18:00:00Z</lastmod>
<lastmod>2013-01-03T07:00:00Z</lastmod>
<rs:md hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6"
length="8876"
type="text/html"
change="updated"/>
<rs:ln rel="memento"
href="http://example.com/20130103070000/res1"
modified="2013-01-02T18:00:00Z"
modified="2013-01-03T07:00:00Z"
hash="md5:1584abdf8ebdc9802ac0c6a7402c03b6"
length="8876"
type="text/html"/>
<rs:ln rel="timegate"
href="http://example.com/timegate/http://example.com/res1"/>
<rs:ln rel="timemap"
href="http://example.com/timemap/http://example.com/res1"
type="application/link-format"/>
</url>
</urlset>

4
tests/testdata/examples_from_spec_v1_0/resourcesync_ex_30.xml → resync/test/testdata/examples_from_spec/resourcesync_ex_8_7.xml vendored
View File

@ -1,10 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="up"
<rs:ln rel="resourcesync"
href="http://example.com/dataset1/capabilitylist.xml"/>
<rs:md capability="changelist"
from="2013-01-03T00:00:00Z"/>
modified="2013-01-03T11:00:00Z"/>
<url>
<loc>http://example.com/res1</loc>
<lastmod>2013-01-03T07:00:00Z</lastmod>

4
tests/testdata/examples_from_spec_v1_1/resourcesync_ex_31.xml → resync/test/testdata/examples_from_spec/resourcesync_ex_8_8.xml vendored
View File

@ -1,10 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:ln rel="up"
<rs:ln rel="resourcesync"
href="http://example.com/dataset1/capabilitylist.xml"/>
<rs:md capability="changelist"
from="2013-01-03T00:00:00Z"/>
modified="2013-01-03T11:00:00Z"/>
<url>
<loc>http://original.example.com/res1.html</loc>
<lastmod>2013-01-03T07:00:00Z</lastmod>

Some files were not shown because too many files have changed in this diff Show More