.. _advexercies:

Exercises
=========

QGIS CSW Client Installation
----------------------------

The HTTP request/response mechanism is not friendly enough to the end user in order to perform queries to the Catalogue Service.
For this workshop, we will use the `QGIS <http://www.qgis.org/>` `OGC Catalogue Service Client <http://sourceforge.net/apps/trac/qgiscommunitypl/wiki/qgcsw>`_ plugin.

To install the plugin in OSGeoLive:

.. code-block:: bash

  $ cd /usr/share/qgis/python/plugins
  $ sudo svn co https://qgiscommunitypl.svn.sourceforge.net/svnroot/qgiscommunitypl/python/plugins/qgcsw/trunk/ qgcsw

.. note::
  Remember that the root password in OSGeoLive is "user".


Data Discovery through QGIS
---------------------------

Start QGIS from the Desktop GIS group and go to "Manage Plugins".

Enable the CSW plugin (version 0.0.9) from the list

  .. image:: ../../_static/pycsw_qgis_install_csw.png
     :scale: 60 %

Then select the CSW button from the toolbar and launch CSW Client

  .. image:: ../../_static/pycsw_qgis_start.png
     :scale: 60 %

Add the pycsw server by pressing the "New" button and type in ``http://localhost/pycsw/csw.py``

  .. image:: ../../_static/pycsw_qgis_add.png
     :scale: 60 %

The user can add some default servers using the "Add default servers" button and also get the capabilities of the server using "Server info" button

  .. image:: ../../_static/pycsw_qgis_capabilities.png
     :scale: 60 %

Raw server responces are available through the "GetCapabilities" button.

  .. image:: ../../_static/pycsw_qgis_capabilities2.png
     :scale: 60 %

.. note::
  The button name is not correct right now. It will show the last server responce even if it is not a GetCapabilities responce.

Perform search using the catalogue, either by a string value or with a bounding box

  .. image:: ../../_static/pycsw_qgis_search.png
     :scale: 60 %

Discovery of data can be also performed through the Tester application by setting the appropriate requests eg. any text search with the string "imagery" leads to discovering XML metadata including the word "imagery". 

  .. image:: ../../_static/pycsw_tester_any_text.png
     :scale: 60 %


Data Discovery through GeoExt
-----------------------------

Another way to use a pycsw server is through a web application, acting like a CSW client.
Such functionality is available through `OpenLayers <http://openlayers.org/>`_ and `GeoExt <http://www.geoext.org/>`_ Javascript libraries.

For this workshop we have hacked a small demonstration in GeoExt (thanks `Bart van den Eijnden <https://github.com/bartvde>`_) using a demo pycsw installation at http://demo.pycsw.org/services/csw: 

- Go to http://demo.pycsw.org/viewer
- Click icon "find layers"
- Enter "airports" (without double quotes)
- Click "search" or hit Enter
- See results
- Click the "add to map" icon beside the last result on that result set ("1 Million Scale - Airports")
- See layer added to map panel

  .. image:: ../../_static/client-gxp.png
     :scale: 60 %


OWSLib CSW client installation
------------------------------

For this exercise we will use the `OWSLib <http://geopython.github.io/OWSLib/>`_ Python library.

OWSLib is a Python package for client programming with Open Geospatial Consortium (OGC) web service (hence OWS) interface standards, and their related content models.

OWSLib is already installed in OSGeoLive as a pycsw dependency.

For installation on another system:

.. code-block:: bash

  $ easy_install OWSLib

or in Ubuntu:

.. code-block:: bash

  $ sudo apt-get install python-owslib


Data Discovery through OWSLib and Python
----------------------------------------

Connect to local pycsw server, and inspect its properties:

.. code-block:: python

  >>> from owslib.csw import CatalogueServiceWeb
  >>> csw = CatalogueServiceWeb('http://localhost/pycsw/csw.py')
  >>> csw.identification.type
  'CSW'
  >>> [op.name for op in csw.operations]
  ['GetCapabilities', 'GetRecords', 'GetRecordById', 'DescribeRecord', 'GetDomain']

You can try to use the following demo deployment for more content:
http://geo.gov.ckan.org/csw

Get supported resultType's:

.. code-block:: python

  >>> csw.getdomain('GetRecords.resultType')
  >>> csw.results
  {'values': ['results', 'validate', 'hits'], 'parameter': 'GetRecords.resultType', 'type': 'csw:DomainValuesType'}
  >>> 

Search for casino data:

.. code-block:: python

  >>> csw.getrecords(keywords=['casino'], maxrecords=20)
  >>> csw.results    
  {'matches': 5, 'nextrecord': 0, 'returned': 5}
  >>> for rec in csw.records:
  ...     print csw.records[rec].title    
  ... 
  CasinoSites
  Parcels_North
  Parcels
  Parcels_East
  Parcels_South
  >>> 

Search for casino data in Canada:

.. code-block:: python

  >>> csw.getrecords(keywords=['casino'],bbox=[-141,42,-52,84])
  >>> csw.results
  {'matches': 0, 'nextrecord': 0, 'returned': 0}
  >>> 

Search for 'casino' or 'bar'

.. code-block:: python

  >>> csw.getrecords(keywords=['casino', 'bar'])
  >>> csw.results
  {'matches': 11, 'nextrecord': 11, 'returned': 10}
  >>> 

Search for a specific record:

.. code-block:: python

  >>> csw.getrecordbyid(id=['http://capita.wustl.edu/DataspaceMetadata_ISO/CIRA.VIEWS.dv.xml'])
  >>> csw.records['http://capita.wustl.edu/DataspaceMetadata_ISO/CIRA.VIEWS.dv.xml'].title
  'VIEWS.dv'

Search with a CQL query

.. code-block:: python

  >>> csw.getrecords(cql='csw:AnyText like "%bar%"')

Transaction: insert

.. code-block:: python

  >>> from urllib2 import urlopen
  >>> content = urlopen('http://demo.pycsw.org/waf/Clause_10.2.5.3.2_Example03_Full.xml').read()
  >>> csw.transaction(ttype='insert', typename='csw:Record', record=content)
  >>> print csw.response
  <?xml version="1.0" encoding="UTF-8" standalone="no"?>
  <csw:TransactionResponse xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:inspire_common="http://inspire.ec.europa.eu/schemas/common/1.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:dct="http://purl.org/dc/terms/" xmlns:ows="http://www.opengis.net/ows" xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" xmlns:gml="http://www.opengis.net/gml" xmlns:dif="http://gcmd.gsfc.nasa.gov/Aboutus/xml/dif/" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:gco="http://www.isotc211.org/2005/gco" xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:srv="http://www.isotc211.org/2005/srv" xmlns:ogc="http://www.opengis.net/ogc" xmlns:fgdc="http://www.opengis.net/cat/csw/csdgm" xmlns:inspire_ds="http://inspire.ec.europa.eu/schemas/inspire_ds/1.0" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:os="http://a9.com/-/spec/opensearch/1.1/" xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope" xmlns:sitemap="http://www.sitemaps.org/schemas/sitemap/0.9" version="2.0.2" xsi:schemaLocation="http://www.opengis.net/cat/csw/2.0.2 http://schemas.opengis.net/csw/2.0.2/CSW-publication.xsd">
    <csw:TransactionSummary>
      <csw:totalInserted>1</csw:totalInserted>
      <csw:totalUpdated>0</csw:totalUpdated>
      <csw:totalDeleted>0</csw:totalDeleted>
    </csw:TransactionSummary>
    <csw:InsertResult>
      <csw:BriefRecord>
        <dc:identifier>00180e67-b7cf-40a3-861d-b3a09337b174</dc:identifier>
        <dc:title>Image2000 Product 1 (at1) Multispectral</dc:title>
      </csw:BriefRecord>
    </csw:InsertResult>
  </csw:TransactionResponse>
  >>> # search
  >>> csw.getrecords(keywords=['image2000'])
  >>> print csw.response
  <?xml version="1.0" encoding="UTF-8" standalone="no"?>
  <csw:GetRecordsResponse xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:inspire_common="http://inspire.ec.europa.eu/schemas/common/1.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:dct="http://purl.org/dc/terms/" xmlns:ows="http://www.opengis.net/ows" xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" xmlns:gml="http://www.opengis.net/gml" xmlns:dif="http://gcmd.gsfc.nasa.gov/Aboutus/xml/dif/" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:gco="http://www.isotc211.org/2005/gco" xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:srv="http://www.isotc211.org/2005/srv" xmlns:ogc="http://www.opengis.net/ogc" xmlns:fgdc="http://www.opengis.net/cat/csw/csdgm" xmlns:inspire_ds="http://inspire.ec.europa.eu/schemas/inspire_ds/1.0" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:os="http://a9.com/-/spec/opensearch/1.1/" xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope" xmlns:sitemap="http://www.sitemaps.org/schemas/sitemap/0.9" version="2.0.2" xsi:schemaLocation="http://www.opengis.net/cat/csw/2.0.2 http://schemas.opengis.net/csw/2.0.2/CSW-discovery.xsd">
    <csw:SearchStatus timestamp="2013-06-13T08:37:07Z"/>
    <csw:SearchResults nextRecord="0" numberOfRecordsMatched="1" numberOfRecordsReturned="1" recordSchema="http://www.opengis.net/cat/csw/2.0.2" elementSet="summary">
      <csw:SummaryRecord>
        <dc:identifier>00180e67-b7cf-40a3-861d-b3a09337b174</dc:identifier>
        <dc:title>Image2000 Product 1 (at1) Multispectral</dc:title>
        <dc:type>dataset</dc:type>
        <dc:subject>imagery</dc:subject>
        <dc:subject>baseMaps</dc:subject>
        <dc:subject>earthCover</dc:subject>
        <dc:format>BIL</dc:format>
        <dct:modified>2004-10-04 00:00:00</dct:modified>
        <dct:abstract>IMAGE2000 product 1 individual orthorectified scenes. IMAGE2000 was  produced from ETM+ Landsat 7 satellite data and provides a consistent European coverage of individual orthorectified scenes in national map projection systems.</dct:abstract>
      </csw:SummaryRecord>
    </csw:SearchResults>
  </csw:GetRecordsResponse>

Transaction: update

.. code-block:: python

  >>> # upload an updated version of that metadata
  >>> content2 = content.replace('Image2000', 'footitle')
  >>> csw.transaction(ttype='update', record=content2)
  >>> print csw.response  # raw
  <?xml version="1.0" encoding="UTF-8" standalone="no"?>
  <csw:TransactionResponse xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:inspire_common="http://inspire.ec.europa.eu/schemas/common/1.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:dct="http://purl.org/dc/terms/" xmlns:ows="http://www.opengis.net/ows" xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" xmlns:gml="http://www.opengis.net/gml" xmlns:dif="http://gcmd.gsfc.nasa.gov/Aboutus/xml/dif/" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:gco="http://www.isotc211.org/2005/gco" xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:srv="http://www.isotc211.org/2005/srv" xmlns:ogc="http://www.opengis.net/ogc" xmlns:fgdc="http://www.opengis.net/cat/csw/csdgm" xmlns:inspire_ds="http://inspire.ec.europa.eu/schemas/inspire_ds/1.0" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:os="http://a9.com/-/spec/opensearch/1.1/" xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope" xmlns:sitemap="http://www.sitemaps.org/schemas/sitemap/0.9" version="2.0.2" xsi:schemaLocation="http://www.opengis.net/cat/csw/2.0.2 http://schemas.opengis.net/csw/2.0.2/CSW-publication.xsd">
    <csw:TransactionSummary>
      <csw:totalInserted>0</csw:totalInserted>
      <csw:totalUpdated>1</csw:totalUpdated>
      <csw:totalDeleted>0</csw:totalDeleted>
    </csw:TransactionSummary>
  </csw:TransactionResponse>
  >>> print csw.results
  >>> {'deleted': 0, 'insertresults': [], 'updated': 1, 'inserted': 0, 'requestid': None}

Transaction: delete

.. code-block:: python

  >>> # delete record we inserted
  >>> csw.transaction(ttype='delete', identifier='00180e67-b7cf-40a3-861d-b3a09337b174')
  >>> csw.results
  {'deleted': 1, 'insertresults': [], 'updated': 0, 'inserted': 0, 'requestid': None}

Harvest a resource

.. code-block:: python

  >>> help(csw.harvest)

  Help on method harvest in module owslib.csw:

  harvest(self, source, resourcetype, resourceformat=None, harvestinterval=None, responsehandler=None) method of owslib.csw.CatalogueServiceWeb instance
      Construct and process a Harvest request

    Parameters
    ----------

      - source: a URI to harvest
      - resourcetype: namespace identifying the type of resource
      - resourceformat: MIME type of the resource
      - harvestinterval: frequency of harvesting, in ISO8601
      - responsehandler: endpoint that CSW should responsd to with response
  (END)
  >>> csw.harvest('http://webservices.nationalatlas.gov/wms/1million', resourcetype='http://www.opengis.net/wms')
  >>> print csw.response
  <?xml version="1.0" encoding="UTF-8" standalone="no"?>
  <csw:HarvestResponse xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:inspire_common="http://inspire.ec.europa.eu/schemas/common/1.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:dct="http://purl.org/dc/terms/" xmlns:ows="http://www.opengis.net/ows" xmlns:apiso="http://www.opengis.net/cat/csw/apiso/1.0" xmlns:gml="http://www.opengis.net/gml" xmlns:dif="http://gcmd.gsfc.nasa.gov/Aboutus/xml/dif/" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:gco="http://www.isotc211.org/2005/gco" xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:srv="http://www.isotc211.org/2005/srv" xmlns:ogc="http://www.opengis.net/ogc" xmlns:fgdc="http://www.opengis.net/cat/csw/csdgm" xmlns:inspire_ds="http://inspire.ec.europa.eu/schemas/inspire_ds/1.0" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:os="http://a9.com/-/spec/opensearch/1.1/" xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope" xmlns:sitemap="http://www.sitemaps.org/schemas/sitemap/0.9" xsi:schemaLocation="http://www.opengis.net/cat/csw/2.0.2 http://schemas.opengis.net/csw/2.0.2/CSW-publication.xsd">
    <csw:TransactionResponse version="2.0.2">
      <csw:TransactionSummary>
        <csw:totalInserted>13</csw:totalInserted>
        <csw:totalUpdated>0</csw:totalUpdated>
        <csw:totalDeleted>0</csw:totalDeleted>
      </csw:TransactionSummary>
      <csw:InsertResult>
        <csw:BriefRecord>
          <dc:identifier>urn:uuid:8ed68189-c33e-43c8-9ac0-59ade5cbf682</dc:identifier>
          <dc:title>1 Million Scale WMS Layers from the National Atlas of the United States</dc:title>
        </csw:BriefRecord>
        <csw:BriefRecord>
          <dc:identifier>urn:uuid:8ed68189-c33e-43c8-9ac0-59ade5cbf682-ports1m</dc:identifier>
          <dc:title>1 Million Scale - Ports</dc:title>
        </csw:BriefRecord>
        <csw:BriefRecord>
          <dc:identifier>urn:uuid:8ed68189-c33e-43c8-9ac0-59ade5cbf682-national1m</dc:identifier>
          <dc:title>1 Million Scale - National Boundary</dc:title>
        </csw:BriefRecord>
        <csw:BriefRecord>
          <dc:identifier>urn:uuid:8ed68189-c33e-43c8-9ac0-59ade5cbf682-impervious</dc:identifier>
          <dc:title>1 Million Scale - Impervious Surface 100 Meter Resolution</dc:title>
        </csw:BriefRecord>
        <csw:BriefRecord>
          <dc:identifier>urn:uuid:8ed68189-c33e-43c8-9ac0-59ade5cbf682-coast1m</dc:identifier>
          <dc:title>1 Million Scale - Coastlines</dc:title>
        </csw:BriefRecord>
        <csw:BriefRecord>
          <dc:identifier>urn:uuid:8ed68189-c33e-43c8-9ac0-59ade5cbf682-cdl</dc:identifier>
          <dc:title>1 Million Scale - 113th Congressional Districts</dc:title>
        </csw:BriefRecord>
        <csw:BriefRecord>
          <dc:identifier>urn:uuid:8ed68189-c33e-43c8-9ac0-59ade5cbf682-landcov100m</dc:identifier>
          <dc:title>1 Million Scale - Land Cover 100 Meter Resolution</dc:title>
        </csw:BriefRecord>
        <csw:BriefRecord>
          <dc:identifier>urn:uuid:8ed68189-c33e-43c8-9ac0-59ade5cbf682-cdp</dc:identifier>
          <dc:title>1 Million Scale - 113th Congressional Districts by Party</dc:title>
        </csw:BriefRecord>
        <csw:BriefRecord>
          <dc:identifier>urn:uuid:8ed68189-c33e-43c8-9ac0-59ade5cbf682-amtrak1m</dc:identifier>
          <dc:title>1 Million Scale - Railroad and Bus Passenger Stations</dc:title>
        </csw:BriefRecord>
        <csw:BriefRecord>
          <dc:identifier>urn:uuid:8ed68189-c33e-43c8-9ac0-59ade5cbf682-airports1m</dc:identifier>
          <dc:title>1 Million Scale - Airports</dc:title>
        </csw:BriefRecord>
        <csw:BriefRecord>
          <dc:identifier>urn:uuid:8ed68189-c33e-43c8-9ac0-59ade5cbf682-one_million</dc:identifier>
          <dc:title>1 Million Scale WMS Layers from the National Atlas of the United States</dc:title>
        </csw:BriefRecord>
        <csw:BriefRecord>
          <dc:identifier>urn:uuid:8ed68189-c33e-43c8-9ac0-59ade5cbf682-states1m</dc:identifier>
          <dc:title>1 Million Scale - States</dc:title>
        </csw:BriefRecord>
        <csw:BriefRecord>
          <dc:identifier>urn:uuid:8ed68189-c33e-43c8-9ac0-59ade5cbf682-treecanopy</dc:identifier>
          <dc:title>1 Million Scale - Tree Canopy 100 Meter Resolution</dc:title>
        </csw:BriefRecord>
      </csw:InsertResult>
    </csw:TransactionResponse>
  </csw:HarvestResponse>