WES Library Components - WES
Data Warehousing for Data, Data Products and Related Information and Content
WES Library - Components
WES Library is made up of the following components:
Discovery Metadata Catalog (DMC)
Metadata Cataloguing and Storage
The WES Discovery Metadata Catalog provides cataloguing and storage of metadata in a relational database for easy discovery and retreiaval via WES Library and WES Search. Structured data from local and external systems can be harvested and the metadata ingested into the WES DMC. Unstructured data ingested by the companion product Meta Manager is also stored in the WES DMC.
WES DMC is available for Oracle, Microsoft SQL Server, and PostgreSQL relational databases.
Federated Search
Federated Access to Remote Libraries
Federated Search provides a means for WES Library to access external systems that may not support or permit data harvesting. WES Library can construct and issue queries directly to a remote system on behalf of the user, using the remote system’s native interface. For remote systems that require authentication, Federated Search can be configured to authenticate using either a dedicated WES account or the user’s individual account.
WES Ingest
Data Ingestion Sub-Component
The primary responsibilities of this module are to:
- receive notifications when incoming data is available for processing;
- validate the incoming data to ensure that all required files and metadata attributes are available;
- extract metadata from the data/data product into a common metadata format;
- create QuickLook images; and,
- utilize the WES Library module to archive the data once processing is complete.
Like all WES service interfaces, WES Ingest functionality is exposed via a Representational State Transfer (REST) API that provides external programs and third parties with the ability to invoke the ingestion process simply by calling the methods available in the REST API.
WES Archive
Data Storage and Retrieval
WES Archive is the sub-component used to store and access data in WES Library. WES Archive may be co-located on the same system as WES or be distributed to different storage systems.
WES Archive provides a REST API to allow the archiving and retrieval of products stored in WES LIbrary. As such, the WES Archive REST API is utilized by multiple subsystems, including:
- WES Ingest to archive incoming products;
- WES Order to retrieve products for delivery, request retrieval status; and,
- Third party applications that require the ability to archive, delete, and update products.
Similar to WES Ingest, the WES Archive REST API supports username/password authentication as well as HTTPS. Also, JSON is used to represent the request on all HTTP POST operations.
WES deliverD
Data Preparation and Processing
WES Data Preparation and Processing (deliverD) is a vital sub-system within WES that provides users with options for instructing WES how data is handled and translated during the data access and delivery actions. This modular infrastructure provides an efficient framework for existing and emerging services that enable efficient and effective retrieval of data, data preparation, validation, transformation, publishing and delivery services for end users.
WES deliverD acts as a middleware component and converts that sometimes difficult to handle data to a format which can then be used by traditional and non-traditional users. The WES deliverD sub-component GeoPackager is based upon the OGC GeoPackage Encoding Starndard that enables the consolidation of data into a new standards-based secure package for use on mobile computing platforms.
WES deliverD is fully integrated with the WES online order and shopping basket modules to provide end-to-end data selection, processing and delivery.
