Extractor DPUs

This DPU downloads one or more files from the defined locations. The files to be downloaded may be located at HTTP URLs, on the local file system, at certain SFTP/FTP servers, etc.

Individual files and also whole directories may be downloaded. If a directory is provided then all files and files in subdirectories are extracted.

If an internal name (file name) is specified for the downloaded entry, this name is then used as a symbolic name to internally identify the given file further on the pipeline.

If you specify a directory as an entry then this file name is used as a prefix for the individual files within that directory.

In cases where you just need to iterate and process each downloaded file in the same way, you do not need to specify a file name.

This DPU also sets virtual path metadata for each file extracted. In case of files it is equal to the file name (local file name from the file path, e.g. example.txt from a/b/c/example.txt).

In case of directories, virtual path metadata for each extracted file is equal to the relative path of the original directory.

The URI of a file may contain macro {{execId}}, which is replaced during pipeline execution with the actual pipeline execution ID.

Name	Type	Data Unit	Description	Required
output	output	FilesDataUnit	Downloaded files
config	input	RdfDataUnit	Dynamic DPU configuration, see Advanced configuration

It is also possible to dynamically configure the DPU over its input `config` data unit using RDF data.

Turtle

<http://localhost/resource/config>  <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://unifiedviews.eu/ontology/dpu/filesDownload/Config>;
        <http://unifiedviews.eu/ontology/dpu/filesDownload/hasFile> <http://localhost/resource/file/0>.


<http://localhost/resource/file/0> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://unifiedviews.eu/ontology/dpu/filesDownload/File>;
        <http://unifiedviews.eu/ontology/dpu/filesDownload/file/uri> "http://www.zmluvy.gov.sk/data/att/117597_dokument.pdf"; 
        <http://unifiedviews.eu/ontology/dpu/filesDownload/file/fileName> "zmluva.pdf".

The following image shows a fragment of a pipeline which downloads an Excel file from the tmp folder of the UnifiedViews server. The data of the Excel file is subsequently converted to RDF and loaded into a Virtuoso triple store. The DPU configuration is illustrated in the image below.

The following image shows the configuration for downloading multiple files at once.

The following image shows the configuration to download all files in a directory, if any subdirectories are located here then those files will be taken as well.

The following image shows a fragment of a pipeline which downloads an Excel file from the tmp folder of the UnifiedViews server. The data of the Excel file is subsequently converted to RDF and serves as input for a SPARQL Construct Query. The purpose of this query is to construct the configuration file of the second Files Download DPU. After the files are downloaded they are uploaded to the tmp folder of the UnifiedViews server using the Files Upload DPU. The DPU configuration is illustrated in the image below; it is empty as the configuration comes from the input RDF file.

The query used in this pipeline creates triples containing the download URI and the file name of the files that are to be downloaded. The query reads as follows:

CONSTRUCT {
<http://localhost/resource/config>  <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://unifiedviews.eu/ontology/dpu/filesDownload/Config>;
        <http://unifiedviews.eu/ontology/dpu/filesDownload/hasFile> <http://localhost/resource/file/0>.

<http://localhost/resource/file/0> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://unifiedviews.eu/ontology/dpu/filesDownload/File>;
        <http://unifiedviews.eu/ontology/dpu/filesDownload/file/uri> ?fileUri; 
        <http://unifiedviews.eu/ontology/dpu/filesDownload/file/fileName> ?fileName.
}
WHERE {
?s <http://localhost/fileuri/fileName> ?fileName.
?s <http://localhost/fileUri> ?fileUri
}

This DPU allows executing HTTP requests (GET, POST methods) to web services and passes the response in form of file data unit.

This DPU targets to enable consuming web services, both. REST and SOAP.

For POST HTTP requests, there are 4 possible modes (type of sent data)

If sent data is multipart or raw, this DPU offers possibility to preview the HTTP response in design time.

The DPU also supports HTTPS requests.

Name	Type	DataUnit	Description	Required
requestOutput	output	FilesDataUnit	File(s) containing HTTP response(s)
requestFilesConfig	input	FilesDataUnit	Files sent as content of raw HTTP POST request
rdfConfig	input	RDFDataUnit	RDF configuration used to configure form-data bodies

It is also possible to dynamically configure the request body over the input config data unit using RDF data. This is available only for raw mode and you can configure only the request.

# to dynamically configure request URL and request body (raw data mode)
<http://localhost/resource/config>
    <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://unifiedviews.eu/ontology/dpu/httpRequest/Config>;
    <http://unifiedviews.eu/ontology/dpu/httpRequest/requestBody> "..." ;
    <http://unifiedviews.eu/ontology/dpu/httpRequest/url> "http://semantic-web.com/service/x".

# two form-param bodies with the same set of three form params
<http://localhost/resource/config>
    <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://unifiedviews.eu/ontology/dpu/httpRequest/Config>;
    <http://unifiedviews.eu/ontology/dpu/httpRequest/formParamBody> <http://unifiedviews.eu/ontology/dpu/httpRequest/FormParamBody/1> ;
    <http://unifiedviews.eu/ontology/dpu/httpRequest/formParamBody> <http://unifiedviews.eu/ontology/dpu/httpRequest/FormParamBody/2> .

<http://unifiedviews.eu/ontology/dpu/httpRequest/FormParamBody/1>  a <http://unifiedviews.eu/ontology/dpu/httpRequest/FormParamBody> ;
    <http://unifiedviews.eu/ontology/dpu/httpRequest/formParam> <http://unifiedviews.eu/ontology/dpu/httpRequest/FormParam1> ;
    <http://unifiedviews.eu/ontology/dpu/httpRequest/formParam> <http://unifiedviews.eu/ontology/dpu/httpRequest/FormParam2> ;
    <http://unifiedviews.eu/ontology/dpu/httpRequest/formParam> <http://unifiedviews.eu/ontology/dpu/httpRequest/FormParam3> .

<http://unifiedviews.eu/ontology/dpu/httpRequest/FormParam1> a <http://unifiedviews.eu/ontology/dpu/httpRequest/FormParam> ;
    <http://unifiedviews.eu/ontology/dpu/httpRequest/param> "corpusId" ;
    <http://unifiedviews.eu/ontology/dpu/httpRequest/value>  "corpus:307b420d-43ad-4771-be41-308199da95b1" .

 <http://unifiedviews.eu/ontology/dpu/httpRequest/FormParam2> a <http://unifiedviews.eu/ontology/dpu/httpRequest/FormParam> ;
    <http://unifiedviews.eu/ontology/dpu/httpRequest/param> "text" ;
    <http://unifiedviews.eu/ontology/dpu/httpRequest/value>  "Test" .

 <http://unifiedviews.eu/ontology/dpu/httpRequest/FormParam3> a <http://unifiedviews.eu/ontology/dpu/httpRequest/FormParam> ;
    <http://unifiedviews.eu/ontology/dpu/httpRequest/param> "title" ;
    <http://unifiedviews.eu/ontology/dpu/httpRequest/value>  "Test title" .


 <http://unifiedviews.eu/ontology/dpu/httpRequest/FormParamBody/2>  a <http://unifiedviews.eu/ontology/dpu/httpRequest/FormParamBody> ;
    <http://unifiedviews.eu/ontology/dpu/httpRequest/formParam> <http://unifiedviews.eu/ontology/dpu/httpRequest/FormParam1> ;
    <http://unifiedviews.eu/ontology/dpu/httpRequest/formParam> <http://unifiedviews.eu/ontology/dpu/httpRequest/FormParam2> ;
    <http://unifiedviews.eu/ontology/dpu/httpRequest/formParam> <http://unifiedviews.eu/ontology/dpu/httpRequest/FormParam3> .

Many times this DPU is used to get RDF data from PoolParty or triple stores using a SPARQL CONSTRUCT query. The following table contains relevant endpoints for this:

The following image shows a fragment of a pipeline which first executes a SPARQL DELETE in a PoolParty project (SPARQL Endpoint Loader) and then creates a snapshot of the PoolParty project with a HTTP Request. The configuration for the snapshot API call can be seen in the image below. For more information on the used Web Service Method see here: https://help.poolparty.biz/x/ySiU

The following image shows another configuration example to make an API call to PoolParty. For this example, the Web Service Method Request Subtree of Concept or Concept Scheme is used. For more information on this Web Service Method see here: https://help.poolparty.biz/x/AimU

The following image shows a fragment of a pipeline which downloads a file, sends it to the PoolParty extractor to be annotated, from the response we will construct the configuration that will be sent as input the the HTTP Request API, and finally the result will be loaded into a SPARQL Endpoint.

The SPARQL Construct configuration can be seen below. This construct will be parsed into the HTTP API Request DPU as configuration parameters following a specific format. The construct creates a unique configuration for each subject, which is essentially the document URI. Attached to each config is the document text encoded to remove whitespaces.

The following image shows a fragment of a pipeline which is used to populate a GraphSearch search space. What is not seen are files are downloaded from a folder on the server and are annotated using the PoolParty Concept Extractor. This fragment shows the start of the configuration based on the annotation results, the configuration is transferred to the PoolParty GraphSearch Content Indexing Request Constructor. Following this HTTP API Requests are sent to drop the current index, create the new content in GraphSearch Space, and finally a refresh of the newly created content's index.

The following image is a configuration of the refresh search index for GraphSearch, this is usually attached to the pipeline where content is created (as seen above).

For further information about this API please check https://help.poolparty.biz/x/FyqU

PoolParty Concept Extractor is a DPU / plugin for UnifiedViews to consume the Concept Extraction service provided by PoolParty Extractor. Given triples with string literal objects representing texts or files containing texts as input, this extractor annotates texts against a thesaurus project in PoolParty and produces annotations in RDF triples as output.

Please refer to the following documentation for more information about PoolParty Extractor.