Transformer DPUs

This DPU transforms Excel files (.XLS, .XLSX) into a CSV file. It is possible to indicate which sheet(s) should be extracted and transformed. Each sheet is extracted to a separate CSV file.

Name	Type	DataUnit	Description	Required
input	input	FilesDataUnit	Input files
output	output	FilesDataUnit	Produced CSV files

The following image shows a pipeline which downloads an Excel file from a server, converts it to CSV, and uploads it to another server. The excelToCSV DPU receives the downloaded Excel file as input and produces a CSV file as output. The configuration of the DPU can be seen in the image below.

This DPU renames files based on a pattern defined in the configuration.

Inputs and Outputs

Name	Type	Data Unit	Description	Required
inFilesData	input	FilesDataUnit	File name to be modified
outFilesData	output	FilesDataUnit	File name after modification

The following image shows a pipeline which downloads a file from a server, renames it, and uploads it again. The Rename Files DPU takes the downloaded file as input and produces the renamed file as output. The configuration of the DPU is shown in the image below.

The following image shows a fragment of a pipeline which downloads a file, unzips it and then uses the Rename Files DPU to replace white spaces in the file names with "_". The configuration of the DPU is shown in the image below.

The following image shows a fragment of a pipeline which downloads a file and uses the Rename Files DPU with the advanced mode to replace the file name with a UUID. The relevant input and output parameters of this DPU are:

Input filename: file.txt

Regex: .*\\.txt

Replacement: struuid()

Output filename: 4528d703-f330-4f51-a993-e09a9e74ec12

The configuration of the DPU is shown in the image below.

The following image shows a fragment of a pipeline which downloads a file and uses the Rename Files DPU with the advanced mode to add a timestamp to the file name. The relevant input and output parameters of this DPU are:

Input filename: file.txt

Regex: (.*)(\\.txt)

Replacement: concat("$1-", str(now()), "$2")

Output filename: file-2018-11-19T09:43:02.590+01:00.txt

This DPU extracts RDF data from input files of any RDF file format and produces RDF graphs as the output.

By default, the RDF format of the input files is estimated automatically based on the extensions of the input file names. The user can also manually specify the RDF format of the input files in the configuration to make sure the correct RDF format is applied.

Based on the selected policy for creation of the output RDF graphs, the output RDF data unit contains either

In the case of one output graph for each processed input file, the symbolic names for output RDF graphs are created based on the symbolic names of input files. When only one single output graph is generated, the symbolic name of the single output RDF graph may be specified in the configuration.

This DPU supports RDF Validation extension.

Name	Type	DataUnit	Description	Required
filesInput	input	FilesDataUnit	Input file containing RDF data
rdfOutput	output	RDFDataUnit	Extracted RDF data

The following RDF formats are available in this DPU:

The following image shows a fragment of a pipeline which downloads a TriG file, converts the file to RDF and then uses the output as basis for a SPARQL Construct query. The image below shows the configuration of the DPU.

This DPU merges RDF input graphs into a single new RDF output graph. This means that all triples from the input graphs are put into a single new output graph.

Name	Type	Data Unit	Description	Required
input	input	RDFDataUnit	Input graphs which are to be merged
output	output	RDFDataUnit	RDF graph which contains all triples from the input graphs

The following image shows a fragment of a pipeline which

It then proceeds to merge both input graphs to one single output graph. This graph is then processed in a SPARQL Construct query. The configuration of the GraphMerger DPU is shown in the image below.

Converts JSON data (e.g. responses to REST services) to XML, so that it can be further processed, e.g., by XSLT.

Name	Type	DataUnit	Description	Required
input	i	FilesDataUnit	Input JSON files
output	o	FilesDataUnit	Produced XML data

The following image shows a pipeline which performs an API request to PoolParty, transforms the JSON response to XML, and converts this XML into RDF/XML.

The configuration of the DPU can be seen in the image below.

The URL Prefix allows you to automatically replace a part of text/URL with a whitespace. This is useful when you work with custom attributes URLs as when these are converted to XML format the first slash from http:/ will then break in the text causing an output of unwanted XML data.

e.g.

JSON Response

{
 "uri": "http://vocabulary.server.org/cocktails/0bbdca98-077b-48d1-99d6-47eca59c442c",
 "prefLabel": "Aviation",
 "http://vocabulary.server.org semantic-web.at/cocktail-ontology/image": [
     "https://upload.wikimedia.org/wikipedia/commons/4/4f/Aviation_Cocktail.jpg"
 ]
}

Unwanted XML Output

<root>
  <http:>https://upload.wikimedia.org/wikipedia/commons/4/4f/Aviation_Cocktail.jpg</http:>
</root>

This is unwanted and is difficult to further transform due to the nature of needing to encode character breaks to handle this cause.

With the usage of the URL Prefix field we are able to have a better generated XML tag.

e.g.

Wanted XML Output

<root>
  <image>https://upload.wikimedia.org/wikipedia/commons/4/4f/Aviation_Cocktail.jpg</image>
</root>

This DPU transforms input using the SPARQL Construct query provided. The result of the SPARQL Construct - the created triples - is stored to the output.

Note: Internally, the query is translated to SPARQL Update query before it is executed.

It supports RDF Validation extension.

Name	Type	Data Unit	Description	Mandatory
input	input	RDFDataUnit	RDF input
output	output	RDFDataUnit	transformed RDF output

The following image shows a fragment of a pipeline which downloads an Excel file from the server and transforms it into RDF. With a SPARQL Construct we add a skos:prefLabel and convert the URI generated by the Tabular Transformer into a UUID. The DPU configuration is illustrated in the image below.

The following image shows a fragment of a pipeline which downloads an Excel file from the tmp folder of the UnifiedViews server. The data of the Excel file is subsequently converted to RDF and serves as input for the SPARQL Construct Query. The purpose of this query is to construct the configuration file of the second Files Download DPU. The query creates triples containing the download URI and the file name of the files that are to be downloaded. With this configuration file the Files Download DPU downloads the indicated files. After the files are downloaded they are uploaded to the tmp folder of the UnifiedViews server using the Files Upload DPU. The DPU configuration is illustrated in the image below.

Transforms SPARQL SELECT query results to CSV. It does not validate query.

Name	Type	DataUnit	Description	Required
input	i	RDFDataUnit	RDF graph
output	o	FilesDataUnit	CSV file containing SPARQL SELECT query result

The following image shows a pipeline which loads all data from a SPARQL Endpoint, selects a subset of the data using the SPARQL Select Query, transforms the CSV response to relational data and loads that into a relational database

This DPU transforms input RDF using a single SPARQL Update query.

It supports RDF Validation extension.

It does not support quads - it is always executed either on top of all input graphs, or, if per-graph execution is checked, successively on each graph.

Name	Type	DataUnit	Description	Required
input	i	RDFDataUnit	RDF input
output	o	RDFDataUnit	RDF output (transformed)

In this pipeline below we are extracting data via a SPARQL Endpoint, then we split the pipeline into two processes to Update and Construct data using the SPARQL Update and SPARQL Construct DPUs. WIth the SPARQL Update Query we

This DPU converts tabular data into RDF data. As an input it takes CSV, DBF and XLS files.

It supports RDF Validation Extension.

There are three different types of mapping available:

The simple mapping tab allows to define how the columns should be mapped to the resulting predicates, including also information about the datatypes. The Advanced mapping tab is equivalent to the Simple mapping tab, but it allows to specify templates for values of the predicates. A sample template is http://localhost/{type}/content/{id}, where "type" and "id" are names of the columns in the input file. The XLS mapping can be used for the static mapping of cells to named cells. Named cells are accessible as extension in every row.

Name	Type	Data Unit	Description	Required
table	input	FilesDataUnit	Input files containing tabular data
triplifiedTable	output	FilesDataUnit	RDF data

The following image shows a fragment of a pipeline which downloads a CSV file from the tmp folder of the UnifiedViews server. The data of the file is subsequently converted to RDF and loaded into a Virtuoso triple store. The DPU configuration is illustrated in the image below.

The following image shows a fragment of a pipeline which downloads an Excel file (.xls) from the tmp folder of the UnifiedViews server. The data of the Excel file is subsequently converted to RDF and serves as input for a SPARQL Construct Query. The purpose of this query is to construct the configuration file of the second Files Download DPU. After the files are downloaded they are uploaded to the tmp folder of the UnifiedViews server using the Files Upload DPU. The DPU configuration is illustrated in the image below.

The query used in this pipeline creates triples containing the download URI and the file name of the files that are to be downloaded. The query reads as follows:

CONSTRUCT {
<http://localhost/resource/config>  <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://unifiedviews.eu/ontology/dpu/filesDownload/Config>;
        <http://unifiedviews.eu/ontology/dpu/filesDownload/hasFile> <http://localhost/resource/file/0>.

<http://localhost/resource/file/0> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://unifiedviews.eu/ontology/dpu/filesDownload/File>;
        <http://unifiedviews.eu/ontology/dpu/filesDownload/file/uri> ?fileUri; 
        <http://unifiedviews.eu/ontology/dpu/filesDownload/file/fileName> ?fileName.
}
WHERE {
?s <http://localhost/fileuri/fileName> ?fileName.
?s <http://localhost/fileUri> ?fileUri
}

The following image shows a fragment of a pipeline which downloads a CSV file from the server and transforms it into RDF. With a SPARQL Construct we convert the URI generated by the Tabular File To RDF Transformer into a UUID. The DPU configuration is illustrated in the image below.

This DPU transforms XML documents to equivalent JSON documents. The input and output is of type file.

Name	Type	DataUnit	Description	Required
filesInput	i	FilesDataUnit	XML documents to be transformed
filesOutput	i	FilesDataUnit	JSON documents produced from transformation

A sequence of XML elements with same element name can be considered as an array. However, there is no array structure in XML similar to JSON array, which means it is not prossible to decide if an element is actually an array with one object when there is only one element. Therefore, StAXON introduces some annotations to explicitly declare the XML elements that should be transformed into JSON arrays. Those annotations can be defined based on a XPath-like syntax path ::= '/'? <localName> ('/' <localName>)*.

For example:

<alice>
        <bob>edgar</bob>
</alice> 

are transformed to:

{
 "alice":{
  "bob":"edgar"
 }
}

By default with an annotation such as /alice/bob the result will become:

{
 "alice":{
  "bob":[
    "edgar"
  ]
 }
}

The core transformation engine of this plugin attributes to StAXON - JSON via StAX from Odysseus Software . For more information related to XML to JSON transformation, please visit StAXON WiKi.

Particularly, the following chapters are relevant to this plugin:

StAXON WiKi - Converting XML to JSON StAXON WiKi - Mapping Convention StAXON WiKi - Mastering Arrays

This DPU performs XSL transformation over input files, using a single static template. The outcome of the transformation will be in file format.

This DPU supports random UUID generation using randomUUID() function in namespace uuid-functions.

Example usage:

<?xml version="1.0" encoding="UTF-8"?> 
<xsl:stylesheet xmlns:uuid="uuid-functions" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0"> 
<xsl:template match="/"> 
<xsl:value-of select="uuid:randomUUID()"/> 
</xsl:template> 
</xsl:stylesheet>

Name	Type	DataUnit	Description	Required
files	i	FilesDataUnit	File/s to be transformed
files	o	FilesDataUnit	Transformed file of given type
config	i	RdfDataUnit	Dynamic DPU configuration, see Advanced configuration

It is also possible to dynamically configure the DPU over its input config using RDF data.

Configuration sample:

<http://localhost/resource/config> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://linked.opendata.cz/ontology/uv/dpu/xslt/Config>; 
<http://linked.opendata.cz/ontology/uv/dpu/xslt/fileInfo> <http://localhost/resource/fileInfo/0>;
<http://linked.opendata.cz/ontology/uv/dpu/xslt/outputFileExtension> “.ttl”.


<http://localhost/resource/fileInfo/0> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://linked.opendata.cz/ontology/uv/dpu/xslt/FileInfo>;
<http://linked.opendata.cz/ontology/uv/dpu/xslt/param> <http://localhost/resource/param/0>; 
<http://unifiedviews.eu/DataUnit/MetadataDataUnit/symbolicName> “smlouva.ttl”.


<http://localhost/resource/param/0> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://linked.opendata.cz/ontology/uv/dpu/xslt/Param>; 
<http://linked.opendata.cz/ontology/uv/dpu/xslt/param/name> “paramName”; 
<http://linked.opendata.cz/ontology/uv/dpu/xslt/param/value> “paramValue”.

The following image shows a fragment of a pipeline which first downloads an XML file, uses XSLT to transform the file into RDF/XML file, followed by transforming this outputted file into RDF Data. The configuration for the XSLT can be seen below.

Using the above sample config I can construct the input for our next example. Allowing me to use RDF config as my input in place of my file, demonstrated below.