Working With the Files Data Unit

Prepare DPU 'MyDpu' as described in Working With the Files Data Unit. To read files from the input Files data unit, you have to define the input Files data unit.

 @DataUnit.AsInput(name = "input")
 public FilesDataUnit input;

All data units must be public with the proper annotation: they must at least contain a name, which will be the name visible in the UnifiedViews administration interface for pipeline developers. The code above goes to the Main DPU class.

Further, we need to iterate over the input data unit in order to get access to files which come through the input files data unit.

The code below goes to the innerExecute() method of the DPU.

Code 1 - Iterating over input files using API classes

Set<File> files = new HashSet<>();
FilesDataUnit.Iteration it = null;
try {
    it = input.getIteration();
    while (it.hasNext()) {
        final String filePathUri = it.next().getFileURIString();
        files.add(new File(java.net.URI.create(filePathUri)));
    }
} catch (DataUnitException ex) {
    throw ContextUtils.dpuException(ctx, ex, "dpuName.error");
} finally {
    if (it != null) {
        try {
            it.close();
        } catch (DataUnitException ex) {
            log.error("Error on close.", ex);
        }
    }
}
 

So after executing Code 1, we have a set of files within the variable files. We may then work with the files as needed.

The code introduced in Code1 can be simplified.

Simplify by using helper: eu.unifiedviews.helpers.dataunit.files.FilesHelper in uv-dataunit-helpers, which automatically stores all the entries to a set of data unit entries. In this case, the DPU developer does not need to manually handle the iteration.

Code 2 - Iterating over input files using helper FilesHelper

try {
        Set<FilesDataUnit.Entry> files = FilesHelper.getFiles(input);
} catch (DataUnitException ex) {
   throw ContextUtils.dpuException(ctx, ex, "dpuName.error");
}

Prepare DPU 'MyDpu' as described in Working With the Files Data Unit. To write files to the output RDF data unit, one has to define output Files data unit.

@DataUnit.AsOutput(name = "output")
public WritableFilesDataUnit output;

All data units must be public with proper annotation: they must at least contain a name, which will be the name visible in the UnifiedViews administration interface for pipeline developers. The code above goes to the Main DPU class.

First, lets introduce the way how to add files using API classes.

There are two methods DPU developers may use to add files to output Files data unit (using the helper):

In order to add existing file, the code below may be used in the innerExecute() method of the DPU.

Code 3 - Adding file to the output data unit using API classes

File file = ...
String fileName = ...
Symbolic symbolicName = output.addExistingFile(fileName, file.toURI().toString());
MetadataUtils.set(output, symbolicName, FilesVocabulary.UV_VIRTUAL_PATH, fileName);

The code introduced in Code3 can be simplified by using helper: eu.unifiedviews.helpers.dataunit.files.FilesHelper in uv-dataunit-helpers.

In general, as a DPU developer you should use helpers, if possible.

There are two methods DPU developers may use to add files to output files data unit (using the helper):

As the methods above return FilesDataUnit.Entry as a result, you may also use public static File asFile(FilesDataUnit.Entry entry) method to convert returned entry to standard Java File object.

To create a new file in the working store of the DPU and further work with that file, you may use the approach as follows :

Code 4a

Entry newEntry = FilesHelper.createFile(output, "myFile");
File f = FilesHelper.asFile(newEntry);
//write to the file 

To add the existing file, you may follow the example below. The sample fragment below also shows how the Code 3 (adding an existing file to the output Files data unit). It may be simplified by using helpers.

Code 4

File file = ...
String fileName = ... 
FilesHelper.addFile(output, file, fileName); 

The advantage of the helpers is cleaner and easier to be used code: compare the Code 3 needed to add existing files to the output File data unit using only API classes:

Further, when the helper is not used, as a DPU developer you must be aware of virtualPath metadata, you must know that best practice is to set virtualPath = symbolicName.

Apart from the FilesHelper, there is also WritableSimpleFiles, which is not a data unit helper, but a DPU extension. Such an extension may be used to write files into the output data unit.

The advantage of such extension is this:

For details about the WritableSimpleFiles extension, please see Working With the Files Data Unit