Pipelines Section

This section contains a short guide on how to work with the pipelines section in UnifiedViews.

The Pipelines section of the main menu is used to create, view and perform specific actions with pipelines.UnifiedViews Glossary

Note

Terms special to UnifiedViews and their meaning are explained in our UnifiedViews Glossary.UnifiedViews Glossary

The Pipelines section is represented as a table that includes:

  • A list of accessible pipelines which can be sorted according to the pipeline name.

    • A user with the role User can see pipelines created by him in this list and pipelines being public (see Visibility option here).

    • A user with the role Administrator can see all pipelines in the system created by all users. Pipelines that the user can see are called accessible pipelines.

  • Action buttons: are placed in each pipeline row and allow to perform specific actions on the specific pipeline. The actions being available:

  • Basic information about the pipeline, its name.

  • Basic information about pipeline execution. Last Run Time, Last Execution Time and Last Status represent the time and status of the last pipeline execution:

    • Last Run Time

    • Last Execution Time

    • Last Status

Pipelines can have the following states:

  • private: only the author or Administrator may use such pipeline),

  • public, read only: everybody can use such pipeline, but only the author or a user with the role Administrator can edit or delete such a pipeline.

  • public, read/write: everybody can use or edit such pipeline.

Privacy of a pipeline is set by the user when he creates or edits a pipeline. Information about the pipeline execution in the table correlates with the privacy settings. If a pipeline is private , its executions are also private and other users (except an Administrator) are not allowed to see them.UnifiedViews Glossary

If a pipeline is public (read only or read/write), its executions are also public and visible for all users in the system.

In case of a private pipeline, this execution was initiated by the pipeline owner or an Administrator. In case of a public pipeline, the execution could be initiated by anybody.

Above the table, there are the following buttons:

  • Create Pipeline: for creating a new pipeline in the system.

  • Import Pipeline: opens the dialogue to import a pipeline.

  • Reset Filers: for clearing filters of the table.

  • Reset Sort: to reset sorting you may have done using the column headers.

24578203.png

Create a UnifiedViews Pipeline

This section describes how to create a pipeline.

To create a pipeline in the system click Create Pipeline in the top right-hand corner of the Pipelines section (1).

24578325.png

After that the page for the Pipeline Detail specification will be opened:

24578326.png

The following topics in this section cover the steps necessary to create pipelines:

At the end of this section you find additional functionality that can be useful during pipeline creation or editing.

Placing DPU Instances on the Canvas

This section contains a short guide on how to create a pipeline in UnifiedViews by placing the individual DPUs on the canvas.

  1. Select a DPU template from the DPU Templates tree and via drag and drop move it onto the pipeline canvas.

  2. Add the rest of the DPUs you need for constructing the pipeline.

    • When you drop a DPU template onto the canvas, a DPU instance based on such a DPU template, is created.

    • DPU templates are logically separated into three groups: Extractors, Transformers, Loaders, which are differentiated by their color.

33751061.png

Note

If there is no suitable DPU template in the tree, you can create your own.

Hover your mouse over the DPU's edge, and click the Copy DPU icon.

33751063.png
Hide the DPU Template Tree to Open Up Space
  • You can hide the DPU Templates Tree panel if you need more space for editing the pipeline. To hide the panel, click the black arrow above the tree (1):

    33751060.png

Entering Properties for a UnifiedViews Pipeline

This section contains a short guide on how to enter properties for a UnifiedViews pipeline.

After you have created a pipeline, you can enter the properties for it as follows.

  • Enter the Name (mandatory), Description (optional) of a pipeline and set Visibility.

    • The name of the pipeline creator is filled in automatically.

      Note

      If the field Name is empty, on the right side of the field an exclamation mark with a notification 'Name must be filled!' is displayed. In this state the pipeline cannot be stored. You will recieve an error message in attempting to save the pipeline without a name.

26411499.png

Visibility: response for the pipeline permission settings. The pipeline has one of three modes:

  • Private (default)

  • Public (ReadOnly)

  • Public

Permissions are described in the following table.

Available Permissions for Pipelines

View/ View in PipelineList

Edit

Delete

Copy

Run

Schedule

On Execution Visible for

Private

ppl-owner

admin

ppl-owner

admin

ppl-owner

admin

ppl-owner

admin

ppl-owner

admin

ppl-owner

admin

ppl-owner

admin

Public (ReadOnly)

all

ppl-owner

admin

ppl-owner

admin

all

all

all

all

Public

all

all

ppl-owner

admin

all

all

all

all

The Pipeline properties panel can be hidden if you need more space for editing the pipeline. To hide the panel you should click the black arrow (1) above the panel.

26411498.png

Setting Configurations of DPU Instances

This section contains a short guide on how to configure DPU instances on the canvas.

You can configure DPU instances after you have placed one on the pipeline canvas.

  1. Open the DPU's Detail dialogue by double clicking the DPU instance.

    • Alternatively, click the Show detail icon on the DPU's tool bar.

    24578285.png
  2. The DPU Detail dialogue has a different layout for different DPUs. Name and Description are always present, but the rest of the dialogue depends on the particular DPU template.

    • Configure this dialogue according to your requirements.

  3. Enter settings in the DPU Detail dialogue and click Save.

You can also save a configuration as a new DPU template.

  • Click Save as New DPU template.

After that a new DPU template will be shown in the DPU Templates Tree on the third level.

Creating and Editing Data Flow Edges for DPUs in UnifiedViews

This section contains a short guide on how to create edges between DPUs on the canvas in UnifiedViews.

Create Data Flow Edges

You can create edges this way:

  1. Create data flow edges by clicking on the Create new edge icon on the DPU instance tool bar.

  2. Drag your mouse to the DPU you want the edge to connect to and click inside that DPU's instance.

24578293.gif

Note

After every modification of the graph edge, the application needs to verify if there is no cycle in the graph, and to deny the modification if there is one.

Find details on how to edit DPU edges in this topic: Edit DPU Edges

Edit DPU Edges

This section contains a short guide on how to edit existing DPU edges and creating mappings between output and input data units.

Every edge has a tool bar that appears after clicking on the edge.

  • Click the Edit edge mapping icon to open the Edge Detail dialogue.

24578304.png

The Edge Detail (1) dialogue allows you to map output data units to input data units.

  • The dialogue contains a list of Output data units of the source DPU (2), which is a source of the edge and a list of Input data units of the target DPU (target of the edge, 3).

    • The application will find out the provided output/input data units via annotations.

Steps to Follow for Mapping Source and Target Data Units
  1. To map Output data units of the source DPU to Input data units of the target DPU use the Edge Detail dialogue.

  2. Select one or more output data units.

  3. Select one input data unit.

  4. Click Map.

    • The dialogue must check that at least one output data unit and exactly one input data unit have been selected.

  5. As a result, a new mapping appears in the Available mappings area, one row represents one mapping.

  6. You can delete a created mapping and also clear the selection of the data units to be mapped.

  7. Click Save to confirm your changes or Cancel to abort.

Example

For example, the Edge Detail dialogue contains the following parameters:

  • Output data units of the source DPU:

    • rdfOutput

  • Input data units of the target DPU:

    • config

    • input

On the left, select rdfOutput, on the right select config. Click Map to create the mapping.

Click the Save button and on the pipeline canvas the mapping will be visible on the edge:

Saving the Details of a UnifiedViews Pipeline

This section is about how to save the details of a DPU pipeline you have created. You have several options to save a pipeline (1). Click either:

  • Save: this only saves the pipeline.

  • Save & Close: leaves also the Pipeline detail section after saving.

  • Close: leave the Pipeline detail section without saving changes

UV_Save_Pipeline.png

Additonal Options on the UnifiedViews Pipeline Canvas

This section contains a short guide on the options on the canvas, deleting DPU instances and edges, debugging, copying of DPUs and pipelines, layout formatting, alignment and validation.

The following topics in this section provide details on additional options on a pipeline canvas:

Alignment and Distribution Options

This section contains a short guide on alignment and distribution options for DPUs.

The pipeline canvas also provides buttons to zoom the canvas and cancel the last operation on the canvas. These buttons are placed in the top right-hand corner (1):

Note

Icons you can use to align multiple DPUs, are greyed out until you select two or more DPUs to align.

Available Icons
24578353.png

Arrows are for aligning DPU instances.

24578354.png
24578355.png

Distribute icons are for creating the same gaps between selected DPU instances (for 3 or more DPU instances) in a given direction.

24578356.png
24578357.png

Use Zoom icons to enlarge or minimize the canvas.

24578358.png

The Undo icon lets you undo the last action you performed with any other icon.

Debugging, Copying and Layout Formatting of DPUs

This section contains a short guide on how to debug, copy, and format the layout of DPUs.

The icons described here are also available in the icon bar of each DPU.

The following options are available:

24578346.png

Debug to this DPU: run the pipeline in debug mode.

24578347.png

Copy DPU: place a copy of this DPU instance on the pipeline canvas.

DPU layout formatting: only selected DPU instances are formatted.

DPU Layout Formatting Steps
  1. Select the first DPU instance by clicking on it.

  2. To select more DPU instances, click the DPU layout formatting icon on the toolbar.

  3. Click on other DPU instances, or hold the CTRL key and click on additional DPU instances.

    • The formatting action bar appears in the top right-hand corner.

Delete DPU Instances and Edges

This section contains a short guide on how to delete DPU instances and edges.

You can delete a DPU instance from a pipeline canvas, if it was placed incorrectly.

To do this, click on the DPU instance you would like to remove and select the Remove DPU icon (1).

You can delete an edge if it was not correctly set. To do this click on the edge and click the Remove the edge icon (2):

24578363.png

Copy a Pipeline

This section contains a short guide on how to copy a pipeline.

Pipeline Copy

To copy the current pipeline click Copy (1).

24578372.png

After that you will automatically work with details of a pipeline copy (2):

24578373.png

Validate DPU Connections in a Pipeline

This section contains a short guide on how to validate connections, input, outputs and mandatory check boxes of a DPU pipeline.

Follow these steps to validate DPU connections in the pipeline, control all mandatory inputs/outputs and mandatory check boxes in the DPU configurations:

  1. Click Validate at the bottom of the pipeline canvas.

  2. A validation message will be shown.

  3. In addition, the respective DPU will display a Warning icon to depict the place of issue.

24578375.png

Defining Pipeline Conflicts

This section contains a short guide on how to set pipeline conflicts.

This function is useful to include or exclude concurrent editing or executing pipelines by multiple users that are active in the system.

For each pipeline you can specify a list of conflicting pipelines, including the current pipeline itself.

Example

A user is editing pipeline A and defines pipelines B and C in the conflict list.

  • In that case if the user wants to start pipeline A, it will wait (in running state) till the end of the executions of the pipelines B and C. Only then it will be executed.

  • If a user sets pipeline A to conflict with pipeline A it means that only one instance of the pipeline A is able to run at one given moment.

  • In case of several instances being launched at the same time, they will be executed one after another.

Setting Pipelines Conflicts
  1. Click Conflicts at the bottom of the canvas.

  2. The Pipeline Conflicts dialogue opens.

  3. In the column All Pipelines select one or more pipelines you want to define conflicts with the current one for. Multi-select is possible.

  4. Use the arrows to add or remove the selected pipelines to or from the column on the right: Pipelines conflicting with the current pipeline

    • In the column on the right the pipelines you selected, will be listed.

  5. Click OK to confirm your changes.

    • Click Cancel to abort.

    • Click Clear Conflicts to remove all pipelines from the column on the right.

Pipeline Actions

This section contains the following topics:

All actions with pipelines are available from the PIPELINES section of the main menu bar.

Run a UnifiedViews Pipeline

This section contains a short guide on how to start a pipeline so it is executed in UnifiedViews.

  1. Click PIPELINES in the main menu.

  2. Click Run in the corresponding row in the pipelines table.

24578239.png
Check Backend Status

Note

In order to be able to run the pipeline, the backend must be online.

You can determine the status of the backend using the icon in the top right-hand corner of the window.

If backend is offline you see the red Disconnected icon (1):

24578243.png

If you try to run a pipeline, when the backend is offline, the dialogue Pipeline Execution will appear.

  • You can click Schedule (4) in order to run the pipeline on a schedule, after the backend is online again.

  • Click Cancel to cancel the execution.

24578248.png

If you try to run the pipeline when the backend is online and the pipeline execution has been started successfully, the system will show a message 'Pipeline execution started..' (5):

24578246.png

After the pipeline execution finishes the information about run time, execution time and execution status will be shown in the pipelines table (6):

24578247.png

More detailed information about the pipeline execution is available in the Execution Monitor section of the main menu bar.

Read more information about how to work with Execution Monitor section here.

Debug a UnifiedViews Pipeline

This section contains a short guide on how to debug a UnifiedViews pipeline.

Debugging of the pipeline means that the pipeline is executed, but you can see the data which flows through the pipeline, you can view that in the browser.

Two ways exist to debug a pipeline.

Debugging From the Pipelines Table
  1. Click PIPELINES.

  2. To run the pipeline in debug mode, click the Debug icon in the corresponding row in the pipelines table.

Note

Debugging of the pipeline as well as running is available only, if the backend is online.

After starting the pipeline execution in debug mode, the system redirects you to the Execution Monitor section where details of the pipeline execution will be shown:

Read more information how to work with the Execution Monitor section here.

It is also possible to debug the pipeline from the pipeline canvas.

  • Click Edit in the pipelines table, then click Debug to this DPU icon on the DPU instance tool bar.

26411506.png

After that the Debug Window with the debug information will appear.

26411507.png

For detailed information about the Debug Window see Execution Monitor Section.

Copy a UnifiedViews Pipeline

This section contains a short guide on how to copy a UnifiedViews pipeline.

  1. Click PIPELINES in the main menu.

  2. To copy the pipeline press the Copy button in the corresponding row in the pipelines table:

26411509.png

After that, the copy of an original pipeline will be created and placed in the pipelines table. The name of the copied pipeline is derived from the name of the original pipeline.

Note

The new pipeline created by Copy will contain all DPUs that the original pipeline contains, with the same settings and data flow edges.

Edit a UnifiedViews Pipeline

This section contains a short guide on how to

  1. Click PIPELINES in the main menu.

  2. To edit the pipeline click the Edit icon in the corresponding row in the pipelines table.

  3. The pipeline detail will be opened. The dialogue is the same as in Create a UnifiedViews Pipeline.

You can modify the pipeline only in the following cases, regarding permissions:

  • you are the owner of the pipeline,

  • you have the Administrator user role,

  • the pipeline is in public,read/write mode.

Otherwise you can only view the pipeline.

Note

If you have permissions to edit the pipeline, the pipeline detail will be opened in a Develop mode.

Without permissions to modify the pipeline it will open in a Standard mode. Also a notification about read-only mode will be shown.

With permissions to modify the pipeline you can:

  • change name and description of the pipeline,

  • add/delete DPU instances on pipeline canvas,

  • change DPU instance configurations,

  • change the data flow edges.

Several users can edit the same pipeline at the same time. In this case the warning that another user is editing the same pipeline will be shown. In that case, you can create a copy of a pipeline and continue with modification in that copy. Additionally you can continue a pipeline modification in parallel with another user.

Read more information how to work with the pipeline detail here.

Delete a UnifiedViews Pipeline

This section contains a short guide on how to delete a UnifiedViews pipeline.

To delete the pipeline, click Delete in the corresponding row in the pipelines table.

26411512.png

After that, a confirmation dialogue will be shown. Click Confirm to delete the pipeline from the system.