Release Notes - PoolParty 7.0

This page contains the release notes for PoolParty release 7.0. In the release notes you can find major developments, improvements and changes made in the respective release.

Details on how to upgrade from your present version find here: PoolParty Upgrades

The release notes contain three chapters, highlights, improvements and fixes:

Highlights

PoolParty GraphEditor

The PoolParty GraphEditor is a completely new addition to PoolParty, a state-of-the-art editor for you to edit, filter and handle any RDF data with ease. Previously you could only create data based on SKOS, and extend it by using ontologies and custom schemes. The PoolParty GraphEditor allows now to create and edit data based on any ontology and custom scheme and the data can be located in all graph databases connected to PoolParty via the Semantic Middleware Configurator supporting the GraphEditor functionality.

As of PoolParty 7.0 the supported graph databases for the GraphEditor functionality are Stardog (5.3.x) and rdf4j.

You can define multiple editors per project. The project and ontology as well as custom scheme permissions also apply to the GraphEditors. That way a very fine-grained access can be granted in different ways, read-only for example to prevent data loss or damage.

Creating new GraphEditors is easy with the editor assistant where you specify basic metadata, URI generation, custom schemes and the graph database to be used. Select from a list which graphs in the selected graph database you want to make editable to store/change data there and which ones shall be searchable only. You can set or change names for graphs as well.

Once you created a GraphEditor using the assistant, a new node will show up below the GraphEditor node and the Details View will show the editor's metadata.

Now you can start and search as well as filter your data extensively in the Search tab, using all classes relations and attributes of the custom schemes you selected.

You can select individual results in the filtered result sets and open them in the Details View. Here you can edit all data of one resource. In addition you can select multiple results and perform batch processing actions, such as adding or removing data or deleting the selected resources.

The GraphEditor offers a history functionality and you have a triples tab for each resource.

Additionally, a host of new API methods for interaction with RDF graphs has been implemented. They allow you to request and set settings as well as details of the RDF graphs stored in your local or remote PoolParty instances and graph databases.

PoolParty Notifications

PoolParty offers an entirely new feature to receive push notifications on changes. You can define multiple notification configurations per project. 

The Notifications feature allows you to:

  • Define Webhook notifications that are pushed to an API defined in the Semantic Middleware Configurator.
  • Configure Logging notifications that are written to the PoolParty log files.

Usage of notifications in PoolParty now includes:

  • You can access the feature in the PoolParty interface using a new icon.
  • Below the Notifications node you see the Logging and Webhook node. Double click one of those to create a new notification configuration.
  • A new node is created for the notification configuration. In the Details View you can review their  Metadata and Status, and more, and a list of Processed as well as Queued ones.

Onboarding Tool Integration

We aim at making the PoolParty experience ever more easy and user friendly. To make sure you find your way about quickly as a first-time user, we integrated a completely new onboarding tool into PoolParty, that you will see on logging in for the first time. It will guide you through all the basic steps that are necessary for creating your first project or interact with the core functionality of PoolParty. You can import and export the content for reusing on other servers, stop the tool or restart it anytime with its separate button.

Depending on user interaction the content will be adapted and made to fit the actions on screen.

PoolParty UnifiedViews

UnifiedViews has been redesigned completely to fit the overall PoolParty interface and workflow even better.

All SWC products now are the same in basic workflow as well as design elements, making orientation even more user friendly and fast.

In addition the configuration is now stored in the built in triple store. Thus no additional relational data base is needed providing you with a more streamlined and smooth installation process.

Improvements - New Features

General Improvements

Browser Support Changed

As of version 7 PoolParty will not support Internet Explorer anymore. Together with Chrome and Firefox now MS Edge will be  supported instead.

OWASP Risks Mitigation Strategies & Password Security Improvements

As has been defined by the Open Web Application Security Project (OWASP) on their website, web applications should comply to certain standards to make them secure . In this version we hardened PoolParty further, using the following measures:

  • Password Change Policy (mandatory)
    New users and legacy users will have to change passwords on their first login. At that point a user after logging in cannot execute any functions, until he or she has changed their password. That way a password is less apt to be abused.   

  • Enforce Password Change Periodically (optional)
    The periodical password change policy is meant to prevent an attacker to guess or otherwise find out a password. The change can be set to a value of your choice in days, default is 90 for as many days.

  • Password History Enforcement (optional)
    This makes sure that users cannot reuse their passwords. Changing this value in the poolparty.properties file sets the number of passwords for a user saved in a history. In this example, the user cannot use the last 10 passwords

  • User Account Locking (optional)
    Allows to define a number of allowed login attempts before a user is lock-out.

  • ReCaptcha Support (optional)
    This makes sure that users have to check the "I am not a robot" checkbox before they can login. Even users with a locked account can then log into PoolParty (the lock will be removed after successful login). To register a new instance, follow this link: https://www.google.com/recaptcha/admin#list

All those settings are configured via the poolparty.properties file. Optional settings have to be enabled explicitly so they become effective.

Taxonomy & Ontology Management Improvements

Remote Storage Architecture

PoolParty version 7 offers the ability to manage the project data and system wide data in all graph databases connected to PoolParty set up in the Semantic Middleware Configurator which support the remote storage functionality.

As of PoolParty 7.0 the supported graph databases are Stardog and rdf4j.

You can select the remote store in the Advanced tab of the create project dialogue:

In addition in the project's Details View the selected graph database (1) as well as the PoolParty version of the project data (2) are being displayed:

With this new feature all data created in PoolParty can be stored in a separate graph database offering the ability to completely separate the application and the data layer and by that provide more support for high availability and scalability scenarios.

Ontology Management Redesigned

The Ontology Management has seen major improvements in this release. The interface has changed and ontologies as well as custom schemes are being presented in a hierarchical tree now, which adds to the consistency in PoolParty. 

Ontologies and custom schemes are grouped into two separate nodes. Below those nodes each ontology or custom scheme has its own node. Expanding them using the Arrow icon beside them lets you look at the child nodes classes, relations and attributes. In the Details View on the ontology or custom scheme level you can see the metadata of the respective ontology or scheme.

Expanding one of those nodes will open the tree of respective classes relations and attributes. Each class, relation or attribute has its own node and Details View.

When you create a new ontology or edit an existing one, a double-click on the respective node will open the Edit Ontology/ New Ontology dialog.

  • The Languages tab allows you to specify the languages for the ontology or custom scheme. You can define labels in all languages defined. That way when you use the ontology to setup e.g. a form to fill in data the forme can be made multilingual.

The User Groups tab allows you to assign user groups to your ontology or custom scheme. Based on the user group assignment you can control which users can edit and change the ontology or custom scheme as well as the users that can create data based on that ontology in the Thesaurus Management or the GraphEditor.

User Management Refactored

The User Administration has seen major improvements in this release. The interface users, roles and groups are displayed in a hierarchical tree now, which adds to the consistency in PoolParty. 

In addition, the User Management has been redesigned so its consistency has been streamlined once more. When you first access the currently logged in user's node, you will see the hierarchy tree displaying all users, roles and permissions this user can access, in this case a superadmin. On the right find the Details View of the user, with the User DetailsGranted AuthorizationsLanguage Settings and Display Settings tabs. In the Granted Authorizations tab you find all roles the user you are looking at has been assigned. 

  • You can change user roles individually on project level allowing a user to be Admin in one project and Read-Only in another.
  • Two additional roles, APIAdmin and APIUser have been added allowing you to define a read-only and a read/write user that only can access PoolParty via APIs.
  • Roles and corresponding permissions are displayed separately. 

neo4j Integration

As of PoolParty 7 thesauri and ontologies created in PoolParty can be exported to neo4j. You can configure connected neo4j stores using the Semantic Middleware Configurator.

DBpedia Linking Service

The Linking Service for DBpedia has seen considerable improvements in this release: 

The SPARQL based DBpedia linking has been replaced to make it more effective. It is now Solr based and uses the spreading activation technique.

  • A button for creating links is now available that lets you easily use the DBpedia linking service.
  • The advantages of the improved functionality are the following:
    • Redirects are reduced to a minimum.
    • The search for linkable data is more effective due to less sensitivity to precise wording.
    • The search now includes labels and abstracts of redirects, as well as titles.
    • The context of parents, children and siblings of data is taken into account as well.
    • A batch linking DPU is available in UnifiedViews(?).

HTTPS for PoolParty's Linked Data Frontend

The Linked Data Frontend in PoolParty will now resolve HTTP as well as HTTPS requests, regardless of the server's settings. This makes it easier and more comfortable to use because there's no additional administration necessary.

Corpus Management & Entity Extraction Improvements

Concept Scoring in Corpus Analysis

As of this release the concept scoring for corpus analysis has been implemented in PoolParty. This means you will have an additional measure to gauge the validity of concepts found in the corpus. The interface in the Extracted Concepts tab now offers the additional column Relevance (1). It contains the scores of concepts found inside the corpus during the Corpus Analysis. The relevance of concepts here is calculated similarly to the terms in the Extracted Terms list.  

You can use these scores as information about the validity of the extracted concepts in the Extracted Concepts tab.

NLP Custom Scheme & Statistics Custom Scheme

The Natural Language Processing ontology (NLP) now included in PoolParty enables additional relations on graph level between concepts, the corpus and the score as well as relevance that has been calculated for them. That way you can use the information later for example for tagging. A remote store configured in the Semantic Middleware Configurator can be used to export the data and then you would use SPARQL queries to check results and reuse the metadata now available.

Exact Matching in Entity Extraction

Via the Add to ExactMatch button you can now enable exact matching for entity extraction for concepts. When enabled, extraction is only done if one of the labels of the concept exactly matches in the text. For the example below the text "SEMANTiCS" or "i-Semantics" has to match exactly in the text so the extraction is triggered.

NLP based Named-entity Recognition

Machine learning based Named-entity Recognition (NER) capabilities have been added to the PoolParty Extractor. As of PoolParty 7.0 an organization, people and location extraction has been added that works based on a trained model using a maximum entropy based classification wich in turn is NLP-based to extract organisations, persons and locations. Below a respective API call is shown:

http(s)://<server>/extractor/api/extract?text=Andreas Blumauer is CEO of SWC&nerParamters[0].method=RULE_BASED&nerParameters[0].type=Person

Result:

namedEntities [
	{
		textValues: "Andreas Blumauer",
		frequency: 1,
		type: "person",
]

Improved Annotate & Annotate and Store Call

The Annotate & Annotate and Store services have been greatly enhanced. It allows now to issue directly in the call a SPARQL query to format the extraction result to a specific format needed as output.

In addition the PoolParty Extractor's RDF data model has been extended to cover all the functionality present in the extract call.

Rule-based Entity Extraction

The improvements in the APIs include the possibility to define extraction rules. Extraction can now be restricted to concepts that have certain attributes or relations. 

UnifiedViews Improvements

New Poolparty DPUs

The following new DPUs have been implemented:

  • DBpedia batch linking

  • RML mapping

  • R2RML mapping

  • JSON to XML transformation

  • XML to JSON transformation

  • NLP extraction

  • GraphSearch content preparator

  • Ontology based tabular data mapping

  • Data fusion

  • Multithreaded concept extraction

API Additions and Improvements

Thesaurus API

New API Calls Description
Notifications
GET /notifications/{project}/setting Returns a notification setting configured for the given project.
GET /notifications/{project}/settings Returns a list of notification settings configured for the given project.
POST /notifications/{project}/settings/create/webhook Creates a new webhook configuration
POST /notifications/{project}/settings/delete/webhook Deletes an existing webhook configuration.
GET /remote/systems/spark Call used by PoolParty Extractor to get central Spark configuration, if PoolParty Thesaurus Server is running and Spark is configured.

PoolParty GraphEditor API Calls


POST /editor/{project}/create Create and configure a new GraphEditor.
GET /editor/{project}/details/{editorId} Get the details of a specific GraphEditor.
POST /editor/{project}/delete/{editorId} Delete the GraphEditor.
POST /editor/{project}/search Filtered search for a GraphEditor.
POST /editor/{project}/createResource

Create a new resource in a GraphEditor.

POST /editor/{project}/addRelation Add a new relation property to a resource.
POST /editor/{project}/addAttribute

Add a new attribute property to a resource.

POST /editor/{project}/deleteAttribute Delete an attribute property from a resource.
POST /editor/{project}/deleteRelation

Delete a relation from a resource.

POST /editor/{project}/updateRelation 

Update the relation of a resource.

POST /editor/{project}/updateAttribute 

Update the attribute of a resource.

POST /editor/{project}/bulkAdd Add attributes or relations to one or more resources in a batch.
POST /editor/{project}/bulkRemove Remove attributes or relations to one or more resources in a batch.
POST /editor/{project}/bulkDeleteAll Delete resources in a bulk operation.

Added Parameters for Existing API Methods


POST /projects/{project}/import Response object added. Previously there was no response (only status code 200) when the import was successful. Instead of an empty response it now always returns a JSON object.
GET | POST /sparql/{project} Changed soving of the project. It is now possible to give either the URI supplement or the ID of the project for the {project} parameter in the request. Previously it would only resolve URI supplements.
GET /projects/{project}/pparExport
 [?exportModules=module1,module2,...]
The new parameter exportModules allows to specify what parts of a project should be exported in the archive. Invalid modules are ignored; if empty, all modules (for example, the full project) will be exported.

Extractor API

Added Parameters for Existing API Methods


[GET | POST] /api/suggest A "conceptSchemeFilters" parameter has been added to filter Results per Concept Scheme.