Distributed data visualization BB – Design Document – Prometheus-X Components & Services

Distributed data visualization BB – Design Document

Distributed Data Visualisation is an on-device building block to ease up building end-user UIs for showing data in an informative way. It allows AI providers to display the results of their analytics, predictions and recommendations.

The building block is a reusable, portable and multi-platform supported HTML5 / JavaSript & D3.js based component that can be displayed by UI provider from any source.

This concept is not only about presenting data in an accessible format, but also about revolutionizing the way data interacts across different platforms, ensuring data privacy and improving user control. With the Prometheus-X project at the forefront, a ground-breaking approach to handling education and career development data will redefine industry standards.

Distributed data visualization is a sophisticated framework designed to seamlessly manage and display data across multiple platforms. It enables AI providers to process datasets from different data providers and gain valuable insights such as recommendations, analysis and predictions. What makes this approach special is the ability to easily integrate these results into different applications and user interfaces (UI), making it a versatile tool for educational and professional development.

Technical usage scenarios & Features

Features/main functionalities

User journey case 1:

Use case description, value and goal: The use case will take as basis Pino a product from Solideos used by Korean citizens to store their diploma and learning credentials. Use information from Korean users' Diploma and Transcript to match job opportunities in the EU (France). Skills data extracted from the documents will be used for service providers in the EU to recommend potential employment opportunities.

Use case functionalities:

User journey case 2:

Use case description, value and goal: LAB aims to prepare individuals for the job market by offering personalized training recommendations worldwide, leveraging skills data and partnerships with service providers like Headai, Inokufu, Rejustify, and Edunao. The goal is to equip learners with relevant skills for career success, enhancing employability and lifelong learning.

Use case functionalities:

Generic user journey:

  1. Matilda has data about herself in several LMS/LXP
  2. She's using a new learning provider and wants to receive personalised learning recommendations based on any/all her data
  3. She can share her data from service provider to another and the results are displayed inside one of those
  4. Service provider only has to integrate the data viz BB and PTX Connector into LMS/LXP

Technical usage scenarios

Distributed Data Visualisation is a on-device building block meant to ease up building end-user UIs for showing data in informative way with a relevant story/journey. Visualiser requires a host system that runs the UI and that host system must configure its components (consent, contract) that are needed to operate this building block.

Because of the nature of this building block being slightly different than others, the reader of this document cannot assume all server-side-building-blocks behaviors to exist in this BB.

Technical usage scenario 1:

User would gives her/his consent, it triggers "API consumption" by sending the data as a payload to this API call to the AI, and the data will be returned by the protocol. In this flow, the connector of the edtech side would send the result of the AI service back to the edtech so that it could be shown to the user.

Technical usage scenario 2:

User gives her/his consent to datasets, data provider(s) gives contracts to datasets that are not personal data. The Host System is the main responsible of triggering the required exchanges. Once Host System fetches the final results of the exchanges, it can load the data into the DDV, specifying custom parameters if required.

Tech usage scenario in short

With Distributed Data Visualisation, Data providers and Data consumers can easily bring the results from their services to end users without developing all the visualization components on their own.

For exapmle, a LMS company using Moodle, can integrate the Distributed Data Visualisation directly to Moodle and so focus on their LMS provider role, while this building block encapsulates all the logic behind the visualization of the results.

Requirements

Functional requirements:

Performance requirements:

Security requirements:

Dependability requirements:

DR01: In case of incomplete rules.json or data the DDV closes the process and returns error message.

Operational requirements:

Integrations

Direct Integrations with Other BBs

There is no direct integrations with other BBs.

Integration with the Data Space Connector

DDV does not have a direct integration with the Data Space Connector.

DDV Building Block provides mechanisms to ease up the process of data visualization. It is designed as an independent component that can be imported and used in multiple platforms. Each Service Consumers must perform their respective data exchanges. Once data is acquired by the service consumer, Distributed Data Visualizer can be used to build a story integrating data from multiple exchanges.

Relevant Standards

The standard data formats are applied in a way that the Building Block specific data model is convertible to any other model with standard tools. This could mean e.g. that 1) graph-data in JSON-LD format can be converted to a CSV table, where nodes and their values are only used. So this node-csv can be used directly in excel for drawing barcharts. 2) JSON-LD can be mapped in PowerBi and applied in any PowerBi report as such. The fundamental idea is to bring a universal skills data visualization building block and at the same time support any data visualization tool.

Data Format Standards

Mapping to DSSC Data Space Reference Architecture Model

The Data Interoperability pillar:

The Data Sovereignty and Trust pillar:

The Data Value Creation pillar:

See full DSSC

GDPR Mapping

AI Act Mapping

Input / Output Data

You can find detailed explanation about the formats that are supported by the building block in the section Data Formats.

Important Clarification: Third Parties and Members of the Data Space must ensure that results are being fetched exchanging data through the Data Space Connectors. Contracts and Consents ensures important quality aspects such as transparency, security and veracity during the data exchange.

Input data format:

Host system can configure the DDV defining a JSON Object called rules.json. This Object defines the data that will be displayed in the Host System, the order in which each visualization will be displayed, and custom visual properties.

Output data format:

DDV will render the different visualizations in a centralized component. The content will be rendered using enriched HTML and SVG.

Use case 1 (Job recommendations)

BB displays a list of Job Recommendations in an interactive and user friendly frame, through the JavaScript web component.

Data Formats / Standard Data Format for Job Recommendations

Use case 2 (Training recommendations)

BB displays a list of Training recommendations in an interactive and user friendly frame, through the JavaScript web component.

Data Formats / Standard Data Format for Course Recommendations

Architecture

---
title: Distributed Data Visualisation
---

classDiagram
    class Api1["API (optional services for building and working with data)"]
    class Api2["API (optional services for running analysis)"]
    Data --> Api1
    Api1 --|> Visualisation
    Api1 --|> Api2
    Api2 --|> Visualisation
    Data : +Consent
    Data : +Contract
    Data : +Snowflake, Data Excahnge()
    Data : +National data sources like Sisu, Peppi, etc.()
    Data : +Free text()
    Data : +Consent services like Visions, Vastuu, Koski()
    Api1 : +Data normalisers like Build Knowlegde Graph
    Api1 : +Data structure builders like Text To Knowledge Graph
    Api1 : +Personal Data Storages like Inokufu, Cozy Cloud, Digital Twin Storage, etc()
    Visualisation: +Hexagon
    Visualisation: +CourseList
    Visualisation: +JobList
    Visualisation: +VisualizationGroup (development)
    Visualisation: +Linechart (development)
    Api2 : +Recommendation services like Headai Compass or MindMatcher recommendation
    Api2 : +Gap analysys services like Headai Score

Dynamic Behaviour

The sequence diagram shows how the component communicates with other components.

Non personal data

For scenarios where personal data is not involved, the consent is not required and only contracts between the participants & the DDV.

---
title: Distributed Visualisation Sequence Diagram (Non Personal Data)
---

sequenceDiagram
    participant ddv as DDV
    participant host as Host System (UI Provider)
    participant dc as Data Consumer
    participant dccon as Data Consumer Connector
    participant con as Contract Service
    participant dpcon as Data Provider Connector
    participant dp as Data Provider
    
    host -) dccon: Get Contract Information (provide contract ID)
    con --) dc: Verified Contract


    loop For n amount of Services that are in the contract
    dc -) dccon: Trigger data exchange<br>BY USING CONTRACT
    dccon -) con: Verify Resources involved <br> in the exchange  (w/ contract & policies)
    con -) dccon : Contract validated
    dccon -) dpcon: Data request + contract
    dpcon -) dp: GET data
    dp --) dpcon: Data Resource <br> Fetched
    dpcon --) dccon: Data Exchange
    dccon --) dc: Data sent to payload <br> representation URL

    end

    host -) dc: Collect results <br> sent by the connector
    dc -) host: Results from each service
    host -) ddv: HTTP request or <br> direct usage of the JS Library
    Note over ddv: rules.json
    ddv -)  host: JavaScript Component

Personal Data

For scenarios where personal data is involved, the consent is required and must be verified on top of the existing contracts between the participants.

---
title: Distributed Visualisation Sequence Diagram (With Personal Data)
---

sequenceDiagram
    participant ddv as DDV
    participant host as Host System (UI Provider)
    participant dc as Data Consumer
    participant dccon as Data Consumer Connector
    participant con as Contract Service
    participant cons as Consent Service
    participant dpcon as Data Provider Connector
    participant dp as Data Provider
    
    host -) dccon: Get Personal information under Consent
    con --) dc: Verified Contract

    dc -) dccon: Verify Consent Validity
    dccon -) con: Verify Consent Validity
    con -) dccon : Consent validated

    host -) dccon: Get Contract Information (provide contract ID)
    con --) dc: Verified Contract

    loop For n amount of Services that are in the contract
    dc -) dccon: Trigger data exchange <br>  with CONTRACT & CONSENT
    dccon -) con: Verify Resources involved <br> in the exchange  (w/ contract & policies)
    con -) dccon : Contract validated
    dccon -) dpcon: Data request + contract
    dpcon -) dp: GET Personal data
    dp --) dpcon: Personal Data
    dpcon --) dccon: Data Exchange of Personal Data
    dccon --) dc: Results from Service sent to <br> payload representation URL
    end

    host -) dc: Collect results <br> sent by the connector
    dc -) host: Results from each service
    host -) ddv: HTTP request or <br> direct usage of the JS Library
    Note over ddv: rules.json
    ddv -)  host: JavaScript Component

Configuration and deployment settings

Configuration of the BB07 can be done in three places

The given parameters follow REST-API type of GET parameters, so only values that could be publicly seen or strongly encrypted values are allowed.

Error Scenarios

Incorrect data format

The configuration file calledrules.json follows a specific format. The absence of a mandatory field aborts the process of visualization and displays a warning message in the screen.

Invalid or Unaccesible Data URL

DDV consumer may define URLs that are not accesible without Authentication methods (E.j HTTP 404 / HTTP 403 Errors) or URLs with incorrect formatting. In these cases, DDV will show a warning message specifying that resources could not be fetched.

Size of the input data incompatible with available computational resources

Complexity to process and render Graphs (Digital Twins) will depend on the number of nodes, and the level of interconectivity between them (edges). Maps with more than 5000 nodes and huge level of interconnectivity may start being difficult to be processed in conventional computers.

Third Party Components & Licenses

Background Component: D3.js Available at D3.js Git https://github.com/d3/d3

D3.js is licensed under the ISC License. https://github.com/d3/d3/blob/main/LICENSE

In order to maximise cyber security we have isolated d3.js online-dependencies in current development version, which may cause small differences on how code behaviors when developed further. This decison may change during the development.

OpenAPI Specification

Full URL Example, development version

URL Parameters

All the parameters that can be passed via URL to the component are described in the document called Properties Documentation.

Test specification

Test plan

The objective of testing the "Distributed Data Visualization" function is two-fold:

  1. To verify that it accurately builds a visualization based on the given parameters, ensuring that the output is correct, reliable, and efficient over varying conditions.

  2. To confirm that the Distributed Data Visualization accurately represents the data provided by the JSON URLs. This involves verifying that all nodes, connections, and groups in the visualization correctly correspond to the data structure and content specified in the JSON file.

Scope of Functional Tests includes:

Resulting users can effectively interact with and derive insights from visualized data, reflecting accurate and meaningful information as intended by the data source.

Technical Description of Test Plan

This test plan outlines a comprehensive approach combining black box testing methodologies with automated testing routines to ensure functional accuracy, performance under various conditions, optimal response times, and resilience against anomalies in the system. The strategy leverages industry-standard tools and methodologies to achieve a high level of software quality and reliability.

Using this approach it is possible to achieve the following:

Methodologies

Black Box Testing: This approach focuses on testing software functionality without knowledge of the internal workings. Test cases will be derived from functional specifications to verify the correctness of output based on varied inputs. This method effectively simulates user interactions and scenarios.

Automated Testing Routines: Automating the execution of repetitive but essential test cases ensures comprehensive coverage, consistency in test execution, and efficient use of resources.

By combining black box testing with automated routines, this test plan will fully meet the requirements of the Distributed Data Visualization (“DDV”) block including but not restricted to its:

A comprehensive evaluation of the Distributed Data Visualization's functionality, performance, resilience, and operational readiness will enhance its robustness in managing anomalies. The use of these specific tools and methodologies enhances the effectiveness of testing efforts, leading to a robust, reliable, and high-performing application ready for production deployment.

Unit tests

You can find details about the unit tests in this link.

Component-level testing

The Distributed Data Visualizer does not integrate its funcionalities with the Dataspace connector or with other Building Blocks in a direct way. For this reason, Component-level tests will be empty.

UI tests

You can find details about the UI tests in this link.

Partnerships & Roles

Headai

Visions

EDUNAO

IMT

Usage in the dataspace

Screenshot 2024-05-22 at 15.53.01

Data Route

1 : Data from the Learning Management System (LMS) is tracked in the Learning Record Store (LRS)

2 : The LRS transmits data to the Learning Record Converter (LRC) in a format other than xAPI

3 : The LRC converts the data into xAPI format and sends it to the Prometheus-X Dataspace Connector (PDC)

4 : The PDC requests validation for transferring data to individual X, which includes their identity, catalogue, contract, and consent

5 : The data intermediary sends the terms of the contract, identity, catalogue, and consent of individual X

6 : The PDC of organization A sends a data set in xAPI format to the PDC of individual X

7 : The PDC of individual X transfers data in xAPI format to its Personal Learning Record Store (PLRS)

4 : The PDC requests validation to transfer data to organization B. This involves confirming the organization's identity, catalogue, contract, and consent

5 : The data intermediary sends the terms of the contract, identity, catalogue, and consent of organization B

8 : PDC of organization A sends a data set in xAPI format to the PDC of organization B

9 : The PDC of individual X requests validation to send data to organization B, which involves identity, catalogue, contract, and consent

10 : The data intermediary sends the terms of the contract, identity, catalogue, and consent of organization B

11 : The PDC of individual X sends a data set in xAPI format to the PDC of organization B

12 : The PDC sends data to the Data Value Chain Tracker (DVCT) in xAPI format and applies the commercial terms of the data-sharing contract

13 : The PDC sends data to the Data Veracity Assurance (DVA) in xAPI format, ensuring the accuracy of specific data exchanges in the database

14 : Organization B sends the data received from the data exchanges of the PDC to the Distributed Data Visualization (DDV) in xAPI format.

15 : The DDV visualizes the received traces, following the custom structure defined in the configuration file *rules.json