Introduction
We live in a connected world today and also the data in your IBM Notes/Domino platform is more connected that you perhaps realise:
I work work for company X at department Y and besides my assignment as “ICS product specialist” I perform in different roles, sometimes as developer, sometimes as a solution architect, sometimes as project manager etc. I own a range of devices to do my work, bought in by different suppliers under different conditions/contracts. Now a external project-leader (they claim they are the best) has became responsible for a new project (Z since it is the final project the company is ever going to run) and to make a huge impact the project-leader want to equip his team (which I have become member of) with the latest and greatest of devices to make our work most convenient and the impact of the project delivery as huge as possible. BUT of course he is unaware about the devices the team owns already and the lifetime of the contracts. He also does not want to make top management angry so he suggest to provide this team with devices just 1 configuration level below top management. So how does the project-leader require the information he need to know (a list of end of lifetime for the current devices the project team owns compared with the list of current devices top management owns) ?
All objects in the story above (organisation, departments, employees, devices, contracts, suppliers) are represented in Notes documents. By connecting them in a Graph DB new information can be gathered!
Graph
Lately I have been exploring Graph data modelling more closely. Mostly because I see many opportunities to connect/model unstructured data in fluid ways. Faster than relational datasources sometimes can.
For now I have been mainly looking at Neo4J because most Graph related education is focussed around this product. But for IBM customers you have IBM Graph (cloud only?) and since recent IBM Notes via the OpenNTF Domino API.
Graph need NoSQL
Graph needs NoSQL databases since NoSQL is capable of describing the variety of objects that are around in enterprise data. As it turns out Lotus Notes is around as one of the first NoSQL databases!
ODA & Tinkerpop
The OpenNTF Domino API has included Tinkerpop and an implementation to store content in a graph database structure.
This is great because as Domino developer you are concerned that your data is stored in an NSF. Probably you want to avoid the hassle of moving your data to another technology first and focus on developing applications directly.
Domino Explorer
Recently Oliver Busse has released Domino Explorer. It scans the Domino Directory and the Catalog application to show the capabilities of Graph in an XPages application. Getting the Notes data into a Graph data model is done via the OpenNTF Domino API.
In the next part of this post I will focus on how this is was implemented…
Document: Exploring the Domino Explorer
This document describes the setup of the Domino Explorer (DE) application (app) available on OpenNTF (link). The DE app demonstrates the Graph db capabilities via the OpenNTF Domino API (ODA) (link).
By exploring the architecture of the app better understanding of the Graph technology is aimed to achieve and how to implement it in XPages applications.
Presumptions
It is presumed that the reader of this document has advanced level of understanding of XPages technology, intermediate level of Java programming language, medium level of understanding of IBM Notes data and low level of the graph data model.
If not turn around or educate yourself on these topics 😉
Graph glossary
The view design element “graph” group it’s containing documents into the two basic units in the Graph Data Model (Graph/GDM): Edges and Vertices.
Vertices
Vertices (singular:vertex) are the objects in a graph.
Edges
Each edge has two (or in hypergraphs, more) vertices to which it is attached, called its endpoints. Edges may be directed or undirected; undirected edges are also called lines and directed edges are also called arcs or arrows.
IBM Notes data
When we translate this to IBM Notes data, vertices are mostly objects that contain values or properties (edge). E.g. a user is member of a group. An order has a date-stamp etcetera. With graph you can make queries to find different types of objects connected via shared properties e.g. A car-leasing company may have a fleet of rental-cars which may have present or past drivers.
General overview
Design elements in scope of the dissection
To have some sort of start we will start easy and give an overview of the design elements involved:
| Design element |
Purpose |
Scope |
| Views |
Used to store documents that are used in the graph data model.
Documents are of type Edge or Vertex(Vertices). |
X |
| XPages |
Used to initiate business logic and present the result to the user in a web interface. |
X |
| Custom controls |
Used to break parts of XPages into reusable pieces. |
X |
| Script libraries |
Client-side javascript libraries used to interact with the web presentation. |
X |
| Java |
Java classes that contain the business logic. |
X |
| Images |
Used to enhance the web presentation. |
|
| Style sheets |
CSS definitions used to enhance the web presentation. |
|
| Themes |
Use to set rules for the application regarding style, behaviour, resources |
|
| faces-config.xml |
Configuration file for the application, mostly used for registering managed beans. |
|
For the rest of this document we will only focus on design elements that are marked with X under the Design elements will most likely be discussed in the logical order how the graph data model is implemented in the DE app.
Managed Beans
Managed Bean (link) is a regular Java Bean class registered by the XPages framework. Important beans registrered in the DE app for Graph are:
| Bean name |
Class |
| config |
net.notesx.domex.controller.ConfigurationController |
| scanner |
net.notesx.domex.controller.ScannerController |
|
|
Config
The ConfigurationController class controls the configuration for the GDM. It sets which databases are in scope for the GDM (Names.nsf, catalog.nsf).
It uses the:
| Java class |
Usage |
| net.notesx.domex.graph.GraphHelper |
Setup of the Graph |
| net.notesx.domex.graph.Configuration |
Configuration of the Graph |
DFramedTransactionalGraph<DGraph> configGraph = GraphHelper.getGraph();
Configuration configuration = configGraph.addVertex(“configuration”, Configuration.class);
configuration.setApplicationName(applicationName);
configuration.setNamesPath(namesPath);
configuration.setCatalogPath(catalogPath);
configGraph.commit();
In the code above the Graph db is loaded and the configuration added as a Vertex.
Configuration class
The configuration class contains the following properties:
- $$Key (??? identifier for current configuration)
- ApplicationName (name for the application, display only)
- namesPath (location of the names.nsf)
- catalogPath (location of the catalog.nsf)
GraphHelper class
The GraphHelper class sets up the Graph in the following order:
- Setup an element store for configuration
- Add the types (Vertices) to the element store
- Create a configuration for the Graph
- Add the element store to configuration and set the default element store (= filepath of current database)
- Setup the Graph by using the configuration
- Return it
// set element store for configuration
DElementStore configStore = new DElementStore();
Database currentDatabase = Factory.getSession(SessionType.CURRENT).getCurrentDatabase();
configStore.setStoreKey(currentDatabase.getFilePath());
// setup the type
configStore.addType(Configuration.class);
configStore.addType(DXDatabase.class);
// create a graph config
DConfiguration config = new DConfiguration();
DGraph graph = new DGraph(config);
// add the config element store
config.addElementStore(configStore);
config.setDefaultElementStore(configStore.getStoreKey());
// setup the graph
DFramedGraphFactory factory = new DFramedGraphFactory(config);
DFramedTransactionalGraph<DGraph> fg = (DFramedTransactionalGraph<DGraph>) factory.create(graph);
// return the graph
return fg;
Scanner
This class adds the database, user and group objects to the Graph according the specified class for each object.
Important methods:
- scanPeopleAndGroups
- scanDatabases
- scanAcl
- scanMember
scanDatabases
- Get the path of the catalog db via the ConfigurationController class
- Initiate the Graph datamodel via the GraphHelper class
- Put all documents from View $ReplicaID into a document collection
- For each document add a vertex to the graph using the DXDatabase class
- Set properties for the vertex
- Commit it to the Graph
- Run the scanAcl method for the databaseVertex object
scanPeopleAndGroups
- Get the name of the names db via the ConfigurationController class
- Initiate the Graph datamodel via the GraphHelper class
- Put all documents from View $VIMPeople into a document collection
- For each document add a vertex to the graph using the DXUser class
- Set properties for the vertex
- Commit it to the Graph
- Put all documents from View $VIMGroups into a document collection
- For each document add a vertex to the graph using the DXGroup class
- Set properties for the vertex
- Commit it to the Graph
- Run the scanMember class for the groupVertex object
scanAcl
- For the given database (vertex object) get the database object
- Grab the ACL
- For each ACL entry add a vertex using the DXACLEntry class
- Set properties for the vertex
- Add the ACL entry vertex as an Edge to the DB vertex (the label “hasAcl” will be used)
- Commit it to the Graph
scanMember
- For the given group (DXGroup class object):
- For each member in groupmembers list:
- Check if it contains a “/” ( This is a user)
- Create a user object with the DXUser class from the Graph
- If the user object is not null (in the Graph) add it to the group
- If not (this is a group)
- Create a group object with the DXGroup class from the Graph
- If the group object is not null add it to the group
- Set a property for the vertex
- Commit it to the Graph
Other classes
DE contains more classes stored under several packages:
Common package
Classes under the common package are used for setting properties for application e.g. navigation, page behaviour etc. They are out of scope for this document.
Domex \ controller package
Classes under the domex \ controller package are ConfigurationController and ScannerController and available as described earlier as managed beans.
Domex \ graph
Classes under the domex \ graph package are directly related to the Graph data model and describe the Graph data objects (vertexes):
- Configuration
- DXDatabase
- DXACLEntry
- DXGroup
- DXUser
Each class describes the type of the object, the properties the object has and how to get & set them, the edges the object has and in which direction they go.
| Annotation |
Description |
Example |
| @TypeValue |
Interface annotation for marking the Element property-value that may contain type information. |
@TypeValue(“DXACLEntry”) |
| @Property |
Property annotations are for getter and setters to manipulate the property value of an Element. |
@Property(“$$Key”)
public String getKey();
@Property(“name”)
public void setName(String s); |
| @AdjacencyUnique |
Adjacency annotate getters and adders to represent a Vertex incident to an Edge. AdjacencyUnique extends this idea to ensure that there is only one instance of an Edge with a specific label between any two specific Vertices. This allows the user of the graph to call an “add” method and return the existing adjacency if it already exists. |
@AdjacencyUnique(label = “hasAcl”, direction = Direction.OUT)
public Iterable<DXDatabase> getDatabases(); |
Below is as example of the definition of the DXUser.java class:
package net.notesx.domex.graph;
import org.openntf.domino.graph2.annotations.AdjacencyUnique;
import org.openntf.domino.graph2.builtin.DVertexFrame;
import com.tinkerpop.blueprints.Direction;
import com.tinkerpop.frames.Property;
import com.tinkerpop.frames.modules.typedgraph.TypeValue;
@TypeValue(“DXUser”)
public interface DXUser extends DVertexFrame {
@Property(“$$Key”)
public String getKey();
@Property(“username”)
public String getUserName();
@Property(“username”)
public void setUserName(String s);
@AdjacencyUnique(label=”member”, direction=Direction.OUT)
public Iterable<DXGroup> getMembershipInGroups();
}
The GraphHelper class is described earlier in this document and is used to setup the Graph.
Domex \ Rest package
The classes under the domex \ rest package create collections of json objects for the objects in the Graph data model. Some classes take in parameters, some don’t. Each class defines what type of graph object is in scope.
The way these classes are initiated is as follow:
- On XPages rest service(s) of type customRestService are defined. The servicebean property directs to which class the service is bound to.
- The XPage contains JavaScript libraries.
- In a library can be defined which rest service should be called when the document is ready e.g. to build a collection of database objects.
- In a library can be defined which rest service should be called when a link is being clicked e.g. a row in a data table control and if a parameter should be send (e.g. the replica Id of the database). Also is defined what should be done with the data returned from the rest service.
XPages
The design elements described above come together in the XPages design elements. We will demonstrate by describing an XPage.
Alldbs.xsp
Elements:
| Type |
Name |
Purpose |
| Custom Control |
_layoutBS3 |
Reusable layout for application. |
| JavaScript library |
alldbs.js |
Calling Rest Services.
On document ready: data
On click table row: acl |
| Rest service control |
Pathinfo: data |
Bound to Java class net.notesx.domex.rest.AllDatabasesService |
| Rest service control |
Pathinfo: acl |
Bound to Java class net.notesx.domex.rest.AclService |
| HTML table |
datatable |
Placeholder for result from rest service under pathinfo data |
| Div |
acl |
Placeholder for result from rest service under pathinfo |
XPage logic explained
On document ready:
- When the document is ready an AJAX call is made to the rest service under pathinfo data. This initiates Java class net.notesx.domex.rest.AllDatabasesService.
- The class gets the Graph data model and collects all objects/vertexes of the type specified in the DXDatabase class.
- For each vertex a JsonJavaObject is created which is being filled by properties from the vertex.
- Each JSON object is a JSONJavaArray.
- The array returned to the rest service which returns it to the AJAX call.
- When the data is returned the HTML table is updated. Note that the table is of type datatable. This is a Bootstrap plugin. When the data is returned to the datatable object it knows how to update the content (rows and columns) of the table.
Below you find the process visualize:
When clicking a row:
- When the document is ready a listener is registered for each time a table row (TR) element is clicked.
- When such a TR is clicked an AJAX call is made to the rest service available under pathinfo acl. As data the replicaid value is send with the call. This value is stored in the data-set for each row.
- The AJAX call initiates class net.notesx.domex.rest.AclService. Here with the value of the given request parameter replicaid once again the Graph datamodel is opened and searched for the DXDatabase vertex that has a matching replicaid property.
- The returned vertex is bound to the DXDatabase class (name = db).
- If the DXDatabase is not null a collection is setup of type Iterable and filled with objects of type DXACLEntry for each item from the method db.getACLEntries.
- Then the collection is walked through and for each item a JSONJavaObject is created and filled with properties from the DXACLEntry object.
- The JSONJavaObject is placed in a JSONJavaArray. This array put as a string in a new JSONJavaObject which is returned to the rest service.
- The rest service returns the data back to the AJAX request.
- The AJAX request clear the DIV with id acl and fills it with the data received from the rest service. For each JSON object in the array a line with values is written in the DIV.
The process is visualized below:
The processes described above are the most basic types. The Graph is called for basic objects, not correlated.
As a result the following information is displayed on your screen:
Summary
Perhaps you are thinking: what the fuzz?! I can do something similar with document collections, perhaps showing a single category from a categorized column or perhaps create a jsonarray containing jsonjavaobjects and iterate through them and find a match.
Perhaps you can for THIS example. But I rather would avoid to create an unnecessary amount of design elements for it. Also in this example a very basic query is performed. Rethink the story in the beginning of this post. How are you gonna search your data under those conditions?
Also the combination to display your result data with the DataTables plugin you can build on the fly custom tables and reduce the amount of Notes views to a minimum.
In a future post I will describe more enhanced queries. Stay “connected”!
Links of interest
Kwintessential Notes 