Building a search function with DataTables plugin (IV)

Introduction

In the previous posts I demonstrated how to setup your application to generate a datatable component from your Notes view(s) and how to apply filtering & search capabilities to the columns.

In this post I will demonstrate how to build an external form to filter the datatable. I guess in most cases you want to provide some sort of search form and displayed separate from your table (left, right, top). For now we restrict the search form with input boxes.

Modifications

I have applied some modifications to the application. The complete and final code you can view in my Github repository.

Modifications:

  • java class that works as a custom service bean
  • javascript library for initializing datatable
  • xpage to display the datatable

Let’s add a form

The first thing we will do is by adding a Bootstrapped styled form to the xpage:

datatables04pre

Notice the following things:

  • The filter class for the input element
  • The data-column-index attribute for the input element

The filter class will be used to register events on:

$(‘.filter’).on(‘keyup change’, function () {
//clear global search values
db.search(”);
db.column($(this).data(‘columnIndex’)).search(this.value).draw();
});

The data-column-index attribute directs to the index of the corresponding column in the datatable.

With the form in place and the script updated we need to extend our java class so the values for the job title are included:

String job = String.valueOf(columnValues.get(10));
if (null!=job){
jo.put(“job”,job);
}

Let’s see the result

For example if I am looking for a person with firstname starting with Car.., from company Firm (something) acting as a host I get presented:

datatables04

Now with the initial search form in place we will extend it with features to improve it’s usability…

Building a search function with DataTables plugin (III)

Introduction

In the previous post about the datatables jquery component I demonstrated how to create input select controls for each column. In some cases an select control may however not be desired, due to the diversity of the entries in the column (e.g. track number, firstname, date).

Text inputs

In such cases you probably wont to use text inputs where a substring is being matched against the string entries in the column.

To achieve this our function to build the table have to be adapted slightly:

function initPersons(){

$(‘#persons tfoot th’).each( function () {
var title = $(this).text();
$(this).html( ‘<input type=”text” placeholder=”Search ‘+title+'” />’ );
} );

var db = $(“#persons”).DataTable();
db.destroy();
localStorage.clear();
var table = $(“#persons”).DataTable( {
stateSave : saveState,
fixedHeader: true,
“language” : {
“lengthMenu” : “Entries per page _MENU_”,
// “info” : “Page _PAGE_ of _PAGES_”,
“infoEmpty” : “No entries found”,
“infoFiltered” : “”
},
scrollY : yScroll,
“ajax” : “api.xsp/Persons” ,
“columns” : [
{
data : “firstname”,
“defaultContent”: “<i>Not set</i>”
},{
data : “lastname”,
“defaultContent”: “<i>Not set</i>”
},{
data : “company”
}
],
initComplete: function(){
this.api().columns().every( function () {
var column = this;

$( ‘input’, this.footer() ).on( ‘keyup change’, function () {
if ( column.search() !== this.value ) {
column
.search( this.value )
.draw();
}
} );

} );
}
});
}

As a result under each column input boxes which column filter functionality are added underneath the columns:

datatables03

Building a search function with DataTables plugin (II)

Introduction

In a previous post the base foundation was set up and a table was drawn on the xpage with the datatables jquery component. The data, residing in a Notes view, was delivered via a CustomServiceBean Rest service.

A datatable comes out of the box with a little search field which performs a search across the whole json data-set.

Individual column searching

In this post we will be adding select inputs for each column so you have a quick filter mechanism matching the values in the columns.

To achieve this we need to make some changes to the html table and our script to initialize the datatable object.

HTML table

We will place the select inputs on the bottom of the table, in the to be added footer section:

<table id=”persons” class=”table table-striped table-bordered”
cellspacing=”0″ width=”100%”>
<thead>
<tr>
<th>Firstname</th>
<th>Lastname</th>
<th>Company</th>
</tr>
</thead>
<tfoot>
<tr>
<th>Firstname</th>
<th>Lastname</th>
<th>Company</th>
</tr>
</tfoot>
</table>

initComplete

In our JS function that initializes the datatable we add a callback function that is triggered when the table is fully loaded:

initComplete: function(){
this.api().columns().every( function () {
var column = this;
var select = $(‘<select><option value=””></option></select>’)
.appendTo( $(column.footer()).empty() )
.on( ‘change’, function () {
var val = $.fn.dataTable.util.escapeRegex(
$(this).val()
);

column
.search( val ? ‘^’+val+’$’ : ”, true, false )
.draw();
} );

column.data().unique().sort().each( function ( d, j ) {
select.append( ‘<option value=”‘+d+'”>’+d+'</option>’ )
} );
} );
}

Progress

As a result of the activities above our table now looks like:

datatables02

The seach field works in conjunction with the select inputs. Not bad for a few lines of code!

navigationPath & selection properties

Introduction

Once in a while you move back in time and understand why you have forgotten certain rules. Mostly because you have stopped using them or you experience the same pitfalls as others do and you just want to forget them to move ahead.

navigationPath and selection properties

One of those things are the navigationPath from the Application Layout control and the selection property in it’s navigation nodes.

navigationPath

The navigationPath allows you to control the currently selected titlebar Tab and the selection appearance of any Navigator control on the XPage.

selection

The selection property is nothing more than a regular expression used to select an item. The expression is matched against the navigationPath, described above.

selected

Selected is a property that indicates if a node is selected or not. This property often causes confusion.

Mix them together

So if you mix these properties above you can establish some nice usability improvement for your app-user. It can visualize (via style classes) when a certain titlebar should be highlighted or not.

Example Aap Noot Mies

The follwoing example will demonstrate it’s basic feature (and shows some oddness in the Dutch education system while Mies is nothing more than a cat).

As titlebar tabs I have: Aap, Noot and Mies.

As Navigator nodes I have Aap, Noot, Mies and Apes/Gorilla, Apes/Chimp.

Each Navigator links points to an individual XSP but the last two nodes belong to the Aap titlebar tab.

ccLayout

I have a custom control with the application layout control on it. For the navigationPath property I have set up a propery definition so I can adjust it for eac XSP:

<xe:this.configuration>
<xe:oneuiApplication
navigationPath=”#{javascript:compositeData.navigationPath}”

The titlebar nodes are defined as follwed:

<xe:this.titleBarTabs>
<xe:pageTreeNode label=”Aap” selection=”/Aap/.*”
page=”/aap.xsp” selected=”true”>
</xe:pageTreeNode>
<xe:pageTreeNode label=”Noot” selection=”/Noot/.*”
page=”/noot.xsp” selected=”true”>
</xe:pageTreeNode>
<xe:pageTreeNode label=”Mies” selection=”/Mies/.*”
page=”/mies.xsp” selected=”true”>
</xe:pageTreeNode>
</xe:this.titleBarTabs>

Here I have set a regex for the selection property.

Navigator control

On the same custom control I also have a Navigator control:

<xe:navigator id=”navigator1″>
<xe:this.treeNodes>
<xe:pageTreeNode label=”Aap” page=”/aap.xsp”></xe:pageTreeNode>
<xe:pageTreeNode label=”Noot” page=”/noot.xsp”></xe:pageTreeNode>
<xe:pageTreeNode label=”Mies” page=”/mies.xsp”></xe:pageTreeNode>
<xe:basicContainerNode label=”Apes”>
<xe:this.children>
<xe:pageTreeNode label=”Gorilla”
page=”/gorilla.xsp”>
</xe:pageTreeNode>
<xe:pageTreeNode label=”Chimp” page=”/chimp.xsp”></xe:pageTreeNode>
</xe:this.children>
</xe:basicContainerNode>
</xe:this.treeNodes>
</xe:navigator>

Here I do not have to define anything. The control will know itself when a selected node needs to be highlighted.

Gorilla XPage

In the app I have multiple XPages, in the gorilla xpage I have set the navigationPath property as follwed:

<?xml version=”1.0″ encoding=”UTF-8″?>
<xp:view xmlns:xp=”http://www.ibm.com/xsp/core&#8221;
xmlns:xc=”http://www.ibm.com/xsp/custom”&gt;

<xc:ccLayout navigationPath=”/Aap/Gorilla”>
<xp:this.facets>
<xp:image url=”/aap.jpg” id=”image1″ xp:key=”facet_1″></xp:image>
</xp:this.facets>
</xc:ccLayout></xp:view>

As a result the Aap titlebar Tab is highlighted:

titlebarTabs

Not really something that opens your mind but at least I have written it down (with a repo) in the hope I won’t have to re-learn it next time I have to use the application layout control.

 

 

 

Building a search function with DataTables plugin

Introduction

In Domino you have multiple options how to provide a search function for an application. I have seen many examples where a FT search query is build for a database, where the form type is defined, which fields should be used in the search etcetera. Most of them are a maintenaince nightmare, layout is little flexible and usability is lacking (e.g. perform a search towards a query which will result in zero hits).

What if you make it easier to maintain, more flexible for user specific desires and add facetted search principles? A win situation?

DataTables

In the following posts I will write how to use the jQuery DataTables plugin to provide a quick search function for your Domino apps. I assume you use XPages.

Stage 1 – Getting & presenting the data

In the initial stage we just try to get and present the data we need. As example I will be using the FakeNames application. I have created a public repo which has at the moment:

  • an xpage to present the data (dtPersons.xsp)
  • an xpage (api.xsp) which will contain rest services
  • a Java class that generates the data in JSON format
  • a CSJS library to transform the HTML table on dtPersons.xsp to  DataTables object
  • the required resources from DataTables.net
  • a Theme resource to have the resources available
  • a Notes view to use as data source.

In order to create some fake documents I have set up an LS agent to do so.

If you bind this all together, you have already a search function:

datatables01

Right on top you have a search field where you can query the columns in the table. The columns are sortable too. Probably it is matching most of the search functions you have seen for Domino. But wait! There is more possible…

 

JavaScript Internationalization & placeholders

Introduction

Some time ago I wrote about moving a Teamroom application up to IBM Bluemix. This month the XPages runtime is now general available.

In the post I mentioned I would come back on some design gotchas I saw while analyzing and preparing the Teamroom application for staging it to Bluemix. So here is (at least9 a first post.

What I noticed for example was the usage of the l18n library in SSJS and placeholders in properties files.

i18n formatting

This is a server script library with methods to format messages for localization, display dates in a locale-specific manner and miscellaneous Internationalization utilities.

So how does this works?

Create two or more strings properties file e.g.

  • strings.properties
  • strings_sv.properties

The strings.properties file content:

compliment=Handsome

hello.world=Hello {0}

The strings_sv.properties file content:

compliment=Snygging

hello.world=Hejsan {0}

Now create an XPage and place e.g. a Computed Field control on it and calculate it’s value:

<xp: text><xp: this.value><![CDATA[#{javascript:compliment = strings.getString(“compliment”);

return i18n.format(strings.getString(“hello.world”), compliment);}]]></xp: this.value></xp: text>

(Do not forget to enable Internationalization for your application in the Application Properties under section International Options)

Placeholders

Notice in the properties file the usage of a placeholder: hello.world=Hello {0} and how it is being referred: strings.getString(“hello.world”), compliment. Here the variable compliment (another string property value) is put in the placeholder.

As a result in the default language I get returned “Hello Handsome” and when I choose Swedish as default language in my browser I get returned “Hello Snygging”.

By wrapping it and formatting it with the L18n library you ensure it is fitted according the active language. This is very handy for dates, amounts etcetera.

Links of interest

More details about JavaScript Internationalization can be found in the IBM Notes Domino Application Development wiki.

 

Graph – a closer look at the data

Introduction

A graph database data is represented as ‘vertices’, sometimes called ‘nodes’. The relationships between vertices are represented by connections called ‘edges’. Graph databases also store metadata or ‘properties’ about vertices and edges.

Domino Explorer

If you look at the data in Domino Explorer after the source databases names.nsf and catalog.nsf are scanned you can group them in the same categories vertices and edges.

If you look further at the Graph related properties on the Vertices documents you notice that the next level of division is the Java class the vertex is added with to the Graph (Java object-type Vertex = graph.addVertex(id, Java.class).

Each class defines specific properties for the object and also the relation(s) to other Vertices. These Edges contain properties like label, direction (in/out).

Adjacency annotate getters and adders to represent a Vertex incident to an Edge. AdjacencyUnique extends this idea to ensure that there is only one instance of an Edge with a specific label between any two specific Vertices. This allows the user of the graph to call an “add” method and return the existing adjacency if it already exists.

So when and where are these relations defined? That depends on your code.

When you “like” a post on Twitter, the relation between you and the tweet is created when you click the “like” icon.

In Domino Explorer relations are created when you run the setup by selecting one of the “Start Scan” buttons.

Screen Shot 2016-05-23 at 13.46.55

Scan Databases

When you select to scan the databases the $ReplicaID view in the catalog.nsf is opened and for each entry found a Vertex is created using the DXDatabase class and committed to the Graph.

View allDbs = catalog.getView(“($ReplicaID)”);
DocumentCollection col = allDbs.getAllDocuments();
for (Document db : col) {
String replicaId = session.evaluate(“@Text(ReplicaID; \”*\”)”, db).elementAt(0).toString();
DXDatabase databaseVertex = graph.addVertex(replicaId, DXDatabase.class);
databaseVertex.setReplicaId(replicaId);
String dbTitle = db.getItemValueString(“Title”);
databaseVertex.setTitle(dbTitle);
databaseVertex.setFilePath(db.getItemValueString(“Pathname”));
databaseVertex.setServer(db.getItemValueString(“Server”));

graph.commit();
// ACL
scanAcl(graph, databaseVertex);
}

Then then Graph is “scanned” with the newly created vertex. Here the database ACL is collected and for each ACL entry a new vertex is created using a specific class. Also the relationship between the database and aclentry vertices is created:

DXACLEntry graphAclEntry = graph.addVertex(aclEntry.getName(), DXACLEntry.class);
db.addAclEntry(graphAclEntry);

The addAclEntry method is defined the DXDatabase class which the db object is created with:

@AdjacencyUnique(label = “hasAcl”, direction = Direction.IN)
public void addAclEntry(DXACLEntry ae);

If you look at the Vertex object in the Domino Explorer NSF you notice these properties for a DXDatabase Vertex object:

#Note-UNID 82753B9B92522221033D1650C7BE35D4
$$Key 85256714:00725208
_ODA_GraphType V
form DXDatabase
filePath AgentRunner.nsf
replicaId 85256714:00725208
server CN=dev1/O=quintessens
title Java AgentRunner
_COUNT_OPEN_IN_hasAcl 5
_OPEN_IN_hasAcl

where

  • note-UNID is the document unique id
  • $$Key is the same as the replicaId of the (target) database (different for Vertices created with other class)
  • _ODA_GraphType, V stands for Vertex
  • form is the Java class the document is created with
  • _COUNT_OPEN_IN_hasAcl, counter. Not sure where it’s used for.
  • _OPEN_IN_hasAcl, for all Vertices empty

If you look at a DXACLEntry Vertex document notice the following properties:

#Note-UNID B9962995F0A05DCBAE35F07C64964A1C
$$Key -Default-
_ODA_GraphType V
form DXACLEntry
_COUNT_OPEN_IN_hasAcl 110
_OPEN_IN_hasAcl 110
level 6
name -Default-
_COUNT_OPEN_OUT_member 1
_OPEN_OUT_member

If you search through the Edges document you will find a document matching the note UNID’s above and carrying the label “hasLabel”:

Screen Shot 2016-05-23 at 15.00.21

and carrying the following properties:

Screen Shot 2016-05-23 at 15.01.06

Hereby the Graph db recognises a relationship between one Vertex of type DXDatabase and  another Vertex of type DXACLEntry.

Because this is not the only relationship the DXDatabase object has when the ACLService remote service is called, provided with the replicaID of the Database more than one matching relationship is returned under db.getAclEntries:

DXDatabase db = graph.getElement(replicaId, DXDatabase.class);
if (db != null) {
int count = 0;
Iterable<DXACLEntry> acl = db.getAclEntries();
for (DXACLEntry entry : acl) {
JsonJavaObject aclEntry = new JsonJavaObject();
aclEntry.put(“name”, entry.getName());
aclEntry.put(“level”, entry.getLevel());
aclEntry.put(“levelName”, ACL_LEVEL[entry.getLevel()]);
data.put(count, aclEntry);
count++;
}
}

Here for each found relationship the DXACLEntry object is collected from the Graph and properties from it placed in a JsonJavaObject, placed in an array and returned.

Wrapup

After taking this deeper look under the hood and analyzing the data I hope you have gained a bit more understanding of the Graph concepts and the implementation of it from it via OpenNTF’s Domino API in Domino Explorer.

 

Another Graph sample from Domino Explorer

Introduction

In a previous post I dove into Domino Explorer, an XPages application that cans the Domino Directory and the Catalog by using the Graph capabilities in the OpenNTF Domino API.

In this post will describe another XPage in that application. Perhaps it helps to get a better picture on the Graph db and how to build an application around it using XPages, JavaScript, & Java.

Hopefully I will be ready writing before a European football final kicks off…

AllACL.xsp

The xpage I discuss is allacl.xsp. Basically it displays at start a table with ACL entries for the entire catalog. When I click on an entry or row in the table I get presented a list of applications where the selected ACL entry resides in the ACL.

The result of the scenario above captured in the following screen:

de_acl

As the image suggests the entry [Anonymous] resides in 4 applications. Let’s dive in a little deeper into the XPage to see how it’s done…

On document ready

The XPage contains a JS library whichs fires an AJAX call to collect the data for the DataTable object. The rest service resides on the XPage and is accessed via it’s pathinfo property. The restservice is bound to a custom servicebean which resides in the application.

The AJAX call does not provide a parameter to the restservice, so the java class will collect all the vertices of type DXACLEntry.class. For each vertice a JSONJavaObject containing name and level and placed in a JSONJAVAArray object. This array is placed in a JSONJavaObject and returned as a string to the restservice.

When the data is received in the DataTable object only the name property is placed.

The routing is visualized in the next image:

de_acl_rest

On row click

In the JS library for the DataTable object is also defined what should happen when a row in the table is clicked.

Here the name value for the selected row is used in a function that initiates a second DataTable object. Again a call is made to the same restservice. However now the name value is send as a parameter (“?name=” + name value).

The servicebean picks this parameter up (String name = request.getParameter(“name”)) and get’s all the vertices from the Graph with the provided name, matching the DXACLEntry (DXACLEntry aclEntry = graph.getElement(name, DXACLEntry.class)).

Similar as in the document ready event for each found DXACLEntry object a JSONJavaObject is created, now with some more properties (title,filepath, replicaId,server) and placed in a JSONJavaArray. This array is returned as a string to the restservice.

When the results are received by the DataTable object the four properties are displayed per column.

de_acl_rest2

Summary

The scenario is that simple. The DataTable plugin is really a great plugin that does a lot for you and can save you multiple design elements (view controls) by defining in your code what you want in it as result and in which order.

Now let me enjoy my evening of football with a cold beer! Happy development =)

 

 

Exploring the Domino Explorer

Introduction

We live in a connected world today and also the data in your IBM Notes/Domino platform is more connected that you perhaps realise:

I work work for company X at department Y and besides my assignment as “ICS product specialist” I perform in different roles, sometimes as developer, sometimes as a solution architect, sometimes as project manager etc. I own a range of devices to do my work, bought in by different suppliers under different conditions/contracts. Now a external project-leader (they claim they are the best) has became responsible for a new project (Z since it is the final project the company is ever going to run) and to make a huge impact the project-leader want to equip his team (which I have become member of) with the latest and greatest of devices to make our work most convenient and the impact of the project delivery as huge as possible. BUT of course he is unaware about the devices the team owns already and the lifetime of the contracts. He also does not want to make top management angry so he suggest to provide this team with devices just 1 configuration level below top management. So how does the project-leader require the information he need to know (a list of end of lifetime for the current devices the project team owns compared with the list of current devices top management owns) ?

All objects in the story above (organisation, departments, employees, devices, contracts, suppliers) are represented in Notes documents. By connecting them in a Graph DB new information can be gathered!

Graph

Lately I have been exploring Graph data modelling more closely. Mostly because I see many opportunities to connect/model unstructured data in fluid ways. Faster than relational datasources sometimes can.

For now I have been mainly looking at Neo4J because most Graph related education is focussed around this product. But for IBM customers you have IBM Graph (cloud only?) and since recent IBM Notes via the OpenNTF Domino API.

Graph need NoSQL

Graph needs NoSQL databases since NoSQL is capable of describing the variety of objects that are around in enterprise data. As it turns out Lotus Notes is around as one of the first NoSQL databases!

ODA & Tinkerpop

The OpenNTF Domino API has included Tinkerpop and an implementation to store content in a graph database structure.

This is great because as Domino developer you are concerned that your data is stored in an NSF. Probably you want to avoid the hassle of moving your data to another technology first and focus on developing applications directly.

Domino Explorer

Recently Oliver Busse has released Domino Explorer. It scans the Domino Directory and the Catalog application to show the capabilities of Graph in an XPages application. Getting the Notes data into a Graph data model is done via the OpenNTF Domino API.

In the next part of this post I will focus on how this is was implemented…

Document: Exploring the Domino Explorer

This document describes the setup of the Domino Explorer (DE) application (app) available on OpenNTF (link). The DE app demonstrates the Graph db capabilities via the OpenNTF Domino API (ODA) (link).

By exploring the architecture of the app better understanding of the Graph technology is aimed to achieve and how to implement it in XPages applications.

Presumptions

It is presumed that the reader of this document has advanced level of understanding of XPages technology, intermediate level of Java programming language, medium level of understanding of IBM Notes data and low level of the graph data model.

If not turn around or educate yourself on these topics😉

Graph glossary

The view design element “graph” group it’s containing documents into the two basic units in the Graph Data Model (Graph/GDM): Edges and Vertices.

Vertices

Vertices (singular:vertex) are the objects in a graph.

Edges

Each edge has two (or in hypergraphs, more) vertices to which it is attached, called its endpoints. Edges may be directed or undirected; undirected edges are also called lines and directed edges are also called arcs or arrows.

IBM Notes data

When we translate this to IBM Notes data, vertices are mostly objects that contain values or properties (edge). E.g. a user is member of a group. An order has a date-stamp etcetera. With graph you can make queries to find different types of objects connected via shared properties e.g. A car-leasing company may have a fleet of rental-cars which may have present or past drivers.

General overview

Design elements in scope of the dissection

To have some sort of start we will start easy and give an overview of the design elements involved:

Design element Purpose Scope
Views Used to store documents that are used in the graph data model.
Documents are of type Edge or Vertex(Vertices).
X
XPages Used to initiate business logic and present the result to the user in a web interface. X
Custom controls Used to break parts of XPages into reusable pieces. X
Script libraries Client-side javascript libraries used to interact with the web presentation. X
Java Java classes that contain the business logic. X
Images Used to enhance the web presentation.
Style sheets CSS definitions used to enhance the web presentation.
Themes Use to set rules for the application regarding style, behaviour, resources
faces-config.xml Configuration file for the application, mostly used for registering managed beans.

For the rest of this document we will only focus on design elements that are marked with X under the Design elements will most likely be discussed in the logical order how the graph data model is implemented in the DE app.

Managed Beans

Managed Bean (link) is a regular Java Bean class registered by the XPages framework. Important beans registrered in the DE app for Graph are:

Bean name Class
config net.notesx.domex.controller.ConfigurationController
scanner net.notesx.domex.controller.ScannerController

Config

The ConfigurationController class controls the configuration for the GDM. It  sets which databases are in scope for the GDM (Names.nsf, catalog.nsf).

It uses the:

 

Java class Usage
net.notesx.domex.graph.GraphHelper Setup of the Graph
net.notesx.domex.graph.Configuration Configuration of the Graph

DFramedTransactionalGraph<DGraph> configGraph = GraphHelper.getGraph();

Configuration configuration = configGraph.addVertex(“configuration”, Configuration.class);

configuration.setApplicationName(applicationName);

configuration.setNamesPath(namesPath);

configuration.setCatalogPath(catalogPath);

configGraph.commit();

In the code above the Graph db is loaded and the configuration added as a Vertex.

Configuration class

The configuration class contains the following properties:

  • $$Key (??? identifier for current configuration)
  • ApplicationName (name for the application, display only)
  • namesPath (location of the names.nsf)
  • catalogPath (location of the catalog.nsf)

GraphHelper class

The GraphHelper class sets up the Graph in the following order:

  • Setup an element store for configuration
  • Add the types (Vertices) to the element store
  • Create a configuration for the Graph
  • Add the element store to configuration and set the default element store (= filepath of current database)
  • Setup the Graph by using the configuration
  • Return it

// set element store for configuration

DElementStore configStore = new DElementStore();

Database currentDatabase = Factory.getSession(SessionType.CURRENT).getCurrentDatabase();

configStore.setStoreKey(currentDatabase.getFilePath());

// setup the type

configStore.addType(Configuration.class);

configStore.addType(DXDatabase.class);

// create a graph config

DConfiguration config = new DConfiguration();

DGraph graph = new DGraph(config);

// add the config element store

config.addElementStore(configStore);

config.setDefaultElementStore(configStore.getStoreKey());

// setup the graph

DFramedGraphFactory factory = new DFramedGraphFactory(config);

DFramedTransactionalGraph<DGraph> fg = (DFramedTransactionalGraph<DGraph>) factory.create(graph);

// return the graph

return fg;

Scanner

This class adds the database, user and group objects to the Graph according the specified class for each object.

Important methods:

  • scanPeopleAndGroups
  • scanDatabases
  • scanAcl
  • scanMember

scanDatabases

  • Get the path of the catalog db via the ConfigurationController class
  • Initiate the Graph datamodel via the GraphHelper class
  • Put all documents from View $ReplicaID into a document collection
  • For each document add a vertex to the graph using the DXDatabase class
    • Set properties for the vertex
    • Commit it to the Graph
    • Run the scanAcl method for the databaseVertex object

scanPeopleAndGroups

  • Get the name of the names db via the ConfigurationController class
  • Initiate the Graph datamodel via the GraphHelper class
  • Put all documents from View $VIMPeople into a document collection
  • For each document add a vertex to the graph using the DXUser class
    • Set properties for the vertex
    • Commit it to the Graph
  • Put all documents from View $VIMGroups into a document collection
  • For each document add a vertex to the graph using the DXGroup class
    • Set properties for the vertex
    • Commit it to the Graph
    • Run the scanMember class for the groupVertex object

scanAcl

  • For the given database (vertex object) get the database object
    • Grab the ACL
    • For each ACL entry add a vertex using the DXACLEntry class
      • Set properties for the vertex
      • Add the ACL entry vertex as an Edge to the DB vertex (the label “hasAcl” will be used)
      • Commit it to the Graph

scanMember

  • For the given group (DXGroup class object):
    • For each member in groupmembers list:
      • Check if it contains a “/” ( This is a user)
        • Create a user object with the DXUser class from the Graph
        • If the user object is not null (in the Graph) add it to the group
      • If not (this is a group)
        • Create a group object with the DXGroup class from the Graph
        • If the group object is not null add it to the group
      • Set a property for the vertex
      • Commit it to the Graph

Other classes

DE contains more classes stored under several packages:

  • Common
  • Domex
    • Controller
    • Graph
    • Rest

Common package

Classes under the common package are used for setting properties for application e.g. navigation, page behaviour etc. They are out of scope for this document.

Domex \ controller package

Classes under the domex \ controller package are ConfigurationController and ScannerController and available as described earlier as managed beans.

Domex \ graph

Classes under the domex \ graph package are directly related to the Graph data model and describe the Graph data objects (vertexes):

  • Configuration
  • DXDatabase
  • DXACLEntry
  • DXGroup
  • DXUser

Each class describes the type of the object, the properties the object has and how to get & set them, the edges the object has and in which direction they go.

Annotation Description Example
@TypeValue Interface annotation for marking the Element property-value that may contain type information. @TypeValue(“DXACLEntry”)
@Property Property annotations are for getter and setters to manipulate the property value of an Element. @Property(“$$Key”)

public String getKey();

@Property(“name”)

public void setName(String s);

@AdjacencyUnique Adjacency annotate getters and adders to represent a Vertex incident to an Edge. AdjacencyUnique extends this idea to ensure that there is only one instance of an Edge with a specific label between any two specific Vertices. This allows the user of the graph to call an “add” method and return the existing adjacency if it already exists. @AdjacencyUnique(label = “hasAcl”, direction = Direction.OUT)

public Iterable<DXDatabase> getDatabases();

Below is as example of the definition of the DXUser.java class:

package net.notesx.domex.graph;

import org.openntf.domino.graph2.annotations.AdjacencyUnique;

import org.openntf.domino.graph2.builtin.DVertexFrame;

import com.tinkerpop.blueprints.Direction;

import com.tinkerpop.frames.Property;

import com.tinkerpop.frames.modules.typedgraph.TypeValue;

@TypeValue(“DXUser”)

public interface DXUser extends DVertexFrame {

@Property(“$$Key”)

public String getKey();

@Property(“username”)

public String getUserName();

@Property(“username”)

public void setUserName(String s);

@AdjacencyUnique(label=”member”, direction=Direction.OUT)

public Iterable<DXGroup> getMembershipInGroups();

}

The GraphHelper class is described earlier in this document and is used to setup the Graph.

Domex \ Rest package

The classes under the domex \ rest package create collections of json objects for the objects in the Graph data model. Some classes take in parameters, some don’t. Each class defines what type of graph object is in scope.

The way these classes are initiated is as follow:

  • On XPages rest service(s) of type customRestService are defined. The servicebean property directs to which class the service is bound to.
  • The XPage contains JavaScript libraries.
    • In a library can be defined which rest service should be called when the document is ready e.g. to build a collection of database objects.
    • In a library can be defined which rest service should be called when a link is being clicked e.g. a row in a data table control and if a parameter should be send (e.g. the replica Id of the database). Also is defined what should be done with the data returned from the rest service.

XPages

The design elements described above come together in the XPages design elements. We will demonstrate by describing an XPage.

Alldbs.xsp

Elements:

Type Name Purpose
Custom Control _layoutBS3 Reusable layout for application.
JavaScript library alldbs.js Calling Rest Services.

On document ready: data

On click table row: acl

Rest service control Pathinfo: data Bound to Java class net.notesx.domex.rest.AllDatabasesService
Rest service control Pathinfo: acl Bound to Java class net.notesx.domex.rest.AclService
HTML table datatable Placeholder for result from rest service under pathinfo data
Div acl Placeholder for result from rest service under pathinfo

XPage logic explained

On document ready:

  1. When the document is ready an AJAX call is made to the rest service under pathinfo data. This initiates Java class net.notesx.domex.rest.AllDatabasesService.
  2. The class gets the Graph data model and collects all objects/vertexes of the type specified in the DXDatabase class.
  3. For each vertex a JsonJavaObject is created which is being filled by properties from the vertex.
  4. Each JSON object is a JSONJavaArray.
  5. The array returned to the rest service which returns it to the AJAX call.
  6. When the data is returned the HTML table is updated. Note that the table is of type datatable. This is a Bootstrap plugin. When the data is returned to the datatable object it knows how to update the content (rows and columns) of the table.

Below you find the process visualize:

Screen Shot 2016-05-16 at 11.09.02

When clicking a row:

  1. When the document is ready a listener is registered for each time a table row (TR) element is clicked.
  2. When such a TR is clicked an AJAX call is made to the rest service available under pathinfo acl. As data the replicaid value is send with the call. This value is stored in the data-set for each row.
  3. The AJAX call initiates class net.notesx.domex.rest.AclService. Here with the value of the given request parameter replicaid once again the Graph datamodel is opened and searched for the DXDatabase vertex that has a matching replicaid property.
  4. The returned vertex is bound to the DXDatabase class (name = db).
  5. If the DXDatabase is not null a collection is setup of type Iterable and filled with objects of type DXACLEntry for each item from the method db.getACLEntries.
  6. Then the collection is walked through and for each item a JSONJavaObject is created and filled with properties from the DXACLEntry object.
  7. The JSONJavaObject is placed in a JSONJavaArray. This array put as a string in a new JSONJavaObject which is returned to the rest service.
  8. The rest service returns the data back to the AJAX request.
  9. The AJAX request clear the DIV with id acl and fills it with the data received from the rest service. For each JSON object in the array a line with values is written in the DIV.

The process is visualized below:

Screen Shot 2016-05-16 at 11.11.07

The processes described above are the most basic types. The Graph is called for basic objects, not correlated.

As a result the following information is displayed on your screen:

Screen Shot 2016-05-16 at 11.17.13

Summary

Perhaps you are thinking: what the fuzz?! I can do something similar with document collections, perhaps showing a single category from a categorized column or perhaps create a jsonarray containing jsonjavaobjects and iterate through them and find a match.

Perhaps you can for THIS example. But I rather would avoid to create an unnecessary amount of design elements for it. Also in this example a very basic query is performed. Rethink the story in the beginning of this post. How are you gonna search your data under those conditions?

Also the combination to display your result data with the DataTables plugin you can build on the fly custom tables and reduce the amount of Notes views to a minimum.

In a future post I will describe more enhanced queries. Stay “connected”!

Links of interest

 

 

Stress testing with ODA Graph

I am very interested in Graph data modelling and with the Graph capabilities in OpenNTF Domino API I decided to setup some demo environments just to get my head around the subject and how you can implement it in XPages and use Notes data.

To my opinion reading Notes data in the Graph database structure can bring interesting new opportunities, far beyond what we can deliver with Views and Collections today.

Oliver Busse has provided a great starting point with his SUTOL demo application so I started with that one.

Besides the implementation and Graph capabilities I am also curious about performance.  So I run some tests on my working demo app. With the help with a simple agent I decided to be gentle and create a set of only 20.000 user documents.

The first test was about returning user profiles (nodes) matching certain properties (relations), presented in a repeat control. Below the list the time to load the filtered set is displayed.

stress01

stress02.JPG

When I compare the result with a normal view filter (by category or FT search) the results where a bit disappointing.

I also noted that navigating through the list was very slow (20 seconds or more returning a new set of rows of 10 documents). More than I expected I received timeouts.

stress03

The reason for this performance is still unknown. I guess there is no index created yet for the user node in the graph db structure. Why navigating through the list is so latent in performance is also a mysterie.

Nevertheless, my demo is up and running so expect more results on Graph in XPages with Notes data in the future on this blog.

Below are sampels of performance using the “traditional” FT search filter capacity in Notes. Notice the difference.

stress04.JPG

stress05.JPG

I would like to thank Oliver Busse for his guidance getting the demo app up and running and for explaining some basic concepts of the implementation.