It’s a long time since my last post. But after my sons heart surgery, my time was consumed by one of the biggest challenge that we as WebGate ever has to face. I will definitely write about this, after the successful closing of the project.

But in the mean time, I have done some experiments with the XPagesToolkit and large datasets. The question that I’ve to answer is very easy. How to process a large dataset like 100’000 documents to a nice List of contact objects, or in java spoken: List contacts = ContactSorageServer.getInstance().getAll().

My first question was of course: Is it really mandatory to have 100’000 Contacts in one List? To be honest, it is mostly no mandatory. Having 100’000 Contacts in a List is a typical “Notes View” case. Most agile Application will not present you a tabular list with 100 results per page and a pager control to browse thru 1000 pages, that you can find the page with the correct dataset. But having such a large dataset helps to find the best solution to read a dataset.

So our test scenario is very easy:

1. 100’000 Documents to transfer to Contact Object add to an ArrayList.
2. The transfer from document to object is fix, no changes will make there, we try only to manipulate the way we get the 100’000 Documents and how we can add them to the ArrayList
3. All tests are instrumented and measured with the XPages Toolbox (see Mastering XPages Second Edition, Chapter 20)

1. Using the current XPT Approach:

The XPagesToolkit uses currently a view and browse the document by getNextDocument(). Each object is added to the ArrayList(). This has the effect that the ArrayList has to extend and realign the internal array each time when we achieve the internal capacity. This is the base value.

2. Using LinkedList instead of ArrayList

The only change to the XPT Approach is, that we use a LinkedList instead of the ArrayList and copy the LinkedList as last opration to an ArrayList. We expect to see a significant speed boost, because the LinkedList can growth in a “one Operation” step and has no reassigning of an internal array.

3. Using a NoteCollection based on the view’s selection formula (suggested by Nathan T. Freeman)

Instead of looping thru a view, we use the views selection formula to build a NoteCollection and access the document directly by “getDocumentById()”. I was personally a bit sceptic about building a selection of NoteID ad hoc. Would this work in such a big dataset?

Each approach was tested 10 times (using the code at the end of the entry). Here the results:

My first finding was, that building a LinkedList and then converting to an ArrayList could be a bit faster than the permanent resizing of the ArrayList, but no so much as expected. That was also the reason to see what was happening in the backend. And as you see here, the time to loop thru the we is long, 10.3 sec, means 0.1ms per document.

But the big surprise was the result with the NoteCollection. It needs only 726ms to build the Collection and other 2.9 sec to browse thru the collection. That’s very fast. This means, even accessing the View to get the selection formula and then building the NoteCollection and browse thru this collection is much faster!

I think that will definitely go in to the XPagesToolkit. But this will also need testing. If you wanna be informed about new SNAPSHOT build of the XPagesTookit, please register here http://mm.wgcdev.ch/mailman/listinfo/xpagestoolkit!

And then, the child in me told me. How will this scale, if you set in the view selection formula a @Today function. We all know that such views has to be rebuilt permanently. And the child wants to play, so I did. Here the same stuff with a view with the following selection formula: SELECT ((Form = “Contact”)) & @Created > @Adjust(@Today;0;0;-10;0;0;0)

The XPT had an average time of 50 sec (that’s what I excepted to see), to get all the document (also 100’000). BUT……..

….. the NoteCollection approach has done this in 32.8 sec! Means near the same speed! The building of the NoteCollection has a time vor 844 ms. That’s definitely great!

Let see what this has for en impact for, small datasets. And a lot of stuff to discuss with the XPagesToolkit Team 🙂

Here my test code:

/*
 * © Copyright WebGate Consulting AG, 2014
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at:
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
 * implied. See the License for the specific language governing
 * permissions and limitations under the License.
 */
 package org.openntf.xpt.demo.bo;

import java.util.ArrayList;
 import java.util.LinkedList;
 import java.util.List;

import lotus.domino.Document;
 import lotus.domino.NotesException;
 import lotus.domino.View;
 import lotus.domino.NoteCollection;

import org.openntf.xpt.core.dss.AbstractStorageService;
 import org.openntf.xpt.core.dss.DominoStorageService;

import com.ibm.commons.util.StringUtil;
 import com.ibm.commons.util.profiler.Profiler;
 import com.ibm.commons.util.profiler.ProfilerAggregator;
 import com.ibm.commons.util.profiler.ProfilerType;
 import com.ibm.xsp.extlib.util.ExtLibUtil;

public class ContactStorageService extends AbstractStorageService {

private static ContactStorageService m_Service;

private static final ProfilerType pt = new ProfilerType("ListPerformance");

private ContactStorageService() {

}

public static ContactStorageService getInstance() {
 if (m_Service == null) {
 m_Service = new ContactStorageService();
 }
 return m_Service;
 }

@Override
 protected Contact createObject() {
 return new Contact();
 }

// Performance TestCode
 public List getAllXPT(String viewName) {
 List lstRC;
 if (Profiler.isEnabled()) {
 ProfilerAggregator pa = Profiler.startProfileBlock(pt, "getAllXPT -> " + viewName);
 long startTime = Profiler.getCurrentTime();
 try {
 lstRC = getAll(viewName);
 } finally {
 Profiler.endProfileBlock(pa, startTime);
 }
 } else {
 lstRC = getAll(viewName);
 }
 return lstRC;
 }

public List getAllLinkedList() {
 if (Profiler.isEnabled()) {
 ProfilerAggregator pa = Profiler.startProfileBlock(pt, "getAllLinkedList");
 long startTime = Profiler.getCurrentTime();
 try {
 return _getAllLinkedList();
 } finally {
 Profiler.endProfileBlock(pa, startTime);
 }
 } else {
 return _getAllLinkedList();
 }
 }

public List<Contact> _getAllLinkedList() {
 List lstRC = new LinkedList();
 try {
 View viw = ExtLibUtil.getCurrentDatabase().getView("AllContacts");
 viw.setAutoUpdate(false);
 Document docNext = viw.getFirstDocument();
 while (docNext != null) {
 Document docProcess = docNext;
 docNext = viw.getNextDocument(docNext);
 Contact con = createObject();
 if (DominoStorageService.getInstance().getObjectWithDocument(con, docProcess)) {
 lstRC.add(con);
 }
 docProcess.recycle();
 }
 viw.recycle();
 } catch (Exception e) {
 e.printStackTrace();
 }
 if (Profiler.isEnabled()) {
 ProfilerAggregator pa = Profiler.startProfileBlock(pt, "copyLinkedList");
 long startTime = Profiler.getCurrentTime();
 try {
 return _copyList(lstRC);
 } finally {
 Profiler.endProfileBlock(pa, startTime);
 }
 } else {
 return _copyList(lstRC);
 }
 }

private List _copyList(List linkedList) {
 return new ArrayList(linkedList);
 }

public List getAllNC(String viewName) {
 if (Profiler.isEnabled()) {
 ProfilerAggregator pa = Profiler.startProfileBlock(pt, "getAllNC " + viewName);
 long startTime = Profiler.getCurrentTime();
 try {
 return _getAllNC(viewName);
 } finally {
 Profiler.endProfileBlock(pa, startTime);
 }
 } else {
 return _getAllNC(viewName);
 }

}

public List getAllNCLinkedList() {
 if (Profiler.isEnabled()) {
 ProfilerAggregator pa = Profiler.startProfileBlock(pt, "getAllNC-LinkedList");
 long startTime = Profiler.getCurrentTime();
 try {
 return _getAllNCLL();
 } finally {
 Profiler.endProfileBlock(pa, startTime);
 }
 } else {
 return _getAllNCLL();
 }

}

private List _getAllNC(String viewName) {
 List lstRC = new ArrayList();
 try {
 View viw = ExtLibUtil.getCurrentDatabase().getView(viewName);
 String formula = viw.getSelectionFormula();
 NoteCollection nc;
 if (Profiler.isEnabled()) {
 ProfilerAggregator pa = Profiler.startProfileBlock(pt, "buildNoteCollection " + viewName);
 long startTime = Profiler.getCurrentTime();
 try {
 nc = _buildNoteCollection(formula);
 } finally {
 Profiler.endProfileBlock(pa, startTime);
 }
 } else {
 nc = _buildNoteCollection(formula);
 }

lstRC = new ArrayList(nc.getCount());
 String notesIDNext = nc.getFirstNoteID();
 while (!StringUtil.isEmpty(notesIDNext)) {
 Document processDoc = ExtLibUtil.getCurrentDatabase().getDocumentByID(notesIDNext);
 notesIDNext = nc.getNextNoteID(notesIDNext);
 if (processDoc.isValid()) {
 Contact con = createObject();
 if (DominoStorageService.getInstance().getObjectWithDocument(con, processDoc)) {
 lstRC.add(con);
 }
 processDoc.recycle();
 }
 }
 nc.recycle();
 viw.recycle();
 } catch (Exception e) {
 e.printStackTrace();
 }
 return lstRC;
 }

private List<Contact> _getAllNCLL() {
 List lstRC = new LinkedList();
 try {
 View viw = ExtLibUtil.getCurrentDatabase().getView("AllContacts");
 String formula = viw.getSelectionFormula();
 NoteCollection nc;
 if (Profiler.isEnabled()) {
 ProfilerAggregator pa = Profiler.startProfileBlock(pt, "buildNoteCollection");
 long startTime = Profiler.getCurrentTime();
 try {
 nc = _buildNoteCollection(formula);
 } finally {
 Profiler.endProfileBlock(pa, startTime);
 }
 } else {
 nc = _buildNoteCollection(formula);
 }
 String notesIDNext = nc.getFirstNoteID();
 while (!StringUtil.isEmpty(notesIDNext)) {
 Document processDoc = ExtLibUtil.getCurrentDatabase().getDocumentByID(notesIDNext);
 notesIDNext = nc.getNextNoteID(notesIDNext);
 if (processDoc.isValid()) {
 Contact con = createObject();
 if (DominoStorageService.getInstance().getObjectWithDocument(con, processDoc)) {
 lstRC.add(con);
 }
 processDoc.recycle();
 }
 }
 nc.recycle();
 viw.recycle();
 } catch (Exception e) {
 e.printStackTrace();
 }
 if (Profiler.isEnabled()) {
 ProfilerAggregator pa = Profiler.startProfileBlock(pt, "copyLinkedList");
 long startTime = Profiler.getCurrentTime();
 try {
 return _copyList(lstRC);
 } finally {
 Profiler.endProfileBlock(pa, startTime);
 }
 } else {
 return _copyList(lstRC);
 }
 }

private NoteCollection _buildNoteCollection(String formula) throws NotesException {
 NoteCollection nc = ExtLibUtil.getCurrentDatabase().createNoteCollection(false);
 nc.setSelectDocuments(true);
 nc.setSelectionFormula(formula);
 nc.buildCollection();
 return nc;
 }
 }