ISM: Creating a Discovery Library Adapter to write IdML docs

If you are interested in creating IdML import files for systems like TADDM then there is a video that shows you how to assemble a DLA using TDI here.

There is also a shorter version on YouTube.

Please let me know if you have any comments, suggestions, corrections or just a bone to pick!

Happy 2009!

Troubleshoot the SyncDBfromSource AL in the Lotus Connections TDI Config: profiles_tdi.xml

Now that I have your attention, let me point you to this forum post where I describe the steps it takes to bring the SyncDBfromSource AssemblyLine -- the AL that forms the entire API into Connections' Profiles registry -- into the TDI Debugger in order to step through it for purposes of troubleshooting and adventure:

Google Group post about how to debug the Connections TDI ALs (near the bottom of the thread)

My wordy response is at the bottom of this thread. If you have an AssemblyLine that you wish to debug, but are still unable to (even after video #6 here) then let me know!

...now I remember

Remember to check out the FormEntry Connector. This is the perfect companion to the AL Debugger, letting you set up an infinite feed while you interactively set up Attribute values in order to exercise every corner of your AssemblyLine logic.

Plus, you can map out to the entryRawData parameter when this component is used in a Connector Loop, letting you use multiple Parsers on the same bytestream.

Very handy indeed!

New Component and Library .jar Files

Generally, adding a new component to TDI's library amounts to either 1) copying the relevant jar files to the [TDI Installation Folder]/jars sub-directory; or 2) leaving these files where they are and instead telling TDI where to find them.

This latter is done by setting the com.ibm.di.loader.userjars property found near the top of the solution.properties file which is located in your Solution Directory.

According to the instructional comments found in the property file, you can set this property to a list of directory paths, or a list of files, or any combination of the two. For example,

com.ibm.di.loader.userjars=C:\Notes;CustomJars;C:\Drivers\jdbcDrv.jar

Using this property lets you keep custom jar  files either in place, or in a common area like the CustomJars folder I use. This can be preferrable to copying them into the TDI product installation folder.

Due to the mystical dance of Java loaders, sometimes these library files may need to be available to other loaders, in which case you end up having to do one or both of the following:

a). Modifying the PATH in the ibmditk and ibmdisrv files to include additional folders*.

b). Modifying the CLASSPATH line in these files for this same purpose.

I have several sub-folders to my own CustomJars directory, each containing the library files associated with a particular data source, like "Maximo", "Lotus" and so forth. TDI's loader checks all sub-folders under root directory specified, making it simple to keep these organized and backed up.

Note: libraries can be in zip files as well.

*Note: For Domino integration you will want to put the Notes path first: SET PATH=c:\Notes;%PATH% - since a little bird told me that only the first 128 chars are used by the Notes loader. At least for older versions.

...and I know there was something else I wanted to mention...

Exceptional Solutions

There are two types: those that go beyond the call of duty to deliver scalable, available and maintainable integration; and those that are defined by the stack dumps of unhandled exceptions.

How do you keep an AssemblyLine in the air? By catching and dealing with exceptions yourself. This is broadly done in two ways:

1) By putting code in Error Hooks. At the very least, you should log the error. I tend to use a script function to handle this.

function logerror(msg) {
  task.logmsg("ERROR", "@@ERROR - " + msg);
  task.logmsg("ERROR", "@@ AL[Component]: " + task.getShortName()
                        + "["  + error.getString("connectorname") + "]");
  task.logmsg("ERROR", "@@     Operation: " + error.getString("operation"));
  task.logmsg("ERROR", "@@       Message: " + error.getString("message"));
  task.logmsg("ERROR", "@@     Exception: " + error.getString("exception"));
}

Of course, no error report would be complete without a dump of the various Entry objects available -- at least work and conn. Unfortunately, the dumpEntry() call does not let you set the log level. So I made my own:

function attList( e ) {
   if (typeof(e) == "undefined" || !e)
      return;

   var attnames = e.getAttributeNames();
   java.util.Arrays.sort(attnames);

   var str = "";
   for (var name in attnames) {
      var att = e.getAttribute(name);
      str += "@@ " + name + ": ";

      for (var i = 0; i < att.size(); i++)
         str += att.getValue(i) + " | ";

      str = str.substring(0, str.length-3) + "\n";
   }

   return str;
}

function dumpEntry( logLevel, e, name ) {
   if (typeof(e) == "undefined" || !e)
      return;

   task.logmsg(logLevel, "\n@@ ******** Entry Dump: " + name + " ********\n"
                           + attList( e )
                           + "@@ ***************************************\n")
}

And then added it to my error function, making sure not to throw any exceptions whilst calling it:

function logerror( msg ) {
  task.logmsg("ERROR", "@@ **** ERROR - " + msg);
  task.logmsg("ERROR", "@@ AL[Component]: " + task.getShortName()
                        + "["  + thisConnector.getName() + "]");
  logval("operation");
  logval("message");
  logval("exception");
  if (typeof(work) != "undefined" && work)
     dumpEntry("ERROR", work, "Work");
  if (typeof(conn) != "undefined" && conn)
     dumpEntry("ERROR", conn, "Conn");
  if (typeof(current) != "undefined" && current)
     dumpEntry("ERROR", current, "Current");
}

Now it's easy to customize the format of the messages, as well as include other functionality if needed; for example, using a Passive Connector to write out error information, or using java.lang.System.out.println() to print messages to the command window where the TDI Server was started from.

2) Surround any script that can throw exceptions (like library calls) with a try-catch block:

try {
  makeSomeCall(); // if this fails, catch below
} catch (exc) {
  task.logmsg("@@ Error: " + exc);
}

If you don't handle it, then your AssemblyLine will stop running.

I'm just saying...

Integrated Service Management (ISM) leverages TDI

TDI is bundled with IBM's ISM offerings to provide integration services for deployment of solutions like TADDM (System/Asset & Relationship Discovery Tool), CCMDB (Change & Configuration Management DB) and TSRM (IBM's Service Desk and Service Catalog product).

Here's a TSRM scenario captured on film:

I didn't always like scripting

Not back when Johan first pulled me into Metamerge. Even though I sometimes just wanted to break free of the Rube Goldberg-esque world of AL construction and do something quick and direct, like:

if (error.getString("status") != "ok")
  task.shutdown(); // Invokes AL Shutdown Request Hook

My issue with scripting was that JavaScript code is "opaque" in TDI; there is no Hook flow through it, making it harder to troubleshoot and test. Furthermore, hiding code away in Hooks doesn't help AL legibility. More importantly, new TDI users often had no previous scripting experience.

And then we added real AL debugging and everything changed.

As far as scripting skills go, trial and error is how I learn. The Debugger turns an AL into an interactive tutorial where I can try my hand at executing any series of code snippets, entered one-by-one into the JavaScript commandline (or selected from the history drop-down) and get immediate feedback in the Log Output window. I can play around with stuff until it feels comfortable. If I do something that bombs with an exception, the Debugger catches and displays it, but the running AL is unharmed

So I found myself doing more scripting...

To make my ALs easier to read and debug, I started using Script components (SCs) to define functions that I called from my Hooks. Larger chunks of code are split up into separate, clearly labeled SCs. For example, the snippet at the start of this article would have appeared in an SC called something like Stop AL On Error.

Then I fire up the Debugger, step from script block to script block (since you can't step into them...yet) and examine work and conn, inspect script variables, call any functions and just generally ensure the sanity of my AssemblyLine logic and the health of the data it's working with.

The QA of my solutions improves as well. Since the AL Debugger lets me change any value or invoke any function, it's easy to trigger and test particular parts of my code. I confess to even using the system.skipTo() call - the infamous TDI GOTO command - to move the "Execute Me Next" relay baton around in my AL.

Hand on heart: TDI's interactive, try-test-refine approach lets me sculpt solutions quickly and visually by tooling around in the Debugger. This is where I learned most of what I do about JavaScript, as well as TDI in general, and REST/APP, XML, Domino, DB2, TDS, TSRM... The list goes on.

So the first time I had to deal with Domino-specific Java types, I configured a Notes Iterator, fired up the Debugger, stepped to the After GetNext Hook and then told the Attribute value in question to tell me its Java class name:

rev = conn.getObject("$Revisions"); task.logmsg(rev.getClass().getName())

Which I then looked it up in some Domino Java Classes documentation that Ken Lin pointed me at (thanks again, Ken). I also used system.dumpJavaClass(rev.getClass().getName()) to quickly get the list of publicly available methods:

[1:29:46 PM CEDT] E: system.dumpJavaClass(work.getObject("$Revisions".getClass().getName()) -> true
CTGDIS827I Class name: lotus.domino.cso.DateTime.
getDateTimeData ();
setJavaDate (java.util.Date);
setJavaDate (java.util.Calendar);
makeRDate ();
doAdjust (int, int, boolean);
getRDateTime ();
markInvalid ();
getParent ();
toJavaDate ();
         ...

Bingo! I could create a java.util.Date object by simply calling the Domino DateTime's toJavaDate() method.

Then I asked the JDBC Connector to query the schema of the DB2 table, learning that the target column for this value was handled as java.sql.Timestamp, and I could easily create one of those like this:

new java.sql.Timestamp(rev.toJavaDate().getTime())

Armed with Google in the one hand and my Norwegian Army Knife in the other, I happily whittled away at the solution, learning and correcting its design as I go.

The same goes for figuring out how to implement a REST-based web service, or scraping web pages for information, or provisioning accounts in Domino, Connections or Content Manager.

Scripted components, e.g. Connectors, Functions or Parsers, are trickier to debug since these execute in their own Script engines- but not impossible. Even though the Debugger gives you full access to the Script environment of the AssemblyLine itself, it can't access those of your scripted components. In order to debug their code, you need to bring it into the AL's environment, preferably as a series of SCs.

Here's an example of one such AL. I make sure my functions get defined first, just in case I have initialization code that makes use of them. Notice how my Test script is empty. That's because I do my testing interactively from the Debugger. I just use Test as a spot to set my breakpoint.

In this screenshot I am exercising the getNextEntry() function of Iterator Mode. Of course, selectEntries() has already been called by the Variables & Init SC. On the same JavaScript commandline I also do a dumpEntry().

e = getNextEntry(); task.dumpEntry(e);

You'd be surprise how much code you can squeeze into a single line :).

For example, I remember one project where an input data source was giving us over 150 Attributes. I used this one-liner to get a sorted list:

atts=work.getAttributeNames(); java.util.Arrays.sort(atts); for (a in atts) task.logmsg(a)

TDI jockey Jon Elwood is currently building an AJAX interace to launch and monitor the progress of his AssemblyLines. You can imagine how handy the Debugger is for examining the incoming HTTP requests and interactively testing out return snippets of XHTML (or XML/JSON) by simply setting the value of the http.body Attribute before End-of-Flow. Jon will be publishing a HowTo write-up on this that I am looking forward to.

So I guess that wraps up what has turned into an Ode to the AL Debugger. Just felt the need to share :)