Sharing Secrets, aka Synchronizing Passwords

There are plenty of good reasons for synchronizing credentials, like migrating a Portal from one identity store to another, or as an alternative to single sign-on - such as making sure that the Windows password gets put into (for example) an LDAP directory. But while transferring user id's is pretty straightforward, synchronizing passwords is not. This is because passwords are encrypted information, and typically encrypted using a one-way algorithm. So even if you can read the password out of a data store, it will be a binary value that cannot be copied as-is to another authentication system. This pretty much prevents you from using standard change detection and propagation techniques.

Unless of course both the source and target systems employ the exact same encryption algorithm. In this case you should be able to read in the password attribute and then write this binary value to the target. Note that the target system must also support doing a direct write to this attributes -- i.e. without encrypting it again. Note also that in order to read in an attribute with a binary value (like userPassword ) using the TDI LDAP Connector, you must first list it in the Binary Attributes parameter.

There are also technologies available for catching passwords in clear text, when they are changed. TDI provides plugins for a number of popular identity stores, like IBM Tivoli Directory Server, Domino (HTTP InternetPassword only), Microsoft Active Directory and SunOne. These plugins securely capture passwords, making them available for processing by TDI AssemblyLines.

But that only deals with passwords as they are changed, so we're back to the original question: how to synchronize (or migrate) existing, already-encrypted passwords?

That's what I love about TDI: if one path is blocked (e.g. decrypting passwords) then pick a protocol or api and try another route. So in this case, instead of trying to pull passwords out of a system, hook TDI into the login/authentication mechanisms and let clear-text passwords be pushed to your AssemblyLines.

One example is with Tivoli Access Manager for e-Business. TAMeb can be configured to securely call out to TDI with id/password during authentication. This not only allows TDI to check these credentials against multiple identity stores thereby extending the configurability of the authentication service, but it lets you write these clear-text passwords to any number of targets, provisioning users on-the-fly with the same set of credentials. The end result is password synchronization on demand as these credentials are applied.

I've also seen similar solutions where callouts were made directly from the Portal login service, again providing the same multi-backend capability plus the opportunity for sync'ing passwords. One creative solution even re-directed the login page to a TDI AssemblyLine using the HTTP Server Connector (mini web server). TDI provided the login page, verifying (and sync'ing) credentials, directing the connection back to Portal once the user was authenticated.

So while you can't pry encrypted passwords out of a store, there are ways of working around this limitation -- given the right tools :)


Addendum: Borrowed from a TDI newgroup post by Christian Chateauvieux (of Metamerge and IBM fame) outlining another solution to this challenge: The Tivoli Directory Server pass-through authentication plugin, shipped with TDS 6.1.

This is how it works: On an authentication failure, TDS can be configured to call out to the plugin which attempts to authenticate the same user in another directory. If the password is validated, TDS stores it so that it won't have to look it up next time. Cool solution, as long as the cached passwords are invalidated regularly enough. Or you synchronize them when they are changed.


Addendum II: You could extend this pass-through feature to support multiple backends, or RDBMS's, Notes db's and even flat files. Simply use the TDI LDAP Server Connector to build a simple LDAP simulator AssemblyLine. Just catch the LDAP bind/rebind operations, then check the credentials (and/or migrate them) against whatever backends you want using the appropriate Connectors and Java calls.

If you want TDI to return mutliple values for an Attribute then you
just need to return a JavaScript array:

   var mValues = Array();
   mValues[0] = "first one";
   mValues[1] = "second one";
   mValues[2] = "you get the picture";

   ret.value = mValues;

This would of course be in a scripted Attribute Map.

If it's just literal values you want to return (like objectClass) you can easily do it like this:

   ret.value = ["top", 
                "person", 
                "organizationalPersion", 
                "inetOrgPerson", 
                "dominoPerson"];

This method does not work when you are adding Attributes to an Entry directly from script. Then you need to either add values to an Attribute and then put it in the Entry:

   myAtt = system.newAttribute("Selection");
   myAtt.addValue("eenie");
   myAtt.addValue("meenie");
   myAtt.addValue("mynie moe");
   work.setAttribute(myAtt); // works with any Entry

...or you add the Attribute and then the values to the Entry:

   ent = system.newEntry();
   ent.setAttribute("Selection","eenie");
   ent.addAttributeValue("Selection","meenie");
   ent.addAttributeValue("Selection","mynie moe");

If instead you want to take a multi-valued Attribute and return it as a comma separated string (as some APIs require), then you could do this using an Attribute Loop and a couple of AttMap components. However, the easiest way is with a snippet of script:

   valStr = "";
   for (i = 0; i < myMultiVarAttribute.size(); i++)
         valStr += ";" + myMultiVarAttribute.getValue(i);
   ret.value = valStr.substring(1); // get rid of the first ";"

If you need the values quoted then the script would be:

   valStr = "";
   for (i = 0; i < myMultiVarAttribute.size(); i++)
         valStr += ";\"" + myMultiVarAttribute.getValue(i) + "\"";
   ret.value = valStr.substring(1); // get rid of the first ";"

Of course, there could be a double quote (") in the value itself, in which case
you'd need to know how the target system wants this encoded. Perhaps simply by
wrapping the value in single quotes(?) I'd ask Google :)

What do JDBC commit/rollback and LDAP rebind have in common?

Both are features of their respective protocols and both are available for use from script. All you have to do is get hold of the Connector Interface, aka the "CI".

var ci = thisConnector.getConnector();

Note that "thisConnector" is a handy variable that always references the current component. Also, for TDI versions prior to 6.1.1 you don't have the getConnector() method and must reference the connector member field directly:

var ci = thisConnector.connector;

Once you have the CI, you have direct access to technology/vendor/platform-specific functionality, as well as the standard methods that all CI's must implement in order to support Connector Modes. However, working with the CI directly will not invoke any Hook flows.

Hook flow execution is initiated when an "AL Component" method is called, like getnext() or lookup(). These standard AL component functions can be found in the TDI JavaDocs (Help > Low Level API) . If look at the class com.ibm.di.server.AssemblyLineComponent you'll see these calls. For example, the update() method invokes the Update Mode Hook flow which performs a a Lookup and then branches to either Add or Modify. The actual read and write operations are provided by calling methods in the attached CI: findEntry(), putEntry() and modEntry(), in that order. Calling any of these CI functions directly from script means bypassing the Hook flow logic provided by the AL Component.

In addition to the methods required to support desired Modes, a CI can contain any number of supplementary functions. The JDBCConnector class offers jdbc-specific calls like commit() and rollback(); The LDAPConnector lets you rebind() and getServerInfo().

But the fun doesn't stop here. Many CI's can also return a handle to the underlying system and vendor libraries. For example, the JDBCConnector offers a getConnection() call to return the underlying driver's Connection object, implemented according to this standard interface:

http://java.sun.com/j2se/1.5/docs/api/java/sql/Connection.html

plus vendor-specific additions.

Another example is the Notes Connector that lets you get a reference to the currently opened Domino Session, Database or View object. Then you can make direct Domino API calls to do stuff like re-certify users or invoke AdminP processes.

And, of course, Functions and Parsers have Interfaces too.

Made-to-order music videos

I found this very cool online service and made TDI ready for MTV ;)

BlueGlue remix #2

Guess I'll need to add a concert t-shirt here as well then:

Prepare to be integrated...

Why throw multiple ALs at a single problem?

I get this one occasionally and it's a very valid question from a simplicity standpoint (a TDI mantra). However, there are good reasons for dividing a workload up among multiple AssemblyLines, and better performance is one of them -- as long as the task at hand allows for multi-threading; For example, data movement/migration/sync where the ordering of updates is not significant as long as all are processed. Here is a whitepaper that outlines a number of patterns that lend themselves to multi-AL solutions:

Performance Best Practices Paper for TDI 6.1 and 6.1.1

Another reason is to improve availability. Having several simultaneous worker ALs as mentioned above also means no single point of failure. Imagine a scenario where one or more AssemblyLines (on one or more TDI Servers) read source data and write to MQ. At the other end of this secure, persistent data pipe is another set of ALs picking up these messages on a first-come-first-serve basis. If an AssemblyLine fails then the integration service is degraded; not dead.

One oft-used multi-AL technique is to have a secondary "launcher" AL with a "while(true)" loop that starts the primary long-running AssemblyLine, waiting for it to complete - which it should never do (in theory). If for some reason the mission-critical AL stops then postmortem failure handling, including logs/alerts can be made by the calling AL; Finally, the AL can be re-started again in the next cycle of the loop.

I have also seen complex logic divided up into smaller, simpler ALs for reasons of simplicity and maintenance. The developer specs out each step in their integration flows - along with constraints, schema and invariants - and them implements these as individual ALs. This allows for incremental upgrades to a solution as well as unit testing.

Finally, entire AssemblyLines can be exposed as components (called "Adapters") facilitating sharing and reuse. You can find more on this topic here:

Introducing Adapters with Tivoli Directory Integrator 6.1

Online backup of the System Store (Derby/Cloudscape)

Follow these steps to build an automated System Store backup utility. Note that your System Store must be set up for Network mode.

1. Create a new AL (call it "BackupSystemStore")


2. Add a JDBC Connector (call it "Derby") in AddOnly mode and put it in Passive state.


3. Configure the JDBC Connector to point at the System Store. The easiest way to do this is to click on the label of the JDBC URL parameter. In the resulting dialog, use the Expression drop-down to choose the com.ibm.di.store.database property. Or you could just enter this into the Expression field:
   
   {property.Solution-Properties:com.ibm.di.store.database}
   
   Now the JDBC URL parameter will be dynamically configured using the named property from your Solution-Properties (solution.properties). This is the same property TDI uses when working with its System Store database.


4. Repeat step 3 for the other mandatory parameters:
   
   
   
   • JDBC Driver - com.ibm.di.store.jdbc.driver
   
   
   • Username - com.ibm.di.store.jdbc.user
   
   
   • Password - com.ibm.di.store.jdbc.password
   
   Since the Connector is not in Iterator mode, we don't have to set Table Name (i.e. no selectEntries).


5. Add a Script component to the AL with this code:
   

   
   // Make sure the path uses forward "/" and ends with one
   // as required by SQL syntax.
   //
   function sqlPath( pth ) {
      pth = system.mapString(pth, "\\", "/");
      if (!pth.endsWith("/"))
         return pth + "/"
      else
         return pth;
   }
   
   // Here is the function for backuping up the System Store
   // to the specified directory.
   //
   function backupDB( backupdir ) {
      // Get today's date as a string:
   
      var todaysDate =  new java.text.SimpleDateFormat("yyyy-MM-dd");
      var backupdirectory = sqlPath(backupdir) +
                 todaysDate.format((java.util.Calendar.getInstance()).getTime());
   
      task.logmsg("Backuping Derby DB to " + backupdirectory + "...");
   
      res = Derby.getConnector().execSQL("CALL SYSCS_UTIL.SYSCS_BACKUP_DATABASE('" +
                                            backupdirectory + "')");
   
   
      if (res == null || res == "")
         task.logmsg("Done!")
      else
         task.logmsg("** Error: " + res);
   }
   
   // Here the function is called to perform the backup
   // to C:/_Backup/ (which is created automagically the
   // first time).
   //
   backupDB("C:\\_Backup\\");
   

Now you can set up a Scheduler AL with the Timer Connector to periodically call your BackupSystemStore AssemblyLine and make a fresh backup.

Thanks to Boli of Ascendant for the link that made my day :) Backing up Derby databases. This is from the admin guide found with the other Online Derby docs.

Buried treasure

A favored pasttime of my youth was treasure hunting. Whether it was scouring a hillside for Easter eggs, rummaging in the attic of my grandparents' house or just cleaning out the pockets of my jeans. I have this same, if somewhat unusual relationship to troubleshooting error messages from TDI. The secret is knowing where to dig.

The first thing you need to do is scan your log output for the very first error message and stack dump, which will be the root cause of the failure. Subsequent error message are likely to be related to this initial fault.

Once you've found the first error then you can start digging. This is done by first splitting the text of the message into two parts: what TDI is reporting, and what the underlying library or system is complaining about. For example, TDI could be giving you a standard message like:

CTGDIJ001E
No default JDBC driver. The 'jdbcDriver' parameter must be set to use the JDBC Connector.

Here the entire prose of the error message comes from TDI. In this case there is no data source involved (yet) since the Connector itself can't be initialized without the JDBC Driver parameter being set correctly. These are also the types of errors that Google may be less than helpful for, and your best bet may be the discussion forum, the community website or the online docs.

To illustrate the type of message that a net search can help you decipher, let's look at a message I helped debug last week (formatted in bold and italics to illustrate my point):

10:12:03 [DB2_Update] CTGDIS810E handleException - cannot handle exception , initialize

Unable to obtain schema: com.ibm.db2.jcc.c.SqlException: DB2 SQL error: SQLCODE: -443, SQLSTATE: 38553, SQLERRMC: SYSIBM.SQLCOLUMNS; COLUMNS;SYSIBM:CLI:-805

In a case like this where the exception originated in a call to some driver or API, the first part of the message (italicized above) will help you find the point in your AssemblyLine where the failure occurred.

At the start of the error message is the name of an AssemblyLine component in brackets: [DB2_Update]. This tells you where the error occured in your AL. Immediately following the component name we can furthermore see TDI telling us that this unhandled exception occurred during initialization of the component. So far so good: we know where and when now. The next step is to discover why we are getting this exception.

That is where the text that I've formatted as bold comes in: this is the error that TDI received from the underlying RDBMS. Since it is a DB2 error (not specific to TDI) there is a much greater chance that someone else has fixed and documented this already.

So when I did a Google search with tidbits gleaned from this part of the message (e.g. "sqlcode -443 sqlstate 38553 cli -805") I uncovered plenty of relevant links, eventually leading me to a newgroup post that described a bind problem easily fixed with "bind db2schema.bnd". The search was over, the problem solved and I am ready for new adventures.