TDI keyboard shortcuts

If you're like me then you're lazy. For example, I'd rather spend the 8 hours making an AssemblyLine than the 1 hour of tedious work it saved me from.

I'm also not fond of mousing about. To limit this while TDI'ing, I've opened Window > Preferences > Keys in TDI and bound the following shortcut keys in All Windows and Dialogs:

• Ctrl+R - Run this AL
  
• Ctrl+D - Debug this AL
• Ctrl + Down Arrow - Step Over
  
• Ctrl+Right Arrow - Step Into
  
• Ctrl+Shift+Down Arrow - Continue
• Ctrl+C - Stop this AL run and/or debug session
  
• ...

And I cycle between AL components by typing the first letter of their name.
So my index fingers never stray far from F and J. Then I can squeeze in a few more dev iterations.

And today it's -10C again. Gulf Stream where are you?!

Loop a Connector today

If you've never tried Connector Loops you should. Especially when you find yourself stuffing initialization scripts into your Prolog Hooks - because that's the only place where you can do something before the Feed Connector (e.g. Iterator) kicks in and reads the first entry; Or if you're parsing data entries out of Attribute values (like XML/SOAP, JSON, ...) being returned by your Feed Iterator.

Let's look at pre-Feed handling first. When your solution requires initial processing to happen before the Feed section engages, why not skip using the Feed section altogether? Instead, you implement your own Feed-Flow loop down in your AL Flow section. You do this with a Connector Loop and attach any Iterator or Lookup Connector you want to it. Now add components under the loop, mix well, and violá you have a mini assemblyline baked into your AL. How handy is that. And you can have as many as you need, with the results of one Loop feeding into the next one. You can also nest them:

FOR-EACH FileInDirectory
   FOR-EACH EntryInFile
      ...

As an observant reader you're thinking, sure, but how does the inner Loop know which file to read for every cycle of the outer Loop? I'm glad you asked that.

One of the cool things about Connector Loops is their "Connector Parameters" tab where you leverage your Attribute mapping skills to configure the connection, for example mapping to the filePath parameter of a File Connector. By default the Connector initializes each time the Loop starts up, and this is perfect for the file example above. However, if the inner loop were talking to a secure LDAP or database server, the cost of negotiating the connection each time is wasteful. So you press the "More..." button and tell the Loop to skip (re)initialization and just do the query. Ok, that was a couple of cool things :)

Digressing slightly, there has been some confusion, particularly among new users, in differentiating between Iterator and Loop modes. We all learn after a while that they are conceptually the same: a search query is made and results returned. But while the Iterator gracefully deals with 0, 1 or many entries returned, Loop mode bails, leaving the handling of 0 and many to the user (through the ever-popular mandatory Hooks: On No Match and On Multiple Found). Despair no longer. The Connector Loop handles this for you. You can still code the On No Match or On Multiple Found Hooks, but these are not mandatory for the Loop. So while Lookup mode is great for checking assumptions about data uniqueness and presence, with a Connector Loop you get the best of both worlds.

Moving to the second part of this lecture: Connector Loops are also how you Iterate entries found in the value of an Attribute. The secret ingredient is the Form Entry Connector which uses a Parser to iterate over entries found in the value of its Raw Data Text parameter. So you add a Connector Loop, attach the Form Entry Connector, map the Attribute with the payload to the 'entryRawData' parameter, choose your Parser and away you go. What more could you ask for?

Seriously, write me if you have more to ask for.

What Is The Solution Directory?

The Solution Directory is the current working directory for an  SDI/TDI Server.

Each TDI Server running on the same machine should have it's own Solution Directory (or SolDir for short). If you don't know where your SolDir is pointing, simply open defaultSolDir.sh (or .bat) found in the bin sub-folder of your installation directory (e.g. V7.2). This script sets the TDI_SOLDIR environment variable that controls Server behavior.

     export/set TDI_SOLDIR="some/place/on/your/disk"

Both the ibmditk script to launch the CE, and ibmdisrv that starts a Server call defaultSolDir, along with other scripts in this folder. Like javaHome.

The Solution Directory holds configuration settings, including Java settings, for a Server. These settings are found in a file named solution.properties.

After reading solution.properties file, the Server then reads the global.properties file in the etc sub-folder of the TDI installation directory. Settings in solution.properties override those in global.properties.

If you open solution.properties in a text editor you will find two important properties that are personal to any running TDI Server:

• The port that the Server listens to for RMI API calls: api.remote.naming.port
• The port that the Server listens to web calls: web.server.port (dashboard, FDS, CURI and REST APIs)

This is also where you provide the paths to keystores and truststores, as well as credentials to access these.

By default, the one for the TDI Server's APIs is named testserver.jks and is found in the SolDir. Certificates used for SSL connections in your solutions are under the serverapi sub-folder and are named testadmin.jks.

NOTE: The default password for testserver.jks is server, and for testadmin.jks is administrator. To import your own certificates use keytool (found under the jvm/jre/bin sub-folder of your TDI installation directory) or the Key Manager button in the TDI CE (Configuration Editor).

NOTE II: If you remove all properties in solution.properties that you are not explicitly setting, then a TDI version upgrade might change global.properties and could then affect settings for your Server.

Leveraging the Solution Properties folder to make your solutions more portable

As mentioned above, the SolDir is the current working directory for TDI, both the CE and the Server. Any Config xml files placed in the configs sub-folder of the SolDir will be automatically loaded when the Server starts up. If they have Schedulers, then these are activated as well.

Another approach is to make a Project folder in the SolDir. Solution filepaths should point in this directory using simply the name of the folder. For example, if you project is named "MyProject" then the path to the Property Store file would be:

    MyProject/MyProject.properties

The next step is to instruct the CE where to write the compiled Config xml, which is written each time you save a change. By default this file is written as:

    MyProject/Runtime-MyProject/MyProject.xml

You can change this easily to have the Config written to your Project folder. Simply right-click on the name of your Project in the CE Navigator View and select Properties.




Choose Tivoli Directory Integrator properties at the bottom and set the Linked File parameter to a relative path that points to a config file under this same folder, like MyProjext/MyProject.xml.

NOTE III: If you use forward-slashes for paths instead of backslashes, your solution runs on Windows and all flavors of *nix (including mainframe).

Make sure you also put any other support files in this folder so you can easily move the solution by simply compressing this directory and then uncompressing it in the SolDir of the target TDI installation.

The ReadMe.txt file you make in your project directory could look something like this:

To install this solution:

1) Unzip QRTrigger.zip to your Solution Directory;

2) Edit QRTrigger.properties to configure connections

3) Run the solution with this commandline:

ibmdisrv -c QRTrigger/QRTrigger.xml -r QRListener

Violá, you have something that you can hand to the operations team for deployment, pass on to a colleague, and even publish it online (and sending me the link :)

Pointing to a specific Solution Directory

Stephen Swann pointed out to me that I should mention the scripts used to start up the CE (ibmdisrv) and the one for the Server (ibmditk). Each accepts the -s argument for specifying the SolDir to use:

    ibmdisrv -s /MyOtherSolDir -c MyOtherProject/MyOtherProject.xml -d

NOTE IV: If the SolDir does not exist then the Server will set it up the first time it starts.

Your Eclipse workspace

So the SolDir is a tool for making solutions portable, and it is independent of where you put your TDI workspace. The workspace is where the Eclipse framework housing TDI organizes your source files, and the directory structure there mirrors what you see in the CE Navigator. It is not recommended that you work directly with source files, apart from tying them to SCM like git.

What if..?

But what if you chose the program directory or "current location" when you installed, or want to move the Solution Directory for some other reason? No problem. Simply edit defaultSolDir.

Changing TDI_SOLDIR is only half the job. You also have to instruct the CE where the new SolDir is when it automatically starts up your local TDI Server. This is visible in the Server view at the bottom left of the CE, and is called 'Default'. Can change the Server Launcher settingsby right-clicking on 'Default' in the Server view and selecting Properties and making the change here too.

Final note about the Servers view

The view is not called Servers for nothing. TDI lets you define new Servers, allowing you to start multiple Servers on your PC, or connect to those running on other platforms. You can point a Project at any Server by editing its settings (right-click > Properties). This lets you test and debug the ALs there. Each Server has its own Solution Directory settings, so using relative paths means you can build your ALs on your laptop and then run them elsewhere. You will also want to use forward slashes for filepaths, since backslash only works on Windows and TDI runs a lot of other places as well.

You can also right-click on a Server and select Debug Server. This opens a blank debugger tab which then catches an ALs that are started on this Server.

And while I'm on the soapbox here are some additional tips: Define Logging for where your task.logmsg() messages are written; Code Error Hooks and log enough info for manual intervention; Write status messages to the command line by using main.logmsg(); And set up Connector Auto-Reconnect (Connection Errors).

Clustering, HA and Scaling, Oh My!

Ok, here comes some ramblings on making better TDI solutions. Risk reduction is always a matter of expense versus exposure, and this approach requires an upfront investment and adds lots of moving parts, so it's not a size that fits all and pays most dividends for larger projects, or for series of similar smaller ones. That said...

You start by deconstructing a solution into a set of individual service AssemblyLines:

1. Error Handler - First consider that all error hooks/scripts in all other ALs dispatch to a central error queue. Then you have this Error Handler AL that continuously iterates this queue and does logging/alerting/reacting as needed. This AL also emits a periodic heartbeat to an event queue*.
   
   * That's right, 2 queues. And note that you will want persistent queueing like MQ, the System Store/DB, files, etc.
   
   
2. Event Handlers (plural) - This set of service ALs catch events - e.g. detecting changes; listening for incoming messages, mail or requests; polling for new/changed files, and so forth. Each AL pushes caught events to the event queue, along with its own heartbeats. As mentioned above, all errors are dispatched to the error queue.
   
   
3. Workers (plural). These each grab the next relevant event off the queue and then performs a required action, like writing detected changes to a single target (e.g. for a sync solution you would have one Worker AL to write to Domino, one for AD, one for SAP, etc.), passing events to other systems (i.e. switching), performing searches and building responses (rss/REST/SOAP/...) or whatever. Each also reports status through the event queue and problems to the error queue.
   
   
4. Heartbeat Monitor - Polls the queues to make sure things are happening, for example that events are being processed in a timely fashion, and that heartbeats are received (and cleaned up). If the Error Handler is down, it does its own alerting and logging (it can even send events to AMC or a backup TDI Server).

This approach is suited for unit testing and provides better solution availability and maintainability than one-stop-shopping ALs tend to do. It also scales easily - just run more ALs, using inter-AL comms for coordination.

But it's definitely not for the lighthearted, or those not comfortable in the AL Debugger. However, if you do it right, you end up with a reusable the AL service framework.

And if you send it to me for publication then I'll send you a limited edition, orange plastic Metamerge pen :)

These really are great pens :)

Why TDI (version 2)

Here comes an updated version based on feedback:

---

All organizations want to reduce risk and maximize return on current investments, especially during tough times when spending for new IT dwindles and resources are redirected towards improving existing infrastructure. Given that infrastructures tend to be the result of evolution - in other words, survival of the highest switching cost and not a grand architectural plan - they seldom accommodate adjustment, much less the introduction of new software or services. Furthermore, the risk of failure or miscalculation can equate to disruption in business operations.

Enter the newer agile development methodologies whose mission is to solve problems incrementally, constantly adjusting design to meet uncovered constraints and correcting project goals to meet discovered requirements. A mindset and approach that bridges the need for executive oversight with the realities of technical innovation/renovation. TDI is designed from the bottom-up for this purpose.

That's why TDI is bundled with a growing number of IBM products and being adopted by more and more of business partners: It uniquely "closes the gap" between product functionality and client expectations. Rapid try/test/refine cycles with TDI reduce projects to manageable steps that provide feedback for stakeholders, enabling risk re-assesment and course correction.

---

In particular, thanks to Avery Salmon for stepping up to the plate here :)

TDI Elevator Pitch

I have been asked many times to write a blurb for a TDI event (e.g. a LabJam) or conference session, with specific instructions to not use much techie-speak. After numerous iterations, and much careful listening - particularly to our architect, Johan Varno - I'm relatively pleased with the following Why TDI? pitch:

---

Most organizations want to reduce risk, especially during tough times when investment in new IT dwindles and resources are redirected towards improving existing infrastructure. Given that infrastructures tend to be the result of evolution -- in other words, survival of the highest switching cost -- and that most evolutionary steps are painful, there is resistance to change.

Enter the agile development methodologies whose mission is to deliver value in stages so ROI can begin sooner and stakeholder feedback shapes each step. A mindset and approach that bridges the need for executive oversight with the uncertain realities of technical innovation/renovation. TDI is built for this purpose.

That's why TDI is bundled with a growing number of IBM products and being adopted by more and companies and integration professionals. It's a customization and deployment tool that uniquely 'closes the gap' between a product's feature list and client expectations. TDI's rapid try/test/refine approach helps reduce complex projects to manageable sprints, enabling continual risk re-assesment and course correction.

---

Comments, suggestions and even ridicule is anxiously awaited :)

Dealing with Errors

Back from summer holiday and starting to remember what I do for a living again. I thought I'd start with a quick post about error handling.

As you already know, the How-To writeup here talks about the mechanics of TDI error flows and Hooks, and from the Configs I've received, people are at the very least enabling Error Hooks so that their AssemblyLines don't stop. A good start.

However, logging enough information to manually recover from a problem is even better.

So what is enough information? That all depends on what your AL is doing and where the error occurs. If the AssemblyLine fails then you get a Java stack dump preceeded with the component name in brackets, along with the operation and underlying error returned from the target system. However if you catch the error by enabling Error Hooks then you don't see this in the log. This information is still available though and you can find it in the error variable.

Just like work and conn, error is an Entry object and it contains a number of Attributes provided by the TDI Server. These are described in the online docs and you can access them - for example, to write to log output, or to implement handling logic - using the getString() method. You can also do a task.dumpEntry(error) to easily get all the information printed to the log.

Sometimes an exception is not an error, but instead an unexpected situation like finding no match for a Lookup or Delete, or finding multiple entries (Lookup, Delete or Update modes). This results in mandatory Hooks like On No Match or On Multiple Found being invoked, and just like with Error Hooks, you have to tell TDI how to proceed or the AL stops. In these cases it is your Link Criteria that is either too narrow or too broad. As a result you will want to log enough information to determine why this is happening, and dumping out error will not get you there.

Instead, you should print out the value of work Attributes used for your Link Criteria: the right-hand operand for simple Link Criteria, usually preceded with a dollar sign ($). That way you can identify which data entries are giving you problems.

For those cases where a lookup is expected to either find no match or multiple then another approach is to add a Loop (Connector Loop) with your Connector attached. The Loop will then cycle once for each Entry found, driving these to components under this Loop. No Hook coding required.

Finally, in the case of connection errors you should note that all Connectors have a Connection Error tab where you can enable the Auto-Reconnect feature. While this is generally not interesting behavior for an initialization failure, if comms drops in the middle of AL processing, Auto-Reconnect can be used to quietly re-establish the connection and your AssemblyLine continues processing as if nothing had happened.

Note that for Iterators this will result in the Select being re-issued as well. As long as Iterator state is maintained - done automatically for all Change Detection Connectors, and you can easily implement this yourself for most other Connectors - processing will resume where it left off. If state is not used to re-align the Iterator then your AL will be reading from the start again.