HTTP Adapter and Apache Keep-Alive issue

Problem.

If you notice a high rate of suspended messages due to communication failures between your BizTalk Server and an HTTP server, you should verify if the HTTP Web server you are accessing is Apache, Resin or another similar server of the UNIX / LINUX / Java family. Note that this could apply to other web servers as well, these are just examples I know of.

BizTalk Server always sends HTTP request with the HTTP header setting Connection: Keep-Alive. Moreover, BizTalk expects that the foreign HTTP server obeys to the header setting and does not check the Connection header setting received in the HTTP response.
To be more precise, the problem is double:

  • Apache is not obeying to the Connection HTTP header setting and always closes the HTTP connection after every request (as in an HTTP 1.0 fashion). The HTTP/1.1 protocol allows this even if it is not the prefered behaviour.
  • BizTalk Server does not react accordingly to the Connection header that is returned back in the HTTP response. This is wrong, following the HTTP/1.1 protocol definition, the client should always check the Connection header returned.

So, on one side there is a Apache closing connections for every HTTP request and on the other side BizTalk is expecting connections to be kept open between requests.

Actually, the source of the problem is not that the Web Server is Apache or not but related to how the Web Server is configured and how it behaves with the connection. Nevertheless, in my experience, this kind of problem only happened with Apache. But I also ever encountered 1 Apache server which kept the HTTP connection open between requests and so no fix was required for that case as no communication problem arose.

For this reason, if you have communication problems between BizTalk and a foreign web server, you should first check if the web server is explicitly closing the HTTP connection after every request.

There is a Microsoft KB that explains the issue and also how to fix it.

Identification of the problem.

As mentioned in the KB, it is important to make sure that your problem is exactly what the KB refers to, so, I will explain how I diagnosed it on my BizTalk Server.

Looking in the BizTalk Administration Console, I saw an abnormal amount of suspended service instances, most of them being of the type “Message”. I also noticed that all the suspended messages belonged to only a subset of our BizTalk applications.
There was clearly something wrong and by checking the Windows Event Viewer, I saw that some BizTalk send ports had an unusual failure rate at the communication level. This explained why so many messages were suspended as messages failing to be sent through a port will be suspended.
The Windows Event Viewer showed messages such as:

A message sent to adapter “HTTP” on send port “XXX” with URI “http://www.zzz.com/Service” is suspended.
Error details: Cannot access a disposed object.
Object name: ‘System.Net.Sockets.NetworkStream’.

And

A message sent to adapter “HTTP” on send port “XXX” with URI “http://www.zzz.com/Service” is suspended.
Error details: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.

After scratching my head on this issue for a while, I found out that all failing messages where those sent to a port using an HTTP adapter configured with an URI pointing to an Apache server. Other ports using the HTTP adapter but configured to access an IIS server were working fine!

I then used a network sniffer to see what was going on exactly and I saw something inconsistent in the HTTP headers:
The header of the HTTP request contained:
Connection: Keep-Alive
But the header of the HTTP response contained:
Connection: close
server: IBM_HTTP_Server/6.0.2.3 Apache/2.0.47

So, even if the HTTP request specifies “Connection: Keep-Alive”, the Apache Server still close the connection and replies with a “Connection: close” in the header of the HTTP response. For reference regarding the Connection HTTP header, see: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.10

Solution.

As it is not possible to change the Connection: Keep-Alive header setting in the HTTP adapter shipped with BizTalk Server 2006, I googled a bit and found out the KB 924638 I mentioned earlier in this article which provides 1 solution and 1 workaround for the problem.

The hotfix mentioned in the KB contains a library that makes it possible to modify the Keep-Alive configuration for the HTTP connection. However, as there is no option to set the header setting from the BizTalk Administration Console, it is required to create a custom pipeline which writes a KeepAlive property in the context of the message going through the pipeline. Later on, the HTTP adapter will read the KeepAlive message context property value and create the HTTP connection accordingly.

You can find here a zip file containing 2 Visual Studio projects, one for the Custom Pipeline Component and another one for the Custom Pipeline. In this custom pipeline, the KeepAlive property is not hard-coded and so its value can be set either from the BizTalk Administration Console at run-time or from Visual Studio at design-time.

To summarize, there are 3 steps to apply the solution:

  • Call Microsoft Support so that they email you the hotfix containing an executable which updates the Microsoft.BizTalk.HttpTransport.dll (as the hotfix is not directly available to download from the KB).
  • Create a custom pipeline which has a property that permits the user to specify the Keep-Alive setting.
  • Use the custom pipeline in the send ports that are pointing to web servers explicitly closing connections, such as some Apache web server.

BizTalk Server 2006 R2 released

BizTalk Server 2006 R2 has just been released.

The key new features differentiating the new release from BizTalk Server 2006 are:

  • Native support for Electronic Data Interchange (EDI).
  • Native support of Radio Frequency Identification (RFID).
  • While BizTalk Server 2006 integrates with Microsoft Office 2003 components and Windows XP (on a client OS), BizTalk Server 2006 R2 integrates with Office 2007 and Windows Vista.
  • Can make use of .Net 3.0 technologies such as Windows Communication Foundation (WCF) for messaging and Windows Workflow Foundation for BAM (Business Activity Monitoring).
  • New edition of BizTalk, called BizTalk Server Branch Edition. This edition is ideal for Hub and Spoke deployment scenarios for which only 1 BizTalk application is running in the Spoke nodes. Its official price is 1,800 USD; making it less than a quarter of the price of BizTalk Server Standard Edition. See BizTalk Server Editions for more details.

Both BizTalk Server 2006 and 2006 R2 supports only Visual Studio 2005 so far.
Visual Studio 2008 will be supported through a BizTalk Server Service Pack so that the BizTalk installation routine will be able to detect VS2008 and run with it.

For some, RFID is the most exciting feature introduced with BizTalk Server 2006 R2. The BizTalk RFID features include:

  • Data management capabilities.
  • Device abstraction and management capabilities.
  • Event processing engine that allows the creation of business rules and manage the execution of event pipelines for RFID events.

For more details about the new features of BizTalk Server 2006 R2, check the BizTalk Server roadmap.

Functoid and BizTalk Application Versioning Problem

1. Problem:

I encountered a problem difficult to diagnose related to deploying a BizTalk Server 2006 application and a functoid that have been compiled together but then deployed with different versions.
For example, let’s say you have developed and compiled a BizTalk application “A” version 1 using a functoid “F” version 1. Later on, you change some of the code inside the functoid and so you have a new version of the functoid, let’s call it functoid “F” version 2.
Note that by version 1 and 2, I do not mean the assembly version number; it is just a nomenclature I use to differentiate the versions. In my case, the assembly’s version number is fixed.
I often deploy new versions of functoids without having to recompile neither redeploy the BizTalk application but this time, the following error message appeared:

Microsoft.XLANGs.Core.XTransformationFailureException: Error encountered while executing the transform XXXX. Error:Unable to create the transform.. —> System.ArgumentNullException: Value cannot be null.
Parameter name: extension
at System.Xml.Xsl.XsltArgumentList.AddExtensionObject(String namespaceUri, Object extension)
at Microsoft.XLANGs.BaseTypes.TransformBase.get_TransformArgs()
at Microsoft.XLANGs.RuntimeTypes.TransformMetaData..ctor(Type transformBaseType)
at Microsoft.XLANGs.RuntimeTypes.TransformMetaData._creator(Type t)
at Microsoft.XLANGs.RuntimeTypes.MetadataCache._slowFor(Type t)
at Microsoft.XLANGs.RuntimeTypes.MetadataCache.For(Type t)
at Microsoft.XLANGs.RuntimeTypes.TransformMetaData.For(Type t)
at Microsoft.XLANGs.Core.Service.ApplyTransform(Type mapRef, Object[] outParams, Object[] inParams)
— End of inner exception stack trace —
at Microsoft.XLANGs.Core.Service.ApplyTransform(Type mapRef, Object[] outParams, Object[] inParams)
at Tourico.Tourico_SmartSearch.segment2(StopConditions stopOn)
at Microsoft.XLANGs.Core.SegmentScheduler.RunASegment(Segment s, StopConditions stopCond, Exception& exp)

2. Solution:

The message was not recorded in the Windows Event Log but as I catch and log exceptions in an exception handler; I could see the exception in my log.
As the message is rather vague, I spent quite a lot of time checking the maps and schemas used in the orchestration but found no clue.
I finally noticed that a functoid’s code had been changed and once I recompiled and re-deployed the BizTalk Application, everything worked fine!

I am not sure of the exact reason as normally you don’t need to redeploy the BizTalk application after having updated and GACed a functoid.
Anyway, if one day you encounter an error message of the same flavor, recompiling and redeploying the application might just work fine. Maybe this hint can help someone in the future 🙂

3. Note about exception handler in BizTalk Server 2006.

For the people who do not know, you just need to use a scope shape and its exception handler to have the equivalent of a try-catch block in a BizTalk orchestration. That gives you the opportunity to take action and/or log errors when exceptions are thrown from the orchestration.

To create an exception handler, drop a scope shape into your orchestration, right click on it and select “New Exception Handler”. Note that only scopes that are non-transactional or long running transactional can have an exception handler.
Here is how it looks like:
Scope shape with its exception handlerEverything in the first square (Scope_1) will be the “try” part and everything in the second square (CatchException_1) will be the “catch” part.