Error configuring BizTalk Server 2006

Today I had quite some troubles installing BizTalk Server 2006. As I have previously installed BizTalk many times without being stuck on a problem for so long, I thought I would blog about it. Who knows, it might help some other soul encountering the same issue.

My configuration was the following:

  • 1 Windows Server 2003 64-bit running SQL Server 2005.
  • 1 Windows Server 2003 64-bit with BizTalk Server 2006 installed but not yet configured.

Both machines had MSDTC and COM+ installed and running, a pre-requisite to install BizTalk on a multiple-server environment.
All the BizTalk Users as well as the Windows Account used to install BizTalk had the necessary rights and belonged to the necessary Groups. See a previous post of mine for more information; Biztalk 2006 multi-server installation guide.

So, after having installed BizTalk Server 2006, the BizTalk Server needs to be configured by using the BizTalk Server Configuration tool and there the weirdest thing happened: it failed, but not completely! Enterprise SSO (Enterprise Single Sign-On) and Business Rules Engine (BRE) components were configured successfully while Group and BizTalk Runtime failed.
Thinking I made some human mistake in the configuration, I retried 2 times but the same problem happened. On a side note, before retrying to configure BizTalk, I had to manually delete the BizTalkMgmtDb as it was created even as the configuration step creating that DB failed.

I then read line by line the log file produced by the configuration tool and searched for the source of the problem. I will paste here the relevant parts:

[1:57:09 PM Error ConfigHelper] [DBNETLIB][ConnectionOpen (Connect()).]SQL Server does not exist or access denied.
[1:57:09 PM Error ConfigHelper] SQL error: 08001 Native error code: 17
[1:57:09 PM Error ConfigHelper] c:\depotsetupv2\private\common\configwizard\confighelper\sqlhelper.cpp(1176): FAILED hr = 80004005
[1:57:09 PM Error ConfigHelper] c:\depotsetupv2\private\common\configwizard\confighelper\sqlhelper.cpp(918): FAILED hr = 80004005
[1:57:09 PM Info ConfigHelper] Did not find existing database: BizTalkMgmtDb on computer:
[1:57:09 PM Info RuleEngineConfig] Management database (server = , database = BizTalkMgmtDb) doesn’t exist

With this information in hand, I suspected something was wrong with the credentials and I checked all the Users and Groups in Active Directory and also checked that I was admin of the SQL Server machine but everything looked ok. I nevertheless re-tried the configuration a few times, being extra careful with every setting but unsuccessfully. I then moved on to next problem in the log file.

[2:08:41 PM Error ConfigHelper] c:\depotsetupv2\private\common\configwizard\confighelper\service.cpp(729): FAILED hr = 80070421
[2:08:41 PM Warning ConfigHelper] The account name is invalid or does not exist, or the password is invalid for the account name specified.
[2:08:41 PM Warning ConfigHelper] Failed to validate service credentials for account: %1
[2:08:41 PM Info BtsCfg] Check Your NT Account Specification

Here the configuration log complains about Windows Credential issues again. Everything was pointing out to Active Directory, this would reveal to actually point me in the wrong direction. So, as I could still not find any problem in AD, I moved on to the next problem I could find in the log:

2008-09-12 14:13:16:0075 [INFO] WMI Deploying ‘C:\Program Files (x86)\Microsoft BizTalk Server 2006\Microsoft.BizTalk.GlobalPropertySchemas.dll’
2008-09-12 14:13:22:0184 [WARN] AdminLib GetBTSMessage: hrErr=80070002; Msg=The system cannot find the file specified.;
2008-09-12 14:13:22:0184 [WARN] AdminLib GetBTSMessage: hrErr=c0c02560; Msg=Failed to read “KeepDbDebugKey” from the registry.
The system cannot find the file specified.;
2008-09-12 14:13:22:0184 [INFO] WMI Error occurred during database creation; attempt to rollback and delete the partially created database’ny-agbdb-301\BizTalkMsgBoxDb’

Two points here:

  • The file “T:\Program Files (x86)\Microsoft BizTalk Server 2006\Microsoft.BizTalk.GlobalPropertySchemas.dll” was present on the file system so it didn’t seem there was anything I could do about it.
  • The key “KeepDbDebugKey” did not exist in the registry of running healthy BizTalk systems so I supposed this registry key exists only temporarily during the BizTalk configuration procedure.

I then moved on the next potential problem in the log file:

2008-09-12 14:14:02:0465 [ERR] WMI Failed in pAdmInst->Create() in CWMIInstProv::PutInstance(). HR=c0c025b3
2008-09-12 14:14:02:0465 [ERR] WMI WMI error description is generated: Exception of type ‘System.EnterpriseServices.TransactionProxyException’ was thrown.

Here the configuration tool log entry complains about a Transaction from the System.EnterpriseServices library. As this package refers to Distributed Transactions (MS DTC), I doubled checked that MS DTC was running well on both machines, checked firewall configurations but all the necessary ports were open.
I then started to try to troubleshoot DTC following the MSDTC troubleshooting guide from Microsoft and found a utility called dtctester that can be downloaded from the KB 293799.

The dtctester test failed and so I was now sure that the problem was related with DTC. I re-checked the firewall settings, checked the dual network adapter and the DNS as advised by Microsoft and found out that the problem lied with the DNS having some wrong reverse mappings (IP address -> hostname mappings).

As a corollary, notice that the only log information that proved to point me to the actual source of the problem were log entries with the string [ERR] in their header. All others had either [INFO] or [WARN].

To conclude, I would advise to:

  • Use dtctester before operating a BizTalk multi-server installation.
  • When running into problems, look first at the log entries having either the [ERR] or Error string in their header – It indeed seems there are two different formats of log entry header in the log file produced by the Configuration tool. While looking first for the Error type of log entries sounds obvious, it seems that not many people are aware of the log entry header formats and the log entry types generated in the log file.

How to purge BizTalkMsgDb?

If the BizTalk Message Box database (BizTalkMsgDb) grows too large, performance of the database subsystem will reduce. As a rule of thumb, the BizTalkMsgDb should never grow larger than 5 Gb for large systems with long running transactions. For high-volume environment with no long running transactions, the BizTalkMsgDb size should be much smaller than 5Gb.

Would your message box have already grown too large and the system become unresponsive, you might want to clean up the BizTalkMsgDb database manually but keep in mind that:

  • The BizTalk Server must be taken down during the procedure.
  • All existing messages will be purged and lost.

There is an article in the MSDN documentation that explains in details how to manually purge data from the Message Box Database.

To make a long story short, here is step by step summary:

  • Stop ALL BizTalk service host instances from the Services console.
  • Restart IIS by running IISRESET from the command prompt if you are running any adapters in isolated hosts (for example HTTP, SOAP, or WCF).
  • Execute the stored procedure bts_CleanupMsgbox on your message box database. If the stored procedure does not exist, create it by running the sql script “msgbox_cleanup_logic.sql” found in the directory “\Schema\”.
    Important note: If you are running BizTalk Server 2006, you first have to get an updated version of the Msgbox_cleanup_logic.sql file to (re)create the bts_CleanupMsgbox stored procedure at http://support.microsoft.com/kb/924715. The hotfix is available for download directly without having to contact MS. If you are running BizTalk Server 2006 R2 then the updated version of the Msgbox_cleanup_logic.sql file is already installed and you do not need to download the hotfix.
  • Execute the stored procedure bts_PurgeSubscriptions on your message box database. If the stored procedure does not exist, you can create it by inspecting the sql script “msgboxlogic.sql” found in the directory “\Schema\”. Copy paste only the part of the script that creates this specific stored procedure and run it.

As deleting data in a database does not reduce the size the database files on the disk; you need to shrink the database files if you want to reduce its physical size. There are 2 simple ways to do it:

  • Through SQL Server Management Studio, right click on the BizTalkMsgDb database, click on Tasks > Shrink > Database
  • Through T-SQL, using the DBCC SHRINKDATABASE command: DBCC SHRINKDATABASE (BizTalkDTADb);

Microsoft does not support this procedure on production systems simply because this will purge production data. So, as long as you keep in mind that all existing messages will be purged, it is safe to run it on a production environment.

Truncating the Database Log File:
There is also another useful command to truncate the database logfile, would it be necessary (such as running out of disk space):
BACKUP LOG BizTalkDTADb WITH TRUNCATE_ONLY (*)
Note that you are not supposed to do this on production but emergency scenarios can call for the necessity.

(*) Edit:
The T-SQL BACKUP command does not support the TRUNCATE_ONLY parameter since SQL Server 2008. See T-SQL BACKUP Reference.
As explained in this post, it was discontinued because it breaks the ability to recover from a certain point in time until a full backup is run again. It was intended to be used only in case of emergency such as when we run out of disk space but apparently it was regularly misused by DBAs and maybe the reason was that it was a misunderstood option of the BACKUP command. It can indeed be a little misleading that an option of the BACKUP command can “break” the backup.
The way to truncate the log file on SQL Server 2008 and above in case of emergency is the following:

  1. Set the Database in Simple Recovery mode.
  2. Shrink the log file.
  3. Set the Database back into Full recovery mode.

Example taken from DBCC SHRINKFILE T-SQL reference:
USE AdventureWorks2012;
GO
-- Truncate the log by changing the database recovery model to SIMPLE.
ALTER DATABASE AdventureWorks2012
SET RECOVERY SIMPLE;
GO
-- Shrink the truncated log file to 1 MB.
DBCC SHRINKFILE (AdventureWorks2012_Log, 1);
GO
-- Reset the database recovery model.
ALTER DATABASE AdventureWorks2012
SET RECOVERY FULL;
GO

Promoted property vs distinguished field – tutorial

First of all while many people talk about Promoted Property, the official term is Promoted Field, as defined in the MSDN documentation and also visible in Visual Studio’s BizTalk Schema Editor. I will nevertheless keep using the term most people are familiar with but I will consider both terms as synonyms.

Part 1 & 2 is a tutorial on how to create promoted properties and distinguished fields through the Schema Editor and the BizTalk’s API while part 3 is discussing about performance and differences between Promoted Properties and Distinguished Fields. You might only be interested in the 3rd part of this article if you already are a seasoned BizTalk developer.

1. Promoted Properties (Promoted Fields)

1.1 What are Promoted Properties?

Promoted Properties are Message Context Properties that are flagged as promoted; being promoted it allows the Message Engine to route messages based on their value, and being in the message context allows doing so without having to look at the message payload (which would be an expensive operation). Promoted Properties are the most common way to enable content-based routing. They are available to the pipelines, adapters, Message Bus and orchestrations.

Promoted Fields (= Promoted Properties) are PROMOTED in the message context by the receive pipeline when a message is received on a port. It is usually the job of the disassembler pipeline component such as the XML and Flat File disassembler but any custom pipeline component can also do it.

1.2 How to promote properties?

As stated in a previous post I wrote, BizTalk Messaging Architecture, message context properties are defined within a property schema and so all promoted properties must be defined in a custom property schema.

The action of promoting a message element to a promoted property creates a message context property that will contain the message element value and flags it as promoted so that it is available for routing.

There are 2 ways to promote a message element:

1. Quick promotion

Quick promotion is the simplest way to create a promoted property. Simply right click on the element’s node and choose Quick Promotion (see Fig 1.1). When choosing this option, Visual Studio will create a property schema called PropertySchema.xsd and add in the message’s schema a reference to the generated property schema.

Each property promoted this way will create a corresponding element in the property schema with the same name and type as defined in the message’s schema.

This means that when using quick promotion, the promoted property element name will always be the same as the message’s element name. If you have several elements with the same name, you might want to use manual promotion instead to avoid confusion or avoid having a property value overridden.

Biztalk promoted properties - quick promotionFig. 1.1 Quick Promotion

2. Manual Promotion

To manually promote a property, a property schema must be created with the elements that will hold the promoted property values. To create a property schema, you need to add a new item in your BizTalk solution, and chose Property Schema as the type of file (see Fig. 1.2). Once all the elements are created in the property schema, you associate the property schema with the message’s schema by right clicking on any node of the message schema and choose Show Promotions (see Fig. 1.3), then click on the Property Fields tab, click on the folder icon and finally select the property schema you just created (see Fig. 1.4). Note that it is actually possible to use more than 1 property schema per message. Anyhow, all promoted properties will end up being written in the message context and available for all BizTalk artifacts having access to message context.

Once the property schema is picked, you can start promoting message elements as promoted properties. To do so, click a message element and click on the add button, the Node Path column will display the XPath to the message element you are promoting and the Property column let you chose the promoted property that will contain the value of the message element at runtime (see Fig. 1.5).

An interesting side effect is that Manual Promotion let you have promoted properties with different names that the original message element name. This might be useful when a same property schema is used to hold promoted properties from different message types or when a message has different element with the same name.

Using Manual Promotion, it is also possible to promote message elements to promoted properties in the system property schemas shipped with BizTalk. To do so, just browse in the References sub-tree when picking the property schema (see Fig. 1.4).

BizTalk - Create a property schemaFig. 1.2 Manually create a custom property schema

BizTalk - Promoted Properties, manual promotionFig. 1.3 Manual Promotion

BizTalk - Promoted Properties, selecting a Property SchemaFig. 1.4 Selecting a Property Schema

BizTalk - Promoted Properties, selecting the promoted propertyFig. 1.5 Selecting the message element to promote and the promoted property that will contain the message element’s value.

1.3 How to promote properties through the BizTalk API?

As I already said in a previous post, BizTalk Messaging Architecture, context properties are stored in a property bag, an object which implements the IBaseMessageContext interface. This interface contains 2 methods which take the same parameters (a property name, an XML namespace, and value for the property).

a. The Write() method is used to write the property value into the message context without actually promoting it. It can be used to write distinguished fields or to write transient values. Calling the Write() method is NOT promoting a property, it is writing a property.

b. The Promote() method is used to write the property value into the message context but also flags the property as promoted so that it is available for routing. This is the method that needs to be called to promote a property.

To promote a property through the API, element name and namespace passed as parameter of the Promote() method must be those of the property defined in the property schema. They are most easily accessed by referencing the property schema assembly and using the properties on the class created for the property.

Example of promoting a property through an API call (copied from the BizTalk 2006 MSDN documentation: Processing the Message):

//create an instance of the property to be promoted
SOAP.MethodName methodName = new SOAP.MethodName();

//call the promote method on the context using the property class for name and namespace
pInMsg.Context.Promote(methodName.Name.Name, methodName.Name.Namespace,
“theSOAPMethodName”);

As I mentioned before, it is possible to promote properties to the BizTalk system property schema using Manual Promotion in Visual Studio’s Schema Editor for BizTalk. It is possible to achieve the same programmatically and as for other promotion, the name and namespace passed in parameter of the Promote() method is the one of the promoted property in the property schema, the system property schema namespace in this case.

//BizTalk system properties namespace
private const string BTSSystemPropertiesNamespace = “http://schemas.microsoft.com/BizTalk/2003/system-properties”;

//Promote the MessageType property
string messageType = “http://” + “schemas.abc.com/BizTalk/” + “#” + “Request”;
message.Context.Promote(“MessageType”, BTSSystemPropertiesNamespace, messageType);

2. Distinguished fields

2.1. What are distinguished fields?

Distinguished fields are message elements that are written into the message context. They differ with promoted properties in 2 main aspects:

  • They are not flagged as promoted in the message context and so are not available for routing by the Messaging Engine (adapters, pipeline…). Their typical use is instead for the orchestration engine.
  • They are not defined using a property schema.

Distinguished Fields are WRITTEN in the message context by the pipeline when a message is received on a port. It is usually the job of the disassembler pipeline component such as the XML and Flat File disassembler but any custom pipeline component can also do it.

Distinguished fields are useful when a message element value needs to be accessed from an orchestration. Instead of having the orchestration engine search through the message to evaluate an XPath expression (which can be resource-intensive on a large message), a distinguished field can be used. Distinguished fields are populated in the message context when the message is first loaded. Consequently, each time the distinguished field needs to be accessed, the orchestration engine will directly read it from the message context (a property bag object) instead of searching for the original value in the message with XPath. Needless to say that retrieving a single value from a property bag object is much faster than evaluating an XPath expression.

Distinguished fields also offer IntelliSense in the orchestration expression editor which enhances code readability.

2.1 How to create a distinguished field?

The main source of confusion between distinguished fields and promoted properties is that they are both created in Visual Studio’s Schema Editor through the Promote -> Show Promotions contextual menu option of a message schema’s element. Once the dialog box is open, make sure that you are on the Distinguished Field tab, select the message elements and click the Add>> and <<Remove buttons to add and remove distinguished fields (see Fig 2.1).

BizTalk - Creating Distinguished FieldsFig 2.1 Creating a Distinguished Field.

2.2 How to create a distinguished field through the BizTalk API.

Distinguished fields are written into the context using the Write() method on the IBaseMessageContext object. To be recognized as a distinguished field, the namespace of the property must be “http://schemas.microsoft.com/BizTalk/2003/btsDistinguishedFields”. Pipeline components delivered with BizTalk do not use those context properties. It is nevertheless possible to read/write distinguished field in the code of custom pipelines, as for any other context properties.

Example of writing a distinguished field through an API call (taken from the BizTalk 2006 MSDN documentation: Processing the Message).
//write a distinguished field to the context
pInMsg.Context.Write(“theDistinguishedProperty”,
“http://schemas.microsoft.com/BizTalk/2003/btsDistinguishedFields”,
“theDistinguishedValue”);

Would you use another namespace, it will result in writing a plain transient value in the property bag and won’t be recognized as a distinguished field by the orchestration engine.

//Write a transient value to the message context
message.Context.Write(“MyVariable”, “SomeNameSpace”, SomeData);

3. Considerations, similarities and differences between promoted properties and distinguished fields.

3.1 Performance considerations:

  • Promoted properties are limited to 255 characters for routing performance reasons. Properties that are simply written in the context (such as distinguished fields) are not limited in size but large properties decrease performance.
  • All Context properties (both Promoted and Distinguished Fields) are stored separately from the message in the Message Box Database. Consequently consuming more space in the BizTalk databases but more importantly incurs more load when persisting/reloading the message in/from the DB. Note that if tracking is turned on, Promoted Properties are also stored in the Tracking database.
  • Distinguished Fields cost less than Promoted Properties in terms of performance. Both Promoted and Distinguished Fields require the same overhead of writing their values to the context Field bag in the Message Box database, but Promoted Fields have the additional overhead of being written in BOTH to the Message Box context tables AND the subscription tables. Distinguished fields are not stored in the subscription table as they do not participate in routing.
    Promoted Fields have an impact every time a message is written in the Message Box because each Promoted Property that exists must be evaluated in a massive T-SQL union statement that builds the list of matching activation subscriptions. In short, the more Promoted Fields you have the more costly the subscription process is.

3.2 Other considerations

  • Both promoted properties and distinguished fields are populated when a Pipeline Disassembler Component parses a message and either Promotes or Writes the value to the message’s context.
  • Empty pipelines such as the Pass-through pipeline do not promote or write anything in the message context as it lacks a disassembler component.
  • Writing a value into the context with the same name and namespace that were used previously to promote a property causes that property to no longer be promoted. The write essentially overwrites the promotion.
  • Writing a property in the context having a null value deletes the context property altogether because null-valued properties are not permitted.

3.3 Distinguished and Property Fields Difference Summary.

  • Promoted Fields should be used for routing, correlation and/or tracking.
  • Distinguished fields should be used when a particular message element is commonly manipulated in one or more orchestration.

Here is a table outlining the main differences between both types of fields:

Promoted Fields (aka Promoted Properties) Distinguished Fields
Used for routing (subscription mechanism)
IsPromoted = true
Do not participate in routing
IsPromoted = false
Used for tracking Not used for tracking
Restricted to 255 characters No size limitation
Available for use in orchestrations Available for use in orchestrations
Require property schema Do not require property schema
Used by standard pipeline components Accessible only by custom pipeline component which would explicitly access them

4. References.

MSDN – The BizTalk Server Message
MSDN – Processing the Message
MSDN – About BizTalk Message Context Properties
Neudesic’s blog – Distinguished fields myths

List of BizTalk prerequesites redistributable CAB files

While I was installing BizTalk Server 2006 the other day, I decided to compile a list of links to the redistributable BizTalk prerequisite CAB file so that I do not have to search for it the next time I install BizTalk Server.

The BizTalk prerequisites CAB file is a compilation of all the prerequisites that are needed to install and run BizTalk Server.
The redistributable CAB file is different for different Windows versions (Windows XP, Vista, Server 2003, 64-bit or 32-bit) and languages (English, French, …). So, when chosing the prerequesite CAB file, make sure that it is the one matching your Windows Operating System version and language.

When installing BizTalk Server, the setup procedure gives you the choice to either automatically install the prerequisites from the web or to automatically install the prerequisites from a redistributable CAB file.
Choosing the redistributable CAB file options helps in speeding up the installation process and is necessary if the BizTalk server has a limited access or no access at all to the internet.

BizTalk Server 2006

For BizTalk Server 2006, the links to the redistributable prerequisites CAB files are listed in the BizTalk Server 2006 Installation and Upgrade Guides. The guides are MS Word documents containing links to all the prerequisites CAB files for all supported versions of the Windows Operating System.

BizTalk Server 2006 R2

For BizTalk Server 2006 R2, the links to the redistributable prerequisites CAB files can be found in the BizTalk Server 2006 R2 Installation and Upgrade Guides. Here again the guides are a bunch of different MS Word documents containing links to all the prerequisites CAB files for all supported versions of the Windows Operating System.

As BizTalk Server 2006 R2 is the current release of BizTalk Server 2006, the links to the prerequisites CAB files can also directly be found on the MSDN documentation site:
http://msdn2.microsoft.com/en-us/library/aa578652.aspx

For information purpose, here is a list of all requirements for BizTalk 2006 R2: http://download.microsoft.com/documents/australia/biztalk/post_event/BizTalkServer2006R2_SystemRequirements.pdf

Biztalk 64 bit high CPU utilization

I started to deploy BizTalk server 2006 64-bit edition on Windows Server 2003 64-bit in our production environment and we noticed an unusual high CPU utilization by the BizTalk host processes.

The symptoms were twofold:

1. The BizTalk 64 bit servers suffered from a high CPU usage (one server had its CPU usage constantly stuck at 100%). This happened even when the load on the server was low, in fact much lower than on other BizTalk Server 32 bit machines running the same applications on a similar hardware.
The reason for the server high CPU usage was that some BizTalk host processes had their CPU usage stuck at a high value even if the BizTalk application(s) run by the host would process very few messages or no message at all. Actually, the CPU usage for those hosts would still be stuck at a high value even if the BizTalk application(s) run by the host were stopped!

2. Even when the CPU utilization was showing 100%, the server was still processing messages as fast as you would expect from a server under little load. No messages were queued for processing and no orchestrations were dehydrated!

Those 2 points gave me the feeling that nothing could be wrong with the BizTalk server itself (by that I mean the BizTalk code and the BizTalk installation) as the same code is running fine on 32 bit servers and as the performance monitor readings did not match the actual facts.

I requested help from Microsoft Support and they guided me to the KB 943165 which has a hotfix solving the high CPU usage problem I encountered on BizTalk Server 64-bit edition.
This hotfix fixes a problem brought by the security bulletin MS07-040, a security update for the .Net framework which brings CPU usage spikes on BizTalk Server 64 bit (meaning that the problem is only for the 64 bit version of the .Net framework). For information, Microsoft Support told me that the problem is suspected to be caused by an infinite loop in CounterManager.RunCacheThread when System.Threading.ThreadAbortException is raised.

I should also add that I could not actually find a Windows update directly related to the bulletin MS07-040 in the list of updates installed on the server running BizTalk Server 64-bit. Nevertheless, my Microsoft Support contact told me that the version of mscorwks.dll (found in c:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727) has a timestamp showing that it is later than the hotfixes issued by bulletin MS07-040 and that the problem should then also occur with any later version of the dll (as it is a code change from that version on that brings the problem).

To conclude, would you encounter the same symptoms I enumerated above, you will want the check the KB 943165 and contact Microsoft Product Support to see if your case is applicable for this hotfix. They will then email you the details to download the hotfix.

I installed this hotfix on all my BizTalk Server 2006 64 bit machines and all of them are running smoothly now 🙂