ASP.Net, .Net, BizTalk, Tech and Life - Francois Malgreve - Bangkok

How to script data – generate insert T-SQL statements from existing data

2008-11-26T11:50:00.001+07:00

I found this script that is able to generate insert T-SQL statements for data already existing in a database table. The script is packaged into a stored procedure and is thus very convenient to use.
It can reveal very useful when creating scripts for database initialization when packaging the release of a database driven application or moving some data to a new environment.

Developers do not always have access to all the different environments but sometimes still need to package data deployment scripts for those environments. Most often this will apply to lookup tables such as a lookup tables containing configuration values of an application which can potentially change with every release of an application. It could also apply to simpler cases such as a country code lookup table. Anyways, with this script, it will just take you seconds to generate all the insert statements into an SQL script that you can release with other deliverables (libraries, stored procedures…) to the release manager in charge of updating the application on the live environments.

The strong points of this script are that it handles identity column as well as computed columns and can script data for both views and tables. As this script is so useful for me, I backed-up the stored procedure scripts would the original link be one day broken:

I do not have SQL 2008 yet so I could check if the script works for SQL 2008. I guess that it might as SQL Server didn’t change as dramatically between the 2005 and 2008 version as it did between the 2000 and 2005 version. Anyway if someone tries it, please feel welcome to let me know about it in a comment :)

Note that commercial products are available would you need fancier data scripting features.

Regular expressions tool designer

2008-09-22T18:36:00.002+07:00

Regular expressions are very useful but as I do not have to use them a lot, I often forget the exact syntax required for my particular need.
So today I would like to share a tool to build regular expressions which I have been using many times over, the Regular Expression Designer of Rad Software.

The tool is free, easy to use but yet powerful and intuitive. The contextual help is very short but just perfect and straight to the point for someone who already has previous regex experience. If you are a total newbie in regular expressions, you should read some short introductory material beforehand.

Error configuring BizTalk Server 2006

2008-09-15T09:40:00.001+07:00

Today I had quite some troubles installing BizTalk Server 2006. As I have previously installed BizTalk many times without being stuck on a problem for so long, I thought I would blog about it. Who knows, it might help some other soul encountering the same issue.

My configuration was the following:

1 Windows Server 2003 64-bit running SQL Server 2005.
1 Windows Server 2003 64-bit with BizTalk Server 2006 installed but not yet configured.

Both machines had MSDTC and COM+ installed and running, a pre-requisite to install BizTalk on a multiple-server environment.
All the BizTalk Users as well as the Windows Account used to install BizTalk had the necessary rights and belonged to the necessary Groups. See a previous post of mine for more information; Biztalk 2006 multi-server installation guide.

So, after having installed BizTalk Server 2006, the BizTalk Server needs to be configured by using the BizTalk Server Configuration tool and there the weirdest thing happened: it failed, but not completely! Enterprise SSO (Enterprise Single Sign-On) and Business Rules Engine (BRE) components were configured successfully while Group and BizTalk Runtime failed.
Thinking I made some human mistake in the configuration, I retried 2 times but the same problem happened. On a side note, before retrying to configure BizTalk, I had to manually delete the BizTalkMgmtDb as it was created even as the configuration step creating that DB failed.

I then read line by line the log file produced by the configuration tool and searched for the source of the problem. I will paste here the relevant parts:

[1:57:09 PM Error ConfigHelper] [DBNETLIB][ConnectionOpen (Connect()).]SQL Server does not exist or access denied.
[1:57:09 PM Error ConfigHelper] SQL error: 08001 Native error code: 17
[1:57:09 PM Error ConfigHelper] c:\depotsetupv2\private\common\configwizard\confighelper\sqlhelper.cpp(1176): FAILED hr = 80004005
[1:57:09 PM Error ConfigHelper] c:\depotsetupv2\private\common\configwizard\confighelper\sqlhelper.cpp(918): FAILED hr = 80004005
[1:57:09 PM Info ConfigHelper] Did not find existing database: BizTalkMgmtDb on computer:
[1:57:09 PM Info RuleEngineConfig] Management database (server = , database = BizTalkMgmtDb) doesn't exist

With this information in hand, I suspected something was wrong with the credentials and I checked all the Users and Groups in Active Directory and also checked that I was admin of the SQL Server machine but everything looked ok. I nevertheless re-tried the configuration a few times, being extra careful with every setting but unsuccessfully. I then moved on to next problem in the log file.

[2:08:41 PM Error ConfigHelper] c:\depotsetupv2\private\common\configwizard\confighelper\service.cpp(729): FAILED hr = 80070421
[2:08:41 PM Warning ConfigHelper] The account name is invalid or does not exist, or the password is invalid for the account name specified.
[2:08:41 PM Warning ConfigHelper] Failed to validate service credentials for account: %1
[2:08:41 PM Info BtsCfg] Check Your NT Account Specification

Here the configuration log complains about Windows Credential issues again. Everything was pointing out to Active Directory, this would reveal to actually point me in the wrong direction. So, as I could still not find any problem in AD, I moved on to the next problem I could find in the log:

2008-09-12 14:13:16:0075 [INFO] WMI Deploying 'C:\Program Files (x86)\Microsoft BizTalk Server 2006\Microsoft.BizTalk.GlobalPropertySchemas.dll'
2008-09-12 14:13:22:0184 [WARN] AdminLib GetBTSMessage: hrErr=80070002; Msg=The system cannot find the file specified.;
2008-09-12 14:13:22:0184 [WARN] AdminLib GetBTSMessage: hrErr=c0c02560; Msg=Failed to read "KeepDbDebugKey" from the registry.
The system cannot find the file specified.;
2008-09-12 14:13:22:0184 [INFO] WMI Error occurred during database creation; attempt to rollback and delete the partially created database'ny-agbdb-301\BizTalkMsgBoxDb'

Two points here:

The file “T:\Program Files (x86)\Microsoft BizTalk Server 2006\Microsoft.BizTalk.GlobalPropertySchemas.dll” was present on the file system so it didn’t seem there was anything I could do about it.
The key “KeepDbDebugKey” did not exist in the registry of running healthy BizTalk systems so I supposed this registry key exists only temporarily during the BizTalk configuration procedure.

I then moved on the next potential problem in the log file:

2008-09-12 14:14:02:0465 [ERR] WMI Failed in pAdmInst->Create() in CWMIInstProv::PutInstance(). HR=c0c025b3
2008-09-12 14:14:02:0465 [ERR] WMI WMI error description is generated: Exception of type 'System.EnterpriseServices.TransactionProxyException' was thrown.

Here the configuration tool log entry complains about a Transaction from the System.EnterpriseServices library. As this package refers to Distributed Transactions (MS DTC), I doubled checked that MS DTC was running well on both machines, checked firewall configurations but all the necessary ports were open.
I then started to try to troubleshoot DTC following the MSDTC troubleshooting guide from Microsoft and found a utility called dtctester that can be downloaded from the KB 293799.

The dtctester test failed and so I was now sure that the problem was related with DTC. I re-checked the firewall settings, checked the dual network adapter and the DNS as advised by Microsoft and found out that the problem lied with the DNS having some wrong reverse mappings (IP address -> hostname mappings).

As a corollary, notice that the only log information that proved to point me to the actual source of the problem were log entries with the string [ERR] in their header. All others had either [INFO] or [WARN].

To conclude, I would advise to:

Use dtctester before operating a BizTalk multi-server installation.
When running into problems, look first at the log entries having either the [ERR] or Error string in their header - It indeed seems there are two different formats of log entry header in the log file produced by the Configuration tool. While looking first for the Error type of log entries sounds obvious, it seems that not many people are aware of the log entry header formats and the log entry types generated in the log file.

How to purge BizTalkMsgDb?

2008-08-08T18:16:00.004+07:00

If the BizTalk Message Box database (BizTalkMsgDb) grows too large, performance of the database subsystem will reduce. As a rule of thumb, the BizTalkMsgDb should never grow larger than 5 Gb for large systems with long running transactions. For high-volume environment with no long running transactions, the BizTalkMsgDb size should be much smaller than 5Gb.

Would your message box have already grown too large and the system become unresponsive, you might want to clean up the BizTalkMsgDb database manually but keep in mind that:

The BizTalk Server must be taken down during the procedure.
All existing messages will be purged and lost.

There is an article in the MSDN documentation that explains in details how to manually purge data from the Message Box Database.

To make a long story short, here is step by step summary:

Stop ALL BizTalk service host instances from the Services console.
Restart IIS by running IISRESET from the command prompt if you are running any adapters in isolated hosts (for example HTTP, SOAP, or WCF).
Execute the stored procedure bts_CleanupMsgbox on your message box database. If the stored procedure does not exist, create it by running the sql script “msgbox_cleanup_logic.sql” found in the directory “<BizTalk installation directory>\Schema\”.
Important note: If you are running BizTalk Server 2006, you first have to get an updated version of the Msgbox_cleanup_logic.sql file to (re)create the bts_CleanupMsgbox stored procedure at http://support.microsoft.com/kb/924715. The hotfix is available for download directly without having to contact MS. If you are running BizTalk Server 2006 R2 then the updated version of the Msgbox_cleanup_logic.sql file is already installed and you do not need to download the hotfix.
Execute the stored procedure bts_PurgeSubscriptions on your message box database. If the stored procedure does not exist, you can create it by inspecting the sql script “msgboxlogic.sql” found in the directory “<BizTalk installation directory>\Schema\”. Copy paste only the part of the script that creates this specific stored procedure and run it.

As deleting data in a database does not reduce the size the database files on the disk; you need to shrink the database files if you want to reduce its physical size. There are 2 simple ways to do it:

Through SQL Server Management Studio, right click on the BizTalkMsgDb database, click on Tasks > Shrink > Database
Through T-SQL, using the DBCC SHRINKDATABASE command: DBCC SHRINKDATABASE (BizTalkDTADb);

There is also another useful command to truncate the database logfile, would it be necessary:
BACKUP LOG BizTalkDTADb WITH TRUNCATE_ONLY

Microsoft does not support this procedure on production systems simply because this will purge production data. So, as long as you keep in mind that all existing messages will be purged, it is safe to run it on a production environment.

How to Handle Unhandled Exception

2008-07-31T22:55:00.007+07:00

When an unhandled exception occurs in the .Net Framework run-time 1.0 & 1.1, the Windows Process will terminate only if the exception occurred on the main application thread. Would an unhandled exception occur in a thread other than the main application thread, it will be caught by the runtime and therefore NOT cause the application to terminate. This means that unhandled exceptions in child threads disappear silently without anyone being able to know about it, implying that bugs are silently swallowed by the runtime without anyone being notified about it. The dangerous scenario is that a user might believe that a unit of work has executed successfully while it is actually not the case. This is in my opinion something not desirable and has been addressed in later version of the .Net Framework.

Indeed, from the .Net Framework 2.0 onwards, unhandled exceptions occurring on ANY thread (thus including any child threads being background threads or not) shuts down the application running the particular thread.

When the runtime terminates an application because of an unhandled exception, it writes an entry in the Windows Event Log which looks like the following:

EventType clr20r3, P1 processname, P2 1.0.0.0, P3 485f85f0, P4 system, P5 2.0.0.0, P6 471ebf0d, P7 3832, P8 bf, P9 system.componentmodel.win32, P10 NIL.

“processname” being the name of the .Net executable.

1. How to resolve unhandled exceptions? The System.AppDomain.UnhandledException event.

If an unhandled exception occurs in a windows application (multi-threaded or not), a windows service named winservice.exe for example, the runtime will terminate the service and write an Event Log entry such as:
“EventType clr20r3, P1 winservice.exe, P2 1.0.0.0, P3 485f85f0, P4 system, P5 2.0.0.0, P6 471ebf0d, P7 3832, P8 bf, P9 system.componentmodel.win32, P10 NIL.”

While it is good to be notified that a service crashed, the information supplied is rather cryptic and has very little added value. It would be nicer to have a way to log unhandled exceptions with a more meaningful message so that it is possible to get enough information to be able to fix the source code and re-deploy the application.

Conveniently, the .Net Framework provides the System.AppDomain.UnhandledException event. This event fires whenever an application domain unloads because of an unhandled exception. Note that registering for the event does not cause unhandled exceptions to be "handled". It simply notifies you that an exception has gone unhandled, so that it is possible to take some action such as logging the exception message and stack trace and eventually do some clean up before the application dies. The Exception is still unhandled and so still causes the application to shut down.

The discussion in this article is not a solution on how to handle exceptions. Exception handling is a strategy on how to handle exceptions and is a different discussion. This article explains how to act with unhandled exceptions so that details about the exception can be logged so that a proper solution can be found to resolve the issue/bug.

Ideally, all exceptions should be handled in the source code with try-catch-finally blocks so that unhandled exceptions do not occur. Nevertheless, there are always unforeseen cases and developer mistakes which, in my opinion, justify caring for unhandled exceptions across the board.

Here is an example on how to register for the event with an event handler that log exception details in the Windows Event Log:

// This event provides notification of uncaught exceptions. Write this in the entry point of your program, like in the OnStart() method of a Windows Service.
AppDomain currentDomain = AppDomain.CurrentDomain;
currentDomain.UnhandledException += new UnhandledExceptionEventHandler(UnhandledExceptionHandler);
static void UnhandledExceptionHandler(object sender, UnhandledExceptionEventArgs args)
{
Exception ex = (Exception)args.ExceptionObject;
EventLog.WriteEntry("WinService.exe", "Unhandled Exception caught: " + ex.Message + ex.StackTrace, EventLogEntryType.Error);
}

2. Note for Windows Form Applications - Application.ThreadException event.

2.1 Application.ThreadException event

For Windows Form Applications, there is another event that is raised when an unhandled exception occurs, the System.Windows.Forms.Application.ThreadException. Nevertheless, this event fires only when unhandled exceptions happen in Window Forms threads (UI threads). This means when an exception is thrown from code that was ultimately called as a result of a Windows Message. Windows Messages are emitted by keyboard hits, mouse clicks or "paint" messages,... in short, nearly all code in a typical Windows Forms application.

While this works perfectly, it lulls one into a false sense of security that all exceptions will be caught by the central exception handler:

Exceptions thrown on worker threads are a good example of exceptions not caught by Application.ThreadException.
Exceptions thrown by the code inside the Main method of the Windows Forms application, including the main form's constructor, executes before the Windows message loop begins and so is another example of exceptions that do not raise the Application.ThreadException event.

In this case, we said the Application.ThreadException event handler to be a “central exception handler” because it is still possible for the application to keep running when this event is raised, depending on what kind of logic is implemented in the handler.

As a reminder, Worker Threads are threads:

Created manually: Thread.Start()
Created by the ThreadPool: ThreadPool.QueueUserWorkItem()
Created by any kind of asynchronous call which internally uses a thread pool thread to execute: Delegate.BeginInvoke(), BeginXXX()

One must attach a handler to the Application.ThreadException event before instantiating the main form of the application by calling Application.Run(). Also, because this is a static event, you must detach the event handler(s) when the application is disposed or memory leaks will result.

The Application.ThreadException as a default event handler which behaves in the following way:

If an unhandled exception occurs in the main application thread, the default exception handler catches it and terminates the application.
If an exception occurs in a thread other than the main application thread, the thread exits, but the application continues to run.

2.2 Application.SetUnhandledExceptionMode

It is possible to instruct the application whether it should catch all unhandled exceptions thrown by Windows Forms components and terminate the application, or whether it should raise an event so that an event handler can be implemented; the event handler could halt execution and expose the unhandled exception to the user. This is setting is done by using the application configuration file or the Application.SetUnhandledExceptionMode() method.
It is possible to instruct the application whether it should catch all unhandled exceptions thrown by Windows Forms components and terminate the application, or whether it should raise an event so that an event handler can be implemented; the event handler could halt execution and expose the unhandled exception to the user. This is setting is done by using the application configuration file or the Application.SetUnhandledExceptionMode() method.

UnhandledExceptionMode.ThrowException never route exceptions to the Application.ThreadException event handler and so the default event handler will terminate the application when an unhandled exception occurs as explained earlier.
UnhandledExceptionMode.CatchException always route exceptions to the Application.ThreadException event handler.

Again, as a reminder, the Application.ThreadException event handler is only for unhandled exception occurring on UI threads and so the SetUnhandledExceptionMode() method affects only the way unhandled exceptions coming from UI threads are treated, it does not affect how non UI threads unhandled exceptions are treated. System.AppDomain.UnhandledException event handlers will always be called when non UI threads unhandled exceptions occur.

2.3 Sample

Hereunder is a code sample on how to register to the Application.ThreadException event in a Windows Form Application. As said earlier the event handler will only be called for unhandled exceptions occurring on the UI thread. Thus, the code sample also has an event handler for unhandled exception on non-UI threads by registering to the System.AppDomain.UnhandledException event:

static class Program
{
///

/// The main entry point for the application.
///

[STAThread]
static void Main()
{
// Add the event handler for handling UI thread exceptions to the event.
Application.ThreadException += new ThreadExceptionEventHandler(Form1_UIThreadException);

// Set the unhandled exception mode to force all Windows Forms errors on UI threads
// to go through our handler regardless of application settings.
Application.SetUnhandledExceptionMode(UnhandledExceptionMode.CatchException);

// Add the event handler for handling non-UI thread exceptions to the event.
AppDomain.CurrentDomain.UnhandledException +=
new UnhandledExceptionEventHandler(CurrentDomain_UnhandledException);

// Default generated code by Visual Studio for WinForms
Application.EnableVisualStyles();
Application.SetCompatibleTextRenderingDefault(false);
Application.Run(new Form1());
}
}

You can find a zip file here containing a complete Visual Studio 2008 solution demonstrating both type of unhandled exception event handler and their effect.

3. Note for ASP.NET applications - Application_Error in Global.asax.

Would an unhandled exception occurs in an ASP.NET application on a worker thread, the runtime will terminate the worker process (w3wpp.exe) and write an Event Log entry such as:
“EventType clr20r3, P1 w3wp.exe, P2 6.0.3790.1830, P3 42435be1, P4 app_web_7437ep-9, P5 0.0.0.0, P6 433b1670, P7 9, P8 a, P9 system.exception, P10 NIL.”

In ASP.Net, there is an Application_Error method in the Global.asax file that can be implemented so that an action can be taken when an unhandled exception occurs (logging the exception details, for example). Again, treating an unhandled exception in such a way is not handling an exception (that should be done in the try-catch-finally block in the code), the web application will still crash and display an error message on the browser but it lets the programmer have a way to log information about the exception and eventually do some other clean-up tasks.

Nevertheless, similarly to Windows Forms Application, the Application_Error will be called only for unhandled exception that occurred on the main thread. Unhandled exceptions occurring on worker threads will not be caught by the Application_Error event handler in the global.asax file. If we want to be notified of unhandled exception occurring on worker threads, we need to register an event handler to the System.AppDomain.UnhandledException event, as we did for Windows Services and Forms Applications. The event handler should at least log the exception and eventually do some clean tasks.

Microsoft has a KB on how to implement an event handler for the System.AppDomain.UnhandledException event within an HTTPModule.

You can find here a zip file with a Visual Studio 2008 solution using the HTTPModule defined in the KB. I registered the HTTP module in the web.config the following way:

<system.web>
<httpModules>
<add name="UnhandledExceptionModule" type="WebMonitor.UnhandledExceptionModule, UnhandledExceptionModule, Version=1.0.0.0,
Culture=neutral, PublicKeyToken=51d169497dc4a81e, processorArchitecture=MSIL"/>
</httpModules>
</system.web>

The only non trivial attribute is the “PublicKeyToken”. The Public Key Token is a 64-bit hash of the public key corresponding to the private key used to sign the assembly. A .Net assembly is said to have a strong name when it is signed.

There are 2 ways to retrieve the Public Key Token from a strongly named assembly:

The lazy way is to add the assembly in the GAC and go to C:\WINDOWS\assembly, one of the column showed in Windows Explorer is the Public Key Token. Only strongly named assemblies can be stored in the GAC.
Run the command line “sn -T UnhandledExceptionModule.dll” which prints the Public Key token of a strongly named assembly.

Contrary to the instructions given in the KB, I did not GAC either NGEN the HTTP module; I just referenced the HTTPModule project from the web project so that the HTTP module library is automatically copied to the web application bin folder. All assemblies in the bin folder are automatically found by the run-time so that is not necessary to GAC all third party libraries.

SQL Server and .Net Data Type mapping

2008-06-26T11:17:00.003+07:00

Today I would like to post a table listing the corresponding .Net Type to use with each SQL Data Type. This comes in handy when writing ADO.NET code.

The following link lists Microsoft SQL Server data types and their equivalent data type in .Net.
To be more exact it lists:

Every SQL Server data types
Their equivalent in the Common Language Runtime (CLR) for SQL Server in the System.Data.SqlTypes namespace
Their native CLR equivalents in the Microsoft .NET Framework

SQL Server Data Types and Their .NET Framework Equivalents

I have also copy-pasted the SQL Server and .Net Data Type mapping table hereunder for my own convenience:

SQL Server data type	CLR data type (SQL Server)	CLR data type (.NET Framework)
varbinary	SqlBytes, SqlBinary	Byte[]
binary	SqlBytes, SqlBinary	Byte[]
varbinary(1), binary(1)	SqlBytes, SqlBinary	byte, Byte[]
image	None	None
varchar	None	None
char	None	None
nvarchar(1), nchar(1)	SqlChars, SqlString	Char, String, Char[]
nvarchar	SqlChars, SqlString SQLChars is a better match for data transfer and access, and SQLString is a better match for performing String operations.	String, Char[]
nchar	SqlChars, SqlString	String, Char[]
text	None	None
ntext	None	None
uniqueidentifier	SqlGuid	Guid
rowversion	None	Byte[]
bit	SqlBoolean	Boolean
tinyint	SqlByte	Byte
smallint	SqlInt16	Int16 (short in C#)
int	SqlInt32	Int32 (int in C#)
bigint	SqlInt64	Int64 (long in C#)
smallmoney	SqlMoney	Decimal (decimal in C#)
money	SqlMoney	Decimal (decimal in C#)
numeric	SqlDecimal	Decimal (decimal in C#)
decimal	SqlDecimal	Decimal (decimal in C#)
real	SqlSingle	Single (float in C#)
float	SqlDouble	Double (double in C#)
smalldatetime	SqlDateTime	DateTime
datetime	SqlDateTime	DateTime
sql_variant	None	Object
User-defined type(UDT)	None	Same class that is bound to the user-defined type in the same assembly or a dependent assembly.
table	None	None
cursor	None	None
timestamp	None	None
xml	SqlXml	None

Regarding CLR Data Types, Data Types are basically divided in 3 categories:

Value Types (CLR-defined structs, Enumerations, bool, user-defined structs)
Reference Types
Pointer Types

Furthermore, 3 types of CLR Data Type structures exist to hold numeric values:

Integral types, holding integers
Floating-point types, holding reals
decimal, holding reals of a smaller range but higher precision

See Data Types (C# Reference) for a complete reference and correspondance between CLR types and C# types.

T-SQL Weighted Round Robin Algorithm

2008-06-19T15:10:00.003+07:00

In a previous post I was discussing of ways to load balance MSMQ messages to a group of servers by implementing a Weighted Round Robin Algorithm in a software module.

I chose to implement the load balancing feature in the database for 2 main reasons:

The application is heavily database driven; stored procedures have to run to retrieve the data that will be used to build each MSMQ message. Therefore, the cost of an additional statement or stored procedure to retrieve the MSMQ path the message should be sent to is much less than the existing cost.
All the application configuration information is also database driven. This is because our organization uses custom tools for administrators so that they can configure servers and applications centrally. In the case at hand, they need to be able to modify queue names, machine relative weights as well as to add and remove machines from the load balanced group of server (the cluster).

The Weighted Round Robin algorithm is implemented as a stored procedure returning the next MSMQ path a message should be sent to. The stored procedure could be bundled or merged with the other stored procedures that have to run before each MSMQ message is sent, so that it would only be one more parameter coming back from the database. This parameter will tell the .Net code to which queue the current MSMQ message build has to be sent to.

T-SQL Weighted Round Robin Algorithm Implementation.

Each machine of a cluster should receive a percentage of messages which is relative to its weight:
%age = ((weight of machine) / (sum of weights for all the machines in the cluster)) * 100
Moreover, the distribution of messages to each machine should be function of the weight in real time, not only in average.

To satisfy these conditions, one way to implement the weighted round robin algorithm and which suits well T-SQL capabilities is to calculate a ratio between the relative number of messages sent to a particular machine of a cluster and the relative weight of that machine within the cluster. The machine having the higher ratio will be the next machine a MSMQ message should be sent to in respect with its relative weight. An ORDER BY clause will easily implement that in T-SQL.

First of all, we need to create a table where we will hold the different server parameters:

Column name	Data type	Purpose
server_name	[nvarchar](20)	Machine name
cluster	[char](10)	Cluster name in which the machine belongs to
queue_path	[nvarchar](100)	MSMQ path
number_requests_sent	[float]	Number of request sent to this machine so far
weight_factor	[float]	Weight factor of this machine
enabled_status	[bit]	Says if the machine is active or inactive in the cluster

Here is the SQL script to create the table:

SET ANSI_NULLS ON

SET QUOTED_IDENTIFIER ON

SET ANSI_PADDING ON

CREATE TABLE [dbo].[msmq_machines](

[server_name] [nvarchar](20) NOT NULL,

[cluster] [char](10) NOT NULL,

[queue_path] [nvarchar](100) NOT NULL,

[number_requests_sent] [float] NOT NULL CONSTRAINT [DF_msmq_machines_number_requests_sent] DEFAULT ((0)),

[weight_factor] [float] NOT NULL CONSTRAINT [DF_msmq_machines_weight_factor] DEFAULT ((100)),

[enabled_status] [bit] NOT NULL,

CONSTRAINT [PK_msmq_machines] PRIMARY KEY CLUSTERED

(

[server_name] ASC,

[cluster] ASC

)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]

) ON [PRIMARY]

SET ANSI_PADDING OFF

Second we need to create a Stored Procedure that will actually implement the algorithm:

SET ANSI_NULLS ON

SET QUOTED_IDENTIFIER ON

CREATE PROCEDURE [dbo].[GetLoadBalancedQueueName]

@cluster as char(10),

@serverName as nvarchar(20) out,

@queuePath as nvarchar(100) out

DECLARE @totalClusterRequestSent as float

DECLARE @totalWeight as float

SELECT @totalClusterRequestSent = sum(number_requests_sent)+1, @totalWeight = sum(weight_factor)

FROM dbo.msmq_machines

WHERE cluster = @cluster

AND enabled_status = 1

SELECT top 1 @serverName = server_name, @queuePath = queue_path

FROM dbo.msmq_machines as A

WHERE A.cluster = @cluster

AND A.enabled_status = 1

order by (number_requests_sent/@totalClusterRequestSent)/(weight_factor/@totalWeight), weight_factor desc

UPDATE dbo.msmq_machines

SET number_requests_sent = number_requests_sent + 1

WHERE server_name = @serverName

AND cluster = @cluster

Note on the T-SQL implementation:

In the definition of the table msmq_machines, I have chosen data types to be float instead of integers (as they actually will hold only integers values) so that no data type conversion is needed when calculating ratios.
@totalClusterRequestSent is calculated as the sum of number_requests_sent + 1 so that if the sum of number_requests_sent is 0 (zero), no division by zero error occurs.

In addition of this, an SQL job will have to run on a regular basis to re-initialize the number_requests_sent column to 0 (zero) so that the value never overflows the data type.

T-SQL weighted Round Robin algorithm Testing

We can test if the algorithm works as expected by creating 3 machines in the msmq_machines table and give them different weights:

If we send 100 messages to the cluster “Bangkok”:

DEV1 should receive (100 / 350) * 100 = 28.57 -> 29 messages
DEV2 should receive (200 / 350) * 100 = 57.14 -> 57 messages
DEV3 should receive (50 / 350) * 100 = 14.28 -> 14 messages

We can create an SQL script that calls the Stored Procedure 100 times and see if the result is what is expected.

declare @counter int

set @counter = 0

while @counter < 100

begin

set @counter = @counter + 1

DECLARE @server as nvarchar(20)

DECLARE @msmq as nvarchar(100)

EXEC [dbo].[GetLoadBalancedQueueName] 'Bangkok' , @serverName = @server out, @queuePath = @msmq out

print @server

print @msmq

end

Opening the msmq_machines table, we can see that the expected number of messages has been sent to each server of the cluster proving that the implementation works.

Load Balancing MSMQ messages

2008-06-17T14:44:00.003+07:00

I am currently writing a database-driven .Net application which needs to send MSMQ messages load balanced across a variable number of machines.

The case scenario is the following:

A server needs to send MSMQ messages to servers A, B and C (we choose 3 MSMQ message recipients for the purpose of this example). The original idea was to put a network load balancer (NLB) between the machine sending MSMQ messages and the recipients.

If the "MSMQ Sender" machine pictured above sends 75 MSMQ messages, the goal is for machines A, B and C to receive 25 messages each. As we know that a network load balancer distributes load on a connection basis, we were hoping that messages would be load-balanced if the .Net code was creating a new System.Messaging.MessageQueue object for each MSMQ messages sent to the NLB. To be more precise, we were hoping that a new connection to the NLB would be created each time we send a MSMQ message with a new instance of the MessageQueue object.

This was pure speculation and a quick test proved that it did not hold; using a simple network load balancer, all the messages were pushed to a single destination server. This happens because connections are re-used within the MSMQ Windows Service, regardless of how the .Net code is written (as a reminder, the .Net class MessageQueue is just a wrapper around the MSMQ Windows service).

Because traffic is load-balanced on a connection basis (not on a message basis) and because the same TCP connection is re-used by the MSMQ service, the NLB forwards all the traffic to the same destination machine. Would the server send 100 MSMQ messages to the NLB, 100 messages would be forwarded to the same target machine as they are all sent using the same underlying TCP connection.

As we have very little control over the way connections are managed within the MSMQ Windows Service, we had to part away with this simple NLB solution and implement an ad-hoc solution.
We chose to implement the load balancing feature in the .Net application itself. The only addition needed to the existing software is a way to configure it so that it can send MSMQ messages to different queue paths. A simple isolated class, module or piece of code could easily do that.

Server Load Balancing: Algorithms

Before choosing how to implement the solution, let’s have a look at different ways (algorithms) on how to implement load balancing.

Random Allocation
In a random allocation, the traffic (MSMQ messages in our case) is assigned to any server picked randomly among the group of destination servers. In such a case, one of the server may be assigned many more requests to process while the other servers are sitting idle. However, on average, each server gets an approximately equal share of the load due to the random selection.
Pros: Simple to implement.
Cons: Can lead to overloading of one server or more while under-utilization of others.
Round-Robin Allocation
In a round-robin algorithm, the traffic is sent to the destination server on a rotating basis. The first request is allocated to a server picked randomly from the group of destination server. For subsequent requests, the algorithm follows the circular order destination servers are listed. Once a server is assigned a request, the server is moved to the end of the list and the next server is chosen for the following request. This keeps all the servers equally assigned.
Pros: Better than random allocation because the requests are equally divided among the available servers in an orderly fashion.
Cons: The round robin algorithm is not good enough for load balancing if technical specification of the servers part of the destination group differs greatly (making that the load each server can handle differs greatly).
Weighted Round-Robin Allocation
Weighted Round-Robin is an advanced version of the Round-Robin that takes in account server capability. In case of a weighted round-robin, one can assign a weight to each server in the destination group. For example, if the server group consists of 2 servers and that one server is capable of handling twice as much load as the other, the powerful server gets twice the weight factor. In such a case, the application would assign two requests to the powerful server for each request assigned to the weaker one. In effect, a server with more weight will receive load proportionally to their weight factor.
Pros: Takes care of the capacity of the servers in the group.
Cons: Does not consider advanced load balancing requirements such as processing time for each individual request.

We have chosen the Weighted Round Robin algorithm as it is the most advantageous load balancing algorithm to implement easily.
We have chosen to implement the algorithm in T-SQL, but it could be easily implemented in a singleton class in any language such as C# or Java. I will explain why and how in a next blog post: T-SQL Weighted Round Robin Algorithm.

SQL Server Execution Plan Tutorial

2008-05-23T11:54:00.003+07:00

Here is a great article written by Grant Fritchey on the basics of analyzing execution plans on SQL Server. It is actually the first chapter of his book on the topic. Yes, a whole book on SQL Server Execution Plans! I haven’t read the whole book but it should probably reveal itself very interesting for DBAs.

Most of us are familiar with the graphical representation of execution plans but Grant Fritchey shows us how to also get execution plans in an XML format and how they actually give more information that their graphical counterpart.

XML executions plans can be saved into files (.sqlplan) and can be viewed either in their XML form or in a graphical form. Being able to save execution plans in a file is very convenient as it makes it easy to share it with DBAs and peers to ask them about their opinion on a slow running query.

Note that on my machine I actually had to edit the .sqlplan file as the first line of the file was a piece of text reading “Microsoft SQL Server 2005 XML Showplan”. This made that the XML file was not well-formed and so could not be loaded by SQL Server Management Studio - this might be fixed in a later Service Pack or hotfix. Once I removed the line of text, I could successfully open the .sqlplan file in SQL Server Management Studio and see the graphical representation of the execution plan. I could also open the file in a text or XML editor and thus see the execution plan in a textual manner. I think that both approaches can prove themselves being complimentary.

The article also shows us how to collect XML execution plans through SQL Server 2005’s Profiler tool. Once execution plans are collected from the server they can be analyzed by DBAs and developers. This can reveal very useful when you want to profile and analyze activities on production environment where you can’t play with data as freely as on a development environment.

For my own sake, I have summarized the introductory materials on the topic hereunder. You might choose to not read it as it is anyway based on the article written by Grant Fritchey.

Execution Plan Introduction.

When you are tuning T-SQL code for performance on SQL Server, the most important information available to you is the Execution Plan. It tells you what kind of JOIN operations and other algorithms are executed as well as what indexes are used. This kind of information will reveal being crucial on poorly performing queries that you need to optimize.

Please keep in mind that this discussion is focused on DML T-SQL.

DML stands for Data Manipulation Language and is aimed at fetching or manipulating data. Basically, it is any SELECT, INSERT, UPDATE and DELETE statement.

DDL stands for Data Definition Language and is aimed at defining data structures. Basically, it is any CREATE, DROP and ALTER statement. DDL statements do not need any query optimization because there is always only 1 way to execute those statements. For example, there is only 1 way to create a table or an index.

When executing a T-SQL query, the T-SQL code is interpreted into instructions understandable by the Database engine. The Database engine is made of multiple processes/sub-engines but 2 are of particular interests regarding Execution Plans: the Relational Engine and the Storage Engine.

Note that in the context of this text, a Process does NOT mean a Windows Process but rather has the more generic meaning of a collection of instructions processing some data. It can be seen as a software module or component.

The Relational Engine.

The Relational Engine is responsible for 3 processes which are of interest in our study:

The Parser. The Parser receives the T-SQL query as input and outputs a parse tree or query tree. The parse tree represents the logical steps necessary to execute the query.
The Algebrizer. The Algebrizer receives the parse tree from the Parser process as input and resolves all datatypes, names, aliases and synonyms. The output is binary information called the query processor tree.
The Query Optimizer. The query optimizer is a piece of software that models the way in which the database relational engine works. Using the query processor tree together with the statistics it has about the data and applying the model, the Query Optimizer works out heuristically what it thinks will be the optimal way to execute the query – that is, it generates an optimized estimated execution plan.

The Storage Engine.

The Storage Engine will execute the estimated execution plan except if it judges that it should be modified. It could be the case if:

The estimated execution plan exceeds the threshold for parallel execution.
The statistics used to generate the plan are out of date.
The estimated execution plan is invalid (for example it creates temp table and so contains DDL statement)

The final execution plan, called the actual execution plan is what is actually executed by the Storage Engine. It might or might not be the same as the estimated execution plan.
Note that generally, there won't be any differences between the esti¬mated and actual execution plans.

Execution Plan Cache.

As it is expensive for the Server to generate execution plans, SQL Server will keep and reuse plans wherever possible. As they are created, plans are stored in a section of memory called the Plan Cache.

Once the estimated execution plan is created, and before it gets passed to the storage engine, the optimizer compares this estimated plan to actual execution plans that already exist in the Plan Cache. This reuse avoids the overhead of creating actual execution plans. This is obviously beneficial for large and complex queries but also for simple queries which could potentially be called very often (hundreds or thousands of time).

Execution Plans can be removed from the cache in the following scenarios:

Memory is required by the system.
The "age" of the plan (its time-to-live) has reached zero.
The plan isn't currently being referenced by an existing connection.

Also, cached execution plans might not be reused if an execution plan needs to be recompiled. Certain events and actions can cause a plan to be recompiled. Grant Fritchey enumerates those events in his article.

New Features in Visual Studio 2008

2008-05-06T22:15:00.003+07:00

In this post I will go through some Visual Studio 2008 tools and features I found interesting.

1. Unit Test Tool

In VS 2008, Unit Testing is facilitated through a unit test class code generator.

Unit Testing is the act of having a piece of code which only purpose is to test another piece of code, this code being part of the end product. It is a particularly tedious task, so having a unit test code generator is very handy.

A Unit Test class is used to test a class that is part of the software being built. If every classes of a software has a matching class used to unit test it, all the code will be unit tested. Unit testing is the lowest level of Quality Assurance, it does not test the software as a whole neither on its external functionalities but rather makes sure that any testable piece of code part of the software is behaving as it should.

You should consider doing Unit Testing as:

It creates test units to check if the code is producing expected results.
It improves code coverage. Code coverage is a number telling how many percent of the code has actually been tested. The higher value the more confident we can be in the quality of the code (as it means that a large part of the code returns expected values).

To generate a Unit Test class in VS 20008, simply right-click on any class definition and select the Create Unit Tests option to call the unit test generator tool.

This will create a Unit Test class in a separated project dedicated for Unit Testing, this way your Unit Test code and the actual code are separated in different projects and so code is neatly kept separated.

Note that if you try to generate a Unit Test Class for a class that has private or internal modifiers, VS2008 will add a special InternalsVisibleTo attribute in your original project so that your Unit Test project (and only that one) has access to all private, internal and protected methods and classes of the original project containing the classes you want to unit test. This means that the attribute is not added at the class level but at the project level in the AssemblyInfo.cs file.

Moreover, as it can be seen hereunder, only the Unit Test project (here called CalculatorTest) will have access to internal classes and methods:

[assembly: System.Runtime.CompilerServices.InternalsVisibleTo("CalculatorTest")]

Once the Unit Test classes are generated, you will have to go into them, inspect the code and set input values that will be used to call the methods that need to be tested and also set the expected result value that should be returned by the method. This way, the actual value returned can be compared with the expected value to decide if the test was conclusive or not. There are TODO sections declared in the generated code so that you can easily locate where to set test values.

In the example hereunder, values are set for the length and width of the rectangle and the variable expected contains the pre-calculated area of the rectangle so that we can check if the output of the method is equal to what should be returned. If the value returned by the method is not what is expected, that means that the unit test fail and that there is a bug in the tested method. Visual Studio will clearly show in the test results what unit test succeeded or failed.

Lastly, do not forget to comment out the Assert.Inconclusive() method call.

The Test project creates a vsmdi file in the solution item folder, named after the solution name, Calculator.vsdmi for a Visual Studio solution called Calculator.

To actually test some or all the unit test methods, open the vsdmi file, select the methods that you want to test, right-click on the list of methods and select Run Checked Tests.

Once the test ran, the result will be displayed showing if the test succeeded, failed or was inconclusive.

2. Object Test Bench

If you do not have the time to create full blown Unit Test but still want to test some of your classes and/or methods, there is a quick and dirty way to do it by using the Object Test Bench tool. The tool can call static methods on classes and create object instances of classes so that instance methods can be called ad-hoc. Methods can thus be tested very simply, in a similar way as when you chose to execute a Stored Procedure in SQL Server Management Studio and that a window pops up to let you input the SP parameters.

To use the Object Test Bench, you need first to create Class Diagram of your project. To do so, click on the Class View of your Visual Studio project, select the root namespace of your project, right-click on it and choose the View Class Diagram option.

Once the class diagram is created you can call the Object Test Bench tool by either calling a static method on the class or one of the class constructor so that you will be able to call an instance methods after the object is instanciated.

In my case I want to test the Add() method to check if my Calculator object correctly adds numbers. To do so I first instanciate a Calculator object which opens the Object Test Bench window under the class diagram. After the object is created in memory and appears in the Object Test Bench window, I can chose to call any method of the object. I will thus dynamically call the Add() method through the Visual Studio IDE and check if the method returns a correct result.

A window will pop up so that you can give values to the input parameters.

Once the parameters are entered and you click on OK, a pop up window will display the result. You can choose to save the result in a variable so that you can re-use it later. For example you could re-use the variable as a parameter to call another method that you want to test-bench. This way, in a few clicks, you can create a bunch of objects and then re-use them later to call methods that take complex-type objects as parameter.

3. Generate Method Stub (only for C#)

It is a code generation feature that creates methods before they exist; the method created is based on the call to that method. Once the call to the method is made, Visual Studio 2008 IntelliSense will give you the option to generate a method stub matching the call to that method, with the matching input parameters and return type.

The generated method will be created in the matching class and its stub implementation will simply throw a NotImplementedException.

Note that this code generation feature is only available for C#.

4. Refactoring Tools (C# only)

Refactoring is making changes to a body of code in order to improve its internal structure without changing its external behavior.

- Martin Fowler

It is useful concept to make code cleaner and more understandable/readable.

A typical refactoring case is to break up a lengthy method into separate methods. To do so, you can highlight a piece of a code, right-click on it (or go to the Refactor menu of Visual Studio) and choose Extract Method. This will generate a method containing the highlighted code as well as calling the generated method from the original location.

Refactor menu in Visual Studio 2008:

Refactoring code by creating a method to shorten the original code, it is the action of extracting a method from a piece of code.

The example here above will create a method containing the highlighted code and replace the original code by a call to the generated method (which I called WriteLogToConsole):

5. The .Net Framework Source Code

It is possible to go through the source code of the .Net Framework while debugging. Here are the steps I did to make it work:

1. Install the hotfix KB 944899 - Visual Studio 2008 performance decreases when you step through source code that you downloaded from Reference Source Server.

2. Configure Visual Studio debugger to be able to step in the .Net Framework Source Code:

Uncheck Enable Just My Code (Managed only).
Check Enable source server support.

3. Configure the Symbols part of the Visual Studio debugger options so that Visual Studio know where to download the .Net Framework debugging symbol (.pdb files) and source code.

Set the Symbol file (.pdb) location to be: http://referencesource.microsoft.com/symbols
Set a Cache location folder where the .Net Framework pdb and source code files will be stored. Make sure it is a location that your account has read/write access to. For example, a folder under your user hive.
Clear the Search the above locations only when symbols are loaded manually if you want that Visual Studio automatically download symbols and source code while you step in .Net Framework code (F11 shortcut key). Note that if your project is big and references many libraries, downloading all the debugging symbols will be slow at the first debug. If you prefer to load symbols only when needed, keep that box checked. You will then have to download debugging symbols and source code on demand by right-clicking the appropriate dll in the stacktrace and choose Load Symbol.

Here is how I configured my Visual Studio:

Visual Studio 2008 is now all set to debug and step in .Net Framework Source Code!

While debugging, we can now see that the debugger call stack contains detailed file and line number information for the .NET Framework classes and methods:

Example of use:

In the following screenshot, I stepped in a line of code that calls the ToString() method on a Double type, this makes that the mscorlib pdb file is downloaded as well as the source code for the Double structure so that I can actually debug into the Double type and see its implementation as written by the .Net team. That is something I find really cool and I think has been missing for a long time!

Modules Window:

While you are debugging, you can bring up the Modules Wwndow by hitting the ALT+CTRL+U keys. This window shows all the dll loaded by the debugger and let you see which dll has debug information loaded and which does not. You can manually load debugging symbols from that window by right clicking on the library you want to load the symbols for and select the Load Symbols option.

List of assemblies currently available at the time of writing for symbol/source loading:

Mscorlib.dll
System.dll
System.Data.dll
System.Drawing.dll
System.Web.dll
System.Web.Extensions.dll
System.Windows.Forms.dll
System.XML.dll
WPF (UIAutomation*.dll, System.Windows.dll, System.Printing.dll, System.Speech.dll, WindowsBase.dll, WindowsFormsIntegration.dll, Presentation*.dll, some others)
Microsoft.VisualBasic.dll

For reference, here is a lengthier blog post by Shawn Burke with more information regarding .Net Framework Source Code debugging.

6. SQL Metal

SQL Metal is used to help implementing LINQ to SQL scenarios. It is a command-line utility (sqlmetal.exe).

SQL Metal can:

Generate source code and mapping attributes or a mapping file from a database.
Generate an intermediate database markup language file (.dbml) for customization from a database.
Generate code and mapping attributes or a mapping file from a .dbml file.

What is a mapping file?
A mapping file is an XML file to specify mapping between the data model of the database and the object model of the .Net code. It keeps the mapping code out of the application code which helps in keeping the code cleaner and leaner. Moreover, since it is XML (like any other .config file) it can be changed without having to rebuild the application code.

Check out MSDN documentation for more information about SQLMetal.

7. Visual Studio 2008 Product Comparison

Great comparison between the different functionalities available on each edition of Visual Studio 2008.
http://msdn2.microsoft.com/en-us/vstudio/products/cc149003.aspx

Promoted property vs distinguished field - tutorial

2008-04-10T11:32:00.012+07:00

First of all while many people talk about Promoted Property, the official term is Promoted Field, as defined in the MSDN documentation and also visible in Visual Studio’s BizTalk Schema Editor. I will nevertheless keep using the term most people are familiar with but I will consider both terms as synonyms.

Part 1 & 2 is a tutorial on how to create promoted properties and distinguished fields through the Schema Editor and the BizTalk’s API while part 3 is discussing about performance and differences between Promoted Properties and Distinguished Fields. You might only be interested in the 3rd part of this article if you already are a seasoned BizTalk developer.

1. Promoted Properties (Promoted Fields)

1.1 What are Promoted Properties?

Promoted Properties are Message Context Properties that are flagged as promoted; being promoted it allows the Message Engine to route messages based on their value, and being in the message context allows doing so without having to look at the message payload (which would be an expensive operation). Promoted Properties are the most common way to enable content-based routing. They are available to the pipelines, adapters, Message Bus and orchestrations.

Promoted Fields (= Promoted Properties) are PROMOTED in the message context by the receive pipeline when a message is received on a port. It is usually the job of the disassembler pipeline component such as the XML and Flat File disassembler but any custom pipeline component can also do it.

1.2 How to promote properties?

As stated in a previous post I wrote, BizTalk Messaging Architecture, message context properties are defined within a property schema and so all promoted properties must be defined in a custom property schema.

The action of promoting a message element to a promoted property creates a message context property that will contain the message element value and flags it as promoted so that it is available for routing.

There are 2 ways to promote a message element:

1. Quick promotion

Quick promotion is the simplest way to create a promoted property. Simply right click on the element’s node and choose Quick Promotion (see Fig 1.1). When choosing this option, Visual Studio will create a property schema called PropertySchema.xsd and add in the message’s schema a reference to the generated property schema.

Each property promoted this way will create a corresponding element in the property schema with the same name and type as defined in the message’s schema.

This means that when using quick promotion, the promoted property element name will always be the same as the message’s element name. If you have several elements with the same name, you might want to use manual promotion instead to avoid confusion or avoid having a property value overridden.

Fig. 1.1 Quick Promotion

2. Manual Promotion

To manually promote a property, a property schema must be created with the elements that will hold the promoted property values. To create a property schema, you need to add a new item in your BizTalk solution, and chose Property Schema as the type of file (see Fig. 1.2). Once all the elements are created in the property schema, you associate the property schema with the message’s schema by right clicking on any node of the message schema and choose Show Promotions (see Fig. 1.3), then click on the Property Fields tab, click on the folder icon and finally select the property schema you just created (see Fig. 1.4). Note that it is actually possible to use more than 1 property schema per message. Anyhow, all promoted properties will end up being written in the message context and available for all BizTalk artifacts having access to message context.

Once the property schema is picked, you can start promoting message elements as promoted properties. To do so, click a message element and click on the add button, the Node Path column will display the XPath to the message element you are promoting and the Property column let you chose the promoted property that will contain the value of the message element at runtime (see Fig. 1.5).

An interesting side effect is that Manual Promotion let you have promoted properties with different names that the original message element name. This might be useful when a same property schema is used to hold promoted properties from different message types or when a message has different element with the same name.

Using Manual Promotion, it is also possible to promote message elements to promoted properties in the system property schemas shipped with BizTalk. To do so, just browse in the References sub-tree when picking the property schema (see Fig. 1.4).

Fig. 1.2 Manually create a custom property schema

Fig. 1.3 Manual Promotion

Fig. 1.4 Selecting a Property Schema

Fig. 1.5 Selecting the message element to promote and the promoted property that will contain the message element’s value.

1.3 How to promote properties through the BizTalk API?

As I already said in a previous post, BizTalk Messaging Architecture, context properties are stored in a property bag, an object which implements the IBaseMessageContext interface. This interface contains 2 methods which take the same parameters (a property name, an XML namespace, and value for the property).

a. The Write() method is used to write the property value into the message context without actually promoting it. It can be used to write distinguished fields or to write transient values. Calling the Write() method is NOT promoting a property, it is writing a property.

b. The Promote() method is used to write the property value into the message context but also flags the property as promoted so that it is available for routing. This is the method that needs to be called to promote a property.

To promote a property through the API, element name and namespace passed as parameter of the Promote() method must be those of the property defined in the property schema. They are most easily accessed by referencing the property schema assembly and using the properties on the class created for the property.

Example of promoting a property through an API call (copied from the BizTalk 2006 MSDN documentation: Processing the Message):

//create an instance of the property to be promoted
SOAP.MethodName methodName = new SOAP.MethodName();

//call the promote method on the context using the property class for name and namespace
pInMsg.Context.Promote(methodName.Name.Name, methodName.Name.Namespace,
"theSOAPMethodName");

As I mentioned before, it is possible to promote properties to the BizTalk system property schema using Manual Promotion in Visual Studio’s Schema Editor for BizTalk. It is possible to achieve the same programmatically and as for other promotion, the name and namespace passed in parameter of the Promote() method is the one of the promoted property in the property schema, the system property schema namespace in this case.

//BizTalk system properties namespace
private const string BTSSystemPropertiesNamespace = “http://schemas.microsoft.com/BizTalk/2003/system-properties”;

//Promote the MessageType property
string messageType = "http://" + "schemas.abc.com/BizTalk/" + "#" + "Request";
message.Context.Promote("MessageType", BTSSystemPropertiesNamespace, messageType);

2. Distinguished fields

2.1. What are distinguished fields?

Distinguished fields are message elements that are written into the message context. They differ with promoted properties in 2 main aspects:

They are not flagged as promoted in the message context and so are not available for routing by the Messaging Engine (adapters, pipeline…). Their typical use is instead for the orchestration engine.
They are not defined using a property schema.

Distinguished Fields are WRITTEN in the message context by the pipeline when a message is received on a port. It is usually the job of the disassembler pipeline component such as the XML and Flat File disassembler but any custom pipeline component can also do it.

Distinguished fields are useful when a message element value needs to be accessed from an orchestration. Instead of having the orchestration engine search through the message to evaluate an XPath expression (which can be resource-intensive on a large message), a distinguished field can be used. Distinguished fields are populated in the message context when the message is first loaded. Consequently, each time the distinguished field needs to be accessed, the orchestration engine will directly read it from the message context (a property bag object) instead of searching for the original value in the message with XPath. Needless to say that retrieving a single value from a property bag object is much faster than evaluating an XPath expression.

Distinguished fields also offer IntelliSense in the orchestration expression editor which enhances code readability.

2.1 How to create a distinguished field?

The main source of confusion between distinguished fields and promoted properties is that they are both created in Visual Studio’s Schema Editor through the Promote -> Show Promotions contextual menu option of a message schema’s element. Once the dialog box is open, make sure that you are on the Distinguished Field tab, select the message elements and click the Add>> and <<Remove buttons to add and remove distinguished fields (see Fig 2.1).

Fig 2.1 Creating a Distinguished Field.

2.2 How to create a distinguished field through the BizTalk API.

Distinguished fields are written into the context using the Write() method on the IBaseMessageContext object. To be recognized as a distinguished field, the namespace of the property must be “http://schemas.microsoft.com/BizTalk/2003/btsDistinguishedFields”. Pipeline components delivered with BizTalk do not use those context properties. It is nevertheless possible to read/write distinguished field in the code of custom pipelines, as for any other context properties.

Example of writing a distinguished field through an API call (taken from the BizTalk 2006 MSDN documentation: Processing the Message).
//write a distinguished field to the context
pInMsg.Context.Write("theDistinguishedProperty",
"http://schemas.microsoft.com/BizTalk/2003/btsDistinguishedFields",
"theDistinguishedValue");

Would you use another namespace, it will result in writing a plain transient value in the property bag and won’t be recognized as a distinguished field by the orchestration engine.

//Write a transient value to the message context
message.Context.Write("MyVariable", "SomeNameSpace", SomeData);

3. Considerations, similarities and differences between promoted properties and distinguished fields.

3.1 Performance considerations:

Promoted properties are limited to 255 characters for routing performance reasons. Properties that are simply written in the context (such as distinguished fields) are not limited in size but large properties decrease performance.
All Context properties (both Promoted and Distinguished Fields) are stored separately from the message in the Message Box Database. Consequently consuming more space in the BizTalk databases but more importantly incurs more load when persisting/reloading the message in/from the DB. Note that if tracking is turned on, Promoted Properties are also stored in the Tracking database.
Distinguished Fields cost less than Promoted Properties in terms of performance. Both Promoted and Distinguished Fields require the same overhead of writing their values to the context Field bag in the Message Box database, but Promoted Fields have the additional overhead of being written in BOTH to the Message Box context tables AND the subscription tables. Distinguished fields are not stored in the subscription table as they do not participate in routing.
Promoted Fields have an impact every time a message is written in the Message Box because each Promoted Property that exists must be evaluated in a massive T-SQL union statement that builds the list of matching activation subscriptions. In short, the more Promoted Fields you have the more costly the subscription process is.

3.2 Other considerations

Both promoted properties and distinguished fields are populated when a Pipeline Disassembler Component parses a message and either Promotes or Writes the value to the message’s context.
Empty pipelines such as the Pass-through pipeline do not promote or write anything in the message context as it lacks a disassembler component.
Writing a value into the context with the same name and namespace that were used previously to promote a property causes that property to no longer be promoted. The write essentially overwrites the promotion.
Writing a property in the context having a null value deletes the context property altogether because null-valued properties are not permitted.

3.3 Distinguished and Property Fields Difference Summary.

Promoted Fields should be used for routing, correlation and/or tracking.
Distinguished fields should be used when a particular message element is commonly manipulated in one or more orchestration.

Here is a table outlining the main differences between both types of fields:

Promoted Fields (aka Promoted Properties)	Distinguished Fields
Used for routing (subscription mechanism) IsPromoted = true	Do not participate in routing IsPromoted = false
Used for tracking	Not used for tracking
Restricted to 255 characters	No size limitation
Available for use in orchestrations	Available for use in orchestrations
Require property schema	Do not require property schema
Used by standard pipeline components	Accessible only by custom pipeline component which would explicitly access them

4. References.

MSDN - The BizTalk Server Message
MSDN - Processing the Message
MSDN - About BizTalk Message Context Properties
Neudesic's blog - Distinguished fields myths

Windows Process, .Net Application Domain and 2 GB limit on 32-bit Windows

2008-03-31T18:08:00.005+07:00

A few weeks ago I heard some comments from a colleague about how .Net applications run and that “all .Net applications run in the same runtime (CLR) so that if you start 10 separate .Net applications, they would share together a single 2 GB limit on Windows 32-bit”. This of course not true and it gave me the idea to blog about the 2GB limit on 32-bit systems, Windows Process, .Net applications and the concept of .Net Application Domain.

2 GB limitation on Windows 32-bit.

32-bit Operating Systems are capped by the number of unique pointers that can exist at a time. On 32-bit processors, only 2^32 distinct addresses can exist. Would all these addresses be used, that would represent 4 GB of memory. On a Windows Operating System the memory address system is not a 1-to-1 relationship to the physical memory of your hardware otherwise you would be stuck with a maximum of 4 GB of addressable memory for the whole machine. This would include all the I/O address space, kernel memory and so leave much less actual memory for programmers to use.

This is why when we talk about memory, it is important to realize the distinction between the physical memory (RAM on the motherboard) and the Virtual Memory accessible through the Virtual Address Space. Note that actually, Virtual Memory is not the same as Virtual Address Space and that there are ways to use Virtual Memory without using the Virtual Address Space. I will nevertheless not go into these details; the important thing to remember is that Windows has a complex memory management system that enables the O.S. to use much more than 4 GB as a whole. The inner workings are not for the faint-hearted and are actually not of interest for most .Net programmers living in the managed world.

Check this blog post for a primer on memory management on Windows Operating System.

When Windows 32 starts a program, a 32 bit process using 32 bit size pointers is created and so the process has a maximum of 4 GB of addressable memory.
Windows will assign to the process a Virtual Address Space of 4 GB (2^32) split in two; 2 GB of user mode virtual address space and 2 GB of kernel mode virtual address space. The user mode virtual address space is the “memory” (read the virtual address space to be correct) available for your program to use.
This 2 GB user mode virtual address space limit is what is commonly called the 2 GB memory limit on Windows 32-bit.

/3 GB switch on 32-bit Windows

The /3GB switch changes the way the 4GB virtual address space is split up. With the /3GB switch, the split is 3GB of user mode virtual address space and 1GB of kernel mode virtual address space. It is nevertheless not recommended to use this option as it can bring unexpected bug from drivers and other kernel-mode processes which might expect to have 2 GB of kernel virtual address space available (not that a driver would ever need 2 GB, just that an older driver might expect to have addresses from 0x80000000 to 0xFFFFFFFF available).
See here and here for other problems that can arise when using the /3GB switch.

AWE

AWE does not give more virtual address space to a process. AWE stands for Address Windowing Extension and is a Microsoft API (Application Programming Interface) that allows a 32-bit software application to access more physical memory that it has virtual address space.
AWE enables programs to reserve physical memory as non-paged memory and then to dynamically map portions of the non-paged memory to the program’s working set of memory. This process enables memory-intensive programs, such as large database systems, to reserve large amounts of physical memory for data without having to be paged in and out of a paging file for usage.
To be clear, AWE can only be available on programs that actually use the AWE API, it is not an OS switch that can be turned on/off on any program.

Windows 64-bit

On windows 64-bit there is not 2 GB limit, the user mode virtual address space limit being 8TB. See here for reference.

Windows Process and Runtime Host

A Windows process is an instance of a program that is executing over the Windows layer. A process contains the executable code and data inside the memory reserved for it by the Operating System. There will be at least one thread executing instructions within the process but more in most cases.

Any program running on Windows is actually working within a process. If you open 2 instances of notepad, you can see that 2 processes running notepad.exe are visible under the Processes tab of the Windows Task Manager.

The concept of a Process exists for two main reasons:

To enable multitasking (time sharing), the different processes a CPU is running will have their states changing between running and waiting very quickly and so give the illusion to the end-user that all processes are running in the same time. This brings multitasking as well as scalability.
To provide boundaries between running programs so that a process cannot peak into another one and that erroneous code inside a process cannot corrupt areas outside of that process (so that a process cannot crash another one). This is brings security and stability.

The isolation between processes is achieved by making sure that any given unique virtual address space runs exactly into one process and not any other.

Runtime Host

.Net applications are compiled in CIL (Common Intermediate Language, formally called MSIL – Microsoft Intermediate Language), and then are JITed (Just-In-Time compiled) by the CLR (Common Language Runtime) into instructions directly understandable by the CPU (native code).
Here is an illustration of this 2 step compilation process:

This means that .Net applications are not Win32 applications and so cannot be executed directly by the Operating System. As any application running on Windows has to run through a Windows Process, a Windows Process called a Runtime Host will actually execute (host) the .Net Application. The Runtime Host first loads the CLR dll (a native Windows library – unmanaged code) which in turn loads the .Net application (managed code), JIT compiles it and runs it. The process thus effectively transitions the control of running the application from itself to the CLR.

There are 2 types of Runtime Host shipped with the .Net Framework, ASP.NET and Shell. Shell runs all Windows-type applications (Windows Form, Windows Service or Console App).

We can see that this concept actually adds a new layer between the .Net application and the Operating System. This layer, implemented by the CLR, is generically called a Virtual Machine and has OS-like features. It is an abstraction layer between the .Net application and the Operating System. As with Java, this permits any .Net Application to run on any Operating System as long as there is a CLR implemented for that OS.

2 GB limit for .Net applications

As the Runtime Host is a Windows Process, the .Net applications run by a Runtime Host is limited to the 2GB barrier on 32-bit Windows OS. Nevertheless, every Runtime Host has a separate 2 GB virtual address space limit. So would you launch 2 instances of a .Net application, each being a separate process in Task Manager, they would each have 2 GB limit.

.Net Application Domain

An Application Domain is the CLR equivalent of an Operating System’s process. As the Windows OS brings logical and physical isolation between Windows applications through the use of Processes, a single Runtime Host Windows Process can run several isolated .Net applications through the use of Application Domains. As explained before, Windows isolate processes by assigning different virtual memory address space to each process. In the .Net world, the memory is actively managed by the CLR and so the CLR can make sure that memory addresses are not shared between application domains, effectively isolating different Application Domains running in the same Runtime Host.

When a Runtime Host starts a .Net application, the CLR will create a default Application Domain to run the .Net application. As multiple Processes can run on a single OS, multiple Application Domains can run within the same Runtime Host.

An Application Domain is cheaper to create than a Windows Process and has relatively less overhead to maintain. It is thus more efficient to isolate .Net Application through Application Domains rather than Windows Processes. Application Domains are sometimes referenced as lightweight processes but strictly speaking, they are NOT processes.

To summarize, here is a list of advantages of having Application Domains within a Runtime Host Process (which are for most of them similar to the advantages of having Processes within an Operating System):

An Application Domain is a more lightweight mean to provide isolation between .Net applications than Processes.
A .Net application in an Application Domain can be stopped without affecting the state of another application running in a separate Application Domain.
A crash in an Application Domain will not affect other Application Domains neither the Runtime Host Process hosting the Application Domains.
Configuration information is part of an Application Domain scope, not the process’ scope.
Each Application Domain can have different security access levels assigned to them, all within the same Runtime Host Process.
Code in one .Net Application Domain cannot directly access memory in another Application Domain. If two .Net applications need to communicate across Application Domains, they need to use .Net Remoting to do so. In .Net 1.x, this kind of inter-process communication was expensive because the TCP/IP stack needed to be involved. In .Net 2.0, .Net Remoting supports named pipe remoting which is much more efficient. WCF in .Net 3.x has this feature as well.

List of BizTalk prerequesites redistributable CAB files

2008-03-24T11:55:00.001+07:00

While I was installing BizTalk Server 2006 the other day, I decided to compile a list of links to the redistributable BizTalk prerequisite CAB file so that I do not have to search for it the next time I install BizTalk Server.

The BizTalk prerequisites CAB file is a compilation of all the prerequisites that are needed to install and run BizTalk Server.
The redistributable CAB file is different for different Windows versions (Windows XP, Vista, Server 2003, 64-bit or 32-bit) and languages (English, French, ...). So, when chosing the prerequesite CAB file, make sure that it is the one matching your Windows Operating System version and language.

When installing BizTalk Server, the setup procedure gives you the choice to either automatically install the prerequisites from the web or to automatically install the prerequisites from a redistributable CAB file.
Choosing the redistributable CAB file options helps in speeding up the installation process and is necessary if the BizTalk server has a limited access or no access at all to the internet.

BizTalk Server 2006

For BizTalk Server 2006, the links to the redistributable prerequisites CAB files are listed in the BizTalk Server 2006 Installation and Upgrade Guides. The guides are MS Word documents containing links to all the prerequisites CAB files for all supported versions of the Windows Operating System.

BizTalk Server 2006 R2

For BizTalk Server 2006 R2, the links to the redistributable prerequisites CAB files can be found in the BizTalk Server 2006 R2 Installation and Upgrade Guides. Here again the guides are a bunch of different MS Word documents containing links to all the prerequisites CAB files for all supported versions of the Windows Operating System.

As BizTalk Server 2006 R2 is the current release of BizTalk Server 2006, the links to the prerequisites CAB files can also directly be found on the MSDN documentation site:
http://msdn2.microsoft.com/en-us/library/aa578652.aspx

For information purpose, here is a list of all requirements for BizTalk 2006 R2:
http://www.microsoft.com/biztalk/techinfo/2006R2/sysreqs.mspx

Biztalk 64 bit high CPU utilization

2008-03-19T12:20:00.002+07:00

I started to deploy BizTalk server 2006 64-bit edition on Windows Server 2003 64-bit in our production environment and we noticed an unusual high CPU utilization by the BizTalk host processes.

The symptoms were twofold:

1. The BizTalk 64 bit servers suffered from a high CPU usage (one server had its CPU usage constantly stuck at 100%). This happened even when the load on the server was low, in fact much lower than on other BizTalk Server 32 bit machines running the same applications on a similar hardware.
The reason for the server high CPU usage was that some BizTalk host processes had their CPU usage stuck at a high value even if the BizTalk application(s) run by the host would process very few messages or no message at all. Actually, the CPU usage for those hosts would still be stuck at a high value even if the BizTalk application(s) run by the host were stopped!

2. Even when the CPU utilization was showing 100%, the server was still processing messages as fast as you would expect from a server under little load. No messages were queued for processing and no orchestrations were dehydrated!

Those 2 points gave me the feeling that nothing could be wrong with the BizTalk server itself (by that I mean the BizTalk code and the BizTalk installation) as the same code is running fine on 32 bit servers and as the performance monitor readings did not match the actual facts.

I requested help from Microsoft Support and they guided me to the KB 943165 which has a hotfix solving the high CPU usage problem I encountered on BizTalk Server 64-bit edition.
This hotfix fixes a problem brought by the security bulletin MS07-040, a security update for the .Net framework which brings CPU usage spikes on BizTalk Server 64 bit (meaning that the problem is only for the 64 bit version of the .Net framework). For information, Microsoft Support told me that the problem is suspected to be caused by an infinite loop in CounterManager.RunCacheThread when System.Threading.ThreadAbortException is raised.

I should also add that I could not actually find a Windows update directly related to the bulletin MS07-040 in the list of updates installed on the server running BizTalk Server 64-bit. Nevertheless, my Microsoft Support contact told me that the version of mscorwks.dll (found in c:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727) has a timestamp showing that it is later than the hotfixes issued by bulletin MS07-040 and that the problem should then also occur with any later version of the dll (as it is a code change from that version on that brings the problem).

To conclude, would you encounter the same symptoms I enumerated above, you will want the check the KB 943165 and contact Microsoft Product Support to see if your case is applicable for this hotfix. They will then email you the details to download the hotfix.

I installed this hotfix on all my BizTalk Server 2006 64 bit machines and all of them are running smoothly now :)

Turn off tracking globally in BizTalk Server 2006

2008-03-13T16:20:00.001+07:00

The BizTalk Tracking (BizTalkDTADb) database grows in size as BizTalk Server processes data on your system. If the size of the BizTalk Tracking database causes poor disk performance or fills up the disk subsystem, you can manually purge the data from the Tracking database. See my previous post about how to purge and maintain the BizTalkDTADb database.
If you repeatedly have issues with the BizTalk tracking database, you may want to configure BizTalk to no longer collect tracking information. This is possible by turning off global tracking for the whole BizTalk Server.

Here is the procedure to turn off tracking globally for BizTalk server 2006 and 2006 R2:

Open SQL Server Management Studio and connect to the Database Server where the BizTalk Management Database is running, BizTalkMgmtDb
Expand the BizTalkMgmtDb, database, expend Tables, right-click the adm_Group table, and then click Open Table.
In the GlobalTrackingOption column, change the value from 1 to 0 and then press ENTER. A value of 0 turns off global tracking for the whole BizTalk server while a value of 1 turns it on.
Restart all your BizTalk hosts for the change to take effect.

As when Tracking is turned off, tracking information is not collected so no information will be available in HAT anymore (Health and Activity Tracking). You should consider this effect against keeping tracking data for a shorter time when thinking about turning off Tracking globally.

An alternative to turn off Tracking globally is to turn it off on an application per application basis.
I think that once a BizTalk application is deployed and running smoothly for a while, there is no reason to still have tracking turned on at all time.
If you deploy new BizTalk applications or keep updating existing ones on your production server regularly, you will probably want to be able to consult HAT after the application is freshly deployed. For that to be possible you would have to keep global tracking on.
In that case, I think that a best practice kind of approach is to have Tracking turned on for the freshly deployed application for a few days until you deem it running fine so that you do not need to consult HAT anymore. Once HAT is not needed regularly anymore, you can use the BizTalk Server Administration Console to disable tracking for all the artifacts belonging to the concerned application (orchestrations, ports, etc.). Would you need to turn it on again, it would just take a few minutes to configure tracking back on for the application's artifacts.

Reference can be found at the MSDN BizTalk documentation How to Turn Off Global Tracking

BizTalk Messaging architecture

2008-03-10T14:15:00.011+07:00

The core of the BizTalk Server product is the BizTalk Messaging subsystem, called the Message Bus. As said in the BizTalk documentation, The Message Bus is a publisher/subscriber model; indeed the Message Bus queries messages published into the BizTalk Message Box database looking for messages that match a particular subscription.

The most important point to understand about the publisher/subscriber model is that publishing and subscription concepts are relative to the Database.

Here is a picture illustrating the concept:

The Messaging infrastructure is composed of the Message Box database and also different software components, called the Messaging Components. The Db and the components together compose the publisher/subscriber BizTalk Messaging subsystem, the Message Bus.

An important side effect of this architecture is that in BizTalk Server, messages are immutable once published (in the Message Box DB). This is because more than one end-point can subscribe to the same message. Would messages be mutable, some end-points might not match the subscriptions rule after the message has been modified. Having the subscription query result vary with time on the same message (due to change in the message payload) would break the publisher/subscription architecture thus making the whole BizTalk product unpredictable (and so useless).

1. The Message Box.

The Message Box is an SQL Server database which stores XML messages as well as metadata related to each messages. The message’s metadata is called the message context. Each metadata item (a key/value pair) of the message context is called a context property. The most important information to know about the message context is that it holds all the necessary information for message routing - message subscription.

2. Messaging Components.

While the Message Box database is the message storage facility of the Message Bus, the messaging components are software components that actually move messages between subscribers and publishers. They receive and send messages in and out of the BizTalk Server system.

2.1 Host Services.

A BizTalk host is a logical container. It provides the ability to structure the BizTalk application into groups that could be spread across multiple processes or machines.
When you create a host in BizTalk server, it creates a logical unit in which you can run different BizTalk applications or different type of BizTalk artifact.
For example, if your BizTalk applications are pretty small, you could create 1 host per BizTalk application you develop. On the opposite if your applications are big, you can create different hosts to separate logical grouping of your application, such adapters, orchestrations, ports and so on. If each host runs on a separate physical machine this help in balancing the load between processors (some sort of manual load balancing).

A host instance is simply a running instance of the host logical grouping. It runs as a Windows Service, each host being a separate windows process; a separate instance of BTSNTSvc.exe (or BTSNTSvc64.exe for BizTalk Server 64 bit).

As explained before, the host instance raison d’être is to provide logical grouping units. It does not implement itself the BizTalk runtime, it is a container where the BizTalk subservices run. These subservices running inside the host instance implements together the actual runtime of the BizTalk Message Bus.

Host instances can run all BizTalk subservices or only some of them depending on what type of BizTalk artifacts they are running. To understand which subservice is used by which type of artifact, here is a list of the different subservices running inside a BizTalk host instance - note that the list of services can be found in the the adm_HostInstance_SubServices table in the Management Database:

Service	Description
Caching	Service used to cache information that is loaded into the host. Examples of cached information would be assemblies that are loaded, adapter configuration information, custom configuration information, etc.
End Point Manager (EPM)	Go-between for the Message Agent and the Adapter Framework. The EPM hosts send/receive ports and is responsible for executing pipelines and BizTalk transformations. The Message Agent is responsible to search for messages that match subscriptions and route them to the EPM.
Tracking	Service that moves information from the Message Box to the Tracking Database.
XLANG/s	Host engine for BizTalk Server orchestrations.
MSMQT	MSMQT adapter service; serves as a replacement for the MSMQ protocol when interacting with BizTalk Server. The MSMQT protocol has been deprecated in BizTalk Server 2006 and should only be used to resolve backward compatibility issues.

2.2 Subscriptions.

In a publish/subscribe design, you have three components:

Publishers
Subscribers
Events

Publishers include:

Receive ports that publish messages that arrive in their receive locations
Orchestrations that publish messages when sending messages (orchestration send shape)
Orchestrations that start another orchestration asynchronously (start orchestration shape). On a side note, the call orchestration shape does not publish the message into the Message Box, the message is just passed as a parameter.
Solicit/response send ports publish messages when they receive a response from the target application or transport.

Subscriptions:

Subscription is the mechanism by which ports and orchestrations are able to receive and send messages within BizTalk server (see picture above).

A subscription is a collection of comparison statements, known as predicates, comparing the values of message context properties and the values specific to the subscription.

There are two types of subscriptions: activation and instance.

An activation subscription is one specifying that a message fulfilling a subscription should create a new instance of the subscriber when it is received. Examples of things that create activation subscriptions include:

Send ports with filters
send ports that are bound to orchestrations
orchestration receive shapes that have their Activate property set to true.

An instance subscription indicates that messages fulfilling the subscription should be routed to an already-running instance of the subscriber. Examples of things that create instance subscriptions are:

Orchestrations with correlated receives.
request/response-style ports waiting for a response.

It is also important to know that when you define filter criteria on a send port, you are actually modifying the subscription of the port. As a reminder, filter expressions determine which messages are routed to the send port from the Message Box.

Enlisting:

The process of enlisting a port simply means that a subscription is written for that port in the Message Box. Consequently, un-enlisted ports do not have subscriptions in the Message Box.
The same is true for other BizTalk artifacts. An un-enlisted orchestration is an orchestration ready to process messages but having no way to receive messages from the Messaging Engine as no subscription is created for it yet.

The difference between an un-enlisted artifacts and a stopped artifacts is that ports and orchestrations that are enlisted, but not started, will have any messages with matching subscription information queued within the Message Box and ready to be processed once the artifact is started. If the port or orchestration is not enlisted, the message routing will fail, since no subscription is available and the message will produce a “No matching subscriptions were found for the incoming message” exception within the Windows Event Log.

Typical port usage with an orchestration:

What happens in an orchestration that as a send shape connected to a logical port which is in turn bound to a physical port, is that the message sent by the send shape will have a TransportID context property set to a value that matches the physical port TransportID. As the TransportID uniquely defines the port, this mechanism assures that the physical port will always receive the messages coming from the orchestration. It does not mean that only that port will receive the message as due to the nature of a publisher/subscriber architecture, any other port having a subscription matching the message context will also receives the message.

2.3 Messages

As said earlier a Message is more than just an XML document. It is actually a message containing both data and context. To be more precise, a message is composed of context properties and zero or more message parts.

Keep in mind that message parts are not always XML document. If the message is received through a port using the pass-through pipeline, the message can be any kind of data including binary data. On a side note, a pass-through pipeline does not promote context properties; this makes sense as the message is not even supposed to be XML in a pass-through pipeline, so it is not possible to evaluate XPath expression on the message to determine the value of the context property.

As said earlier a message is immutable once it is published. This means that once stored in the MessageBox DB, it can’t be changed. A message can nevertheless be changed once it is out of the database. In a receive pipeline component, a message can be modified before it is published in the MessageBox. In a send pipeline component, a message can be modified after being received from the MessageBox. A typical place to create or modify a message is also inside an orchestration.

2.4 Message Context Properties

Message context properties are used for the subscription mechanism (routing the message to its appropriate end point). They are defined in a property schema. At runtime, the property values are stored into a context property bag.

The property schema is associated with the message schema within BizTalk so that every inbound schema-based message has a schema and a property schema attached to it.

The property schema consists of a global property schema that every message can use by default and of an optional custom property schema which can be created to define application-specific properties. Both types of properties are essentially the same at runtime and both are stored in the context property bag.

So, both types of properties can be used by the subscription mechanism to evaluate which endpoints have a subscription matching the message. The most common subscription is based on a global property called the messageType which is a combination of the XML namespace of the message and the root node name separated by a # character. Ex: http://www.abc.com#RootElementName.

Using subscription to route documents to the proper endpoint is called Content Based Routing (CBR).
For information, if the message is not schema-based, there will be no MessageType property value. Such is the case for binary data message.

Message context properties are populated by the BizTalk runtime in 2 artifacts:

The adapter writes and promotes into the message context properties related to the location, adapter type, and others properties related to the adapter.
The Receive Pipeline can write and promote properties into the message context in any of its pipeline components. Disassembling components are of particular interest because they promote the messageType property which is commonly used for Content Based Routing.

Property bag.

It is possible to use the BizTalk API in pipeline component code to read/write context properties from the property bag. The property bag is an object implementing the IBasePropertyBag interface. If you intend to use that interface in a custom pipeline to write properties that will be used for routing, you have to keep in mind that properties that are simply written into the property bag using the Write() method are not available for routing. To have a property available for routing, you need to promote the property with a different API call, the Promote() method. This method writes the property and its value in the property bag but ALSO flag the property as promoted and so make it available for routing.

BizTalkDTADb grows too large – How to purge and maintain the database?

2008-02-19T18:02:00.009+07:00

The BizTalkDTADb is a BizTalk database that stores health monitoring data tracked by the BizTalk Server tracking engine. It is commonly called the “BizTalk Tracking Database”. This database can grow relatively quickly in size depending of the kind of load your server is under.

I will explain first what should be done to keep the database healthy (by which I mean to keep it under a reasonable size) and after how to clean up the database if it grew up so large that the normal clean up method doesn't work anymore.

1. How to maintain the BizTalkDTADb?

Each BizTalk service instance running is processing data and while the data is processed, BizTalk tracks it and saves it in the BiztTalk Tracking Database. This means that the Tracking Database will grow indefinitely over time which is obviously not a viable option.
There is an SQL job called “DTA Purge and Archive (BizTalkDTADb)” that is installed on the BizTalk SQL Server which is used for cleaning up (deleting old tracking information) the BizTalkDTADb. That job is not enabled by default so the first thing that should be done after installing BizTalk server is to configure and enable the job. See here for information about how the cleanup process works and here for information on how to configure the SQL job. Basically, the job calls a single stored procedure on the BizTalkDTADb and once edited should looks like the following:

exec [dbo].[dtasp_BackupAndPurgeTrackingDatabase] 1, 0, 1, '\\MyBizTalkServer\backup', null, 0

The 4 first parameters are the one that you need to know about. The 2 first are the number of hours and days for which completed instances will be cleaned up. The third one is the number of days after which even non completed instance will be cleaned up. The fourth is the location of the backup folder.
This means that the SQL job will back up the BizTalkDTADb each time it runs, making that backup files will fill up your disk subsystem pretty quickly if nothing is done about it! Backups are important in case of a Database crash and that the Tracking Database needs to be restored.

If you do not consider the Tracking Database to be of enough importance to be backed up and have the extra burden to manage the backups, you can modify the “DTA Purge and Archive (BizTalkDTADb)” SQL job as explained here. This way, the job will only purge the tracking database without backing it up. It is especially applicable for development and QA environments and might also apply to your production environment.
In short, the only change that needs to be done in the SQL job is to modify the T-SQL statement run by the job. It needs to execute the SP dtasp_PurgeTrackingDatabase instead of dtasp_BackupAndPurgeTrackingDatabase.

The final T-SQL statement executed by the SQL job will be similar to the following:

declare @now as datetime
set @now = GetUTCDate()
exec [dbo].[dtasp_PurgeTrackingDatabase] 0, 3, 6, @now

In this case I keep complete instances in the Tracking DB for 3 days and incomplete one for 6 days, everything older should be purged. As you see there is no path to specify for the backup location as no database backup is executed.

Instead of modifying the original SQL job you could alternatively disable it and create a new job with the appropriate T-SQL call. That is how I have done it myself and consider it to be a best practice.
Moreover, I scheduled the job to run every 5 minutes. This has proven to be a good time interval. I used to run the job every 30 minutes only but I ever encountered cases where the clean up procedure did not keep up with the amount of tracked data and I ended up with a huge tracking database which I had to purge manually, as I will explain next.
So, a 5 minutes interval to run the job seems to also be a best practice from my experience.

2. How to manually purge the BizTalkDTADb?

You will have to manually purge the BizTalkDTADb database if it grew too large either because the clean up procedure was not started or the clean up procedure could not keep up with the amount of data saved in the Tracking Database.
This is explained in details here but, in short, the important points are:

- All the BizTalk services used by BizTalk needs to be stopped. This means all the BizTalk host service instances, Enterprise SSO, BizTalk Rules Engine, EDI service, BAM, BAS and IIS if they are used.

- Open Microsoft SQL Server Management Studio and run the following SQL statement on the BizTalkDTADb: exec dtasp_PurgeAllCompletedTrackingData

Once the procedure is executed, a lot of space will have been freed in the Tracking Database. The database will nevertheless still take the same amount of space on the disk subsystem because deleting data in a database does not reduce the size the database takes on the disk. If you want to reduce its size on the disk, you need to shrink it. You can do that in 2 ways:

1. Through SQL Server Management Studio, right click on the BizTalkDTADb database, click on Tasks > Shrink > Database

2. Through T-SQL using the DBCC SHRINKDATABASE command:
DBCC SHRINKDATABASE (BizTalkDTADb);
The reference of the T-SQL DBCC SHRINKDATABASE command can be found here.

AdWords API and .Net client

2008-01-17T17:04:00.001+07:00

I have been recently given the task to be in charge of a project using the AdWords API.

The AdWords API actually consists of a serie of Web Services which let you access to your AdWord account through an API instead of a normal user interface. The documentation regarding the Web Services can be found at: http://www.google.com/apis/adwords/developer/index.html
To access the AdWords API using a .Net client, there is a .Net solution with pre-generated Web Service proxies that you can find at http://www.google.com/apis/adwords/net.html.

The documentation regarding the .Net client is pretty scarce but there is an AdWords API forum where you can find solutions to your problems and also post your own questions.

Please note that the problem I describe hereunder applies to the latest .Net client available at time of writing (mid-January). This blog post applies only if you encounter the same problem with your version of the .Net client code. Would a future release not show the problem, no fix will be needed.

The only problem I have had so far with the proxy classes is that they constantly threw me an error message saying that “The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.”
This problem is similar to what I have previously encountered in BizTalk projects when .Net HTTP requests access a Web Server which ignores the keepAlive instruction. This is often the case with Apache servers (or should I say their administrators? – I am not sure if this is a setting they can change). Anyway, for people interested, I have posted more details about the problem in a previous post: BizTalk HTTP Adapter and Apache Keep-Alive issue.

So, the first thing I had to do to fix the issue is to modify every Web Service proxy classes so that the HTTP request used to make the Web Service call has its KeepAlive setting set to false.
All Web Service proxy classes that are using SOAP extends from the SoapHttpClientProtocol class. Web methods belonging to the WebService are called using the Invoke() method. Nevertheless, we, as client programmers, do not call the Invoke method directly, it is instead the proxy class which wraps the Invoke() call into a method matching the WebService method signature. For example, the ReportService proxy class delivered in the AdWords API .Net client has a method validateReportJob() which matches the WebService method defined on the server and which encapsulates the call to the Invoke() method:
public void validateReportJob(ReportJob job)
{
this.Invoke("validateReportJob", new object[] {job});
}

The SoapHttpClientProtocol.Invoke() method is implemented using the protected method GetWebRequest () which creates an instance of a WebRequest object that the Invoke() method uses to establish an HTTP connection with the web server.

With this in mind, all we have to do is to override the GetWebRequest() method in our ReportService proxy class so that we can intercept the WebRequest object created and modify the properties we want to customize.

Note that the WebRequest class is in fact an abstract classs and that the actual object returned at runtime is a class implementing the protocol used. In cases of Web Services and everything that runs over HTTP, it is the HttpWebRequest class.
So in our case, I have added the following code in the ReportService proxy class:

protected override System.Net.WebRequest GetWebRequest(Uri uri)
{
System.Net.HttpWebRequest webRequest = (System.Net.HttpWebRequest)base.GetWebRequest(uri);
webRequest.KeepAlive = false; //set the HTTP Header to not keep connections alive between requests
webRequest.Timeout = 1200000; // The length of time, in milliseconds, until the request times out (20 minutes here)
return webRequest;
}

With this piece of code in each of the Web Service classe proxies, “the connection closed” exception is not thrown anymore as both the .Net client and the Apache server that runs on http://adwords.google.com have the same KeepAlive setting.

Custom Pipeline Component deployment gotcha.

2007-11-11T00:26:00.001+07:00

When you create custom pipeline, you might want to compile and install the custom pipeline component in the GAC first before adding the pipeline component in the toolbox. That way, the custom pipeline solution will reference the custom pipeline component from the GAC instead of its location on the disk.
The reason behind this is that if you install the pipeline component library on your production servers only in the GAC and if you did not GACed the pipeline component before using it in the custom pipeline, an exception is raised saying that the custom pipeline cannot be found (as the custom pipeline refers to a custom pipeline component on the local drive instead of the GAC). See a full explanation of the problem on Stephen W. Thomas' blog.

Note that if you follow the BizTalk documentation on Deploying Pipeline Components, it is advised that the assembly should be deployed in both the GAC and the <installation directory>\Pipeline Components folder. I would advise to follow this way so it won’t matter how you or your developers create and compile the pipeline and their components; it will always work when the code is moved to the production environment and thus avoid unnecessary stress in the case it would have gone wrong because one developer referenced his component in a different way.

EDIT for BizTalk Server 2006 R2:
I have just notice after having published the post that the documentation has been changed for BizTalk Server 2006 R2 and that now the documentation says:

All the .NET pipeline component assemblies (native and custom) must be located in the \Pipeline Components folder to be executed by the server.

and:

You do not need to add a custom pipeline component to be used by the BizTalk Runtime to the Global Assembly Cache (GAC).

So now, Microsoft officially advises to put the custom pipeline component libraries only in the \Pipeline Components and not in the GAC anymore. If you still have BizTalk Server 2006 installed on your machine like I do, you will see that the local BizTalk documentation still says as mentioned in my original post.

BizTalk Server 2006 Custom Functoid Documentation Mistake

2007-11-09T18:14:00.000+07:00

When you develop a custom functoid, there are 5 important tasks to do:

Create a resource file with the various resources such as functoid’s name, description, tooltip and icon.
Create the class that will implement the functoid. This class must derive from the Microsoft.BizTalk.BaseFunctoids.BaseFunctoid class.The constructor must override the bas class constructor and call a certain number of methods and properties so that the custom functoid can run properly at run-time as well as integrate well into the Visual Studio’s mapper.
Create a member method in the functoid class that will actually implement the functionality of the custom functoid (its business logic).
Compile and sign the functoid project into an assembly with a strong name key file (so that the assembly containing the custom functoid can be deployed in the GAC).
Copy the assembly to C:\Program Files\Microsoft BizTalk Server 2006\Developer Tools\Mapper Extensions and add the functoid in Visual Studio’s ToolBox. You must also install the assembly into the GAC so that the functoid is available to BizTalk at runtime.

Today I found out that the BizTalk Server 2006 documentation has a mistake in the custom functoid development section: Using BaseFunctoid, which relates to the point number 2 in my previous list of tasks.
That section says that the constructor of the functoid must make a call to the SetupResourceAssembly method to point to the resource assembly. It details:

Include a resource file with your project. If building with Visual Studio, the resource assembly must be ProjectName.ResourceName.

In my experience this is not true, when I build my custom functoid assembly in Visual Studio, I have to call the SetupResourceAssembly method with FunctoidNameSpace.ResourceName as parameter instead of ProjectName.ResourceName. If I follow the documentation’s advice, the resource information such as functoid’s name and icon do not appear in Visual Studio’s toolbox, proving that the resources could not be found by the IDE.

The only case that it would work as mentioned in the documentation would be if the project name and the namespace of the functoid were identical, which happens by default in Visual Studio. Indeed, when you create a new project with Visual Studio 2005, a default namespace will be created for the project, its value being the project name. You can see the default namespace value by right-clicking on your project files and click “properties”.

BizTalk Expression Shape IntelliSense

2007-11-06T13:29:00.000+07:00

Whenever you want to call IntelliSense from the Expression Shape editor, you can do so by just hitting the keys CTRL + SPACE. You will see the different namespaces, classes and objects (such as local variables and messages) available from the scope of your Expression Shape.

Note that this shortcut key combination actually comes from the Visual Studio .Net IDE. Thus, it is available for any “traditional” .Net project types (C#, VB.NET, Windows form, Web App…)

HTTP Adapter and Apache Keep-Alive issue

2007-09-14T12:38:00.000+07:00

Problem.

If you notice a high rate of suspended messages due to communication failures between your BizTalk Server and an HTTP server, you should verify if the HTTP Web server you are accessing is Apache, Resin or another similar server of the UNIX / LINUX / Java family. Note that this could apply to other web servers as well, these are just examples I know of.

BizTalk Server always sends HTTP request with the HTTP header setting Connection: Keep-Alive. Moreover, BizTalk expects that the foreign HTTP server obeys to the header setting and does not check the Connection header setting received in the HTTP response.
To be more precise, the problem is double:

Apache is not obeying to the Connection HTTP header setting and always closes the HTTP connection after every request (as in an HTTP 1.0 fashion). The HTTP/1.1 protocol allows this even if it is not the prefered behaviour.
BizTalk Server does not react accordingly to the Connection header that is returned back in the HTTP response. This is wrong, following the HTTP/1.1 protocol definition, the client should always check the Connection header returned.

So, on one side there is a Apache closing connections for every HTTP request and on the other side BizTalk is expecting connections to be kept open between requests.

Actually, the source of the problem is not that the Web Server is Apache or not but related to how the Web Server is configured and how it behaves with the connection. Nevertheless, in my experience, this kind of problem only happened with Apache. But I also ever encountered 1 Apache server which kept the HTTP connection open between requests and so no fix was required for that case as no communication problem arose.

For this reason, if you have communication problems between BizTalk and a foreign web server, you should first check if the web server is explicitly closing the HTTP connection after every request.

There is a Microsoft KB that explains the issue and also how to fix it.

Identification of the problem.

As mentioned in the KB, it is important to make sure that your problem is exactly what the KB refers to, so, I will explain how I diagnosed it on my BizTalk Server.

Looking in the BizTalk Administration Console, I saw an abnormal amount of suspended service instances, most of them being of the type “Message”. I also noticed that all the suspended messages belonged to only a subset of our BizTalk applications.
There was clearly something wrong and by checking the Windows Event Viewer, I saw that some BizTalk send ports had an unusual failure rate at the communication level. This explained why so many messages were suspended as messages failing to be sent through a port will be suspended.
The Windows Event Viewer showed messages such as:

A message sent to adapter "HTTP" on send port "XXX" with URI "http://www.zzz.com/Service" is suspended.
Error details: Cannot access a disposed object.
Object name: 'System.Net.Sockets.NetworkStream'.

And

A message sent to adapter "HTTP" on send port "XXX" with URI "http://www.zzz.com/Service" is suspended.
Error details: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.

After scratching my head on this issue for a while, I found out that all failing messages where those sent to a port using an HTTP adapter configured with an URI pointing to an Apache server. Other ports using the HTTP adapter but configured to access an IIS server were working fine!

I then used a network sniffer to see what was going on exactly and I saw something inconsistent in the HTTP headers:
The header of the HTTP request contained:
Connection: Keep-Alive
But the header of the HTTP response contained:
Connection: close
server: IBM_HTTP_Server/6.0.2.3 Apache/2.0.47

So, even if the HTTP request specifies “Connection: Keep-Alive”, the Apache Server still close the connection and replies with a “Connection: close” in the header of the HTTP response. For reference regarding the Connection HTTP header, see: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.10

Solution.

As it is not possible to change the Connection: Keep-Alive header setting in the HTTP adapter shipped with BizTalk Server 2006, I googled a bit and found out the KB 924638 I mentioned earlier in this article which provides 1 solution and 1 workaround for the problem.

The hotfix mentioned in the KB contains a library that makes it possible to modify the Keep-Alive configuration for the HTTP connection. However, as there is no option to set the header setting from the BizTalk Administration Console, it is required to create a custom pipeline which writes a KeepAlive property in the context of the message going through the pipeline. Later on, the HTTP adapter will read the KeepAlive message context property value and create the HTTP connection accordingly.

You can find here a zip file containing 2 Visual Studio projects, one for the Custom Pipeline Component and another one for the Custom Pipeline. In this custom pipeline, the KeepAlive property is not hard-coded and so its value can be set either from the BizTalk Administration Console at run-time or from Visual Studio at design-time.

To summarize, there are 3 steps to apply the solution:

Call Microsoft Support so that they email you the hotfix containing an executable which updates the Microsoft.BizTalk.HttpTransport.dll (as the hotfix is not directly available to download from the KB).
Create a custom pipeline which has a property that permits the user to specify the Keep-Alive setting.
Use the custom pipeline in the send ports that are pointing to web servers explicitly closing connections, such as some Apache web server.

BizTalk Server 2006 R2 released

2007-09-12T01:41:00.001+07:00

BizTalk Server 2006 R2 has just been released.

The key new features differentiating the new release from BizTalk Server 2006 are:

Native support for Electronic Data Interchange (EDI).
Native support of Radio Frequency Identification (RFID).
While BizTalk Server 2006 integrates with Microsoft Office 2003 components and Windows XP (on a client OS), BizTalk Server 2006 R2 integrates with Office 2007 and Windows Vista.
Can make use of .Net 3.0 technologies such as Windows Communication Foundation (WCF) for messaging and Windows Workflow Foundation for BAM (Business Activity Monitoring).
New edition of BizTalk, called BizTalk Server Branch Edition. This edition is ideal for Hub and Spoke deployment scenarios for which only 1 BizTalk application is running in the Spoke nodes. Its official price is 1,800 USD; making it less than a quarter of the price of BizTalk Server Standard Edition. See BizTalk Server Editions for more details.

Both BizTalk Server 2006 and 2006 R2 supports only Visual Studio 2005 so far.
Visual Studio 2008 will be supported through a BizTalk Server Service Pack so that the BizTalk installation routine will be able to detect VS2008 and run with it.

For some, RFID is the most exciting feature introduced with BizTalk Server 2006 R2. The BizTalk RFID features include:

Data management capabilities.
Device abstraction and management capabilities.
Event processing engine that allows the creation of business rules and manage the execution of event pipelines for RFID events.

For more details about the new features of BizTalk Server 2006 R2, check the BizTalk Server roadmap.

Functoid and BizTalk Application Versioning Problem

2007-09-05T23:48:00.001+07:00

1. Problem:

I encountered a problem difficult to diagnose related to deploying a BizTalk Server 2006 application and a functoid that have been compiled together but then deployed with different versions.
For example, let’s say you have developed and compiled a BizTalk application “A” version 1 using a functoid “F” version 1. Later on, you change some of the code inside the functoid and so you have a new version of the functoid, let’s call it functoid “F” version 2.
Note that by version 1 and 2, I do not mean the assembly version number; it is just a nomenclature I use to differentiate the versions. In my case, the assembly’s version number is fixed.
I often deploy new versions of functoids without having to recompile neither redeploy the BizTalk application but this time, the following error message appeared:

Microsoft.XLANGs.Core.XTransformationFailureException: Error encountered while executing the transform XXXX. Error:Unable to create the transform.. ---> System.ArgumentNullException: Value cannot be null.
Parameter name: extension
at System.Xml.Xsl.XsltArgumentList.AddExtensionObject(String namespaceUri, Object extension)
at Microsoft.XLANGs.BaseTypes.TransformBase.get_TransformArgs()
at Microsoft.XLANGs.RuntimeTypes.TransformMetaData..ctor(Type transformBaseType)
at Microsoft.XLANGs.RuntimeTypes.TransformMetaData._creator(Type t)
at Microsoft.XLANGs.RuntimeTypes.MetadataCache._slowFor(Type t)
at Microsoft.XLANGs.RuntimeTypes.MetadataCache.For(Type t)
at Microsoft.XLANGs.RuntimeTypes.TransformMetaData.For(Type t)
at Microsoft.XLANGs.Core.Service.ApplyTransform(Type mapRef, Object[] outParams, Object[] inParams)
--- End of inner exception stack trace ---
at Microsoft.XLANGs.Core.Service.ApplyTransform(Type mapRef, Object[] outParams, Object[] inParams)
at Tourico.Tourico_SmartSearch.segment2(StopConditions stopOn)
at Microsoft.XLANGs.Core.SegmentScheduler.RunASegment(Segment s, StopConditions stopCond, Exception& exp)

2. Solution:

The message was not recorded in the Windows Event Log but as I catch and log exceptions in an exception handler; I could see the exception in my log.
As the message is rather vague, I spent quite a lot of time checking the maps and schemas used in the orchestration but found no clue.
I finally noticed that a functoid’s code had been changed and once I recompiled and re-deployed the BizTalk Application, everything worked fine!

I am not sure of the exact reason as normally you don’t need to redeploy the BizTalk application after having updated and GACed a functoid.
Anyway, if one day you encounter an error message of the same flavor, recompiling and redeploying the application might just work fine. Maybe this hint can help someone in the future :)

3. Note about exception handler in BizTalk Server 2006.

For the people who do not know, you just need to use a scope shape and its exception handler to have the equivalent of a try-catch block in a BizTalk orchestration. That gives you the opportunity to take action and/or log errors when exceptions are thrown from the orchestration.

To create an exception handler, drop a scope shape into your orchestration, right click on it and select “New Exception Handler”. Note that only scopes that are non-transactional or long running transactional can have an exception handler.
Here is how it looks like:
Everything in the first square (Scope_1) will be the “try” part and everything in the second square (CatchException_1) will be the “catch” part.

Biztalk 2006 multiserver installation

2007-08-27T21:23:00.000+07:00

This article explains how to install BizTalk Server 2006 on multiple servers for a configuration with 1 SQL Server and 1 BizTalk Server.

Doing a multiserver BizTalk installation should be done once for practice before doing it on the production servers so that all the specificities to a multiserver install are well understood.
I will here share my experience in a practice multiserver BizTalk installation done on 2 Virtual Machines.

Simplified BizTalk Server Installation scenario.

For a first multiserver installation, I will keep a simplified scenario with only 2 servers and neither BAS nor BAM features installed. Only the following components will be installed:
- Enterprise Single Sign-On (SSO).
- BizTalk Group.
- BizTalk Runtime.
- MSMQT.
- Business Rule Engine.

There will be 2 virtual servers:
- 1 Virtual Machine to host SQL Server and act as a Domain Controller
- 1 Virtual Machine to host BizTalk Server.

This is the most basic multi-server install possible. It will nevertheless help you to understand how a multi-server BizTalk installation works.
For more complex installation scenarios with BAM, BAS and/or clustered servers, Microsoft provides a complete installation guide document, “BizTalk Server 2006 Installation Guide - Multiserver”. See in the reference section of this article for a link to the documentation.

Note that to follow this article as a lab, you will need the following software:
- Microsoft Virtual PC 2007 (Virtual PC is now free and you can download it here)
- Windows Server 2003
- SQL Server 2005
- BizTalk Server 2006

The need for a Domain Controller.

A very important difference between a single server and a multi-server installation is that the multi-server configuration requires you to use domain users and groups to run the various BizTalk services making having a domain controller a necessity.
On the other hand, a single server configuration can run with local windows groups and users.
To be more precise, these domain accounts and groups are used for the security configuration of the BizTalk Server databases. Indeed, the user account which runs the BizTalk server service must have access to the SQL Server. As SQL Server is installed on a separate machine, the use of a domain user account is therefore a necessity so that the account can have access rights on both the BizTalk machine and the SQL Server machine.

The important point to remember is that when you plan to install a multi-server configuration of BizTalk in your production environment, you will have to check with your hosting company to have access to a domain controller or to have them create all the necessary domain users and groups.

Create Domain Groups and Users.

For a single server installation, the setup procedure is able to create the Windows Groups and Users necessary for BizTalk as they are created on the local machine.
On a multi server installation, BizTalk Windows Groups and Users must be created manually on the Domain Controller. The setup procedure is not able to create the Windows Groups and Users on a Domain Controller.

BizTalk Installation, Configuration and Applications Deployment considerations.

To be able to install and configure BizTalk Server 2006, you have to log on the server using a Domain User who has Administrator rights on both the SQL Server and the BizTalk Server. Moreover, that domain user must also be part of the BizTalk Server Administrators group and the SSO Administrators group.
The user must also have the sysadmin role on the SQL Server.

I found out something that is, as far as I know, NOT mentionned in the BizTalk documentation: the user installing BizTalk Server must also have the Domain Administrator role. Technically, this privilege does not have to be granted when installing BizTalk Server but only when configuring it. It is because when the BizTalk Server Configuration tool runs, the tool must be able to check if the Windows accounts assigned to the various BizTalk services belong to the Windows groups set for these services. Only a user being domain administrator has the right to check other users properties.

After BizTalk Server is configured and installed, the user does not have to be part of the Domain Administrators group anymore. Moreover, its privileges on the SQL Server can be lowered to dbo instead of administrator. That way the user can be used to install BizTalk Applications without having unnecessary privileges.
FYI, installing BizTalk application creates tables, and executes SP on the BizTalk DBs.

To simplify things I advise to create a “BTSAdmin” domain user who has these rights. Only the person who will install, configure and manage BizTalk applications should have that user login information.

The BizTalk Server 2006 installation guide provides a list of all the necessary Groups and User that need to be created. Microsoft also provides a best practice guide on how to implement the Windows Groups and Users for BizTalk Server 2006.

Microsoft Distributed Transaction Coordinator (MS DTC) considerations.

Network DTC access and Network COM+ access must but enabled for all BizTalk and SQL servers. It is turned off by default on Windows Server 2003.

SQL Server Considerations.

SQL Server client tools must be installed on all BizTalk servers. The client Tools installed must be of the same version as the SQL Server installed.
SQL server must be started and running when running the BizTalk Server Configuration tool.

1. Creating the Virtual Servers.

To install the Virtual Servers, you will need Microsoft Virtual PC 2007 and Windows Server 2003.

In this step, we will create 2 Virtual Machines, one named SQLServer and another one named BizTalkServer. Windows Server 2003 will be the OS installed on both of them.
The machine named SQLServer will run SQL Server and also act as a Domain Controller while BizTalkServer will host BizTalk Server.

As installing Windows Server 2003 on a virtual machine is particularly slow, please refer to a previous post that explains how to install and clone virtual servers efficiently.

Once the 2 virtual machines are ready, set their networking setting to “Local only”. This option gives you networking support between virtual machines only. For a description of the different network settings available for Virtual PC, check the following link: http://support.microsoft.com/kb/833134

2. Installing the Domain Controller and creating Windows User and Groups.

2.1. Complete the setup of the SQLServer Virtual Machine.

1. Rename the VM. If you created the SQLServer Virtual Machine by using a base virtual machine and “sysprepped” it, you will need to rename it first. To do so, go to Control Panel > System > Computer Name and click “Change”. Input “SQLServer” in the machine name field. When prompted, do not restart the machine yet.
2. Set the Virtual Machine IP address. Set the SQLServer VM to have a static IP address of 192.168.0.1 - The 192.168.x.x address range is a typical IP range for private networks.
To modify the IP address of the VM, go to Control panel -> Network Connections -> Local Area Connection -> Properties -> TCP/IP -> Properties and use 192.168.0.1 for the IP address as well as the DNS server. In the subnet mask field type in 255.255.255.0 and leave the default gateway blank.

You must now reboot the SQLServer virtual machine.

2.2. Installing the Domain Controller.

Start the SQLServer virtual machine as it is the machine chosen to act as a Domain Controller (DC).
For this step, have the Windows Server 2003 installation disk at hand as it might be needed.

1. Make sure that you restarted the server after it has been renamed. This is because Active Directory can’t be installed after you changed the computer name without rebooting first.
2. Run “Manage Your Server”, and click “Add or Remove a Role” (next to “Adding Roles to Your Server” area).
3. Select the radio button “Typical configuration for a first server” and click “Next”.
4. Type in “biztalk.local” for your Active Directory domain name and click “Next”.
5. In the NetBIOS Name screen, accept the default setting and click “Next”.
6. In the Forwarding DNS Queries screen, select “No, do not forward queries” and click “Next”.
7. Click “Next” again and “OK” in the pop up window. If prompted to insert Windows Server 2003 disk, click on “Use Physical Drive” in the “CD” menu of the windows containing the Virtual Machine. You might additionally have to point to some files in the \I386 folder located on the Windows Server installation disk.
8. When the installation is almost complete, the VM will restart and finish the DC setup. Once the virtual machine has rebooted itself, a “Server Configuration Progress” window will open; click “Next” and “Finish”. The domain controller installation is now completed.

Note: Once your server is promoted to be a Domain Controller, the local administrator is promoted to become the domain administrator.
Moreover, it is not possible to log on the local machine anymore; you can only log on the domain. The concept of local Users and Groups doesn’t exist anymore and if you try to run lusrmgr.msc, a message will be displayed saying that is not possible to manage local groups and users on a Domain Controller. Only domain groups and users can be created and managed.

2.3. Create BizTalk Domain Users and Groups.

Start the Domain Controller, the SQLServer VM.
As we don’t install all the features of BizTalk Server, we create only a subset of the Windows Users and Groups defined for BizTalk Server 2006. This subset of Groups and Users is tailored for the features installed in this article.

1. Run “Manage Your Server” and click on “Manage users and computers in Active Directory”, located next to the “Domain controller (Active Directory)” area. This will open the Active Directory Users and Computers MMC (Microsoft Management Console). Alternatively, you can open it by going to: Control Panel > Administrative Tools > Active Directory Users and Computers.
2. Expand biztalk.local
3. Create a BizTalk Organizational Unit. To keep things tidy, we place the BizTalk Users and Groups in an Organizational Unit (OU). To do so, right click on “biztalk.local” and click New -> Organizational Unit. For the Organizational Unit name, enter “Biztalk”.
4. Create the Windows Groups. These groups will be used for the security configuration of the BizTalk Server databases. Right click on the Organizational Unit “BizTalk” and click New -> Group. For each group created, enter the group name specified in the table bellow, keep the default settings and click “OK”.

Group Name

SSO Administrators

SSO Affiliate Administrators

BizTalk Server Administrators

BizTalk Server Operators

BizTalk Application Users

BizTalk Isolated Host Users

There are 6 Groups created in total.
5. Create the BizTalk Windows Service Accounts (Windows Users). Windows Users are required for the various BizTalk services. You might want to set all the users to use the same password so that managing them is easier. Right click on the Organizational Unit “BizTalk” and click New -> User. For each user created, enter the full name and the user logon name specified in the following table (under the “Full Name” and “User logon name” columns) and click “Next”. In the following screen, enter the user’s password, uncheck “User must change password at next logon”, check “User cannot change password” and “Password never expires”. Finally, click “Next” and “Finish”. Repeat the procedure for every Windows User.

Full Name	User logon name
Enterprise Single Sign-On Service	SSOService
BizTalk Administrator	BTSAdmin
BizTalk Host Instance Account	BTSAppHost
BizTalk Isolated Host Instance Account	BTSIsolatedHost
Business Rule Engine Update Service Account	ReuService

6. Add the Windows Service Accounts to their respective Groups.

Windows Group	Windows User
Server Administrators	biztalk\BTSAdmin
SSO Administrators	biztalk\BTSAdmin and biztalk\SSOService
BizTalk Application Users	biztalk\BTSAppHost
BizTalk Isolated Host Users	biztalk\BTSIsolatedHost

7. Close the Active Directory Users and Computers MMC.

Remember that the Windows Users will be used to run the different BizTalk services and the Windows Groups are used to manage security and access rights to the SQL Server.

In case you would install all the BizTalk server features, you can find a complete list of Windows Users and Groups needed for BizTalk Server in the BizTalk Server 2006 installation guide. You should also read the Microsoft guideline to implement Windows Groups and Users for BizTalk Server 2006.

2.4. Chose the Domain User performing the BizTalk multiserver installation.

If you continue the installation procedure logged on the virtual servers with the Domain Administrator, you don’t need to do this step. Nevertheless, in real environments, you will most likely not want or have the possibility to use the Domain Administrator to install BizTalk Server. In that case, you will have to create a domain user who is Administrator on both BizTalk and SQL Server machines and also member of the “BizTalk Server Administrators” and “SSO Administrators” groups.

In this practice installation, we will choose the BTSAdmin domain user that we have just created to execute the BizTalk Server 2006 multi-server installation. As we added BTSAdmin in the “BizTalk Server Administrators” and “SSO Administrators” groups already, the remaining task is to give the user administrator privileges on both the SQLServer and the BizTalkServer VMs.

2.5. Configure the SQLServer VM Administrator and SQL Server sysadmin Role.

To give BTSAdmin administrator privileges of the SQLServer VM:
1. Run “Manage Your Server” and click on “Manage users and computers in Active Directory”, next to the “Domain controller (Active Directory)” zone.
2. Under the “Builtin” folder, add “biztalk\BTSAdmin” to the “Administrators” group. Note that in our lab this will unintentionally make the BTSAdmin user a domain administrator, but in a normal scenario, SQL Server would not be a Domain Controller and so BTSAdmin would just be administrator of the local machine.

As by default the Windows Builtin\Administrators group has the sysadmin SQL role on the SQL Server, the BTSAdmin user will also have the sysadmin role as BTSAdmin is now part of the Administrators group. Once SQL Server is installed, you will be able to check that by opening the SQL Server Management Console, go to “Security\Server Roles” and double click on the “sysadmin” role. The pop up window will show all the windows and SQL logins having the sysadmin role.

3. Installing SQL Server.

3.1. Install SQL Server.

In this step, we will install SQL Server 2005 on the virtual machine that will host the BizTalk databases.

Only a subset of the SQL Server 2005 components are needed for the BizTalk features we have chosen to install. These are: “SQL Server Database Services” and “Workstation components, Books Online and Development tools”.
If you plan to install the BAM component of BizTalk Server, you will have to additionally install the following SQL Server 2005 components: “Analysis Services”, “Notification Services” and “Integration Services”.

The SQL Server service should use the Local System account as the Service Account, and you should configure the SQL Server Agent to start at the end of the setup.

SQL Server 2005 installation procedure:
1. Start the Virtual Machine SQLServer. Log on using the domain user you have chosen to use for the BizTalk multiserver installation (see paragraph 2.4).
2. Start the setup of SQL Server from its installation disk.
3. On the SQL Server Installation welcome screen, click “Server components, tools, Books Online, and samples”.
4. Accept the license terms and conditions and click “Next”.
5. On the Installing Prerequisites screen, click “Install”. Note that all the necessary prerequisites, such as the .Net Framework 2.0, are present on the installation disk.
6. Once the prerequisites are installed, click “Next”.
7. On the Welcome to the Microsoft SQL Server Installation wizard, click “Next”.
8. On the System Configuration Check, click “Next”.
9. Enter the registration information on the Registration screen, click “Next”.
10. On the Components to Install screen, chose the 2 components “SQL Server Database Services” and “Workstation components, Books Online and Development tools”, click “Next”.
11. Leave the “default instance” radio button selected on the Instance Name screen, click “Next”.
12. On the Service Account screen, make sure that the check box “Customize for each service account” is cleared; select the radio button “Use the built-in System account” with “Local System” selected in the dropdown list. Note that you could also use a Windows User account instead and if you do so, enter a local account or domain account and its password. Lastly, tick the check box “SQL Server Agent” in the “Start Services” section so that the SQL Server Agent starts once the setup is completed. Click “Next”.
13. In the Authentication mode select “Windows Authentication Mode”. Note that Mixed Mode can also be used but it is potentially less secure and not advised by Microsoft. Click “Next”.
14. On the Collation Settings screen leave the default and click “Next”.
15. Click “Next” again and then “Install”.

3.2. Configuring the SQL Server Virtual Machine.

Once SQL Server is installed, you need to configure it.

3.2.1 Configure SQL Server for Remote Connections.

In order for the BizTalk servers to connect to SQL Server, remote connections need to be enabled.
To do so you need to:
1. Run SQL Server Surface Area Configuration, found in Programs -> Microsoft SQL Server 2005 -> Configuration Tools
2. Once the tool opens, click on Surface Area Configuration for Services and Connections.
3. Locate “Remote connection” under the Database Engine tree and enable local and remote connections for both protocols by selecting “Local and remote connections” and “Using both TCP/IP and named pipes” radio buttons.

3.2.2. Configure Microsoft Distributed Transaction Coordinator (MS DTC).

In order for BizTalk to run against its databases located on a remote SQL server, you will need to turn on MS DTC options (both on the SQL Server and BizTalk Server).
Follow the next steps on the SQLServer VM:
1. Open the Control Panel and open “Add or Remove Programs”. Once open, click “Add/Remove Windows Components”.
2. Once the Windows Components Wizard open, select “Application Server” and click on de “Details” button.
3. Select the check boxes in front of “Enable network COM+ access” and “Enable network DTC access”. Click “OK”
4. You are now back to the Windows Components Wizard window; click “Next” and then “Finish”.
5. Restart the server.

If your SQL Server is installed on the Domain Controller like in this practice installation, enabling Network DTC Access will not work as there is a bug between DTC and the Domain Controller service. I posted a previous article explaining the problem and how to solve it, see: Enabling network DTC access fails on a Domain Controller server.

3.3 Install and Configure Enterprise Single Sign-On Master Secret Server.

I chose to install the Enterprise Single Sign-On Master Secret Server on the SQL Server virtual machine so that I could try it out.
Nevertheless, it is more common to install the Master Secret Server on the BizTalk Server machine when there is only 1 BizTalk machine.
In case you have a cluster of several BizTalk Server machines, you can chose to either install the Master Secret Server on a separate machine (such as in this example) or install it on one of the BizTalk Server.
The points to remember are:
- There can be only 1 machine being the Master Secret Server. It can either be one of the machines hosting BizTalk Server or any other.
- All machines running BizTalk and the machine being the Master Secret Server will have the Enterprise Single Sign On Windows Service running.

See the Microsoft documentation on installing BizTalk Server 2006 for more details on how to cluster SSO.

1. Start up the BizTalk Server 2006 Installation.
2. On the Microsoft BizTalk Server 2006 Installation Wizard page, click “Install Microsoft BizTalk Server 2006 on this computer”.
3. On the Customer Information page, type your user name and organization, and then click “Next”.
4. Accept the License Agreement and click “Next”.
5. On the Component Installation page, select ONLY “ESSO Administration Module” and “ESSO Master Secret Server”, found under “Additional Software”. Unselect all other components.
6. After the installation, make sure that the checkbox “Launch BizTalk Server Configuration” is selected and click on “Finish”. The BizTalk Server Configuration will start.
7. In the BizTalk Configuration, select the “Custom Configuration” radio button. In the “Service Credential” zone, type in “BIZTALK\SsoService” and its password. It is the domain account that will run the SSO Windows Service. Click “Configure”.
8. In the summary window, click “Next”.
9. On the final window, click “Finish”
10. Use Event Viewer to verify the ENTSSO service has started successfully.

4. Installing BizTalk Server.

In this step, we will install the BizTalk prerequisites, BizTalk Server 2006 and operate the necessary configuration.
Before starting the BizTalk virtual machine, make sure that its network configuration setting is set to “local only”.

4.1. Complete the setup of the BizTalkServer VM.

1. Start the virtual machine BizTalkServer.

2. Change machine name and join the domain. If you created the BizTalkServer Virtual Machine by using a base virtual machine and “sysprepped” it, you will have to rename the VM first. To do so, go to Control Panel -> System -> Computer Name and click “Change”. Change the computer name to “BizTalkServer”.
In the same window, join the domain by selecting the “Domain” radio button and type “biztalk.local” for the Domain name. Click “OK”. You will be prompted to enter the Domain Administrator user name and password.
Before restarting the Virtual Machine you should execute the next step.

3. Set static IP address. To modify the IP address of the VM, go to Control panel -> Network Connections -> Local Area Connection -> Properties -> TCP/IP -> Properties and use 192.168.0.2 for the IP address, 255.255.255.0 for the subnet mask, leave the default gateway blank and set the DNS server IP to 192.168.0.1

You can now reboot the BizTalkServer virtual machine. Note that for the BizTalkServer VM to boot and run properly as being part of the Domain, the virtual machine acting as the Domain Controller will have to be running as well. This means that from now on, the SQLServer VM must already be running when you start the BizTalkServer VM.

4.2. Configure BizTalkServer VM Administrator.

From now on, you will always log on the BizTalkServer VM using a domain account so that you are logged on the domain.

If you chose to execute the installation of Biztalk Server 2006 for a multi-server environment using the Domain Administrator, you don’t need to do this step and you will log on the BizTalkServer VM using the domain administrator account to you install BizTalk Server. Otherwise, continue.
See paragraph 2.4 for more details about choosing a domain account to install BizTalk Server 2006.

Once you have rebooted the BizTalkServer VM, log on the machine using the Domain Administrator account, so you have to log on the domain, not the local machine which is the default choice.

We have previously chosen to give the domain user BTSAdmin administrator privileges on both the SQLServer and the BizTalkServer VMs so that it can execute the BizTalk setup procedure.

To add BTSAdmin user as an Administrator of the BizTalkServer VM;
1. Open “Computer Management” found under Control Panel -> Administrative Tools.
2. Under the “Builtin” folder, add “biztalk\BTSAdmin” to the “Administrators” group.

You can know log off or reboot the BizTalkServer VM and log on again using the BTSAdmin domain user which is now administrator of the BizTalkServer machine.

4.3. Installing prerequisites.

4.3.1 Install IIS 6.0.

BizTalk server requires IIS for the HTTP Adapter, SOAP Adapter, Windows SharePoint Services Adapter, SSL encryption, Business Activity Services (BAS), Human Workflow Services (HWS).

In this step you might need access to the Windows Server 2003 installation disk.

IIS 6.0 installation procedure:
1. Click Start, point to Control Panel, and then click “Add or Remove Programs”.
2. In the Add or Remove Programs dialog box, click “Add/Remove Windows Components”.
3. In the Windows Components Wizard, select “Application Server” and then click “Details”.
4. Select “ASP.NET” and make sure that “Internet Information Services (IIS)”and “Enable network COM+ access” are selected as well. Click “OK”, and then click “Next”.
5. On the Completing the Windows Components Wizard page, click “Finish”.

4.3.2 Install the .Net framework 2.0 or Visual Studio 2005.

If the machine is used for the BizTalk runtime-only (such as on a production environment), you should install the .Net framework 2.0. You can download a redistributable version of .Net 2.0 framework here.
If the machine is used for the BizTalk runtime and/or development, Visual Studio 2005 or the .Net Framework 2.0 SDK should be installed instead.

4.3.3. Install SQL Server 2005 Client Tools.

1. Insert the SQL Server 2005 DVD in the DVD-ROM drive and browse to the \Tools folder and double click "setup.exe".
2. Accept the licensing agreement on the End User License Agreement page and click “Next”.
3. On the Installing Prerequisites page, click “Install”. Once the prerequisites are installed, click “Next”.
4. On the Welcome to the Microsoft SQL Server Installation Wizard page, click “Next”.
5. On the System Configuration Check page, ensure that there are no errors before you continue. Note that as we are running on a virtual machine there will be a warning for the “Minimal Hardware Requirement”, ignore it and click “Next”.
6. On the Registration Information page, click “Next”.
7. On the Feature Selection page, under “Client Components”, select “Will be installed on local hard drive” for the following Components: “Connectivity Components”, “Management Tools”, “SQLXML Client Features” and “Legacy Components”. Click “Next”.
8. Once the components are installed, click “Next” and then “Install”.
9. Once the setup is finished, click “Next” and then “Finish”.
10. Restart the virtual machine.

4.3.4. Configure Microsoft Distributed Transaction Coordinator (MS DTC).

To be able to install BizTalk server (run the BizTalk Configuration tool, to be more precise), install BizTalk applications and run BizTalk applications, network DTC access must be enabled.

Follow the next steps to do so:
1. Open the Control Panel and open “Add or Remove Programs”. Once open, click “Add/Remove Windows Components”.
2. Once the Windows Components Wizard open, select “Application Server” and click on the “Details” button.
3. Select the check box in front of “Enable network DTC access” and make sure that “Enable network COM+ access” is also selected. Click “OK”
4. You are now back to the Windows Components Wizard window; click “Next” and then “Finish”.
5. Restart the VM.

4.4. Installing BizTalk Server 2006.

4.4.1. Considerations:

Before you install BizTalk Server 2006, consider the following:
- If your computer name is longer than 15 characters, BizTalk Server Configuration will not work.
- The account you are logged on as must be part of the local administrators group and have DBA rights on SQL Server. After configuration is complete, you can lower the privileges to DBO.
- When installing BizTalk Server, there will be prerequisites components to install such as ADOMD.NET, SQLXML 3.0 SP3... You can either let the BizTalk Installation Wizard download them and install them for you or you can pre-download a cab file that contains all the prerequisites. You can see a list of all the prerequisite components required by BizTalk server here.
As we don’t have access to the internet from the Virtual Machine in this practice installation (we set the VM Network settings to "Local only"), we first need to download the BizTalk Server 2006 redistributable prerequisites cab file for Windows Server 2003 (32-bit) (BtsRedistW2k3EN32.cab) and then copy it on the virtual machine hard disk.

4.4.2 Install BizTalk Server 2006.

1. Start the BizTalkServer Virtual Machine and log on using the domain user you have chosen to use for the BizTalk multiserver installation (see paragraph 2.4).
2. Start up the BizTalk Server 2006 Installation; Double click on “Setup.exe” located on the root of the BizTalk Server 2006 installation disk.
3. On the Microsoft BizTalk Server 2006 Installation Wizard page, click “Install Microsoft BizTalk Server 2006 on this computer”.
4. On the Customer Information page, type your user name and organization, and then click “Next”.
5. Accept the License Agreement and click “Next”.
6. On the Component Installation page, select the following components: “Documentation”, “Server Runtime (without Base EDI Adapter)”, “Administration Tools” and “Business Rules Components”. Click “Next”.
7. On the Redistributable Prerequisites page, select “Automatically install the redistributable prerequisites from a CAB file” and specify the cab file location. This is the cab file that we pre-downloaded in paragraph 4.4.1. Click “Next”.
8. On the Summary page, verify that the components you selected to install are correct. Click “Install”.
9. On the Installation Completed page, unselect the “Launch BizTalk Server Configuration” check box, and then click “Finish”.
10. You can eventually stop your virtual machine and commit your change if you are using undo disk so that you start a new session when you are running the BizTalk Server Configuration tool.

4.5. Check that MS DTC network access is enabled on all servers participating in the BizTalk multiserver installation.

The last step to install BizTalk Server 2006 is to run the Configuration Tool. For the BizTalk Server Configuration Tool to run, all SQL Server and BizTalk Server machines must have Network DTC access and Network COM+ access enabled. So, before continuing and configure BizTalk, verify that this is the case on all the machines participating in the BizTalk multiserver installation.
In paragraph 3.2.2, I highlighted that there is a bug in turning on Network DTC access on a machine which is a Domain Controller so, double checking that Network DTC access is enabled on all machines can save you from a lot of trouble.

4.6. Configure BizTalk Server and other components.

The BizTalk configuration is the most likely point of failure of the BizTalk multiserver installation. If you use Virtual Machine with undo disks, you should commit your changes to the VM’s virtual hard disk before running the BizTalk Server Configuration Tool so that you have a fresh session when starting the configuration process. In that case, if the configuration fails, you will be able to cancel the change and go back to the state the Virtual Machine was before starting the BizTalk configuration. This means that the only thing you would have to re-do is to run the BizTalk Configuration again (after fixing the cause of the problem) but all previous installation tasks will be preserved.

1. Starts the BizTalk Server Configuration from the Microsoft BizTalk Server 2006 menu (Programs -> Microsoft BizTalk Server 2006).
2. Select the “Custom configuration” radio button. “Basic configuration” can be used for single server installation only. For the “Database server name”, type in the name of the machine hosting SQL Server: "SQLServer". For the “User Name” type in "BIZTALK\BTSAdmin" and its password. Click “Configure”.
3. You are now on the Overview Panel of the BizTalk Configuration tool. You will now configure all the BizTalk components per section.
4. Enterprise SSO. Click on Enterprise SSO; tick the checkbox “Enable Enterprise Single Sign-On on this computer”. Select the radio Button “Join an existing SSO Sytem”. For the Database configuration, make sure the Server Name is “SQLServer” and the DB Name is “SSODB”. Set the account name to “BIZTALK\SsoService” for running the SSO Windows Service.
5. Enterprise SSO Secret Backup. No change as the Master Secret Server is the SQLServer machine, not the BizTalkServer machine. See paragraph 3.3 in which we installed the Enterprise SSO Master Secret Server. Once BizTalk is fully configured, you will be able to verify that by running the SSO Administration snap-in and check the value of "SSO Secret Server" field under the "system" sub-tree.
6. Group. Tick the checkbox “Enable BizTalk Server Group on this computer”. Select the radio button “Create a new BizTalk Group”. Under “Data Stores”, check that the Database Server Name is set to “SQLServer” for the 3 BizTalk databases that will be created. Under “BizTalk Administrative Roles”, change the Windows Group to use “BIZTALK\BizTalk Server Administrators” and “BIZTALK\BizTalk Server Operators”. Be careful that the default values are Windows Groups with the same name but local to the BizTalk machine (not the Groups defined in the Domain).
7. BizTalk Runtime. Tick the checkbox “Register the BizTalk Server runtime components”. Make sure that both checkboxes “Create BizTalk In-Process Host and Instance” and “Create BizTalk Isolated Host and Instance” are checked. There is no need to tick the “Trusted” checkboxes. Select “Trusted” only if you want to pass the credentials (SSID and/or Party ID) of the sender when submitting messages to the MessageBox database. This is equivalent to creating a trust relationship between the servers.
Under “Windows Service”, Change the account names to “BIZTALK\BTSAppHost” for the BizTalk Host Instance Account and “BIZTALK\BTSIsolatedHost” for the BizTalk Isolated Host Instance Account.
Under “Windows Group”, change the Windows Group to “BIZTALK\BizTalk Application Users” and “BIZTALK\BizTalk Isolated Host Users”. Be careful that the default values are Windows Groups with the same name but local to the BizTalk machine.
8. MSMQT. No changes.
9. Business Rule Engine. Tick the checkbox “Enable Business Rules Engine on this computer”. Make sure that the Server Name is “SQLServer” and the Database Name is “BizTalkRuleEngineDB”.
Under “Windows Service”, set the Account name to “BIZTALK\ReuService”.
10. HWS Runtime. No changes.
11. BAM Tools. No changes.
12. BAM Alerts. No changes.
13. The configuration is now finished. Click on “Apply configuration”. Review the configuration settings and click “Next”. This is the most likely point of failure in this lab, hence the usefulness of undo disks. If things go bad, look at the problem carefully (you can find clues from the log file generated by the BizTalk Server Configuration tool) and check that you have completed every step of the setup correctly on each server.

4.7. Verify Configuration.

You can check if the configuration went well by having a look at the Windows Event Log, there should be a lot of Information entries and no errors or warning related to BizTalk.

You should also start the BizTalk Server administration console and verify the BizTalk Server host instance is started (found under BizTalk Group\Platform Settings\ Host Instances). Try stopping and starting the host instance.

Finally, deploy and test a simple orchestration, such as the HelloWorld orchestration found in the BizTalk SDK samples.

5. References.

MS documentation - BizTalk Server 2006 Installation and Upgrade Guides.

Simplified BizTalk Server 2006 Installation.

Guidelines for Implementing Active Directory Permissions on Multi Server BizTalk Installations.