Promoted property vs distinguished field - tutorial

First of all while many people talk about Promoted Property, the official term is Promoted Field, as defined in the MSDN documentation and also visible in Visual Studio’s BizTalk Schema Editor. I will nevertheless keep using the term most people are familiar with but I will consider both terms as synonyms.

Part 1 & 2 is a tutorial on how to create promoted properties and distinguished fields through the Schema Editor and the BizTalk’s API while part 3 is discussing about performance and differences between Promoted Properties and Distinguished Fields. You might only be interested in the 3rd part of this article if you already are a seasoned BizTalk developer.


1. Promoted Properties (Promoted Fields)

1.1 What are Promoted Properties?

Promoted Properties are Message Context Properties that are flagged as promoted; being promoted it allows the Message Engine to route messages based on their value, and being in the message context allows doing so without having to look at the message payload (which would be an expensive operation). Promoted Properties are the most common way to enable content-based routing. They are available to the pipelines, adapters, Message Bus and orchestrations.

Promoted Fields (= Promoted Properties) are PROMOTED in the message context by the receive pipeline when a message is received on a port. It is usually the job of the disassembler pipeline component such as the XML and Flat File disassembler but any custom pipeline component can also do it.


1.2 How to promote properties?

As stated in a previous post I wrote, BizTalk Messaging Architecture, message context properties are defined within a property schema and so all promoted properties must be defined in a custom property schema.

The action of promoting a message element to a promoted property creates a message context property that will contain the message element value and flags it as promoted so that it is available for routing.

There are 2 ways to promote a message element:

1. Quick promotion

Quick promotion is the simplest way to create a promoted property. Simply right click on the element’s node and choose Quick Promotion (see Fig 1.1). When choosing this option, Visual Studio will create a property schema called PropertySchema.xsd and add in the message’s schema a reference to the generated property schema.

Each property promoted this way will create a corresponding element in the property schema with the same name and type as defined in the message’s schema.

This means that when using quick promotion, the promoted property element name will always be the same as the message’s element name. If you have several elements with the same name, you might want to use manual promotion instead to avoid confusion or avoid having a property value overridden.

Fig. 1.1 Quick Promotion


2. Manual Promotion

To manually promote a property, a property schema must be created with the elements that will hold the promoted property values. To create a property schema, you need to add a new item in your BizTalk solution, and chose Property Schema as the type of file (see Fig. 1.2). Once all the elements are created in the property schema, you associate the property schema with the message’s schema by right clicking on any node of the message schema and choose Show Promotions (see Fig. 1.3), then click on the Property Fields tab, click on the folder icon and finally select the property schema you just created (see Fig. 1.4). Note that it is actually possible to use more than 1 property schema per message. Anyhow, all promoted properties will end up being written in the message context and available for all BizTalk artifacts having access to message context.

Once the property schema is picked, you can start promoting message elements as promoted properties. To do so, click a message element and click on the add button, the Node Path column will display the XPath to the message element you are promoting and the Property column let you chose the promoted property that will contain the value of the message element at runtime (see Fig. 1.5).

An interesting side effect is that Manual Promotion let you have promoted properties with different names that the original message element name. This might be useful when a same property schema is used to hold promoted properties from different message types or when a message has different element with the same name.

Using Manual Promotion, it is also possible to promote message elements to promoted properties in the system property schemas shipped with BizTalk. To do so, just browse in the References sub-tree when picking the property schema (see Fig. 1.4).

Fig. 1.2 Manually create a custom property schema


Fig. 1.3 Manual Promotion


Fig. 1.4 Selecting a Property Schema


Fig. 1.5 Selecting the message element to promote and the promoted property that will contain the message element’s value.



1.3 How to promote properties through the BizTalk API?

As I already said in a previous post, BizTalk Messaging Architecture, context properties are stored in a property bag, an object which implements the IBaseMessageContext interface. This interface contains 2 methods which take the same parameters (a property name, an XML namespace, and value for the property).

a. The Write() method is used to write the property value into the message context without actually promoting it. It can be used to write distinguished fields or to write transient values. Calling the Write() method is NOT promoting a property, it is writing a property.

b. The Promote() method is used to write the property value into the message context but also flags the property as promoted so that it is available for routing. This is the method that needs to be called to promote a property.

To promote a property through the API, element name and namespace passed as parameter of the Promote() method must be those of the property defined in the property schema. They are most easily accessed by referencing the property schema assembly and using the properties on the class created for the property.

Example of promoting a property through an API call (copied from the BizTalk 2006 MSDN documentation: Processing the Message):

//create an instance of the property to be promoted
SOAP.MethodName methodName = new SOAP.MethodName();

//call the promote method on the context using the property class for name and namespace
pInMsg.Context.Promote(methodName.Name.Name, methodName.Name.Namespace,
"theSOAPMethodName");

As I mentioned before, it is possible to promote properties to the BizTalk system property schema using Manual Promotion in Visual Studio’s Schema Editor for BizTalk. It is possible to achieve the same programmatically and as for other promotion, the name and namespace passed in parameter of the Promote() method is the one of the promoted property in the property schema, the system property schema namespace in this case.

//BizTalk system properties namespace
private const string BTSSystemPropertiesNamespace = “http://schemas.microsoft.com/BizTalk/2003/system-properties”;

//Promote the MessageType property
string messageType = "http://" + "schemas.abc.com/BizTalk/" + "#" + "Request";
message.Context.Promote("MessageType", BTSSystemPropertiesNamespace, messageType);


2. Distinguished fields

2.1. What are distinguished fields?

Distinguished fields are message elements that are written into the message context. They differ with promoted properties in 2 main aspects:

• They are not flagged as promoted in the message context and so are not available for routing by the Messaging Engine (adapters, pipeline…). Their typical use is instead for the orchestration engine.
• They are not defined using a property schema.

Distinguished Fields are WRITTEN in the message context by the pipeline when a message is received on a port. It is usually the job of the disassembler pipeline component such as the XML and Flat File disassembler but any custom pipeline component can also do it.

Distinguished fields are useful when a message element value needs to be accessed from an orchestration. Instead of having the orchestration engine search through the message to evaluate an XPath expression (which can be resource-intensive on a large message), a distinguished field can be used. Distinguished fields are populated in the message context when the message is first loaded. Consequently, each time the distinguished field needs to be accessed, the orchestration engine will directly read it from the message context (a property bag object) instead of searching for the original value in the message with XPath. Needless to say that retrieving a single value from a property bag object is much faster than evaluating an XPath expression.

Distinguished fields also offer IntelliSense in the orchestration expression editor which enhances code readability.


2.1 How to create a distinguished field?

The main source of confusion between distinguished fields and promoted properties is that they are both created in Visual Studio’s Schema Editor through the Promote -> Show Promotions contextual menu option of a message schema’s element. Once the dialog box is open, make sure that you are on the Distinguished Field tab, select the message elements and click the Add>> and <<Remove buttons to add and remove distinguished fields (see Fig 2.1).

Fig 2.1 Creating a Distinguished Field.


2.2 How to create a distinguished field through the BizTalk API.

Distinguished fields are written into the context using the Write() method on the IBaseMessageContext object. To be recognized as a distinguished field, the namespace of the property must be “http://schemas.microsoft.com/BizTalk/2003/btsDistinguishedFields”. Pipeline components delivered with BizTalk do not use those context properties. It is nevertheless possible to read/write distinguished field in the code of custom pipelines, as for any other context properties.

Example of writing a distinguished field through an API call (taken from the BizTalk 2006 MSDN documentation: Processing the Message).
//write a distinguished field to the context
pInMsg.Context.Write("theDistinguishedProperty",
"http://schemas.microsoft.com/BizTalk/2003/btsDistinguishedFields",
"theDistinguishedValue");

Would you use another namespace, it will result in writing a plain transient value in the property bag and won’t be recognized as a distinguished field by the orchestration engine.

//Write a transient value to the message context
message.Context.Write("MyVariable", "SomeNameSpace", SomeData);


3. Considerations, similarities and differences between promoted properties and distinguished fields.

3.1 Performance considerations:

• Promoted properties are limited to 255 characters for routing performance reasons. Properties that are simply written in the context (such as distinguished fields) are not limited in size but large properties decrease performance.
• All Context properties (both Promoted and Distinguished Fields) are stored separately from the message in the Message Box Database. Consequently consuming more space in the BizTalk databases but more importantly incurs more load when persisting/reloading the message in/from the DB. Note that if tracking is turned on, Promoted Properties are also stored in the Tracking database.
• Distinguished Fields cost less than Promoted Properties in terms of performance. Both Promoted and Distinguished Fields require the same overhead of writing their values to the context Field bag in the Message Box database, but Promoted Fields have the additional overhead of being written in BOTH to the Message Box context tables AND the subscription tables. Distinguished fields are not stored in the subscription table as they do not participate in routing.
  Promoted Fields have an impact every time a message is written in the Message Box because each Promoted Property that exists must be evaluated in a massive T-SQL union statement that builds the list of matching activation subscriptions. In short, the more Promoted Fields you have the more costly the subscription process is.

3.2 Other considerations

• Both promoted properties and distinguished fields are populated when a Pipeline Disassembler Component parses a message and either Promotes or Writes the value to the message’s context.
• Empty pipelines such as the Pass-through pipeline do not promote or write anything in the message context as it lacks a disassembler component.
• Writing a value into the context with the same name and namespace that were used previously to promote a property causes that property to no longer be promoted. The write essentially overwrites the promotion.
• Writing a property in the context having a null value deletes the context property altogether because null-valued properties are not permitted.

3.3 Distinguished and Property Fields Difference Summary.

• Promoted Fields should be used for routing, correlation and/or tracking.
• Distinguished fields should be used when a particular message element is commonly manipulated in one or more orchestration.

Here is a table outlining the main differences between both types of fields:

Promoted Fields (aka Promoted Properties)	Distinguished Fields
Used for routing (subscription mechanism) IsPromoted = true	Do not participate in routing IsPromoted = false
Used for tracking	Not used for tracking
Restricted to 255 characters	No size limitation
Available for use in orchestrations	Available for use in orchestrations
Require property schema	Do not require property schema
Used by standard pipeline components	Accessible only by custom pipeline component which would explicitly access them

4. References.

MSDN - The BizTalk Server Message
MSDN - Processing the Message
MSDN - About BizTalk Message Context Properties
Neudesic's blog - Distinguished fields myths

BizTalk Messaging architecture

The core of the BizTalk Server product is the BizTalk Messaging subsystem, called the Message Bus. As said in the BizTalk documentation, The Message Bus is a publisher/subscriber model; indeed the Message Bus queries messages published into the BizTalk Message Box database looking for messages that match a particular subscription.

The most important point to understand about the publisher/subscriber model is that publishing and subscription concepts are relative to the Database.

Here is a picture illustrating the concept:


The Messaging infrastructure is composed of the Message Box database and also different software components, called the Messaging Components. The Db and the components together compose the publisher/subscriber BizTalk Messaging subsystem, the Message Bus.

An important side effect of this architecture is that in BizTalk Server, messages are immutable once published (in the Message Box DB). This is because more than one end-point can subscribe to the same message. Would messages be mutable, some end-points might not match the subscriptions rule after the message has been modified. Having the subscription query result vary with time on the same message (due to change in the message payload) would break the publisher/subscription architecture thus making the whole BizTalk product unpredictable (and so useless).


1. The Message Box.

The Message Box is an SQL Server database which stores XML messages as well as metadata related to each messages. The message’s metadata is called the message context. Each metadata item (a key/value pair) of the message context is called a context property. The most important information to know about the message context is that it holds all the necessary information for message routing - message subscription.


2. Messaging Components.

While the Message Box database is the message storage facility of the Message Bus, the messaging components are software components that actually move messages between subscribers and publishers. They receive and send messages in and out of the BizTalk Server system.


2.1 Host Services.

A BizTalk host is a logical container. It provides the ability to structure the BizTalk application into groups that could be spread across multiple processes or machines.
When you create a host in BizTalk server, it creates a logical unit in which you can run different BizTalk applications or different type of BizTalk artifact.
For example, if your BizTalk applications are pretty small, you could create 1 host per BizTalk application you develop. On the opposite if your applications are big, you can create different hosts to separate logical grouping of your application, such adapters, orchestrations, ports and so on. If each host runs on a separate physical machine this help in balancing the load between processors (some sort of manual load balancing).

A host instance is simply a running instance of the host logical grouping. It runs as a Windows Service, each host being a separate windows process; a separate instance of BTSNTSvc.exe (or BTSNTSvc64.exe for BizTalk Server 64 bit).

As explained before, the host instance raison d’être is to provide logical grouping units. It does not implement itself the BizTalk runtime, it is a container where the BizTalk subservices run. These subservices running inside the host instance implements together the actual runtime of the BizTalk Message Bus.

Host instances can run all BizTalk subservices or only some of them depending on what type of BizTalk artifacts they are running. To understand which subservice is used by which type of artifact, here is a list of the different subservices running inside a BizTalk host instance - note that the list of services can be found in the the adm_HostInstance_SubServices table in the Management Database:

Service	Description
Caching	Service used to cache information that is loaded into the host. Examples of cached information would be assemblies that are loaded, adapter configuration information, custom configuration information, etc.
End Point Manager (EPM)	Go-between for the Message Agent and the Adapter Framework. The EPM hosts send/receive ports and is responsible for executing pipelines and BizTalk transformations. The Message Agent is responsible to search for messages that match subscriptions and route them to the EPM.
Tracking	Service that moves information from the Message Box to the Tracking Database.
XLANG/s	Host engine for BizTalk Server orchestrations.
MSMQT	MSMQT adapter service; serves as a replacement for the MSMQ protocol when interacting with BizTalk Server. The MSMQT protocol has been deprecated in BizTalk Server 2006 and should only be used to resolve backward compatibility issues.

2.2 Subscriptions.

In a publish/subscribe design, you have three components:

• Publishers
• Subscribers
• Events

Publishers include:

• Receive ports that publish messages that arrive in their receive locations
• Orchestrations that publish messages when sending messages (orchestration send shape)
• Orchestrations that start another orchestration asynchronously (start orchestration shape). On a side note, the call orchestration shape does not publish the message into the Message Box, the message is just passed as a parameter.
• Solicit/response send ports publish messages when they receive a response from the target application or transport.

Subscriptions:

Subscription is the mechanism by which ports and orchestrations are able to receive and send messages within BizTalk server (see picture above).

A subscription is a collection of comparison statements, known as predicates, comparing the values of message context properties and the values specific to the subscription.

There are two types of subscriptions: activation and instance.

An activation subscription is one specifying that a message fulfilling a subscription should create a new instance of the subscriber when it is received. Examples of things that create activation subscriptions include:

• Send ports with filters
• send ports that are bound to orchestrations
• orchestration receive shapes that have their Activate property set to true.

An instance subscription indicates that messages fulfilling the subscription should be routed to an already-running instance of the subscriber. Examples of things that create instance subscriptions are:

• Orchestrations with correlated receives.
• request/response-style ports waiting for a response.

It is also important to know that when you define filter criteria on a send port, you are actually modifying the subscription of the port. As a reminder, filter expressions determine which messages are routed to the send port from the Message Box.


Enlisting:

The process of enlisting a port simply means that a subscription is written for that port in the Message Box. Consequently, un-enlisted ports do not have subscriptions in the Message Box.
The same is true for other BizTalk artifacts. An un-enlisted orchestration is an orchestration ready to process messages but having no way to receive messages from the Messaging Engine as no subscription is created for it yet.

The difference between an un-enlisted artifacts and a stopped artifacts is that ports and orchestrations that are enlisted, but not started, will have any messages with matching subscription information queued within the Message Box and ready to be processed once the artifact is started. If the port or orchestration is not enlisted, the message routing will fail, since no subscription is available and the message will produce a “No matching subscriptions were found for the incoming message” exception within the Windows Event Log.


Typical port usage with an orchestration:

What happens in an orchestration that as a send shape connected to a logical port which is in turn bound to a physical port, is that the message sent by the send shape will have a TransportID context property set to a value that matches the physical port TransportID. As the TransportID uniquely defines the port, this mechanism assures that the physical port will always receive the messages coming from the orchestration. It does not mean that only that port will receive the message as due to the nature of a publisher/subscriber architecture, any other port having a subscription matching the message context will also receives the message.


2.3 Messages

As said earlier a Message is more than just an XML document. It is actually a message containing both data and context. To be more precise, a message is composed of context properties and zero or more message parts.

Keep in mind that message parts are not always XML document. If the message is received through a port using the pass-through pipeline, the message can be any kind of data including binary data. On a side note, a pass-through pipeline does not promote context properties; this makes sense as the message is not even supposed to be XML in a pass-through pipeline, so it is not possible to evaluate XPath expression on the message to determine the value of the context property.

As said earlier a message is immutable once it is published. This means that once stored in the MessageBox DB, it can’t be changed. A message can nevertheless be changed once it is out of the database. In a receive pipeline component, a message can be modified before it is published in the MessageBox. In a send pipeline component, a message can be modified after being received from the MessageBox. A typical place to create or modify a message is also inside an orchestration.


2.4 Message Context Properties

Message context properties are used for the subscription mechanism (routing the message to its appropriate end point). They are defined in a property schema. At runtime, the property values are stored into a context property bag.

The property schema is associated with the message schema within BizTalk so that every inbound schema-based message has a schema and a property schema attached to it.

The property schema consists of a global property schema that every message can use by default and of an optional custom property schema which can be created to define application-specific properties. Both types of properties are essentially the same at runtime and both are stored in the context property bag.

So, both types of properties can be used by the subscription mechanism to evaluate which endpoints have a subscription matching the message. The most common subscription is based on a global property called the messageType which is a combination of the XML namespace of the message and the root node name separated by a # character. Ex: http://www.abc.com#RootElementName.

Using subscription to route documents to the proper endpoint is called Content Based Routing (CBR).
For information, if the message is not schema-based, there will be no MessageType property value. Such is the case for binary data message.

Message context properties are populated by the BizTalk runtime in 2 artifacts:

• The adapter writes and promotes into the message context properties related to the location, adapter type, and others properties related to the adapter.
• The Receive Pipeline can write and promote properties into the message context in any of its pipeline components. Disassembling components are of particular interest because they promote the messageType property which is commonly used for Content Based Routing.

Property bag.

It is possible to use the BizTalk API in pipeline component code to read/write context properties from the property bag. The property bag is an object implementing the IBasePropertyBag interface. If you intend to use that interface in a custom pipeline to write properties that will be used for routing, you have to keep in mind that properties that are simply written into the property bag using the Write() method are not available for routing. To have a property available for routing, you need to promote the property with a different API call, the Promote() method. This method writes the property and its value in the property bag but ALSO flag the property as promoted and so make it available for routing.