In this post I will introduce the pipeline technology in BizTalk Server.
1. What is a BizTalk Server Pipeline?
BizTalk server being an integration platform it is most likely that it will have to communicate with systems using various document formats. Because the BizTalk Server 2006 engine works with XML documents internally, it must provide a way to convert other formats to and from XML. Other services may also be required when converting formats, such as authentication of the sender of a message, encryption or decryption, property promotion and so on.
To handle these tasks in a modular and customizable way, a pipeline is constructed from some number of stages; each stage contains one or more .NET or COM (Component Object Model) components. Each component handles a particular part of message processing. It is because all these components are run in sequence that the whole process is called a pipeline.
A stage is a container of components, each stage is itself a component with metadata. Stages have no execution code, as opposed to pipeline components, which do have execution code.
The BizTalk Server 2006 engine provides several standard components that address the most common cases. If these aren’t sufficient, developers can also create custom components for both receive and send pipelines.
To summarize, pipelines enable the developer to define a series of transformations that will be performed on a message as it is being received or sent.
Message processing workflow:
The message is passed from the adapter to the receive pipeline where it is transformed to XML. The message can then be used by orchestrations, or passed to a send pipeline, and then to a send adapter.
As seen in the picture above, there are 2 types of pipelines:
– Receive Pipeline, its purpose is to prepare a message for being processed by the server after being received by an adapter.
– Send Pipeline, its purpose being to prepare a message for sending to an adapter after being processed by the server.
Execution mode
Each pipeline stage can have its own Execution Mode setting, so different stages within a pipeline can have different execution modes.
When the Execution Mode property is set to All, all the components within the stage are run in the configured sequence. A run-time error occurs if any component of the stage encounters an error while processing a message. Use this mode when several components must be run to complete a logical task.
When the Execution Mode property is set to FirstMatch, only the first component that recognizes the message is run. If no components in the stage recognize the message, a run-time error results. Use this mode when the pipeline stage receives messages in several formats.
All the stages execution mode of the standard pipelines delivered with BizTalk Server 2006 are set to All, except for the Disassemble stage of the receive pipeline which is set to FirstMatch.
2. Receive Pipeline.
The receive pipeline is the pipeline executed by a receive port; it is executed after an adapter reads the message from a physical location through a particular protocol (FTP, HTTP, MSMQT …)
The receive pipeline takes the initial message, performs some transformations, and disassembles the raw data into zero, one, or multiple messages which are then processed by the BizTalk server engine.
Details of tasks executed by a Receive Pipeline:
As shown, there are 4 stages executed by a receive pipeline:
2.1 Decode Stage.
– This stage is used for components that decode or decrypt the message. Decoding is the process of transforming information from one format into another.
– This stage takes one message and produces one message.
– This stage can contain between zero and 255 components.
– All components in this stage are run.
2.2 Disassemble Stage.
– This stage is used for components that parse and disassemble the message.
– Disassembling is a process of breaking up a large interchange message into smaller messages by removing the Envelopes. It is also often called “debatching”
For example, instead of sending n messages of the same type sequentially, a system might send 1 large message containing n messages which are the actual purpose of the communication. The message containing the sub-messages is what is called the envelope.
For this feature to take place, the use of an Envelope Schema is necessary. The Envelope schema is a special type of schema that can be created within BizTalk. It defines the schema of the enveloping message.
– This stage creates a BizTalk Message Object and a Message Context Object.
– This stage promotes properties from the interchange message (envelope and individual messages) to the Message Context of each individual messages generated.
– The components within this stage probe the message to see if the format of the message is recognized. Based on the recognition of the format, one of the components disassembles the message.
– If this stage contains more than one component, only the first component that recognizes the message format is run. If none of the components within the stage recognize the message format, the message processing fails.
– This stage should include any custom components that implement special behavior to disassemble the message contents.
– This stage can contain between zero and 255 components. If there are no components in the stage, the message is passed through.
– This stage takes one message and can produce one or more messages.
2.3 Validate Stage.
– This stage is used for components that validate the message format – XML document validation against a XML schema (XSD).
A pipeline component processes only messages that conform to the schemas specified in that component.
– Components in this stage are used to validate the XML messages produced by the Disassemble stage. The XML is validated through the use of XML schemas defined in the component.
– This stage can contain between zero and 255 components.
– All components in this stage are run.
– This stage may be run more than once. This is because it runs once per message created by the Disassemble Stage. So if the disassemble stage produce 10 messages, it will run 10 times.
2.4 Resolve Party Stage.
– This stage is a placeholder for the Party Resolution Pipeline Component.
– This stage may be run more than once. It runs once per message created by the Disassemble stage.
– This stage can contain between zero and 255 components.
– All components in this stage are run.
3. Send Pipeline.
A send pipeline is responsible for processing documents before sending them to their final destinations. The send pipeline takes one message and produces one message to send.
Details of tasks executed by a Send Pipeline:
There are 3 stages executed by a send pipeline:
3.1 Pre-assemble Stage
– This stage is a placeholder for custom components that should perform some action on the message before the message is serialized.
– This stage is run once per message.
– This stage can contain between zero and 255 components.
– All components in this stage are run.
3.2 Assemble Stage
– Components in this stage are responsible for assembling or serializing the message and converting it to or from XML. It is the inverse operation of the disassemble stage in a receive pipeline.
– Assembles the message and prepares it to be transmitted by taking steps such as adding envelopes, move context properties to the message and other tasks complementary to the disassemble stage in a receive pipeline
– This stage accepts zero components or one component.
– All components in this stage are run.
3.2 Encode Stage
– This stage is used for components that encode or encrypt the message. Encoding is the process of transforming information from one format into another. The encoding stage is the inverse operation of the decoding stage in a receive pipeline.
– This stage is run once per message.
– This stage can contain between zero and 255 components.
– All components in this stage are executed.
4. Standard Pipelines and Pipeline Components shipped with BizTalk.
BizTalk is shipped with a series of existing pipeline components ready to use.
BizTalk is also shipped with 2 pairs of default pipelines, which are composed of standard pipeline components.
4.1 Standard Pipeline Components
4.1.1 Standard components provided for each stages of a receive pipeline:
Decode Stage: BizTalk Server 2006 provides one standard component for this stage, the MIME/SMIME Decoder. This component can handle messages and any attachments they contain in either MIME or Secure MIME (S/MIME) format. The component converts both types of messages into XML, and it can also decrypt S/MIME messages and verify their digital signatures.
Disassemble Stage: BizTalk Server 2006 provides three standard components for this stage.
1. The Flat File Disassembler component turns flat files into XML documents. These files can be positional, where each record has the same length and structure, or delimited, with a designated character used to separate records in the file. A typical example of such flat file is the CSV (Comma Separated Values) file format.
2. The XML Disassembler parses incoming messages that are already described using XML.
Main role of the XML disassemble component:
– Removes envelopes.
– Disassembles the interchange. The interchange is the envelope and the documents contained in it. Disassembling is the process of creating individual messages from the messages contained in an Envelope.
– Responsible for promoting properties from the interchange (the envelope) and the individual documents to the message object context.
3. The BizTalk Framework Disassembler. It accepts messages sent using the reliable messaging mechanism defined by the BizTalk Framework, which was implemented in BizTalk Server 2000.
Validate Stage: BizTalk Server 2006 provides one standard component for this stage, the XML Validator. This component validates an XML document produced by the Disassemble stage against a specified schema or group of schemas, returning an error if the document doesn’t conform to one of those schemas.
Resolve Party: BizTalk Server 2006 provides one standard component for this stage, Party Resolution, which attempts to determine an identity for the message’s sender. If the message was digitally signed, the signature is used to look up a Windows identity in the Management database of BizTalk Server 2006. If the message carries the authenticated security identifier (SID) of a Windows user, this identity is used. If neither mechanism succeeds, the message’s sender is assigned a default anonymous identity.
4.1.2 Standard components provided for each stages of a send pipeline:
Pre-assemble Stage: No standard components are provided but custom components can be inserted here as needed.
Assemble Stage: As the Disassemble stage in a receive pipeline, this stage also has three standard components. They implement the inverse operation of the receive pipeline standard components.
1. The Flat File Assembler converts an XML message into a positional or delimited flat file.
2. The XML Assembler supports adding an envelope and making other changes to an outgoing XML message.
Main role of the assembler:
– The assembler creates the envelope by using a specified envelope.
– The component copy the context properties values to the XML message by using the predefined XPaths coded as annotations in the XSD schemas of the message.
– The component copy the context properties to the envelope by using the predefined XPaths coded as annotations in the XSD schemas associated with envelopes.
– The component appends the message to the envelope.
3. The BizTalk Framework Assembler packages messages for reliable transmission using the BizTalk Framework messaging technology.
Encode Stage: The MIME/SMIME Encoder. It implements the inverse operation as the MIME/SMIME decoder component of the receive pipeline.
This component packages outgoing messages in either MIME or S/MIME format. If S/MIME is used, the message can also be digitally signed and/or encrypted.
4.2 Standard Pipelines
The standard pipelines are shipped with the BizTalk server product. They cannot be modified in the Pipeline Designer. The pipeline designer is a tool to create new pipelines from within Visual Studio 2005.
These default pipelines are ready to use and can be selected when configuring a send port or receive location in BizTalk Explorer or during the configuration of a send or receive shape within the orchestration designer.
The two pairs of default pipelines available are the pass-through pipelines and the XML pipelines.
Pass-Through Pipelines.
The pass-through pipelines have no components. They are used for simple pass-through scenarios when no message payload processing is necessary. These pipelines are generally used when the source and the destination of the message are known, and the message requires no validation, encoding, or disassembling.
Particularity of the pass-through receive pipeline:
Because it does not contain a disassembler, the pass-through receive pipeline cannot be used to route messages to orchestrations (as it is the disassemble stage that does the property promotion).
The pass-through receive pipeline does not support property promotion.
Particularity of the pass-through send pipeline:
No particularity, it just sends the message to the adapter as it was received from the MessageBox.
XML Pipelines.
The XML pipelines are the default pipelines that should be used when the message sent or received are already in XML and must be fully functional within the BizTalk system (for example, orchestration must be able to subscribe to them, properties promotion must be possible).
Particularity of the XML receive pipeline:
The XML receive pipeline consists of the following stages:
1. Decode. Empty
2. Disassemble. Contains the XML Disassembler component.
3. Validate. Empty
4. Resolve Party. Runs the Party Resolution component.
Particularity of the XML send pipeline:
The XML send pipeline consists of the following stages:
1. Pre-assemble. Empty
2. Assemble. Contains the XML Assembler component
3. Encode. Empty
Pictures in this post were taken from the BizTalk documentation. Hope that’s alright!