Solving Error : BAM deployment failed : The locale identifier (LCID) 8192 is not supported by SQL Server

When trying to deploy a BAM activity on a BizTalk Server 2020 machine, I had the following error: BAM deployment failed : The locale identifier (LCID) 8192 is not supported by SQL Server

Error Description

The following BAM command:

bm deploy-all -DefinitionFile:MyActivity.xml

Returns the following error:

Deploying Activity… ERROR: The BAM deployment failed.
A .NET Framework error occurred during execution of user-defined routine or aggregate "deploy_project_internal":
System.Data.SqlClient.SqlException: The locale identifier (LCID) 8192 is not supported by SQL Server.
System.Data.SqlClient.SqlException:
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.SqlInternalConnectionSmi.EventSink.DispatchMessages(Boolean ignoreNonFatalMessages) at System.Data.SqlClient.SqlDataReaderSmi.InternalNextResult(Boolean ignoreNonFatalMessages) at System.Data.SqlClient.SqlDataReaderSmi.NextResult() at System.Data.SqlClient.SqlCommand.RunExecuteReaderSmi(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteScalar()
at Microsoft.SqlServer.IntegrationServices.Server.ServerConnectionControl.GetServerProperty(String propertyName)
at Microsoft.SqlServer.IntegrationServices.Server.ServerConnectionControl.GetSchemaVersion()
at Microsoft.SqlServer.IntegrationServices.Server.ISServerExecArgumentBuilder.ToString()
at Microsoft.SqlServer.IntegrationServices.Server.ServerApi.DeployProjectInternal(SqlInt64 deployId, SqlInt64 versionId, SqlInt64 projectId, SqlString projectName)

After a little bit of Googling, I found someone having a different issue but with what looked like the same root cause (see Marc van der Wielen blog post here).
I detail the solution here again in case that one day Marc’s blog goes down. All credit goes to him.

Error root cause

The root cause of the problem is that the Service Account running the SQL Server instance is using an (apparently) unsupported locale setting.

In my case, my Windows 10 machine is configured with en-BE (English-Belgium) locale. I installed MS SQL Server with mostly all default options. By doing so, the Service Account created by the installer took the en-BE locale setting.
Once I changed the locale setting of the Service Account to en-US, the error hereabove disappeared.
It looks like while SQL Server did run fine for everything so far, it needs to have the local settings of the Service Account set to en-US for some specific functionalities to work properly (such as publishing a BizTalk BAM Activity in my case).

Resolution Procedure

1. Find what is the Service Account running the MS SQL Server service.
We can find that easily by running services.msc from a prompt.
When we find the SQL Server service, we can just look at its properties to see the name of the Service Account used to run it:

Finding SQL Server Service Account Name

2. Find the SID (Security Identifier) of the Service Account.
One way to do this is through the registry:
– Open the registry by running regedit from a prompt.
– Once in the registry editor, navigate to the following registry node: Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList
– The registry node contains a list of nodes having SIDs as name. Find the node for which the ProfileImagePath‘s entry contains the MSSQLSERVER value. Write down the SID, in my case it is S-1-5-80-3880718306-xxxxx.

Another way to find the SID is to use PowerShell:
The PowerShell command is slightly depending of the type of account running the SQL Service (local account or domain account), see details here.

For local accounts:
>$objUser = New-Object System.Security.Principal.NTAccount("NT Service\MSSQLSERVER")
>$strSID = $objUser.Translate([System.Security.Principal.SecurityIdentifier])
>$strSID.Value

For Domain account we get the NTAccount object in the following way:
>$objUser = New-Object System.Security.Principal.NTAccount(DOMAIN_NAME, USER_NAME)

3. Change the Service Account locale setting.
Navigate to the following registry node folder (using your own SID from the previous step): Computer\HKEY_USERS\S-1-5-80-3880718306-xxx\Control Panel\International.
– Set the Locale entry value to “00000409
– Set the LocalName entry value to “en-US
– Restart the SQL Server Service.

Set SQL Server Service Account Locale

Setting the local setting for new user accounts to be of a particular locale

To avoid having this kind of issue, it is possible to tell Windows to set the default locale for new accounts. This option is in the Control Panel -> Regions -> Administrative -> Welcome screen and new user accounts.

This is also scriptable. See : https://docs.microsoft.com/en-US/troubleshoot/windows-client/deployment/automate-regional-language-settings

BizTalk WCF Metadata Only MEX Endpoint Error: Root element is missing

In a BizTalk 2016 application, I have a receive location using the WCF-NetTcp adapter. The receive location is using an in-process Receive Handler and so I used the BizTalk WCF Service Publishing Wizard to publish a Service Metadata Endpoint (MEX) hosted in IIS so that clients can retrieve the WSDL of the web service exposed.
Note that typically, a BizTalk isolated host is an IIS Application Pool instance: w3wp.exe.

Endpoint error: Root element is missing

Once the MEX endpoint is deployed in an IIS application, I saw the following error when browsing to the endpoint:

Server Error in ‘xxx’ Application.
Root element is missing. 
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. 
Exception Details: System.Xml.XmlException: Root element is missing.
Source Error: 
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Windows Event Log Analysis

When looking at my Windows Event Log, I saw the following error:

Server Error in ‘xxx’ Application.
WebHost failed to process a request.
Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/4032828
Exception: System.ServiceModel.ServiceActivationException: The service ‘xxx.svc’ cannot be activated due to an exception during compilation. The exception message is: Root element is missing.. —> System.Xml.XmlException: Root element is missing.
at System.Xml.XmlTextReaderImpl.Throw(Exception e)
….
Process Name: w3wp

I looked further down in my event log and saw a series of warnings from the Enterprise SSO service with messages of the likes:

SSO AUDIT
Function: GetConfigInfo ({CB480FD2-902B-4F1E-A2DB-43B3954A341B})
Tracking ID: 1c44a765-61a3-4679-901d-f1853fb2f497
Client Computer: BTS2016 (wmiprvse.exe:8432)
Client User: IIS APPPOOL\BizTalkIsolatedHostAppPool
Application Name: {315B6926-BF0C-462D-A8FD-5512F5E41456}
Error Code: 0x80070005, Access is denied.

And:

Access denied. The client user must be a member of one of the following accounts to perform this function.
SSO Administrators: SSO Administrators
SSO Affiliate Administrators: SSO Affiliate Administrators
Application Administrators: BizTalk Server Administrators
Application Users: –
Additional Data: IIS APPPOOL\BizTalkIsolatedHostAppPool {315B6926-BF0C-462D-A8FD-5512F5E41456} WCF-NetTcp_RL_BizTalkServerApplication_{315B6926-BF0C-462D-A8FD-5512F5E41456}

The first thing I noticed in the SSO warnings here above is that they refer to the user “IIS APPPOOL\BizTalkIsolatedHostAppPool” which in my case is the identity of the Application Pool running my WCF Service metadata endpoint.
In IIS 7.5 and above, each Application Pool is by default assigned it’s own virtual account and it’s named using the following pattern: “IIS AppPool\<ApplicationPoolName>” (i.e. this is the name of the virtual account in Windows). In the Application Pool setting, the virtual account name is simply referenced to by setting the identity property to “ApplicationPoolIdentity“.

I already had this account part of the BizTalk Isolated Host Users Windows Group but it seems to not be enough as Enterprise SSO is complaining.

At this stage I did some research and found a clue here but, for security reason, I was not satisfied with having the App Pool Service Account being part of BizTalk Server Administrators Windows Group.
I dug a little deeper and found this interesting blog post. This made me check the BizTalk documentation and it is indeed now documented that BizTalk Host Instance Accounts and BizTalk Isolated Host Instance Accounts must be part of the SSO Affiliate Administrators Windows Group.

Final Solution

Finally, what I did to solve the issue was to add the Application Pool identity to the SSO Affiliate Administrators Windows Group (as it is actually instructed in the BizTalk’s documentation!) and rebooted my machine.

Nevertheless, my personal opinion is that while it is the supported solution documented by Microsoft, the root of the problem lies in the implementation of ISSOConfigStore::GetConfigInfo and that Microsoft should fix that or change the way MEX endpoints retrieve data. Indeed, to me, it does not really make sense that an IIS App Pool should run under an account being an SSO Afiliate Administrator.

Note: I did not have to modify the web.config of the WCF Service MEX endpoint web service generated by the BizTalk WCF Service Publishing Wizard as was mentioned in one of the blog I referenced to:
<system.web>
    <trust level=”Full” originUrl=”” />
</system.web>

Failure Importing WCF Extension configuration to BizTalk WCF-Custom Handler

One of the BizTalk application that I am working on has to access web services from a third-party which does not play nicely with WCF. Nothing too uncommon if the third-party is a little exotic.
The solution was that a Custom Message Encoder was written to be able to read the messages coming from the third-party. To configure the WCF runtime stack, a Custom Message Encoding Binding Element was also created. As we want this custom encoder to only be used by that BizTalk application, we created a specific WCF-Custom Adapter Handler for the application and only that handler is configured with the binding extension configuration (instead of doing a change in machine.config or maybe the BTSNTSvc.exe.config).

As I was recreating a new development environment, I tried to import the WCF binding extension configuration file (a .config file containing a bindingElementExtensions element) to the WCF-Custom Adapter handler and got the following error: “Unable to import configuration from file “xxx.config”. (System.UnauthorizedAccessException) Access to path ‘xxx’ is denied.

WCF Extension configuration fails BizTalk Handler

I found out that this was simply because the configuration file was read-only (as it was coming from TFS source control) and that the BizTalk Admin Console first copies the config file to a temp folder and then tries to delete it. As the file is read-only, it cannot be deleted and importing the binding configuration fails!

The solution was simple, I just had to make the file writeable.
I chose to check-out the file temporarily, import the WCF binding extension configuration file and undo the check-out once the configuration was imported through the BizTalk Admin Console.

Notes:

If you are unfamiliar with WCF extensibility, a sample of a Custom Message Encoder sample is available on MSDN at Custom Message Encoder: Custom Text Encoder and the source code of the sample (and other samples) can be downloaded at Windows Communication Foundation (WCF) and Windows Workflow Foundation (WF) Samples for .NET Framework 4.

To import WCF Extension to a WCF-Custom Adapter Handler, open the BizTalk Admin Console, navigate to BizTalk Group\Platforms Settings\Adapters\WCF-Custom and right click the Handler of choice. Click Properties and then click Import.

Import WCF Extension bindings for BizTalk WCF-Custom Adapter Handler

How to modify WCF services previously published with the BizTalk WCF Service Publishing Wizard?

The BizTalk WCF Service Publishing Wizard is the tool used to easily publish a WCF Service implemented in BizTalk (typically through an orchestration). See the Publish WCF Services section of the BizTalk documentation for some background information if you are not used to the process of publishing WCF services in BizTalk.

In short, the main output of the Wizard is a Web Application containing the Web Services defined by the user in the Wizard plus a bunch of definition files, and schemas. See Publishing WCF Services with the Isolated WCF Receive Adapters for details of the output produced.

The main problem when using the Wizard from the Program Files menu is that it always start empty. If you have previously published a WCF service through the wizard and wish to modify it, you will always have to redefine the existing services and methods from scratch. This is quite inefficient as it is common at development time to have to either:

  • Add new services to the site (in case it is hosted in IIS).
  • Add new web methods on an existing service.
  • Change a schema (which is probably the change we do the most often).

It would thus be particularly tedious to have to redefine completely a WCF service each time we need to modify it.

One of the files produced by the Wizard is WcfServiceDescription.xml (located under \App_Data\Temp). As explained in Publishing WCF Services with the Isolated WCF Receive Adapters on MSDN, it is an XML file that summarizes the settings used when defining the WCF services in the Wizard.

Luckily, it is possible to feed this file back to the Wizard when running it again so that all the existing services and methods can be pre-populated. This is a great time saver at development time as more often than not, methods and contracts (schemas) are changing regularly.

Nevertheless the tool is far from perfect and I had to deploy the BizTalk assembly containing the updated schema to the GAC so that the Wizard would see it. Having the assembly compiled and picked up by the wizard’s file dialog box did not seem to work properly as I could only see the schemas that were already in the GAC from a previous deployment.

The way to do that is to launch the wizard from the command line by using the following syntax:
BtsWcfServicePublishingWizard.exe -wcfServiceDescription=C:\FolderPath\App_Data\Temp\WcfServiceDescription.xml

BtsWcfServicePublishingWizard.exe is located right in the folder where BizTalk is installed: “C:\Program Files (x86)\Microsoft BizTalk Server 2010” on my 64 bit machine.
The only shortcomings I have noticed so far are:

  1. In the wizard, if you chose to create receive locations in a BizTalk application, it will attempt to create all the receive locations defined in the wizard. If any of the receive locations already exist in the BizTalk application (from a previous run), the creation of ALL the receive locations will fail. Therefore, none of the new receive locations will be created while the already existing receive locations obviously still exist in the BizTalk Application. This does not mean that the wizard fails, it still succeeds. We can thus grab the new BindingInfo.xml generated by the wizard, extract the new ports and import them separately through the BizTalk Admin console. Alternatively, it is also possible to simply delete the pre-existing receive locations before running the wizard.
  2. The wizard does not repopulate the target namespace of the generated WCF services, it will default back to http://www.tempuri.org/. The work around is to pick it up beforehand from the service’s wsdl. When opening the wsdl, just look for the “targetNamespace” attribute in the element <wsdl:definitions> , take its value and paste it back in the Wizard.

Anyhow, even with these shortcomings, reusing the WcfServiceDescription.xml is still a great time saver!

BtsWcfServicePublishing.exe

On a side note, there is another tool similarly named, BtsWcfServicePublishing.exe which can be downloaded here (notice that there is no “Wizard” at the end of the name). As this tool does not have any GUI, it can be used to script and automate creation of WCF services for BizTalk. That can be useful for automated deployment for example. See the tool reference. As the tool was made available for BizTalk 2006 R2 (.Net 2.0 runtime), the following <startup> configuration section must be added to the tool’s config file so that it can run against BizTalk 2010 assemblies (.Net 4.0 assemblies).

<configuration>
<startup>
<supportedRuntime version="v4.0" />
</startup>
</configuration>

I have actually wrote a note about it in the MSDN documentation (see the Community Content section).

Datetime XML element converted to UTC – How to read the original time of a different time zone?

I noticed that time information is converted to UTC or to the local time zone when converting XML messages elements of the datetime XML type to a DateTime .Net type. The side effect is that the original time is lost and can’t be recovered.

I will demonstrate this through a scenario and then postulate conclusion and best practice to keep in mind. As this post got a little lengthier than expected, you can jump right to the summary section if you just want the facts.

 

Scenario

I had an orchestration for which I had to read a datetime XML element from an incoming message and put its value in a user friendly string message which would ultimately be visible in an application front-end. The datetime element was made a distinguished field so that it would be easier to access.  When I did a ToString() on the distinguished field, the time part was modified to reflect UTC time instead of the original time from the XML datetime element. This was a problem as the log message had to reflect the actual time of the original message regardless of the time zone.

I wrote a little application to study what was going on through a few illustrating cases. The application has an Order Message containing a <ProviderTime> datetime element with a value using the time zone of Bangkok (UTC+7).

 

Case 1: Get the value from a distinguished field.

In this case I simply mark the datetime XML element as a distinguished field (in the message’s schema) and assign it to a DateTime .Net variable in an expression shape (* – See the footnote on a remark about this).

Input XML:

<ProviderTime>2012-09-22T21:30:00.000+07:00</ProviderTime>

Result:

Calling the ToString() method on the DateTime variable prints: 9/22/2012 2:30:00 PM. This corresponds to UTC time and it means that BizTalk’s runtime created a UTC System.DateTime structure when reading the distinguished field and assigning to the variable.

While it is “correct” in the sense that both the XML datetime element and the .Net DateTime structure represent the same instant in time, it was not good for me as the user expected to see  9/22/2012 9:30:00 PM.

The reason this happens is that as the System.DateTime structure does not contain any information regarding the time zone, the BizTalk runtime converts the time to UTC time. To be exact, the BizTalk runtime calls the .Net framework XmlConvert.ToDateTime(String, XmlDateTimeSerializationMode) method which creates a DateTime object and loses the original time zone information. BizTalk then converts the resulting DateTime to UTC. There is thus no way to display the time as it was in the original message.

We would not want to use a promoted property just for reading a value out of a message but if we had a promoted property, using it would produce the same result.

 

Case 2: Get the value from XPath and convert it to a System.DateTime structure

In this case I use Xpath to get the element value and parse the resulting string into a System.DateTime structure with the following mothod: System.Xml.XmlConvert.ToDateTime(String). I also tried to various overload of the ToDateTime() method.

Input XML:

<ProviderTime>2012-09-22T21:30:00.000+07:00</ProviderTime>

Result:

Calling the DateTime.ToString() method on the DateTime structure would now print: 9/22/2012 3:30:00 PM. This is UTC+1, Dublin’s time zone (+0) with Daylight Time Savings (+1 in summer). It means that the XmlConvert.ToDateTime(String) method creates a System.DateTime structure reflecting the Local Time. Note that this particular overload is deprecated and other exists, which I tried, but basically all they let you chose for is if you want to create a DateTime reflecting Local Time or UTC.

 

Using the DateTime structure is a lost cause as it is not time zone aware and I would thus never be able to hold anything else than local time or UTC time. To solve my problem, I had to do something else but did not want to do some ugly manual string parsing.

I did some research about Date and Time in the .Net framework and read from MSDN that:

DateTimeOffset should be considered the default date and time type for application development

So yes you read it correctly, System.DateTime is “sort of” deprecated! See for yourselves directly from the horse’s mouth: http://msdn.microsoft.com/en-us/library/bb384267.aspx

As The DateTimeOffset structure contains an Offset property (a TimeSpan) which represents the time difference between the time stored in the DateTime structure and UTC time, it can represent other times than local time or UTC time.

 

Case 3: Get the value from XPath and convert it to a System.DateTimeOffset structure

In this case I use Xpath to get the element value and parse the resulting string into a System.DateTimeOffset structure with the following mothod: System.Xml.XmlConvert.ToDateTimeOffset(String).

Input XML:

<ProviderTime>2012-09-22T21:30:00.000+07:00</ProviderTime>

Result:

Bingo! Calling the ToString() method on the DateTimeOffset variable would print 9.30 PM as in the original message! Now all I had to do was to use an overload of the ToString() method taking a format string to display that in a user-friendly manner.

 

Here is a screenshot of the result of my investigations with cases 1,2&3 highlighted:

datetime timezone xml element parsing

And here is a Visual Studio solution if you want to play around yourself (or for myself in the future).

 

Summary:

  1. A distinguished field on a datetime xml element creates a System.DateTime structure adjusted for UTC Time. The DateTime.Kind property is DateTimeKind.Utc. So if in a different scenario than mine, the distinguished field is always Local Time, you can use DateTime.ToLocalTime () to convert the value back to local time. In that case, the DateTime.Kind property will have the value DateTimeKind.Local. Time in original time zone is lost.
  2. Reading a datetime xml element into a string by using XPath and then converting it to a System.DateTime structure by using one of the XmlConvert.ToDateTime() method overloads creates a Local Time or UTC time structure (depending of a parameter on one of the overload). Time in original time zone is lost.
  3. Reading a datetime xml element into a string by using XPath and then converting it to a System.DateTimeOffset structure by using one of the XmlConvert.ToDateTimeOffset() method overloads keeps the original time as it holds the time zone offset information (i.e. +7 hours). We can thus either display the time of the original time zone or convert it to another time zone offset.

 

Type of XML datetime read .Net Type created Default Time zone Time in original time zone available?
Distinguished field System.DateTime UTC NO
Promoted Property System.DateTime UTC NO
XPath and  XmlConvert.ToDateTime() System.DateTime Local Time (for default overload) NO
XPath and  XmlConvert.ToDateTimeOffset() System.DateTimeOffset Original time zone YES

 

Conclusion:

  1. Do not use distinguished fields on datetime elements; use XPath and a DateTimeOffset variable instead. This is a tip I will keep in mind so that I don’t have to worry about the original time value being lost. If you decide to use a distinguished field anyway, you must be aware of its limitation and if it impacts you or not.
  2. When using a promoted property on a datetime element, be aware that its value will be converted to UTC timezone. This might be of importance on subscriptions (port filters and so on). There is no work around on this at its part of the BizTalk engine.

 

I like to use distinguished field when it makes sense because it avoids having to use xpath query all over the place which can be an annoyance when a schema change. As Microsoft advises to use the System.DateTimeOffset structure, would it not be nice to have a distinguished field assignable to DateTimeOffset variable instead of DateTime? It would make a lot of sense as the XML datetime type is time zone aware while DateTime is not but DateTimeOffset is. Making the latter type is a much better match. Anyhow it is definitely something I would put in my wish list of BizTalk features!

It might of course not be straightforward to implement this feature as the code generated by the BizTalk compiler would depend of the type of the variable you assign to the distinguished field (either DateTime or DateTimeOffset). Or maybe that some automatic/easy casting is possible, there is a lengthy article on the conversion between DateTime and DateTimeOffset on MSDN but I did not try to play around with it.

 

(*) Footnote:

While the distinguished field looks like a .Net variable accessed like an object’s member, it actually is not, it is just how it is displayed in the expression shape editor. If you have in the editor something like myVar = MyMsg.MyDistinguishedField, it is just the syntax to access the distinguished field in the expression shape. In real, the code generated from the expression shape will be: myVar = MyMsg.part.GetDistinguishedField(“MyDistinguishedField”). The method will return the correct .Net type depending of the XML type of the distinguished field.

This is why you can’t think of the distinguished field as an object member and can’t call any method such as ToString() directly on it, it will confuse the code generator and the orchestration won’t compile. Hence, we must always assign the distinguished field to a variable.