Integration Experts - Dipesh Avlani

The Future of Integration with MS Technologies

DipeshA — Sun, 08 Jan 2012 23:47:01 +0000

With the announcement of BizTalk Server 2010 R2, it is clear that there is great commitment from MS for supporting on-premise solutions. However, one other smart move from MS was they are now catering to all the three areas below:

Area: On-Premises Integration
Products:

BizTalk Server
Windows Server AppFabric

Area: Hybrid Integration
Products:

BizTalk Server
Windows Azure AppFabric
Windows Server AppFabric

Area: Cloud Integration
Products:

Windows Azure AppFabric

BizTalk 2010 R2 boasts improvements in the following areas:

Development Environment Support:
1) Visual Studio 2011
2) SQL Server 2012
3) Windows 8
4) Easy migration from BizTalk Srv 2010

Updates & Improvments:
1) Updated schemas and accelarators. Eg: HIPPA, HL7, SWIFT
2) Support for faster batch processing.
3) Improved performance with ordered send ports.
4) Support for new data sources. Eg IBM informix.

Cloudification:
1) Easily extend on-premise apps to the cloud.
2) Tighter integration with Windows Azure Service Bus.
2) Licensing geared towards cloud licensing.

BizTalk 2010 R2 Due for Release

DipeshA — Fri, 09 Dec 2011 00:07:12 +0000

BizTalk 2010 R2 is due for release soon…

http://blogs.msdn.com/b/biztalk_server_team_blog/archive/2011/12/08/biztalk-server-2010-r2.aspx

Mocking up XLANGMessage and XLANGPart to unit test BizTalk components

DipeshA — Thu, 17 Nov 2011 00:23:18 +0000

BizTalk developers have often felt the need to unit test any dotnet components called from within an orchestration. However, messages passed to dotnet components are of type XLANGMessage and this class is an abstract class, therefore rendered useless when it comes to initialising these types in your unit tests. The way around this is to mock both the XLANGMessage and XLANGPart classes.

The best way to mock these classes is to run reflector on the xlang base types dll and look at the definition of these classes.

As you can see, both these classes are abstract classes and can be inherited and the behavior added. Below is the code the these mock classes. The key here is the constructor of these classes which will help us use them like any other class.




using System;

using Microsoft.XLANGs.BaseTypes;

using System.Xml.Schema;
namespace IntegrationExperts.BusinessObjects.Tests.MockClasses

{

    public class MockXLANGPart : XLANGPart

    {

        readonly T _obj;
        #region ctor
        public MockXLANGPart(T obj) { _obj = obj; }
        #endregion
        #region props
        public override string Name

        {

            get { return "MockXLANGPart"; }

        }
        public override XmlSchema XmlSchema

        {

            get { throw new NotImplementedException(); }

        }
        public override XmlSchemaCollection XmlSchemaCollection

        {

            get { throw new NotImplementedException(); }

        }
        #endregion
        #region funcs
        public override object GetPartProperty(Type propType)

        {

            throw new NotImplementedException();

        }
        public override Type GetPartType()

        {

            throw new NotImplementedException();

        }
        public override string GetXPathValue(string xpath)

        {

            throw new NotImplementedException();

        }
        public override void LoadFrom(object source)

        {

            throw new NotImplementedException();

        }
        public override void PrefetchXPathValue(string xpath)

        {

            throw new NotImplementedException();

        }
        public override object RetrieveAs(Type t)

        {

            if (t == typeof(T))

            {

                return _obj;

            }

            return null;

        }
        public override void SetPartProperty(Type propType, object value)

        {

            throw new NotImplementedException();

        }
        #endregion
        #region dispose
        public override void Dispose() { }
        #endregion

    }

}
//----------------------------------------------------------------------
using System;

using System.Collections;

using System.Collections.Generic;

using Microsoft.XLANGs.BaseTypes;
namespace IntegrationExperts.BusinessObjects.Tests.MockClasses

{

    public class MockXLANGMessage : XLANGMessage

    {

        readonly List _parts = new List();
        #region ctor
        public MockXLANGMessage(T obj)

        {

            _parts.Add(new MockXLANGPart(obj));

        }
        #endregion
        #region props
        public override int Count { get { return _parts.Count; } }
        public override string Name

        {

            get { return "MockXLANGMessage"; }

        }
        public override XLANGPart this[int partIndex]

        {

            get { return _parts[partIndex]; }

        }
        public override XLANGPart this[string partName] { get { return _parts[0]; } }
        #endregion
        #region funcs

        public override void AddPart(XLANGPart part)

        {

            throw new NotImplementedException();

        }
        public override void AddPart(object part, string partName)

        {

            throw new NotImplementedException();

        }
        public override void AddPart(XLANGPart part, string partName)

        {

            throw new NotImplementedException();

        }
        public override object GetPropertyValue(Type propType)

        {

            throw new NotImplementedException();

        }
        public override void SetPropertyValue(Type propType, object value)

        {

            throw new NotImplementedException();

        }
        public override IEnumerator GetEnumerator()

        {

            return _parts.GetEnumerator();

        }
        #endregion
        #region dispose
        public override void Dispose() { }
        #endregion

    }

}

Add these classes to your unit test project and in the test class, you can write tests as shown below:




  [TestMethod]

          public void DoActionTest()

          {
                    var msg = new MySchemaClass(); //schema class can be generated via tools like XSD to code.

                    var xlang = new MockXLANGMessage(msg); // Create a new mock XLANGMessage to wrap the message

                     DoAction(xlang); // Call the method to be tested by passing in the mock XLANGMessage

                    //Assert.IsTrue(....);

  }

  internal static void DoAction(XLANGMessage message)

  {             
               var myObj = new MySchemaClass();//schema class can be generated via tools like XSD to code.

                 myObj = (MySchemaClass)message[0].RetrieveAs(MySchemaClass.GetType()); //retrieve parts of the message.

//var val1 = myObj.prop1; //var val2 = myObj.prop2; //perform additional tasks as per your testing requirements. }

Troubleshooting SQL Azure connectivity

DipeshA — Sat, 05 Nov 2011 02:32:08 +0000

Below is very good compilation of tips to troubleshoot sql azure connectivity:
http://www.microsoftblog.co.in/?cat=414

Accessing archived (& purged) tracking data in BizTalk 2009/2010

DipeshA — Wed, 19 Oct 2011 07:16:20 +0000

It is known that the BizTalkDTADb database can grow over a period of time if not purged regularly. For this there is a sql job in BizTalk which can be configured to archive and purge tracked data at regular intervals. The job is called DTA Purge and Archive. Once this job is configured and enabled, it archives and purges tracking data as per the schedule in the job.

Everytime the job runs, it archives this database which can be restored with a different name in SSMS.

Database restored in SQL Server Management Studio with a diff name:

Once the the archived copy is restored, it can be accessed from BTS Admin Console.

You now get to select the tracking database as shown below:

On connecting to the archived database, the database now appears in BTS Admin Console which can be queried to display tracked events and messages.

Restoring BizTalk Databases (Disaster Recovery)

DipeshA — Wed, 19 Oct 2011 00:22:31 +0000

I found an interesting article on msdn which explains the backup and restore strategies for BizTalk databases. Setting this up is fairly straight forward but requires a fair knownledge of BizTalk Server and SQL Server 2008.

BizTalk Server makes available a SQL Server job (Backup BizTalk Server) that you can use to back up all of the databases in your BizTalk Server source system, except for some databases used by Business Activity Monitoring (BAM). The source system is the server or group of servers that contain live data. Because BAM databases have different backup and restore requirements, these databases are backed up and restored using other methods.

The following BizTalk databases are backed up by the sql job:

BAMPrimaryImport
BAMAlertsApplication
BAMAlertsNSMain
BizTalkDTADb
BizTalkMgmtDb
BizTalkMsgBoxDb
BizTalkRuleEngineDb
SSODB

Backing up the BizTalk Server databases and restoring them involves the following steps:

1. Configuring the Backup BizTalk Server job

This is the first step required to be performed before you can back up the BizTalk Server databases. You must first configure the Backup BizTalk Server job on the source system, which directs backups to be automatically written to a folder where they can then be used to restore the databases on the destination system. The destination system is the server or group of servers that will be used to restore the database backups produced by the source system. For more information about this step, see link.

2. Configuring the destination system for log shipping

You must also configure the destination system for log shipping, which provides standby server capabilities and reduces downtime in the event of a system failure. For more information about this step, see link.

The result of this step is three new SQL Server agent jobs as described under:

BTS Log Shipping Get Backup History
The BizTalk Server Log Shipping Get Backup History job moves backup history records from the source to the destination. It is scheduled by default to run every minute. This job runs as frequently as possible in order to move history records from the source to the destination. In the event of a system failure to the source system, the server that you identified as the destination system will continue to process the history records that have already been imported.

BTS Server Log Shipping Restore Databases
The BizTalk Server Log Shipping Restore Databases job restores backup files for the given databases for the source to the destination server. It is scheduled by default to run every minute. This job runs continuously without completing as long as there are backup files to restore. As an extra precaution, you can run this job an additional time to ensure that it is complete.

BTS Log Shipping Restore To Mark
The BizTalk Server Log Shipping Restore To Mark job restores all of the databases to a mark in the last log backup. This ensures that all of the databases are in a transactionally consistent state. In addition, this job re-creates all of the SQL Server Agent jobs on the destination system that had been on the source system.

3. Restoring the databases

When a hardware failure occurs, you can restore your databases by using the backups and logs sent to your destination system. For more information about this step, see link.

After completing this step, your BizTalk Server should be restored and running. If you are using the Rules Engine, after restoring the databases, you must restart the Rule Engine Update Service on every server in the BizTalk Server group.

BizTalk Service Packs and Cummulative Updates List

DipeshA — Fri, 02 Sep 2011 00:12:13 +0000

You can get a list of all SPs and CUs for BizTalk from kb article below:
KB Article

BizTalk Maintenance Jobs in SQL Server

DipeshA — Mon, 22 Aug 2011 23:32:04 +0000

I recently came across a situation where I had to identify bottlenecks on a BizTalk server which was performing like a snail. I have written a blog post about this here.

However, I also realised the importance of various BizTalk jobs in SQL Server during this process. I would like to share with you what I learnt in relation to this and how one should keep an eye on these sql jobs as they play a pivotal role in highly (& poorly) performing BizTalk setups.

There are about 12 sql jobs in BT 2009 and 13 jobs in BT 2010 which play their own little role when it comes to ensuring good BizTalk performance.

The DTA purge job is disabled by default and can be enabled if the tracking database gets too big. The messagebox cleanup job is also disabled by default as it is triggered by the ManageRefCountLog job as needed.

It is very important to note that if these jobs dont run on a timely manner especially when the tracking and spool tables get bigger than the set throttling limit, BizTalk starts throttling. The moment BizTalk starts throttling, performance starts to degrade.

Lets look at what each job is meant to do (referenced from MS Support):

Job name	Description	Enabled by default
Backup BizTalk Server	Consists of three steps. Step 1 performs full database backups of the BizTalk Server databases. Step 2 backs up the BizTalk Server database logs. Step 3 specifies for how long the backup history is kept.	No
CleanupBTFExpiredEntriesJob_BizTalkMgmtDb	Deletes expired BizTalk Framework entries from the BizTalk Management (BizTalkMgmtDb) database.	Yes
DTA Purge and Archive	Automates the archiving of tracked messages and the purging of the BizTalk Tracking database to maintain a healthy system and to keep the tracking data archived for future use.Note On BizTalk Server 2004, this job is created after you install BizTalk Server 2004 Service Pack 2.	No
MessageBox_DeadProcesses_Cleanup_BizTalkMsgBoxDb	Detects when a BizTalk Server host instance (BTSNTSvc.exe) has stopped responding. The job then releases the work from the host instance so a different host instance can finish the tasks.	Yes
MessageBox_Message_Cleanup_BizTalkMsgBoxDb	Removes all messages that are not referenced by any subscribers in the BizTalkMsgBoxDb database tables.Important This job is also started by the MessageBox_Message_ManageRefCountLog_BizTalkMsgBoxDbjob. Therefore, we recommend that you disable this job.Note On BizTalk Server 2004, this job is enabled by default. Therefore, we recommend that you disable this job.	No
MessageBox_Message_ManageRefCountLog_BizTalkMsgBoxDb	Manages the reference count logs for messages and determines when a message is no longer referenced by a subscriber.This job runs in an infinite loop and deletes the entries from the two individual message reference count logs. This job also calls the MessageBox_Message_Cleanup_BizTalkMsgBoxDbjob.Important At first, the MessageBox_Message_ManageRefCountLog_BizTalkMsgBoxDb job status icon displays a status of Success. However, there will be no corresponding success entry in the job history. If one of the jobs in the MessageBox_Message_ManageRefCountLog_BizTalkMsgBoxDb job fails, a failure entry appears in the job history and the status icon displays a status of Failure. The job will always display a status of Failure after the first failure. To verify that the other BizTalk Server SQL Server Agent jobs run correctly, check the status of the other BizTalk Server SQL Server Agent jobs. Note On BizTalk Server 2004, this job is created after you install BizTalk Server 2004 Service Pack 2.	Yes
MessageBox_Parts_Cleanup_BizTalkMsgBoxDb	Removes all message parts that are no longer referenced by a message in the BizTalkMsgBoxDb database tables. All messages are composed of one or more message parts that contain the message data.	Yes
MessageBox_UpdateStats_BizTalkMsgBoxDb	Updates the statistics for the BizTalkMsgBoxDb database.Note This job does not exist on BizTalk Server 2004.	Yes
Monitor BizTalk Server	Scans for any known issues with the BizTalkMgmtDb, BizTalkMsgBoxDb, and BizTalkDTADb databases. This includes orphaned instances.Note This job is created on BizTalk Server 2010.	Yes
Operations_OperateOnInstances_OnMaster_BizTalkMsgBoxDb	Used for multiple BizTalkMsgBoxDb database deployment. It asynchronously performs operational actions. For example, it asynchronously performs bulk terminates on the master BizTalkMsgBoxDb database after those changes are applied to the subordinate BizTalkMsgBoxDb database. Note This job does not exist on BizTalk Server 2004.	Yes
PurgeSubscriptionsJob_BizTalkMsgBoxDb	Purges unused subscription predicates from the BizTalkMsgBoxDb database.	Yes
Rules_Database_Cleanup_BizTalkRuleEngineDb	Purges old audit data from the Rule Engine (BizTalkRuleEngineDb) database every 90 days. This job also purges old history data (deploy/undeploy notifications) from the Rule Engine (BizTalkRuleEngineDb) database every 3 days.Note This job is created on BizTalk Server 2009.	Yes
TrackedMessages_Copy_BizTalkMsgBoxDb	Copies the message bodies of tracked messages from the BizTalkMsgBoxDb database to the Tracking (BizTalkDTADb) database.	Yes
TrackingSpool_Cleanup_BizTalkMsgBoxDb	Purges inactive tracking spool tables to free database space.Note This job exists only on BizTalk Server 2004.

Improving BizTalk Server throughput and identifying bottlenecks

DipeshA — Fri, 29 Jul 2011 03:10:48 +0000

Recently one of our BizTalk 2009 systems started to suffer in terms of perfomance due to the sudden influx of extra messages that were fed into the system. My team was given the task to identify the bottleneck and provide reccomendations to improve performance.

With curiousity, I jumped onto the BizTalk Server and opened up the performance monitor to observe a few counters. I was interested in the following performance counter:

Message publishing delay (ms) under the category BizTalk:MessageAgent

Our observation was this delay kept increasing over a period of time which led us to believe that this “might” be due to BizTalk throttling. Throttling is a mechanism to control the flow of the workload associated with a host instance.

The next performance counter to confirm our suspicions was the following:

Message Publishing Throttling State under category BizTalk:Message Agent

This counters tells us if throttling is occuring and the likely causes based on the value in the graph.

Average was 6.

Now going back to the msdn document (found here), this is what the value 6 represented:

A flag indicating whether the system is throttling message publishing (affecting XLANG message processing and inbound transports).
0: Not throttling
2: Throttling due to imbalanced message publishing rate (input rate exceeds output rate)
4: Throttling due to process memory pressure
5: Throttling due to system memory pressure
6: Throttling due to database growth
8: Throttling due to high session count
9: Throttling due to high thread count
11: Throttling due to user override on publishing

Possible reasons for this condition include:
A. The SQL Agent jobs used by BizTalk Server to maintain the BizTalk Server databases not running or are running slowly.
B. Down-stream components are not processing messages from the in-memory queue in a timely manner.
C. Number of suspended messages is high.
D. Maximum sustainable load for the system has been reached.

After the process of elimination, we were able to eliminate A,C and D which left B for further investigation. We were able to eliminated A by checking the record counts in the Spool and TrackingData tables in the messagebox database. The number of records in these tables was less than 500,000 or mostly 0 which proved that BizTalk wasnt throttling due to tracked messages as defined under:

“The Message count in database setting also indirectly defines the threshold for a throttling condition based on the number of messages in the spool table or tracking table. If the number of messages in the spool table or tracking table exceeds 10 times this value then a throttling condition will be triggered. By default the Message count in database value is set to 50,000, which will cause a throttling condition if the spool table or the tracking table exceeds 500,000 messages.”

We could eliminate C and D because the suspended messages queue was almost empty and the CPU & memory (physical and process) consumption on the server was almost negligible.

Apparently, it was the downstream system (WCF service in our case) which was queueing requests and slowing things down due to too many requests to the service.

We then wanted to control the flow of messages going to the WCF port and we could do that by checking “Ordered Delivery” in our custom WCF solicit-response sendport. Once this was done, we saw a significant improvement in the throughput and BizTalk wasnt throttling anymore.

In cases where the reason for throttling is the database, I would highly recommend reading the throttling whitepapers on msdn before making any changes to the defaults or disabling throttling altogether.

Cheers!

Dipesh