Apache Storm on Windows

In a release in February Apache Storm community added Windows platform support for Storm 0.9.1

I for one have been very excited to see this.  The Hortonworks distribution of Hadoop (HDP) is the only one that runs on both Windows and Linux and this gives a lot more choice to traditional enterprise clients.  I’ve been working with HDP for about a year and a half now and really like the experience – both on Linux and Windows. 

Storm is a very exciting development in real time data processing using a Hadoop cluster.  This is useful for running models that you’ve created by more traditional batch processing and map reduce within Hadoop.  Storm uses a simple spout and bolt topology for processing tuples of information at scale and in real time.  More information can be found at the storm site: http://storm.incubator.apache.org/

I am now wondering if this technology, now running on Windows, will make it into the Windows Azure HDInsight service.  I certainly don’t have any inside information on this, but I’d be interested to see it. 

More messaging like Rabbits

After playing around some with RabbitMQ I decided I’d really let BizTalk message like rabbits.  I modified the file send port so that it sent back to the same queue instead of to the file location.  This creates a feedback loop so a message will keep propagating.  Then to make it more “rabbit like” I added a send port group that had the filter: BTS.ReceivePortName == RMQI_Q_Receive Or BTS.ReceivePortName == RMQI_Receive_Start

This way starting with the first message received from the file drop; every message receive results in two sends to the queue (which just feeds back and results in another doubling of messages). 

Two things became readily apparent.  BizTalk scales well and works fast and RabbitMQ will certainly keep pace.  In moments I had thousands of messages flying through my system. 

Checking the Queues section of the RabbitMQ management portal I was easily able to see the volume flowing through.  I almost think this could be a good benchmarking tool for BizTalk, but the BizTalk Benchmark Wizard is already so good I don’t really think I need a RabbitMQ based one. 

Messaging like Rabbits

I have been all kinds of excited about AMQP lately and yesterday’s news from David Ingham about AMQP on Service Bus got me pretty excited.  Obviously I’m always excited about BizTalk and my work lately has pushed me far heavier into the WCF-Adapters than I had previously gone (including using WCF 4.5 features from within BizTalk 2010). 

Eager to try some AMQP out and not willing to wait I decided I’d give RabbitMQ a try.  I knew they had a WCF binding for their implementation and decided it would probably be a good place to start.  This way I could placate my “open source” critics and also get some good AMQP fun going on Windows… quickly (I’m told AMQP works on Linux too). 

The setup was pretty easy for the server aspect, but the client part was not so much.  Leonid Ganeline’s awesome TechNet article on this helped me very much: http://social.technet.microsoft.com/wiki/contents/articles/7401.biztalk-and-rabbitmq.aspx.

I knew I had to GAC the assemblies and register the binding in machine.config, but I couldn’t find the assemblies anywhere (neither could Leonid), so I ended up building them locally (though you can strong name assemblies after they’re compiled I instead just added a key and strong named them).  I also changed the target to .NET 4.0.  You need to compile RabbitMQ.Client and RabbitMQ.ServiceModel.

I followed a very similar pattern with a file drop that Leonid did and it worked great.  I’m playing around with the graphical tools (management console) for RabbitMQ and I still don’t really know what many of these defaults are, but I do plan to work on it a bit more. 

Windows Azure 1.6/1.5 SDK Service Bus Authentication issue: 40102: Missing Authorization Token

I’ve been playing  a lot with the Azure 1.6 SDK and have been enjoying it quite a bit.  I’m in the process of modifying some of the examples in BizTalk 2010 Patterns to use Azure and ran into an interesting issue that seems to have only affected Service Bus.  I was trying to use queues as a destination and kept receiving this error:

The adapter failed to transmit message going to send port “OP_File_Send_CRM_SalesOrder” with URL “sb://XXXXXX.servicebus.windows.net/salesorders”. It will be retransmitted after the retry interval specified for this Send Port. Details:”System.UnauthorizedAccessException: 40102: Missing Authorization Token.TrackingId:8ac56df4-4094-495b-b302-ddf3c842efe7_6,TimeStamp:3/7/2012 6:38:19 PM —> System.ServiceModel.FaultException: 40102: Missing Authorization Token.TrackingId:8ac56df4-4094-495b-b302-ddf3c842efe7_6,TimeStamp:3/7/2012 6:38:19 PM

I assumed that I must have been using ACS incorrectly and decided to try the owner token.  Still no luck – though I did learn a lot about claims in ACS.  I also tried many permutations switching between transportClientEndpointBehavior settings for ClientCredentials and TokenProvider.

After far too long I remembered something from last week’s MVP Summit.  There had been discussion about versions of the SDK in use and the process of changing to another.  Another MVP said something about having to cut over.  I’m sure this is common knowledge to many (and probably in the SDK readme / docs), but I thought, well maybe I need to uninstall the old SDK.  Before I did this I thought I’d run the RelayConfigurationInstall and uninstall all the previous 1.5 bits that had been on my system (I vaguely remember running this ages ago when I installed 1.5).  I opened a command line to C:\Program Files\Windows Azure AppFabric SDK\V1.5\Assemblies and ran:

RelayConfigurationInstaller.exe /u

I then uninstalled the 1.5 SDK.  Finally I ran the RelayConfigurationInstall from the 1.6 SDK (located at C:\Program Files\Windows Azure SDK\v1.6\ServiceBus\bin).

RelayConfigurationInstaller.exe /i

Amazingly everything worked after that with no other changes.  I was set!