Serverless Big Data

It’s been a long time since I’ve blogged here, but figured it’s overdue. I’ve been busy in the Azure world and am really excited to see how messaging has really started to shape the modern cloud landscape. Creating and shipping Azure Event Grid was perhaps the highlight of my career thus far, certainly at Microsoft. I’ll write a long over due blog about that shortly.

Seeing Azure Event Hubs grow to 2 trillion requests per day in the time I’ve been with it has also been a great experience. In this time not only has messaging become core to the cloud in general and Serverless in particular, but new patterns are starting to emerge that are really exciting. One of the most exciting this month has been seeing the traction our quietly released new Apache Kafka endpoint for Event Hubs is gaining and the new directions it is driving. The Kafka endpoint feature is available in some regions of Azure today and you can give it a try following this quick start. This allows you to use Kafka producers and consumers to read and write from an Event Hub.

By combining this with Azure Functions Event Hubs Binding you can create a Serverless Kafka processor in just a few minutes. The blog Processing 100,000 Events Per Second on Azure Functions shows you how easily you can scale this Serverless processing to pretty high scale very easily. This means you can really start to create truly Serverless Big Data solutions on Azure today. This is an exciting time and we will see more development in this space as users of these platforms drive innovation on the platforms which we and other cloud providers are starting to provide.

Give this stuff a try and tell me what you think!


About danrosanova
I am a Principal Program Manager for Messaging at Microsoft and product owner for Azure Messaging: Service Bus, Relay, and Event Hubs. I have a long history in distributed computing on a variety of platforms and have focused on large scale messaging and middleware implementations from inception to implementation. I was a five time Microsoft MVP before joining Microsoft and author of the book Microsoft BizTalk Server 2010 Patterns.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: