Azure Stream Analytics not processing data from Event Hubs to storage

The issue I ran into with Azure Stream Analytics is a little bit embarrassing but I think we have all been there before. You know the story… it is late, you have that due date tomorrow and you can’t figure out why it is not working. At least not until you start to explain it to someone else.

As I was doing the initial set up for Stream Analytics for my woodstove project I found that I was able to send data to Event Hubs but it was not being processed by the Stream Analytics job. A couple hours later, after checking to make sure all my code was correct and the Stream Analytics tests passed, I found the issue: I did not start the Stream Analytics Job. All that was need was to press the start button at the bottom of the job:

I know it happens to all of us but hopefully this will short circuit your debugging process and get you on to developing awesome applications with Azure Stream Analytics.

Using Azure Python SDK with Event Hubs

Python is a beautiful and powerful language. Many times in just a few lines of code you can accomplish amazing things. Following the Python mindset, the Azure team has provided an easy to use Python API for the Azure Service Bus.

But until recently it only worked with Queues and Topics. Interacting with the Event Hubs using Python required us to write pure rest calls, which worked fine but is not as clean and simple as python libraries usually are. But rejoice becuase this is no longer the case!

Azure SDK with Event Hubs

After some digging for my wood stove project, I found the latest release of the Python Azure SDK (version 0.11 on May 13, 2015) now includes a simple SDK to send events to the Event Hub:

from azure.servicebus import ServiceBusService

sbs = ServiceBusService(api_key["namespace"], shared_access_key_name=api_key["policy_name"], shared_access_key_value=api_key["policy_secret"])

sbs.send_event('eventhubname', '{ "DeviceId":"deviceId1", "Temperature":"10.0" }')

Conclusion

In just three lines of code we have enabled a python client to send data to an incredibly powerful Event Hubs ingestion service. You can see a full Internet of Things (IoT) example using the Azure Python library at my wood stove github project and you can find the documentation for the Azure Python API here.

A Wood Stove monitoring system built using ASP.NET MVC on Azure with Eventhubs, Stream Analytics and SignalR

The problem

I live in Maine where wood stoves are plentiful. The problem I encountered was when I would get involved in my development work, I would often let the fire burn out and the house would get cold (and more importantly my wife).

In response, last year I built a prototype for a woodstove monitor, with Floyd Hilton. The real-time monitoring system allowed me to sit in my office upstairs and see the temperature of my wood stove. I was able to get reminders, and keep the house warmer for the family.

The plan

This year I want to take the system to the next level. I would like to be able to get notification through different means (email, text, application alerts…), view the data in whatever context I am in (computer, mobile phone, smart watch) and have an easily deployable solution. I will be building this system out over the next few months and would enjoy it if you came along for the ride.

There are a lot of powerful tools to enable IoT devices. The main technologies I will be using are:

• Eventhubs - Azure Event ingestion service that allows for millions of events per second
• Stream Analytics - Azure Real-time stream processing service similar to Apache Storm
• SignalR Real-time websocket library that allows for bi-directional communication between applications.
• ASP.NET MVC - C# Web application framework
• RasberryPi - Small inexpensive computer used for prototyping hardware solutions.

I am going to be trying out new types of deployment and hosting options. I will also be trying out the new design craze called [Microservices] (http://martinfowler.com/articles/microservices.html). I will be documenting my progress, success and failures so please be patient and understanding.

Check out the code repository at https://github.com/jsturtevant/woodstove.

Top Podcasts for May 2015

In no particular order, the following are the top episodes that stood out to me during the month of May. Check out past selections here.

• MS Dev Show - Azure vs AWS with Kevin Wittkopf
• DotNetRocks - Akka.NET V1 with Aaron Stannard
• Herding Code - Rob Eisenberg on Aurelia
• Run As Radio - Tuning SQL Server Stored Procedures with Kim Tripp
• DotNetRocks - Shepherding Novice Developers with Jeff Casimir

DevCamp - Office and Office365 in Cambridge MA

I will be at DevCamp - Office this Monday May 15, 2015. Come learn how to build apps using standard web technologies that extend the familiar Office and Office 365 experience. And since there are over 1 billion Office users there is a great opportunity in side the Office Store to reach your customers.

The agenda from the Event page:

1. Intro to day (40 mins)

2. Standing up the environment (40 mins) Hands on lab (40 mins)

3. Hooking into Apps for SharePoint (40 mins) Hands on lab (40mins)

4. Hook into Office 365 APIs via Standalone web application (40 mins) Hands on lab (40 mins)

5. Hooking into Apps for Office (40 mins) Hands on lab (40 mins)

6. Building native Android App (40 mins) Hands on lab (40mins)

DevCamp must-haves

Note – this is a hands-on workshop. To get the most out of your session, please don’t forget to bring your laptop. In addition:

• Please watch the free, on-demand Introduction to Office 365 Development video.
• Activate a free 30-day trial Azure account
• If you subscribe to MSDN, activate your free Azure MSDN subscriber benefits
• Install Visual Studio 2013 Community with the latest updates, Office 365 Developer Tools

Fees

This free event is brought to you by your local Microsoft office. Delegates are responsible, however, for booking and funding their own travel and accommodation, as required.

See you there!