Introduction to Real-Time Analytics in Microsoft Fabric

Real-Time Analytics is one of the data and analytical workloads/experiences available in Microsoft Fabric, the new platform currently in Public Preview at Microsoft. With Real-Time Analytics, companies and developers can gain valuable insights and analysis from real-time data streams.

A unified analytics solution for the era of AI

Microsoft Fabric brings a unified SaaS-based solution that stores all organizational data where analytics workloads operate. Microsoft Fabric brings together existing offerings such as Data Factory, Azure Synapse Analytics, and Power BI into one unified product for all data and analytics workloads.

Key pilars:

• Complete analytics platform
• Lake centric and open
• Empower every Office user
• AI Powered

When Microsoft Fabric is not yet activated in your tenant, you can activate it in the Admin Portal. Please note that Microsoft Fabric Capacity(Trial) or Power BI Premium Capacity is required to get started with Microsoft Fabric.

Microsoft Build

Now that we have seen the initial sessions during Microsoft Build, it’s time to delve deeper into a topic. But what an announcement! We have all worked hard on this in the last couple of months. We have done a lot of testing and provided a lot of feedback. And personally, I can say that all feedback has been listened to carefully.

In this blog, I will delve deeper into Real-Time Analytics, one of the available experience in Microsoft Fabric. An experience is a look and feel of various Fabric Artifacts for a specific role such as a Data Engineer, Data Analyst or Data Scientist. For all available experiences see picture above.

Real-Time Analytics

Real-Time Analytics is critical in today’s fast-paced business environment. It enables organizations to react immediately to events and trends as they happen, rather than reacting to historical data afterwards. The Real-Time Analytics workload allows users to monitor, analyze, and visualize data in real-time to make fast and data-driven decisions.

Here are some key features and functionalities of Real-Time Analytics in Microsoft Fabric:

1. Real-time data processing: The workload supports processing large amounts of data in real-time, giving users instant access to up-to-date information.
2. Advanced analytics: Built-in analytics capabilities enable users to apply complex calculations and statistical models to real-time data for deep insights.
3. Flexible visualizations: The app offers a wide range of visualization options, such as graphs, charts, and dashboards, to present data in a clear and understandable manner.
4. With Data Activator(coming soon): Users can set up custom notifications and alerts based on predefined criteria, keeping them informed of important events or anomalies in real-time.

As you can see, you can use Real-Time Analytics for a range of solutions, such as IoT analytics, Telemetry data, human and system logs and in many scenarios including manufacturing operations, cybersecurity, oil and gas, automotive and many more.

Benefits

One of the great benefits of using Real -ime Analytics in Microsoft Fabric is that you have a seamless integration with other artifacts in Fabric such as Lakehouse, Data Warehouse and Machine Learning Models for Predictive Analytics. One of the other benefits in Microsoft Fabric is that you don’t have to start from scratch, is very easy to connect to existing Event Hubs to load your streaming events into Fabric. Which I will explain in my next blog.

Real-Time Analytics Artifacts

Currently the Real-Time Analytics workload supports 3 different artifacts:

KQL Database:          A Kusto database exactly the same as you were used to in Azure Data Explorer

KQL Queryset:          Collection of queries which you can run on top of your KQL Database

Eventstream:            Capture, transform and route real-time event stream to various destinations with a no-code experience. Similar to Azure Stream Analytics

OneLake: The foundation for Microsoft Fabric

OneLake eliminates today’s pervasive and chaotic data silos by providing a data lake as a service without you needing to build it yourself. OneLake is the OneDrive for data and like OneDrive, OneLake is provisioned automatically with every Fabric tenant with no infrastructure to manage. All Fabric Artifacts, such as mentioned above for Real-Time Analytics are deployed/ provisioned automatically into the Onelake upon on creation. How easy is that?

Having a closer look at the picture above, you see an end to end workflow for a Real-Time Analytics scenario.

• Ingest the data from Event Hub, custom apps, structured and Unstructured data source with pipelines and Dataflows.
• Store the data in a KQL Database or Lakehouse.
• Expose the data in Power BI and/or make available in Notebooks and KQL Queriesets.
• Train and test the data with Machine Learning Models and Experiments.

With this end to end workflow you can directly see which artifacts you need to use to build your Real-Time Analytics Solution.

Public Preview

It’s important to note that as Microsoft Fabric is currently in Public Preview, additional functionality is still being developed, and feedback is being incorporated. This presents a great opportunity for users to get involved early, provide feedback, and contribute to the further development of Microsoft Fabric.

When you decide to start using Microsoft Fabric and encounter any issues with the Real-Time Analytics workload, please don’t hesitate to reach out to me. I’m here to assist and appreciate your feedback to further enhance the platform.

Click below to read more about Microsoft Fabric and Real-Time Analytics.

Microsoft Fabric Real -Time Analytics documentation

Exploring the Fabric technical documentation

OneLake in Fabric blog

More information about Microsoft Fabric can be found at:

Microsoft Fabric Content Hub

In my next blog I will get a bit deeper how easily you can connect existing Event Hubs to Microsoft Fabric. So stay tuned(published on may 26th 2023)

Note:

Please be aware that Microsoft Fabric is currently not authorized for production use as it is still in the Public Preview phase. It’s important to consider this when planning deployments or making critical business decisions.

In the video below, Tzvia Gitlin Troyna, a Principal Manager with Synapse Real-Time Analytics experience in Microsoft Fabric, shares a first look at what’s included in the first release of Real-Time Analytics in Microsoft Fabric.