Introduction to Real-Time Analytics in Microsoft Fabric

by May 24, 2023

Introduction to Real-Time Analytics in Microsoft Fabric

Real-Time Analytics is one of the data and analytical workloads/experiences available in Microsoft Fabric, the new platform currently in Public Preview at Microsoft. With Real-Time Analytics, companies and developers can gain valuable insights and analysis from real-time data streams.

A unified analytics solution for the era of AI

Microsoft Fabric brings a unified SaaS-based solution that stores all organizational data where analytics workloads operate. Microsoft Fabric brings together existing offerings such as Data Factory, Azure Synapse Analytics, and Power BI into one unified product for all data and analytics workloads.

Key pilars:

  • Complete analytics platform
  • Lake centric and open
  • Empower every Office user
  • AI Powered

When Microsoft Fabric is not yet activated in your tenant, you can activate it in the Admin Portal. Please note that Microsoft Fabric Capacity(Trial) or Power BI Premium Capacity is required to get started with Microsoft Fabric.

Microsoft Build

Now that we have seen the initial sessions during Microsoft Build, it’s time to delve deeper into a topic. But what an announcement! We have all worked hard on this in the last couple of months. We have done a lot of testing and provided a lot of feedback. And personally, I can say that all feedback has been listened to carefully.

Microsoft-Fabric-Workloads

In this blog, I will delve deeper into Real-Time Analytics, one of the available experience in Microsoft Fabric. An experience is a look and feel of various Fabric Artifacts for a specific role such as a Data Engineer, Data Analyst or Data Scientist. For all available experiences see picture above.

Real-Time Analytics

Real-Time Analytics is critical in today’s fast-paced business environment. It enables organizations to react immediately to events and trends as they happen, rather than reacting to historical data afterwards. The Real-Time Analytics workload allows users to monitor, analyze, and visualize data in real-time to make fast and data-driven decisions.

Here are some key features and functionalities of Real-Time Analytics in Microsoft Fabric:

  1. Real-time data processing: The workload supports processing large amounts of data in real-time, giving users instant access to up-to-date information.
  2. Advanced analytics: Built-in analytics capabilities enable users to apply complex calculations and statistical models to real-time data for deep insights.
  3. Flexible visualizations: The app offers a wide range of visualization options, such as graphs, charts, and dashboards, to present data in a clear and understandable manner.
  4. With Data Activator(coming soon): Users can set up custom notifications and alerts based on predefined criteria, keeping them informed of important events or anomalies in real-time.

As you can see, you can use Real-Time Analytics for a range of solutions, such as IoT analytics, Telemetry data, human and system logs and in many scenarios including manufacturing operations, cybersecurity, oil and gas, automotive and many more.

Benefits

One of the great benefits of using Real -ime Analytics in Microsoft Fabric is that you have a seamless integration with other artifacts in Fabric such as Lakehouse, Data Warehouse and Machine Learning Models for Predictive Analytics. One of the other benefits in Microsoft Fabric is that you don’t have to start from scratch, is very easy to connect to existing Event Hubs to load your streaming events into Fabric. Which I will explain in my next blog.

Real-Time Analytics Artifacts

Currently the Real-Time Analytics workload supports 3 different artifacts:

KQL Database:          A Kusto database exactly the same as you were used to in Azure Data Explorer

KQL Queryset:          Collection of queries which you can run on top of your KQL Database

Eventstream:            Capture, transform and route real-time event stream to various destinations with a no-code experience. Similar to Azure Stream Analytics

OneLake: The foundation for Microsoft Fabric

OneLake eliminates today’s pervasive and chaotic data silos by providing a data lake as a service without you needing to build it yourself. OneLake is the OneDrive for data and like OneDrive, OneLake is provisioned automatically with every Fabric tenant with no infrastructure to manage. All Fabric Artifacts, such as mentioned above for Real-Time Analytics are deployed/ provisioned automatically into the Onelake upon on creation. How easy is that?

Happy Path - Real-time Analytics

Having a closer look at the picture above, you see an end to end workflow for a Real-Time Analytics scenario.

  • Ingest the data from Event Hub, custom apps, structured and Unstructured data source with pipelines and Dataflows.
  • Store the data in a KQL Database or Lakehouse.
  • Expose the data in Power BI and/or make available in Notebooks and KQL Queriesets.
  • Train and test the data with Machine Learning Models and Experiments.

With this end to end workflow you can directly see which artifacts you need to use to build your Real-Time Analytics Solution.

Public Preview

It’s important to note that as Microsoft Fabric is currently in Public Preview, additional functionality is still being developed, and feedback is being incorporated. This presents a great opportunity for users to get involved early, provide feedback, and contribute to the further development of Microsoft Fabric.

When you decide to start using Microsoft Fabric and encounter any issues with the Real-Time Analytics workload, please don’t hesitate to reach out to me. I’m here to assist and appreciate your feedback to further enhance the platform.

Click below to read more about Microsoft Fabric and Real-Time Analytics.

Microsoft Fabric Real -Time Analytics documentation

Exploring the Fabric technical documentation

OneLake in Fabric blog

Exploring the Fabric technical documentation

In my next blog I will get a bit deeper how easily you can connect existing Event Hubs to Microsoft Fabric. So stay tuned(published on may 26th 2023)

Note:

Please be aware that Microsoft Fabric is currently not authorized for production use as it is still in the Public Preview phase. It’s important to consider this when planning deployments or making critical business decisions.

In the video below, Tzvia Gitlin Troyna, a Principal Manager with Synapse Real-Time Analytics experience in Microsoft Fabric, shares a first look at what’s included in the first release of Real-Time Analytics in Microsoft Fabric.

Feel free to leave a comment

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

five × three =

This site uses Akismet to reduce spam. Learn how your comment data is processed.

How to create a Azure Synapse Analytics Workspace

Creating your Azure Synapse Analytics Workspace In the article below I would like to take you through,  how you can configure an Azure Synapse Workspace and not the already existing Azure Synapse Analytics SQL Pool(formerly Azure SQL DW): In de Azure Portal search for...

Provision users and groups from AAD to Azure Databricks (part 4)

In this blog, you will learn how to assign Users and Groups to an Azure Databricks Workspace and define the correct entitlements.

Azure SQL Data Warehouse: Reserved Capacity versus Pay as You go

How do I use my Reserved Capacity correctly? Update 11-11-2020: This also applies to Azure Synapse SQL Pools. In my previous article you were introduced, how to create a Reserved Capacity for an Azure SQL Datawarehouse (SQLDW). Now it's time to take a look at how this...

Create Virtual Machines with Azure DevTest Lab

A while ago I had to give a training. Normally I would roll out a number of virtual machines in Azure. Until someone brought my attention to an Azure Service, Azure DevTest Labs. With this Azure service you can easily create a basic image and use this image to roll...

SSMS 18.1: Schedule your SSIS Packages in Azure Data Factory

Schedule your SSIS Packages with SSMS in Azure Data Factory(ADF) This week SQL Server Management Studio version 18.1 was released, which can be downloaded from here. In version 18.1 the Database diagrams are back and from now on we can also schedule SSIS Packages in...

Azure Data Factory: How to assign a Customer Managed Key

Customer key With this new functionality you can add extra security to your Azure Data Factory environment. Where the data was first encrypted with a randomly generated key from Microsoft, you can now use the customer-managed key feature. With this Bring Your Own Key...

Blog Serie: Provision users and groups from AAD to Azure Databricks

This blog post series contains topics on how to Provision users and groups from Azure Active Directory to Azure Databricks using the Enterprise Application(SCIM). This is a summary of the all the blogs I posted the last couple of days. I am very happy with all the feedback and tips I have received about this blog series.

Azure DevOps and Azure Feature Pack for Integration Services

Azure Feature Pack for Integration ServicesAzure Blob Storage A great addition for SSIS is using extra connectors like  Azure Blob Storage or Azure Data Lake Store which are added by the Azure Feature Pack. This Pack needs to be installed on your local machine. Are...

Using Azure Automation to generate data in your WideWorldImporters database

CASE: For my test environment I want to load every day new increments into the WideWorldImporters Azure SQL Database with Azure Automation. The following Stored Procedure is available to achieve this. EXECUTE DataLoadSimulation.PopulateDataToCurrentDate...

Connect Azure Synapse Analytics with Azure Purview

How do you integrate Azure Purview in Azure Synapse Analytics? This article explains how to integrate Azure Purview into your Azure Synapse workspace for data discovery and exploration. Follow the steps below to connect your Azure Purview account in your Azure Synapse...