My virtual session at Data Toboggan

My virtual session at Data Toboggan

Month: January 2022

Month: January 2022

by Erwin | Jan 29, 2022 | 0 comments

Data Toboggan

This Saturday I’ve joined the Data Toboggan to talk about Azure Synapse Analytics.

 

Azure Synapse Analytics

Today I’ve been talking on how to deal with all the different roles in Azure Synapse Analytics during Data Toboggan. An event 100% focussed on Azure Synapse Analytics.

Synapse-Access-Control

You can find my slides below on Slideshare:

Some useful links:
 
Azure-Synapse-Role-Actions
 
 
 

 

 

In case you have any questions left please feel free to ask them via the comment or Socials

Feel free to leave a comment

Microsoft Build 2022 Book of News

Month: January 2022 Microsoft Build 2022 MSBuild 2022 has just started. A lot of new Announcement will be released or demonstrated the upcoming days. Like every year or event most of these announcement are added to the Book of News. The book of news can be downloaded…

Goodbye 2020 Hello 2021

Month: January 2022 Goodbye 2020  Started to work for InSpark Last year was certainly an eventful year. Started with a new job at InSpark and after 10 weeks we all know what happened, the first intelligent lockdown. The Netherlands was partially locked, but our office…

New Microsoft Azure Certifications

Month: January 2022 Microsoft Certification by Solution Area Handy overview of the new Microsoft Azure Certifications. More details can be found here Feel free to leave a comment

LEAP 2018

Month: January 2022 Lead Enterprise Architect Program From the 4th of December until the 6th of December I’ve joined the LEAP (Lead Enterprise Architect Program) 2018 Data/AI from Microsoft NL. Three  days on the Microsoft Campus in Redmond joining business and…

Rerun Pipeline activities in Azure Data Factory

Month: January 2022 Rerun Pipeline activities in ADF! As of today you can rerun or partially, yes you’re reading it correct partially, rerun you Azure Data Factory pipeline.Where you previously had to run the entire Pipeline again, you can now run a part of the…

My First Blog Post SQLSatHolland 790

Month: January 2022 Yeah, m y first blog is LIVE After a good talk with Reza Rad from RADACAD during SQLSatHolland, I decided to start my first blog! Knowledge sharing is very important, it gives me a lot of energy. But it also gives others people in the community…

My Sessions at DataSaturdayNL 2019

Month: January 2022 DataSaturday NL 2019 Date: 5 th October Location: Pathé Leidsche Rijn Data Saturday Holland is the new name for SQL Saturday Holland. A new start a new location and wow what a top location, Pathé Leidsche Rijn. A cinema with 7 screens and ranging…

Goodbye 2021, Hello 2022

Month: January 2022 Goodbye 2021 Recap First of all, I would like to wish everyone a very beautiful and healthy 2022. We are now 3 days on the road into the new year and it is always good to look back at what happened last year. It’s certainly been an eventful year,…

SQLBits session: Microsoft Purview Data Policy App

Month: January 2022 SQLBits 2023 Thanks everyone for visiting my session during SQLBits. It’s great to see such a full room and that so many people have started using Microsoft Purview.   SLIDES The slides can be downloaded via the link below, so that you can…

Azure Synapse Analytics Code Repository has arrived

Month: January 2022 Azure Synapse Analytics Code repository ‎I just opened my Azure Synapse Analytics Workspace and got a great surprise, the option Git Configuration is available as of today‎.     After a long wait, today the Git Configuration option became available…

How to use concurrency in Azure Synapse pipelines?

How to use concurrency in Azure Synapse pipelines?

Month: January 2022

Month: January 2022

by Erwin | Jan 12, 2022 | 6 comments

How to prevent concurrent pipeline execution?

Concurrency

This week I had a discussion with a colleague about how we can now make sure that a Pipeline does not start when it’s already started.

He then indicated, have you ever thought of the concurrency option?  I’ve seen this option before but never paid attention to it.

How does the concurrency work?

If you read the Microsoft documentation it says the following:
The maximum number of concurrent runs the pipeline can have. By default, there is no maximum. If the concurrency limit is reached, additional pipeline runs are queued until earlier ones complete.

The concurrency option is working in Azure Synapse Analytics and in Azure Data Factory.

I started to test this functionality and there are certainly some nice use cases for that:

  • If the Pipeline was started via a Schedule and someone else triggers this Pipeline Manually, the Pipeline is placed in a queue.
  • Sometimes it happens that there is a delay in the processing of data or that more data is delivered. If you process this data every 30 minutes and the 1st run is not yet ready and the 2nd starts again, this could result in incorrect data. Also in this case the to be executed run is placed in a queue and only starts when the previous one is ready.

It is a fairly simple process but can be quite useful especially in the case of short loading windows.

Azure-Synapse-Concurrency

Please pay attention, running the pipeline in a Debug modus has no effect on this and will run directly.
Check the monitoring regularly to check if this situation is not happening all the time, if so,  you better change the recurrence ​of your Triggered Pipeline. You still have the option to cancelled a queued pipeline.

How to enable concurrency?

 

To enable concurrency in an Azure Synapse pipeline, you can use the Concurrency property in the pipeline settings. The default value is 1, which means that only one copy of the pipeline will run at a time. By default, there is no maximum. If the concurrency limit is reached, additional pipeline runs are queued until earlier ones complete. Setting the concurrency level to a higher value will cause multiple copies of the pipeline to run concurrently, which can improve performance if the pipeline is CPU-bound or if the data source can handle the increased load. If you leave the property blank the pipeline will not be queued. 

Enable-concurrency-Azure-Synapse

When you have any questions regarding concurrency, please let me know.

Feel free to leave a comment

Restore a accidentally deleted Azure SQL Database

Month: January 2022 Help I deleted my Logical Server ​OOPS Have you ever experienced that you accidentally deleted your Logical Server in Azure?  Because, for example, you made your Pipeline wrong. Surely. And of course you didn’t have a backup in your storage…

Creating Notebook Connections in Microsoft Fabric Just Became a Lot Easier (and Automatable!)

Month: January 2022 For a long time, creating a Notebook connection in Microsoft Fabric was surprisingly limited. If you wanted a Notebook to run under a Workspace Identity or Service Principal, you only could configure this directly in the DataPipeline UI. In fact,…

New Microsoft Azure Certifications

Month: January 2022 Microsoft Certification by Solution Area Handy overview of the new Microsoft Azure Certifications. More details can be found here Feel free to leave a comment

Enable Pattern Rules in Azure Purview

Month: January 2022 How can I enable Pattern Rules? ​Pattern Rules Last night I was preparing for a demo with Azure Purview. As always, I walk through all the activity hubs to see if there are any new options. This time I noticed that the Pattern Rules option was…

SQLBits session: Microsoft Purview Data Policy App

Month: January 2022 SQLBits 2023 Thanks everyone for visiting my session during SQLBits. It’s great to see such a full room and that so many people have started using Microsoft Purview.   SLIDES The slides can be downloaded via the link below, so that you can…

10 Days of Azure Synapse Analytics

Month: January 2022 10 Days of Azure Synapse Analytics For the next 10 days, every day a different topic is explained about Azure Synapse Analytics. The shortest and easiest way to see how Azure Synapse Analytics can help you, to make decisions within your data…

Azure Purview MSIgnite Spring 2021 Announcements

Month: January 2022 Azure Purview Ignite 2021 Annoucement Pricing This week the Azure Purview announced that they will extend the Azure Purview offer to provision 4 Capacity Units of the Data Map for free till May 31, 2021! Charging will start on June 1, 2021. Great…

Book of News Microsoft Ignite 2019

Month: January 2022 Microsoft Ignite 2019 MSIgnite 2019 has almost come to an end. A lot of new Annoucement have been released and presented.  Would you also like to know which announcements have been made? Microsoft has released during MSIgnite a “Book of News”. This…

Are you using Azure DevOps?

Month: January 2022 Azure DevOps More and more users are starting with Azure DevOps. Azure DevOps can be used for up to 5 users free of charge and is therefore a great start to start a project.     Build and Release I will not describe what Azure DevOps can do,…

Service Health Alerts in Azure

Month: January 2022 Maintenance and Service Healths in Azure I get often questions, is there no maintenance in Azure. Like every data center, the Azure data center also needs maintenance. You can even be notified when a specific service, in a specific data center, has…

My first Virtual session in 2022 for Dataminds

My first Virtual session in 2022 for Dataminds

Month: January 2022

Month: January 2022

by Erwin | Jan 12, 2022 | 0 comments

DataMinds

This Tuesday I’ve joined the DataMinds user Group to talk about Azure Purview.

 

Feel free to leave a comment

Restore a accidentally deleted Azure SQL Database

Month: January 2022 Help I deleted my Logical Server ​OOPS Have you ever experienced that you accidentally deleted your Logical Server in Azure?  Because, for example, you made your Pipeline wrong. Surely. And of course you didn’t have a backup in your storage…

Azure Purview Public Preview Starts billing

Month: January 2022 Billing for Azure Purview(Public Preview) As of January 20th 2021 0:00 UTC Azure Purview will starts billing. Preview From January 20 ,2021 Azure Purview will start billing. During the Public Preview, you will only be billed if you exceed the 4…

Updated competency exams and certifications Data Platform and Data Analytics for 2020

Month: January 2022 Retiring and new exams and certifications as of June 30 2020 A lot of Exams and certifications for Data Platform and Data Analytics are retiring on June 30 2020. All retired Exams and certifications will remain eligible for competency attainment…

Microsoft Build 2022 Book of News

Month: January 2022 Microsoft Build 2022 MSBuild 2022 has just started. A lot of new Announcement will be released or demonstrated the upcoming days. Like every year or event most of these announcement are added to the Book of News. The book of news can be downloaded…

Azure Synapse Analytics Code Repository has arrived

Month: January 2022 Azure Synapse Analytics Code repository ‎I just opened my Azure Synapse Analytics Workspace and got a great surprise, the option Git Configuration is available as of today‎.     After a long wait, today the Git Configuration option became available…

Are you using Azure DevOps?

Month: January 2022 Azure DevOps More and more users are starting with Azure DevOps. Azure DevOps can be used for up to 5 users free of charge and is therefore a great start to start a project.     Build and Release I will not describe what Azure DevOps can do,…

Data Factory Pricing

Month: January 2022 Data Factory pricing Are you also having problems to understand the Pricing Model for Azure Data Factory?  After some research on the internet I came across an article which I wanted to share with you. ADFV2 Pricing Examples Feel free to leave a…

Collection of all ADF Mapping Data Flow videos

Month: January 2022 ADF Mapping Flow Did you use the Dataflow preview functionality in Azure Data Factory? This has recently be renamed to Mapping Data Flows.   All video’s which the ADF team has created, are collected. Start Here: ADF Data Flow: Overview ADF…

Enable Pattern Rules in Azure Purview

Month: January 2022 How can I enable Pattern Rules? ​Pattern Rules Last night I was preparing for a demo with Azure Purview. As always, I walk through all the activity hubs to see if there are any new options. This time I noticed that the Pattern Rules option was…

Azure Data Factory: New functionalities and features

Month: January 2022 New functionalities and features Last week, a number of great new functionalities and features were added within Azure Data Factory. I would like to take you in some details in the blog below: Customer key With this new functionality you can add…

Goodbye 2021, Hello 2022

Goodbye 2021, Hello 2022

Month: January 2022

Month: January 2022

by Erwin | Jan 3, 2022 | 0 comments

Goodbye 2021

Recap

First of all, I would like to wish everyone a very beautiful and healthy 2022.

We are now 3 days on the road into the new year and it is always good to look back at what happened last year. It’s certainly been an eventful year, topped off with my MVP Award, which I’m super proud of.
Within InSpark there were a number of changes within our Management Team and that had a significant effect, partly expected but sometimes not completely. Everyone has now been able to find their way and the various Teams are making quite a lot of progress.
Our office is currently closed due to the Lock down in the Netherlands and we communicate through Teams again. I certainly look back to the months that our office was open and that you also saw other colleagues outside your own team. I continue to find meetings via teams very difficult and I regularly have trouble finding the right drive and inspiration there, but unfortunately it is no different and let’s hope for better.

Managed Oxygen

POTY_Analytics_2021

With Managed Oyxgen, our Data Platform as a Service, we have once again made such major improvements that I did not think it was possible at the beginning of this year, but confirmation came in July 2021. We submitted Managed Oxygen for the Microsoft Partner of the Year awards, we just didn’t win the award but we did become a finalist in the Category Analytics and that out of 4400 entries.
Wow we were so happy with this appreciation, then you know were you worked hard for every day. Compliments to all my colleagues who work every day with such a drive and energy on the development of our Managed Oxygen.

In addition to our Managed Oxygen, we continued working with the whole team on our Nitrogen Accelerator.

  • Metadata-driven Framework for Azure Data Factory and Azure Synapse Analytics which allows us to automatically extract data from various sources and  building a Lakehouse.
  • Monitoring, Logging and Audit Pipelines
  • Build and release pipelines for all the necessary Azure Data services and Power BI in DevOps.
  • Data quality and privacy patterns.
  • Automated Documentation and other best practices.

An Accelerator that greatly benefits our customers and to which we as a team provide input from all disciplines.

Cool and innovative projects

As InSpark, we are the Cloud Incubator for our mother company KPN, which has the advantage that we can work a lot on innovation. We have done a lot of connected projects this year, such as Connected Vehicles, Connected Ships, Connected containers, with some projects processing more than 70 million messages per day. Still pretty cool to see how easy it all goes and fit within Managed Oxygen. In addition to these Connected projects, we have done projects in which we help cities and local governments with their ever-growing demand for data and data solutions, the Urban Data Platform. We have made the first steps with Azure Percept and I’m looking forward to start our first Azure Percept project this year.

We are still looking for new colleagues to help us with these cool projects. If you want to know more about InSpark and what cool projects we do even more, let me know.

Blog

Just like last year, I wanted to write more blogs and articles, but unfortunately the counter has stopped at 20 this time. My blogs and articles were mostly about Azure Synapse Analytics and Azure Purview. It was good to see that the community is finding my blogs and articles better and better and that’s what it’s all about in the end.

MVP Award

In October last year I became a Data Platform MVP, a great appreciation from Microsoft for all the input and feedback I provide on the various Azure Data Services and my contribution to the Community. When I saw the message in my mailbox I didn’t know what I saw, so happy, I immediately called my colleagues to share the news. They have always supported me in everything I do.

ADF Hackathon

I submitted a ADF Pipeline Template “Scale Dedicated SQL Pool Dynamically using Azure Data Factory control flow“ to the ADF Hackathon in March and my submission was marked as WINNER. I am very proud that a simple template where you can easily save costs has won. This template will help you to scale up and down a Dedicated SQL Pool in Azure Synapse Analytics. See full post of the announcement here.

Events

This year I regularly spoke at Virtual Events such as SQL Bits, Scottisch Summit, DataWeekender, Data Toboggan, Cloud Lunch and Learn Marathon and various DataSaturdays. In October I helped as a volunteer during DataMinds Connect a physical event(my only one in 2021) which was held in Mechelen. This event gave me so much energy again because I saw so many great sessions and talked to so many people again. The event was perfectly organized according to the then applicable conditions in Belgium. My brain and body were no longer used to that and I was completely exhausted after these 2 super fantastic days.

SQL Bits, Scottish Summit, DataMinds and Datagrillen are already planned for this year. I look forward to seeing everyone again.

Whatever a year looks like, the most important thing is that everyone is healthy and safe. I look forward to a great collaboration with everyone.

Feel free to leave a comment

Proud to be an ADF Hackathon winner!

Month: January 2022 Azure Data Factory Hackathon Around the clock During the event ‘Around the clock Azure SQL and Azure Data Factory’ event on Feb 3, 2021, they kicked-off the Azure Data Factory (ADF) Hackathon. Recording of this event can be found here . Winner I…

Calculate Workingdays including Holidays with T-SQL

Month: January 2022 Calculate Workingdays between 2 Date columns Recently I have been getting some questions from my customers, can I calculate the number of workdays between 2 dates? Of course my answer was, yes you can. But I do want certain closing dates and…

Azure Purview March Updates

Month: January 2022 Azure Purview updates Announcements Last week during SQLBITS, quite a few new updates were announced. I would like to include you in these announcements. March updates Support for SAP Business Warehouse (Preview) Blogpost:…

Parameterize Linked Services in ADF

Month: January 2022 Parameterize Linked Services  For my Azure Data Factory solution I wanted to Parameterize properties in my Linked Services. Not all properties are Parameterized by default through the UI. But there’s another way to achieve this.     Linked Service…

ADF: Get Metadata Activity stopped working

Month: January 2022 Meta Data Activity Today my pipelines in Azure Data Factory (ADF) suddenly stopped working. The output structure was not found. Quit strange while these pipelines have been running for weeks.     Invalid Template After debugging my Pipeline I found…

Collection of all ADF Mapping Data Flow videos

Month: January 2022 ADF Mapping Flow Did you use the Dataflow preview functionality in Azure Data Factory? This has recently be renamed to Mapping Data Flows.   All video’s which the ADF team has created, are collected. Start Here: ADF Data Flow: Overview ADF…

Azure Purview MSIgnite Spring 2021 Announcements

Month: January 2022 Azure Purview Ignite 2021 Annoucement Pricing This week the Azure Purview announced that they will extend the Azure Purview offer to provision 4 Capacity Units of the Data Map for free till May 31, 2021! Charging will start on June 1, 2021. Great…

My Sessions at DataSaturdayNL 2019

Month: January 2022 DataSaturday NL 2019 Date: 5 th October Location: Pathé Leidsche Rijn Data Saturday Holland is the new name for SQL Saturday Holland. A new start a new location and wow what a top location, Pathé Leidsche Rijn. A cinema with 7 screens and ranging…

Microsoft Fabric pricing (Preview)

Month: January 2022 Microsoft Fabric Pricing Microsoft Fabric pricing in Public Preview is announced as of the 1 st of June. These are currently the Pay as You go pricing, later this year the Azure Reservation will follow. OneLake storage pricing is comparable to…

SQLBits session: Microsoft Purview Data Policy App

Month: January 2022 SQLBits 2023 Thanks everyone for visiting my session during SQLBits. It’s great to see such a full room and that so many people have started using Microsoft Purview.   SLIDES The slides can be downloaded via the link below, so that you can…