Azure Data Factory updates June

by Jun 30, 2022

Azure Data Factory updates

There have been quite a few updates in Azure Data Factory and Azure Synapse Analytics in the last few days.
Below is a summary of these updates:

 

Time-To-Live (TTL) on Integration Runtime with managed virtual network enabled

The new TTL functionality, currently in Preview, will save you a lot of time. Instead of provision compute for every copy activity execution, with this new option the provisioned compute will be reserved. You can set this new option on every Integration Runtime with Managed Virtual Network enabled. Very handy and will advise you to do this direclty. This option is also available in Azure Synapse Analytics.

adf-ttl

More can be found on the blog below:

Time-To-Live (TTL)

CI/CD improvement using Global Parameters in Azure Data Factory

We used to have a PowerShell script to add the Global parameters to our CI/CD pipelines. Now we can include these as Override Parameters in our Deployment/Release Pipeline. Very handy and easy.

Deprecated:

adf-gloabal-parameters-depr

New:

adf-gloabal-parameters-new

If you are using the arm-template-parameters-definition.json file to customize your parameters make sure you add the global parameters to this file. Otherwise you cannot override the parameters in the Deployment/Release Pipeline.

adf-gloabal-parameters-template

When we have created a Global Parameters Environment, we have now have the option to override this parameter option during the release:

-default_properties_Environment_value “Development

More can be found on the blog below:

CI/CD improvement using Global Parameters in Azure Data Factory

Public Preview of the SAP CDC solution in Azure Data Factory

The new SAP ODP connector leverages SAP Operational Data Provisioning (ODP) framework, which is an established best practice for data integration within SAP landscapes. ODP provides access to a wide range of sources across all major SAP applications and comes with built-in CDC capabilities. In combination with the predefined data flow templates to process and update the changed records to any sink, this makes SAP data integration into Azure very much straight forward.

More can be found on the blog below:

Public Preview of the SAP CDC solution in Azure Data Factory

Azure Data Factory studio preview experience

You can now choose whether to enable preview experiences/functionality in your Azure Data Factory.

There are 2 ways how you can enable this preview experience:

When opening the portal you will see banner, just click on the Open settings to learn more and opt in. link

adf-preview-update-portal

Or open the setting pane on the right top and enable the toggle Azure Data Factory Studio preview update.

Current Preview Updates

Currenlty the following options are available as preview features:

Dataflow data experimental view

Configuration panel

The configuration panel will only have Data Preview that will automatically refresh when changes are made to transformations.

Before:

adf-preview-before

After:

adf-preview-after
Transformation settings

A pop-up will now appear when you want to change your settings

adf-preview-transformation-settings
Data preview

Data preview now includes Elapsed time (seconds) to show how long your data preview took to load.

adf-preview-update-data-preview

Pipeline experimental view

Adding activities

You now have the option to add an activity using the Add button in the bottom right corner of an activity in the pipeline editor canvas. Clicking the button will open a drop-down list of all activities that you can add.

 adf-preview-adding-activitities
Iteration & conditionals container view

And the last one is really awesome, you now view the activities in the contained iteration and conditional activities like For each, Switch, Until and Switch

adf-preview-activitities

More can be found on the detailed page below:

Azure Data Factory studio preview experience

Please visit the blog below for all the latest update on Synapse Analytics in June 2022:

Azure Synapse Analytics June Update 2022

 

If you have any questions left, please leave them in the comments below.

Feel free to leave a comment

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

nineteen − four =

This site uses Akismet to reduce spam. Learn how your comment data is processed.

SSMS 18.1: Schedule your SSIS Packages in Azure Data Factory

Schedule your SSIS Packages with SSMS in Azure Data Factory(ADF) This week SQL Server Management Studio version 18.1 was released, which can be downloaded from here. In version 18.1 the Database diagrams are back and from now on we can also schedule SSIS Packages in...

Provision users and groups from AAD to Azure Databricks (part 6)

Configure the Enterprise Application(SCIM) for Azure Databricks Workspace provisioning In one of my previous blogs, I explained how to setup the Enterprise Application for Azure Databricks account level  provisioning. This blog is related to the Azure Databricks...

Blog Serie: Provision users and groups from AAD to Azure Databricks

Blog Series This blog post series contains topics on how to Provision users and groups from Azure Active Directory to Azure Databricks using the Enterprise Application(SCIM). This is a summary of the all the blogs I posted the last couple of days. I am very happy with...

Provision users and groups from AAD to Azure Databricks (part 1)

Blog Serie: Provisioning identities from Azure Active Directory to Azure Databricks. Instead of adding users and groups manual to your Azure Databricks environment, you can also sync them automatically from your Azure Active Directory to your Azure Databricks account...

Provision users and groups from AAD to Azure Databricks (part 5)

Add Service principals to your Azure Databricks account using the account console In the previous blog, you assigned Users and Groups to an Azure Databricks Workspace. In this blog, you will learn how to assign Service Principals to an Azure Databricks Workspace and...

Create Virtual Machines with Azure DevTest Lab

A while ago I had to give a training. Normally I would roll out a number of virtual machines in Azure. Until someone brought my attention to an Azure Service, Azure DevTest Labs. With this Azure service you can easily create a basic image and use this image to roll...

Provision users and groups from AAD to Azure Databricks (part 2)

Assign and Provision users and groups in the Enterprise Application In the previous blog you learned how to configure the Enterprise Application. In this blog, you will learn how to assign and Provision Users and Groups. Once the Users and groups are assigned to the...

How to create a Azure Synapse Analytics Workspace

Creating your Azure Synapse Analytics Workspace In the article below I would like to take you through,  how you can configure an Azure Synapse Workspace and not the already existing Azure Synapse Analytics SQL Pool(formerly Azure SQL DW): In de Azure Portal search for...

Connect Azure Databricks to Microsoft Purview

Connect and Manage Azure Databricks in Microsoft Purview This week the Purview team released a new feature, you’re now able to Connect and manage Azure Databricks in Microsoft Purview. This new functionality is almost the same as the Hive Metastore connector which you...

Get control of data loads in Azure Synapse

Load Source data to DataLake There are several ways to extract data from a source in Azure Synapse Analytics or in Azure Data Factory. In this article I'm going to use a metadata-driven approach by using a control table in Azure SQL in which we configure the...