Azure Data Factory updates June

by Jun 30, 2022

Azure Data Factory updates

There have been quite a few updates in Azure Data Factory and Azure Synapse Analytics in the last few days.
Below is a summary of these updates:

 

Time-To-Live (TTL) on Integration Runtime with managed virtual network enabled

The new TTL functionality, currently in Preview, will save you a lot of time. Instead of provision compute for every copy activity execution, with this new option the provisioned compute will be reserved. You can set this new option on every Integration Runtime with Managed Virtual Network enabled. Very handy and will advise you to do this direclty. This option is also available in Azure Synapse Analytics.

adf-ttl

More can be found on the blog below:

Time-To-Live (TTL)

CI/CD improvement using Global Parameters in Azure Data Factory

We used to have a PowerShell script to add the Global parameters to our CI/CD pipelines. Now we can include these as Override Parameters in our Deployment/Release Pipeline. Very handy and easy.

Deprecated:

adf-gloabal-parameters-depr

New:

adf-gloabal-parameters-new

If you are using the arm-template-parameters-definition.json file to customize your parameters make sure you add the global parameters to this file. Otherwise you cannot override the parameters in the Deployment/Release Pipeline.

adf-gloabal-parameters-template

When we have created a Global Parameters Environment, we have now have the option to override this parameter option during the release:

-default_properties_Environment_value “Development

More can be found on the blog below:

CI/CD improvement using Global Parameters in Azure Data Factory

Public Preview of the SAP CDC solution in Azure Data Factory

The new SAP ODP connector leverages SAP Operational Data Provisioning (ODP) framework, which is an established best practice for data integration within SAP landscapes. ODP provides access to a wide range of sources across all major SAP applications and comes with built-in CDC capabilities. In combination with the predefined data flow templates to process and update the changed records to any sink, this makes SAP data integration into Azure very much straight forward.

More can be found on the blog below:

Public Preview of the SAP CDC solution in Azure Data Factory

Azure Data Factory studio preview experience

You can now choose whether to enable preview experiences/functionality in your Azure Data Factory.

There are 2 ways how you can enable this preview experience:

When opening the portal you will see banner, just click on the Open settings to learn more and opt in. link

adf-preview-update-portal

Or open the setting pane on the right top and enable the toggle Azure Data Factory Studio preview update.

Current Preview Updates

Currenlty the following options are available as preview features:

Dataflow data experimental view

Configuration panel

The configuration panel will only have Data Preview that will automatically refresh when changes are made to transformations.

Before:

adf-preview-before

After:

adf-preview-after
Transformation settings

A pop-up will now appear when you want to change your settings

adf-preview-transformation-settings
Data preview

Data preview now includes Elapsed time (seconds) to show how long your data preview took to load.

adf-preview-update-data-preview

Pipeline experimental view

Adding activities

You now have the option to add an activity using the Add button in the bottom right corner of an activity in the pipeline editor canvas. Clicking the button will open a drop-down list of all activities that you can add.

 adf-preview-adding-activitities
Iteration & conditionals container view

And the last one is really awesome, you now view the activities in the contained iteration and conditional activities like For each, Switch, Until and Switch

adf-preview-activitities

More can be found on the detailed page below:

Azure Data Factory studio preview experience

Please visit the blog below for all the latest update on Synapse Analytics in June 2022:

Azure Synapse Analytics June Update 2022

 

If you have any questions left, please leave them in the comments below.

Feel free to leave a comment

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

five + 14 =

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Migrate Azure Storage to Azure Data Lake Gen2

Migrate Azure Storage to Storage Account with Azure Data Lake Gen2 capabilities Does it sometimes happen that you come across a Storage Account where the Hierarchical namespace is not enabled or that you still have a Storage Account V1? In the tutorial below I...

Scale your SQL Pool dynamically in Azure Synapse

Scale your Dedicated SQL Pool in Azure Synapse Analytics In my previous article, I explained how you can Pause and Resume your Dedicated SQL Pool with a Pipeline in Azure Synapse Analytics. In this article I will explain how to scale up and down a SQL Pool via a...

Azure Purview announcements and new functionalities

This week the Azure Purview Product team added some new functionalities, new connectors(these connectors where added during my holiday), Azure Synapse Data Lineage, a better Power BI integration and the introduction of Elastics Data Map. Slowly we are on our way to a...

Azure Synapse Analytics Power BI Integration

Creating a Linked Service for Power BI Open your Synapse Studio and select the Management Hub. Add a new Linked Service If you haven't connect to Power BI before, you will see the screen above. If you want to add another Power BI Linked Service(Workspace). Search for...

Azure Purview Pricing example

Azure Purview pricing? Note: Billing for Azure Purview will commence November 1, 2021. Updated October 31st, 2021 Pricing for Elastic Data Map and Scanning for Other Sources are changed and updated in the blog below. Since my last post on Azure Purview announcements...

Using Azure Automation to generate data in your WideWorldImporters database

CASE: For my test environment I want to load every day new increments into the WideWorldImporters Azure SQL Database with Azure Automation. The following Stored Procedure is available to achieve this. EXECUTE DataLoadSimulation.PopulateDataToCurrentDate...

Azure SQL Data Warehouse: How to setup Reserved Capacity

Purchase your Azure SQL Datawarehouse Reservation   Since a few weeks you can buy Reserved Capacity for an Azure SQL Datawarehouse (SQLDW). This Reservation can save you up to 65% on the normal Pay as You go rates with a 3 year pre-commit. A pre-commit of 1 year...

Change your Action Group in Azure Monitoring

Change a Action GroupPrevious Article In my previous artcile I wrote about how to create Service Helath Alerts. In this article you will learn how to change the Action Group to add, change or Remove members(Action Group Type Email/SMS/Push/Voice) Azure Portal In the...

SSMS 18.xx: Creating your Azure Data Factory SSIS IR directly in SSMS

Creating your Azure Data Factory(ADF) SSIS IR in SSMS Since  version 18.0 we could see our Integration Catalog on Azure Instances directly. Yesterday I wrote an article how to Schedule your SSIS Packages in ADF, during writing that article I found out that you can...

Create an Azure Synapse Analytics Apache Spark Pool

Adding a new Apache Spark Pool There are 2 options to create an Apache Spark Pool.Go to your Azure Synapse Analytics Workspace in de Azure Portal and add a new Apache Spark Pool. Or go to the Management Tab in your Azure Synapse Analytics Workspace and add a new...