ERWIN & BUSINESS ANALYTICS

Azure Data Factory: New functionalities and features

by May 22, 2020Azure Data Factory, Blogs

New functionalities and features

Last week, a number of great new functionalities and features were added within Azure Data Factory. I would like to take you in some details in the blog below:

ADF DEBUG Report

Customer key

With this new functionality you can add extra security to your Azure Data Factory environment. Where the data was first encrypted with a randomly generated key from Microsoft, you can now use the customer-managed key feature. With this Bring Your Own Key (BYOK) you can add extra security to your Azure Data Factory environment. If you use the customer-managed key functionality, the data will be encrypted in combination with the ADF system key. You can create your own key or have it generated by the Azure Key Vault API.

You can read more in this Article which I wrote.

Pipeline Consumption Report

Last week the Azure Data Factory added the Pipeline Consumption Report.

The report can be used for your Triggered runs, just go to your Triggered runs and click on the new Icon.

ADFMonitor

The consumption of the selected Pipeline will be displayed. The data shown is only from this Pipeline and not from other Pipelines fired by this Pipeline. Would be a nice addition if the report shows the aggregation of the complete Triggered Run.

For your debug run, click on right site of your Output pane:

ADF DEBUG button

ADF DEBUG Report

The ADF consumption report is only surfacing Azure Data Factory related units. There may be additional units billed from other services that you are using and accessing which are not accounted for here including Azure SQL Database, Synapse Analytics, CosmosDB, ADLS, etc. More detailed can be found here.

Parameters from Execute Pipeline Activity

When calling a Pipeline you first had to add the parameters yourself, now they are automatically taken over from the Pipeline you select. Very handy and saves time again if you use a lot of parameters.

Define a Parameter in one of your Pipelines:

ADF Parameter

 

Create another Pipeline and add the Execute Pipeline activity. On the settings tab where you have to select the Pipeline you want to execute, you will discover that the option to add Manually the parameters is not there anymore. But, all the Parameters you had defined in your Pipeline are directly shown. Very handy and it reduces errors.

Old Situation:

ADF Parameter 3

New Situation:

ADF Parameter Pipeline

General Tab moved to new Properties Pane

Your General tab is now moved to the right site of the Canvas.

ADF Pane General

To edit it your properties, click on the pane icon located in the top-right corner of the canvas.

ADF Pane Properties

So these were some nice and useful addition to Azure Data Factory. I am very happy with it and what do you think?

Feel free to leave a comment

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

20 − 13 =

This site uses Akismet to reduce spam. Learn how your comment data is processed.

A new year with a new job

Changing jobsAfter almost 11 years and 4 months I have decided to leave Axians and to start a new adventure in the new year. On January 2, 2020 I will start my day as Lead Data and AI at InsparkThe past years have flown by. I started at Eniac BI, which was...

Data Factory Pricing

Data Factory pricingAre you also having problems to understand the Pricing Model for Azure Data Factory? After some research on the internet I came across an article which I wanted to share with you. ADFV2 Pricing ExamplesFeel free to leave a comment

Working from Home: How do I get my energy and focus back on track?

Working from home It has now been exactly 7 weeks since I started working from home. In the beginning this went with full energy, but now gradually the energy and the focus on the work is starting to drain.I'm the only one in here? I'm sure, I'm not. I regularly hear...

Goodbye 2020 Hello 2021

Goodbye 2020 Started to work for InSpark Last year was certainly an eventful year. Started with a new job at InSpark and after 10 weeks we all know what happened, the first intelligent lockdown. The Netherlands was partially locked, but our office was immediately...

Azure Synapse Analytics Code Repository has arrived

Azure Synapse Analytics Code repository‎I just opened my Azure Synapse Analytics Workspace and got a great surprise, the option Git Configuration is available as of today‎.    After a long wait, today the Git Configuration option became available in Azure Synapse...

Service Health Alerts in Azure

Maintenance and Service Healths in AzureI get often questions, is there no maintenance in Azure. Like every data center, the Azure data center also needs maintenance. You can even be notified when a specific service, in a specific data center, has scheduled...

Are you using Azure DevOps?

Azure DevOpsMore and more users are starting with Azure DevOps. Azure DevOps can be used for up to 5 users free of charge and is therefore a great start to start a project.    Build and Release I will not describe what Azure DevOps can do, because that is too much to...

Calculate the Last Day of the Month using SQL

Calculate the Last DayToday I needed to calculate the last day of the previous month for a Customer.Ever heard from the function EOMONTH? Searching on the web I came across a function I never heard from before EOMONTH, this function can be used as of SQL Server 2012....

How to check your SQL Server Quota in Azure?

Azure Subscription Usages for SQL Server  Last week we reached our Logical server Quota in Azure. By default you're only allowed to add 20 Logical Servers, but we wanted to have some more for testing purposes.Microsoft Support You can submit a support ticket trough...

LEAP 2018

Lead Enterprise Architect ProgramFrom the 4th of December until the 6th of December I've joined the LEAP (Lead Enterprise Architect Program) 2018 Data/AI from Microsoft NL. Three  days on the Microsoft Campus in Redmond joining business and technical sessions from a...