Rerun Pipeline activities in Azure Data Factory

Rerun Pipeline activities in Azure Data Factory

by | Mar 7, 2019 | Azure, Azure Data Factory, Azure Synapse Analytics

Erwin

by Erwin | Mar 7, 2019

Rerun Pipeline activities in ADF!

As of today you can rerun or partially, yes you’re reading it correct partially, rerun you Azure Data Factory pipeline.
Where you previously had to run the entire Pipeline again, you can now run a part of the Pipeline. This can save a lot of time if many different activities are created within one pipeline. Another nice step forward, I'm curious what else is coming in the next months.

 

Visualized

Besides that you can rerun your Pipeline in Azure Data Factory in a easy way, you also have the possibilities to see your run,  visualized in the Azure Data Factory Monitoring. This is a big improvement in my opinion.

Rerun a Pipeline

If you want to partially rerun a Pipeline, follow the steps below:
Select the Pipeline which has failed, go to the view activity runs and select the activity which failed.

Click on the Rerun Icon

 

 

 

 

You need to confirm that you want to rerun this activity.

The Pipeline will start and will first skip all the activities(the grey new icons in the upper right corner of each activity) in the Pipeline before your selected Activity.
Your Pipeline will now finalize all the activities from your newly defined starting point.

 

What else is new?

Monitor Rerun History

You can now view all the history reruns by clicking on the toggle to ‘View All Rerun History’.

By clicking on the red marked action, you can see all the History from an particular Pipeline run.

 

Thanks for reading.

 

 

Updated 10th of March:

 

Found a video on Channel9 which explains how to  "Rerun activities inside your Azure Data Factory pipelines"

https://channel9.msdn.com/Shows/Azure-Friday/Rerun-activities-inside-your-Azure-Data-Factory-pipelines?ocid=player

Feel free to leave a comment

Are you using Azure DevOps?

Are you using Azure DevOps?

by | Feb 21, 2019 | Azure DevOps

Azure DevOps

More and more users are starting with Azure DevOps. Azure DevOps can be used for up to 5 users free of charge and is therefore a great start to start a project.

 

 

Build and Release

I will not describe what Azure DevOps can do, because that is too much to mention. In my future articles I want to take you into the challenges that I had in my projects during building and releasing of f.e. DataFactory, SSIS Packages, KeyVault, a number of recognizable Azure Services that are regularly used within Business Analytics projects.

If you want to use Azure Feature Pack components like Azure Blob Storage or Azure Data Lake Store  in your SSIS Packages, I would definitely read the following article:

Azure DevOps and Azure Feature Pack for Integration Services

Azure DevOps and Azure Feature Pack for Integration Services

by | Feb 21, 2019 | Azure Data Factory, Azure DevOps, Azure Synapse Analytics

Azure Feature Pack for Integration Services

Azure Blob Storage

A great addition for SSIS is using extra connectors like  Azure Blob Storage or Azure Data Lake Store which are added by the Azure Feature Pack. This Pack needs to be installed on your local machine. Are you running your SSIS packages in Azure?  You don't have to install anything, this pack is installed by default.

SSIS Package

 

 

Building your SSIS Packages in Azure DevOps

After I started to use Azure Dev Ops to build my SSIS packages on a hosted VS2017, I got some strange error messages running these packages.

SSIS error

Microsoft Support

After contacting support we found out that the Azure Feature Pack is not installed on a Hosted VS2017 instance and that you need to add this installation to your build processes.

 

Install Azure Feature Pack on your Hosted VS2017 machine

Follow the steps to download and install the Azure Feature Pack:

  • Open  your dev.azure.com/instance.
  • Create a new Build Pipeline or use an existing one.
  • Select the correct Sources and after that you can add a new build task.
  • Add a Powershell Task.
    • This task needs to be added before the build process of your SSIS project.
  • Define the Display name “Install Azure Feature Pack”.Azure Dev Ops Pipeline Install Feature Pack
  • Type => Inline.
  • Add the script which you can find below.
  • Save and Queue the Pipeline.
  • Check the Results.

 

Powershell script

The script will take care of downloading and installing the Azure Feature Pack for SSIS2017 on your hosted 2017 machine.

The File SsisAzureFeaturePack_2017_x64.msi will be downloaded to the system variable Build.StagingDirectory.

Inline script:

[code lang="ps"]
# Erwin de Kreuk
# February 2019
# PURPOSE: Install Azure Feature pack on Hosted VS2017 machine in Azure DevOps

Write-Information 'Starting ADF ARM Transform'

#Define Filename
$Filename = 'SsisAzureFeaturePack_2017_x64.msi'
$Arguments=' /qn'
Write-Host 'Downloading...$Filename'
#Define download link including filename and output directory with filename
Invoke-WebRequest -Uri 'https://download.microsoft.com/download/E/E/0/EE0CB6A0-4105-466D-A7CA-5E39FA9AB128/SsisAzureFeaturePack_2017_x64.msi' -OutFile '$(Build.StagingDirectory)$Filename'

Write-Host 'Installing...$Filename'
Invoke-Expression -Command '$(Build.StagingDirectory)$Filename $Arguments'
Write-Host 'Finished Installing...$Filename'

[/code]

Azure Dev Ops Build

The next time you build your SSIS Packages with the Azure Components, these packages are build correctly. Create a Release Pipeline to Deploy the SSIS Packages to the SSIS server and to test your Package.

Thanks for reading today and if there're some questions left do not hesitate to ask them.

Collection of all ADF Mapping Data Flow videos

Collection of all ADF Mapping Data Flow videos

by | Feb 19, 2019 | Azure Data Factory

Erwin

by Erwin | Feb 19, 2019

ADF Mapping Flow

Did you use the Dataflow preview functionality in Azure Data Factory? This has recently be renamed to Mapping Data Flows.

 

All video's which the ADF team has created, are collected.

Start Here:

Debug and Prep:

Transformations:

Optimize:

Patterns:

Expressions:

 

Do you want to know more or you want to try out some examples, you can find them here

Feel free to leave a comment

Azure Data Factory: Generate Pipeline from the new Template Gallery

by | Feb 12, 2019 | Azure, Azure Data Factory

Erwin

Last week I mentioned that we could save a Pipeline to GIT. But today I found out that you can also create a Pipeline from a predefined Solution Template.

Template Gallery

These template will make it easier to start with Azure Data Factory and it will reduce development time when you start a new project.

Currently Microsoft has released the following templates:

Copy templates:

  • Bulk copy from Database
  • Copy multiple file containers between file-based stores
  • Delta copy from Database

Copy from <source> to <destination>

  • From Amazon S3 to Azure Data Lake Store Gen 2
  • From Google Big Query to Azure Data Lake Store Gen 2
  • From HDF to Azure Data Lake Store Gen 2
  • From Netezza to Azure Data Lake Store Gen 1
  • From SQL Server on premises to Azure SQL Database
  • From SQL Server on premises to Azure SQL Data Warehouse
  • From Oracle on premises to Azure SQL Data Warehouse

SSIS templates

  • Schedule Azure-SSIS Integration Runtime to execute SSIS packages

Transform templates

  • ETL with Azure Databricks

These templates can be found directly in the Azure Data Factory Portal:

Now you can select the option Create Pipeline from Template.

After selecting this option, all templates from the gallery but also the templates you saved yourselves, are visible.

Create a Pipeline from a Template

To start the creating of the template, click on the template you want to create. For this example, I have chosen for the template Bulk Copy from Database. A wizard will open which you have to follow.

The only thing you need to do right know is selecting the correct inputs. You can also create a new input from the Template wizard.

After selecting all the correct inputs you can finalize the template and the template will be added to your Factory.

Do you want to follow the detailed steps of creating this pipeline? The details can be found here.

Thanks so much for reading through this article today, and I hope you all take some time to try it out. It's will make your life easier.