More and more users are starting with Azure DevOps. Azure DevOps can be used for up to 5 users free of charge and is therefore a great start to start a project.
Build and Release
I will not describe what Azure DevOps can do, because that is too much to mention. In my future articles I want to take you into the challenges that I had in my projects during building and releasing of f.e. DataFactory, SSIS Packages, KeyVault, a number of recognizable Azure Services that are regularly used within Business Analytics projects.
If you want to use Azure Feature Pack components like Azure Blob Storage or Azure Data Lake Store in your SSIS Packages, I would definitely read the following article:
A great addition for SSIS is using extra connectors like Azure Blob Storage or Azure Data Lake Store which are added by the Azure Feature Pack. This Pack needs to be installed on your local machine. Are you running your SSIS packages in Azure? You don’t have to install anything, this pack is installed by default.
Building your SSIS Packages in Azure DevOps
After I started to use Azure Dev Ops to build my SSIS packages on a hosted VS2017, I got some strange error messages running these packages.
Microsoft Support
After contacting support we found out that the Azure Feature Pack is not installed on a Hosted VS2017 instance and that you need to add this installation to your build processes.
Install Azure Feature Pack on your Hosted VS2017 machine
Follow the steps to download and install the Azure Feature Pack:
Open your dev.azure.com/instance.
Create a new Build Pipeline or use an existing one.
Select the correct Sources and after that you can add a new build task.
Add a Powershell Task.
This task needs to be added before the build process of your SSIS project.
Define the Display name “Install Azure Feature Pack”.
Type => Inline.
Add the script which you can find below.
Save and Queue the Pipeline.
Check the Results.
Powershell script
The script will take care of downloading and installing the Azure Feature Pack for SSIS2017 on your hosted 2017 machine.
The File SsisAzureFeaturePack_2017_x64.msi will be downloaded to the system variable Build.StagingDirectory.
Inline script:
[code lang="ps"]
# Erwin de Kreuk
# February 2019
# PURPOSE: Install Azure Feature pack on Hosted VS2017 machine in Azure DevOps
Write-Information 'Starting ADF ARM Transform'
#Define Filename
$Filename = 'SsisAzureFeaturePack_2017_x64.msi'
$Arguments=' /qn'
Write-Host 'Downloading...$Filename'
#Define download link including filename and output directory with filename
Invoke-WebRequest -Uri 'https://download.microsoft.com/download/E/E/0/EE0CB6A0-4105-466D-A7CA-5E39FA9AB128/SsisAzureFeaturePack_2017_x64.msi' -OutFile '$(Build.StagingDirectory)$Filename'
Write-Host 'Installing...$Filename'
Invoke-Expression -Command '$(Build.StagingDirectory)$Filename $Arguments'
Write-Host 'Finished Installing...$Filename'
[/code]
The next time you build your SSIS Packages with the Azure Components, these packages are build correctly. Create a Release Pipeline to Deploy the SSIS Packages to the SSIS server and to test your Package.
Thanks for reading today and if there’re some questions left do not hesitate to ask them.
Last week I mentioned that we could save a Pipeline to GIT. But today I found out that you can also create a Pipeline from a predefined Solution Template.
Template Gallery
These template will make it easier to start with Azure Data Factory and it will reduce development time when you start a new project.
Currently Microsoft has released the following templates:
Copy templates:
Bulk copy from Database
Copy multiple file containers between file-based stores
Delta copy from Database
Copy from <source> to <destination>
From Amazon S3 to Azure Data Lake Store Gen 2
From Google Big Query to Azure Data Lake Store Gen 2
From HDF to Azure Data Lake Store Gen 2
From Netezza to Azure Data Lake Store Gen 1
From SQL Server on premises to Azure SQL Database
From SQL Server on premises to Azure SQL Data Warehouse
From Oracle on premises to Azure SQL Data Warehouse
SSIS templates
Schedule Azure-SSIS Integration Runtime to execute SSIS packages
Transform templates
ETL with Azure Databricks
These templates can be found directly in the Azure Data Factory Portal:
Now you can select the option Create Pipeline from Template.
After selecting this option, all templates from the gallery but also the templates you saved yourselves, are visible.
Create a Pipeline from a Template
To start the creating of the template, click on the template you want to create. For this example, I have chosen for the template Bulk Copy from Database. A wizard will open which you have to follow.
The only thing you need to do right know is selecting the correct inputs. You can also create a new input from the Template wizard.
After selecting all the correct inputs you can finalize the template and the template will be added to your Factory.
Do you want to follow the detailed steps of creating this pipeline? The details can be found here.
Thanks so much for reading through this article today, and I hope you all take some time to try it out. It’s will make your life easier.
Another great improvement in Azure Data Factory. Saving you template!
How can you save your template?
First of all you need to connect your Azure Data Factory to a GIT integration. Both Azure DevOps GIT and GitHub are supported. Please follow thislinkif you don’t have a Git Integration.
Select the template you want to save:
Name your template and save it to a Git Location of your choice:
The template you have saved can be found in Factory Resources.
With this new option it will be easier to share Pipeline Activities with other Azure Data Factory instances.
And most important of saving your template it will save a lot of development time in your new instance.
