Parameterize Linked Services in ADF

by Jul 9, 2020

Parameterize Linked Services 

For my Azure Data Factory solution I wanted to Parameterize properties in my Linked Services. Not all properties are Parameterized by default through the UI. But there’s another way to achieve this.

 

 

Linked Service Azure Data Factory

Linked Service

Open your existing Linked Services.

Linked Services ADF

In this situation I want to Parameterize my FTP connection so that I can change the Host name based on a Azure Key Vault Secret.

By default is this not possible through the UI but in the Bottom of your Linked Service there is a Advanced box

Linked Services Advanced

 

If you enable this box you can start building your own connection, but also create your own Parameters for this connection.

How to start:

As a base we will use the default code or our connection

Linked Services Code

{
    "name": "LS_FTP_SOURCE",
    "properties": {
        "annotations": [
            "stage: none",
            "scenario: demo",
            "environments: dvlm"
        ],
        "type": "FtpServer",
        "typeProperties": {
            "host": "ftp.erwindekreuk.com",
            "port": 21,
            "enableSsl": true,
            "enableServerCertificateValidation": true,
            "authenticationType": "Basic",
            "userName": "ftp_down",
            "password": {
                "type": "AzureKeyVaultSecret",
                "store": {
                    "referenceName": "LS_AKV_OXGN",
                    "type": "LinkedServiceReference"
                },
                "secretName": "secretname"
            }
        }
    },
    "type": "Microsoft.DataFactory/factories/linkedservices"
}
Copy this code to your advanced box and enable the option Specify dynamic contents in JSON format/

Now you can start adding new parameters.

Linked Services Advanced

If you want to Parameterize your HOST name  connection you have to add in the top of the code a new Parameter, under the type of your connection

    "properties": {
        "type": "FtpServer",
        "parameters": {
            "ConnectionKeyvaultSecret": {
                "type": "string"
            }

After you have done this, you need to specify for which properties you want to use this parameter. In my case I want to read the parameter form my Azure Key Vault for my HOST propertie.

The JSON code below will now use above parameter as an input.

        "typeProperties": {
            "host": {
                "type": "AzureKeyVaultSecret",
                "store": {
                    "referenceName": "LS_AKV_OXGN",
                    "type": "LinkedServiceReference"
                },
                "secretName": {
                    "value": "@linkedService().ConnectionKeyvaultSecret",
                    "type": "Expression"
                }
            }

Save your connection and you will see that your UI is changed and that you have to define all your setting through the Advanced Editor.

If you test your connection you will now see that you have to fill in a parameter.

Linked Service Test Connection

And now you can create parameters of every TypeProperties within your connection.

The code below will create Parameters for your Host, Username and Password entries with Azure Key Vault enabled. For the authenticationType you have to choose between Basic and Anonymous. But can also at this to your Azure Key Vault.

{
    "name": "LS_FTP_SOURCE",
    "properties": {
        "type": "FtpServer",
        "parameters": {
            "ConnectionKeyvaultSecret": {
                "type": "string"
            },
            "UsernameKeyvaultSecret": {
                "type": "string"
            },
            "PasswordKeyvaultSecret": {
                "type": "string"
            },
            "authenticationType": {
                "type": "string"
            }
        },
        "annotations": [ ],
        "typeProperties": {
            "host": {
                "type": "AzureKeyVaultSecret",
                "store": {
                    "referenceName": "LS_AKV_OXGN",
                    "type": "LinkedServiceReference"
                },
                "secretName": {
                    "value": "@linkedService().ConnectionKeyvaultSecret",
                    "type": "Expression"
                }
            },
            "port": 21,
            "enableSsl": false,
            "enableServerCertificateValidation": false,
            "authenticationType": "@linkedService().authenticationType",
            "userName": {
                "type": "AzureKeyVaultSecret",
                "store": {
                    "referenceName": "LS_AKV_OXGN",
                    "type": "LinkedServiceReference"
                },
                "secretName": {
                    "value": "@linkedService().UsernameKeyvaultSecret",
                    "type": "Expression"
                }
            },
            "password": {
                "type": "AzureKeyVaultSecret",
                "store": {
                    "referenceName": "LS_AKV_OXGN",
                    "type": "LinkedServiceReference"
                },
                "secretName": {
                    "value": "@linkedService().PasswordKeyvaultSecret",
                    "type": "Expression"
                }
            }
        }
    }
}

Thanks for reading my blog post and have fun with Parameterization of your Linked Services in ADF.

Feel free to leave a comment

2 Comments

  1. David Laplante

    Thanks for the article! Just a note, proper indentation of the json goes a long way in helping everyone understand the logic 🙂

    Reply
    • Erwin

      Thanks for the Feedback David, looks like my editor removed the indentation. Going to look if I can better integrate/visualize this within my webpage.

      Reply

Submit a Comment

Your email address will not be published. Required fields are marked *

thirteen + five =

This site uses Akismet to reduce spam. Learn how your comment data is processed.

My First Blog Post SQLSatHolland 790

Yeah, my first blog is LIVEAfter a good talk with Reza Rad from RADACAD during SQLSatHolland, I decided to start my first blog!Knowledge sharing is very important, it gives me a lot of energy. But it also gives others people in the community energies to pick up new...

Azure Purview Public Preview Starts billing

Billing for Azure Purview(Public Preview)As of January 20th 2021 0:00 UTC Azure Purview will starts billing.Preview From January 20 ,2021 Azure Purview will start billing. During the Public Preview, you will only be billed if you exceed the 4 capacity units for Azure...

Rerun Pipeline activities in Azure Data Factory

Rerun Pipeline activities in ADF! As of today you can rerun or partially, yes you’re reading it correct partially, rerun you Azure Data Factory pipeline.Where you previously had to run the entire Pipeline again, you can now run a part of the Pipeline. This can save a...

Service Health Alerts in Azure

Maintenance and Service Healths in AzureI get often questions, is there no maintenance in Azure. Like every data center, the Azure data center also needs maintenance. You can even be notified when a specific service, in a specific data center, has scheduled...

How to use concurrency in Azure Synapse pipelines?

How to prevent concurrent pipeline execution?Concurrency This week I had a discussion with a colleague about how we can now make sure that a Pipeline does not start when it's already started. He then indicated, have you ever thought of the concurrency option?  I've...

Updated competency exams and certifications Data Platform and Data Analytics for 2020

Retiring and new exams and certifications as of June 30 2020 A lot of Exams and certifications for Data Platform and Data Analytics are retiring on June 30 2020. All retired Exams and certifications will remain eligible for competency attainment and renewal until June...

Azure Purview March Updates

Azure Purview updatesAnnouncements Last week during SQLBITS, quite a few new updates were announced. I would like to include you in these announcements.March updates Support for SAP Business Warehouse (Preview) Blogpost:...

Calculate Workingdays including Holidays with T-SQL

Calculate Workingdays between 2 Date columnsRecently I have been getting some questions from my customers, can I calculate the number of workdays between 2 dates? Of course my answer was, yes you can. But I do want certain closing dates and holidays of our company not...

LEAP 2018

Lead Enterprise Architect ProgramFrom the 4th of December until the 6th of December I've joined the LEAP (Lead Enterprise Architect Program) 2018 Data/AI from Microsoft NL. Three  days on the Microsoft Campus in Redmond joining business and technical sessions from a...

ADF: Get Metadata Activity stopped working

Meta Data ActivityToday my pipelines in Azure Data Factory (ADF) suddenly stopped working. The output structure was not found. Quit strange while these pipelines have been running for weeks.    Invalid Template After debugging my Pipeline I found out the...