Use Global Parameters to Suspend and Resume your Analysis Services in ADF

Use Global Parameters to Suspend and Resume your Analysis Services in ADF

Suspend or Resume your Azure Analysis Services in Azure Data Factory

Last week one of my customer asked me if they could start or stop his Azure Analysis Services within Azure Data Factory. After a search on the internet I came across a blog from Joost, I’m using that blog as input for this post. Most of the credits goes to him. For me the focus was more on making it parameterized so that I can reuse these Pipelines for all of my customers. A couple of weeks ago the ADF team released the Global Parameters and in this post I’m going to use these parameters.

Global Parameters

Global parameters are constants across a data factory that can be consumed by a pipeline in any expression. They are useful when you have multiple pipelines with identical parameter names and values.

Creation and management of global parameters is done in the management hub.

ADF and GlobalParameter

ADF and GlobalParameter

Create above Global Parameters to build the Pipeline.

The following parameters can now be used across all your Data Factory Activities:

@pipeline().globalParameters.AAS_ResourceGroupName
@pipeline().globalParameters.AAS_ServerName
@pipeline().globalParameters.SubscriptionId

Build Pipeline

Create a new Pipeline PL_ACT_AAS_SUSPEND_GP 

Add a Parameter to the Pipeline Action to easily reuse this Pipeline to Resume our AAS.

ADF and GlobalParameters

Add a Web Activity.

Name = Suspend_AAS  (depends on the Action).

As Joost Mentioned in his blog we first have to define the Rest API Url in the Settings Tab.

https://management.azure.com/subscriptions/<xxx>/resourceGroups/<xxx>/providers/Microsoft.AnalysisServices/servers/<xxx>/<ACTION>?api-version=2017-08-01

The  <xxx> we need to replace with the Global Parameters and the <Action> with the Pipeline Parameter. The final Result will be:

https://management.azure.com/subscriptions/@{pipeline().globalParameters.SubscriptionId}/resourceGroups/@{pipeline().globalParameters.AAS_ResourceGroupName}/providers/Microsoft.AnalysisServices/servers/@{pipeline().globalParameters.AAS_ServerName}/@{pipeline().parameters.Action}?api-version=2017-08-01

Method = POST

Body = Create a dummy json message, it is not used by the Rest API.

ADF and GlobalParameters

Add Azure Data Factory as Contributor to Azure Analysis Services

Before you can debug or test your Pipelines you should add your ADF Instance with Contributor Role to your Azure Analysis Services.

ADF and GlobalParameters

After you have done this, you can Debug your Pipeline.

ADF and GlobalParameters

Error

I got an error because my AAS is already Suspend Or Resumed. We can solve this by adding a check, to check  what the Status of Analysis Services is.

Check Analysis Services Status

To check if our Analysis Services is already Suspended or Resumed we can at Web Activity to check the Status.

Add a Web Activity to your Pipeline  or make a copy of the existing Web Activity

Name = Check_Status_AAS

URL= Https://management.azure.com/subscriptions/@{pipeline().globalParameters.SubscriptionId}/resourceGroups/@{pipeline().globalParameters.AAS_ResourceGroupName}/providers/Microsoft.AnalysisServices/servers/@{pipeline().globalParameters.AAS_ServerName}?api-version=2017-08-01

Method = GET

ADF and GlobalParameters

Add IF Condition Activity (Check if AAS is Running)

ADF and GlobalParameters

Add an Expression on the If Condition Activity  @bool(startswith(activity(‘Check_AAS_Status’).output.properties.state,’Paused’))

This expression will check if the Analysis Services is Running or not. If we want to to Suspend our Analysis Services we have to add to the Web Activity Suspend_AAS  to False (Cut from the main frame and Paste in the False Activity). In case the Analysis Services is already Suspended we do nothing(True).

ADF and GlobalParameter

Debug your Pipeline, to see what is happening

Analysis Services was running, Web Activity Suspend AAS is called:

ADF and GlobalParameter

Analysis Services was already Paused/Suspended, no action required:

ADF and GlobalParameter

Create Pipeline to Resume your Analysis Services

Clone your PL_ACT_AAS_SUSPEND_GP and rename it to PL_ACT_AAS_RESUME_GP. Change your action Parameter to “Resume”.

ADF and GlobalParameter

Within the IF Condition move the Web Activity Suspend AAS from False to True and rename to Resume AAS

Debug to see if everything is working fine:

ADF and GlobalParameter

You have now learned how to Suspend and Resume your Azure Analysis Services Dynamically with the use of Global Parameters. Both Pipelines can be easily transferred to different customers.

Please feel free to download the Pipeline Templates here

If you’re already using a database where you store your Meta Data, then you have also the possibility to store the necessary parameters in the database. The only thing you need to do is to add a Lookup Activity to get the parameters from your database(and replace the global parameters with the output from the lookup activity)

ADF and GlobalParameter

 

Hopefully this article has helped you a step further. As always, if you have any questions, leave them in the comments.

 

Based on above article you should now also be able to build a Pipeline to Process your Analysis Services Model with some help from this blog or you download the Pipeline Template from here.

 

Parameterize Linked Services in ADF

Parameterize Linked Services in ADF

Parameters

by Erwin | Jul 9, 2020

Parameterize Linked Services 

For my Azure Data Factory solution I wanted to Parameterize properties in my Linked Services. Not all properties are Parameterized by default through the UI. But there's another way to achieve this.

 

 

Linked Service

Open your existing Linked Services.

Linked Services ADF

In this situation I want to Parameterize my FTP connection so that I can change the Host name based on a Azure Key Vault Secret.

By default is this not possible through the UI but in the Bottom of your Linked Service there is a Advanced box

Linked Services Advanced

 

If you enable this box you can start building your own connection, but also create your own Parameters for this connection.

How to start:

As a base we will use the default code or our connection

Linked Services Code

{
    "name": "LS_FTP_SOURCE",
    "properties": {
        "annotations": [
            "stage: none",
            "scenario: demo",
            "environments: dvlm"
        ],
        "type": "FtpServer",
        "typeProperties": {
            "host": "ftp.erwindekreuk.com",
            "port": 21,
            "enableSsl": true,
            "enableServerCertificateValidation": true,
            "authenticationType": "Basic",
            "userName": "ftp_down",
            "password": {
                "type": "AzureKeyVaultSecret",
                "store": {
                    "referenceName": "LS_AKV_OXGN",
                    "type": "LinkedServiceReference"
                },
                "secretName": "secretname"
            }
        }
    },
    "type": "Microsoft.DataFactory/factories/linkedservices"
}
Copy this code to your advanced box and enable the option Specify dynamic contents in JSON format/

Now you can start adding new parameters.

Linked Services Advanced

If you want to Parameterize your HOST name  connection you have to add in the top of the code a new Parameter, under the type of your connection

    "properties": {
        "type": "FtpServer",
        "parameters": {
            "ConnectionKeyvaultSecret": {
                "type": "string"
            }

After you have done this, you need to specify for which properties you want to use this parameter. In my case I want to read the parameter form my Azure Key Vault for my HOST propertie.

The JSON code below will now use above parameter as an input.

        "typeProperties": {
            "host": {
                "type": "AzureKeyVaultSecret",
                "store": {
                    "referenceName": "LS_AKV_OXGN",
                    "type": "LinkedServiceReference"
                },
                "secretName": {
                    "value": "@linkedService().ConnectionKeyvaultSecret",
                    "type": "Expression"
                }
            }

Save your connection and you will see that your UI is changed and that you have to define all your setting through the Advanced Editor.

If you test your connection you will now see that you have to fill in a parameter.

Linked Service Test Connection

And now you can create parameters of every TypeProperties within your connection.

The code below will create Parameters for your Host, Username and Password entries with Azure Key Vault enabled. For the authenticationType you have to choose between Basic and Anonymous. But can also at this to your Azure Key Vault.

{
    "name": "LS_FTP_SOURCE",
    "properties": {
        "type": "FtpServer",
        "parameters": {
            "ConnectionKeyvaultSecret": {
                "type": "string"
            },
            "UsernameKeyvaultSecret": {
                "type": "string"
            },
            "PasswordKeyvaultSecret": {
                "type": "string"
            },
            "authenticationType": {
                "type": "string"
            }
        },
        "annotations": [ ],
        "typeProperties": {
            "host": {
                "type": "AzureKeyVaultSecret",
                "store": {
                    "referenceName": "LS_AKV_OXGN",
                    "type": "LinkedServiceReference"
                },
                "secretName": {
                    "value": "@linkedService().ConnectionKeyvaultSecret",
                    "type": "Expression"
                }
            },
            "port": 21,
            "enableSsl": false,
            "enableServerCertificateValidation": false,
            "authenticationType": "@linkedService().authenticationType",
            "userName": {
                "type": "AzureKeyVaultSecret",
                "store": {
                    "referenceName": "LS_AKV_OXGN",
                    "type": "LinkedServiceReference"
                },
                "secretName": {
                    "value": "@linkedService().UsernameKeyvaultSecret",
                    "type": "Expression"
                }
            },
            "password": {
                "type": "AzureKeyVaultSecret",
                "store": {
                    "referenceName": "LS_AKV_OXGN",
                    "type": "LinkedServiceReference"
                },
                "secretName": {
                    "value": "@linkedService().PasswordKeyvaultSecret",
                    "type": "Expression"
                }
            }
        }
    }
}

Thanks for reading my blog post and have fun with Parameterization of your Linked Services in ADF.

Feel free to leave a comment