My Virtual Session at SQLBits 2020

by Erwin | Oct 5, 2020 | Events, SQLBits

SQL BITS 2020, the greatest data show

Last week was SQL BITS week.

After the event was moved from April to September, it eventually became a Virtual event. Setting up a Virtual event requires a lot of adjustments in the Organization.

Recording

All regular sessions had to be recorded in advance so that during the event itself it could not go wrong.
For some of us this was new and others have done it before. In any case, it was new to me, but the organization did everything it could, to help us with various sessions in which everything was explained and in which we could ask all kinds of questions. Thanks for that.

Is it strange to pre-record a session?

Yes, it is, you are trying to find an environment in which you have no ambient noise, a good microphone and a good camera. But you don't always have an influence on ambient sounds. And presenting to a webcam is strange.

After practicing my session again, I recorded my session in one go and did not edit anything in the session. After all, something can always go wrong or go in a different way in a session, even if you record it in advance. Once you start adjusting or editing that, the end is lost and a lot of time goes into it. But also the charm of a session is gone. After all, we are data professionals and not movie stars.

My Session

Back to the day itself, half an hour in advance I could log in to my session and must say that it was quite exciting. Would the video start, how do I come across and some more questions where in my head?
But all nerves for nothing, the session started right on time. In the meantime I had created a few polls which you could have the audience answer in between.
But also being able to answer the questions live during the session and sometimes even with a link to some extra information was now easy.

I was delighted to see at the outcome of the last poll, most of the people who attended my session will now start using Azure Key Vault in their day-to-day work. In the end that's why we do it for(help or advice others).

My presentation can me found here.

Conclusion

I saw a very great event, the quality of the sessions were very high. And there was so much choice, but luckily they will be soon available to watch On-Demand.
A big round of applause to the entire organization, you have organized a fantastic event with a super nice portal, including exhibitor hall, networking, chat rooms and much more. Thank you for having me and see you next year.

Watch the MS Ignite sessions on-demand

by Erwin | Oct 5, 2020 | Events, Microsoft

MS Ignite Sessions

MS Ignite 2020 was this year a virtual event.

Most of the sessions were live in the evenings and the other sessions were available at different times in different time zones. Compliments to the MS Ignite team for organizing such a great event

Most of the sessions are ready to watch on demand. If you have not done this yet, here is the link to be able to do this.

Use Global Parameters to Suspend and Resume your Analysis Services in ADF

by Erwin | Sep 16, 2020 | Azure, Azure Analyis Services, Azure Data Factory

Suspend or Resume your Azure Analysis Services in Azure Data Factory

Last week one of my customer asked me if they could start or stop his Azure Analysis Services within Azure Data Factory. After a search on the internet I came across a blog from Joost, I'm using that blog as input for this post. Most of the credits goes to him. For me the focus was more on making it parameterized so that I can reuse these Pipelines for all of my customers. A couple of weeks ago the ADF team released the Global Parameters and in this post I'm going to use these parameters.

Global Parameters

Global parameters are constants across a data factory that can be consumed by a pipeline in any expression. They are useful when you have multiple pipelines with identical parameter names and values.

Creation and management of global parameters is done in the management hub.

Create above Global Parameters to build the Pipeline.

The following parameters can now be used across all your Data Factory Activities:

@pipeline().globalParameters.AAS_ResourceGroupName
@pipeline().globalParameters.AAS_ServerName
@pipeline().globalParameters.SubscriptionId

Build Pipeline

Create a new Pipeline PL_ACT_AAS_SUSPEND_GP

Add a Parameter to the Pipeline Action to easily reuse this Pipeline to Resume our AAS.

Add a Web Activity.

Name = Suspend_AAS (depends on the Action).

As Joost Mentioned in his blog we first have to define the Rest API Url in the Settings Tab.

https://management.azure.com/subscriptions/<xxx>/resourceGroups/<xxx>/providers/Microsoft.AnalysisServices/servers/<xxx>/<ACTION>?api-version=2017-08-01

The <xxx> we need to replace with the Global Parameters and the <Action> with the Pipeline Parameter. The final Result will be:

https://management.azure.com/subscriptions/@{pipeline().globalParameters.SubscriptionId}/resourceGroups/@{pipeline().globalParameters.AAS_ResourceGroupName}/providers/Microsoft.AnalysisServices/servers/@{pipeline().globalParameters.AAS_ServerName}/@{pipeline().parameters.Action}?api-version=2017-08-01

Method = POST

Body = Create a dummy json message, it is not used by the Rest API.

Add Azure Data Factory as Contributor to Azure Analysis Services

Before you can debug or test your Pipelines you should add your ADF Instance with Contributor Role to your Azure Analysis Services.

After you have done this, you can Debug your Pipeline.

Error

I got an error because my AAS is already Suspend Or Resumed. We can solve this by adding a check, to check what the Status of Analysis Services is.

Check Analysis Services Status

To check if our Analysis Services is already Suspended or Resumed we can at Web Activity to check the Status.

Add a Web Activity to your Pipeline or make a copy of the existing Web Activity

Name = Check_Status_AAS

URL= Https://management.azure.com/subscriptions/@{pipeline().globalParameters.SubscriptionId}/resourceGroups/@{pipeline().globalParameters.AAS_ResourceGroupName}/providers/Microsoft.AnalysisServices/servers/@{pipeline().globalParameters.AAS_ServerName}?api-version=2017-08-01

Method = GET

Add IF Condition Activity (Check if AAS is Running)

Add an Expression on the If Condition Activity @bool(startswith(activity('Check_AAS_Status').output.properties.state,'Paused'))

This expression will check if the Analysis Services is Running or not. If we want to to Suspend our Analysis Services we have to add to the Web Activity Suspend_AAS to False (Cut from the main frame and Paste in the False Activity). In case the Analysis Services is already Suspended we do nothing(True).

Debug your Pipeline, to see what is happening

Analysis Services was running, Web Activity Suspend AAS is called:

Analysis Services was already Paused/Suspended, no action required:

Create Pipeline to Resume your Analysis Services

Clone your PL_ACT_AAS_SUSPEND_GP and rename it to PL_ACT_AAS_RESUME_GP. Change your action Parameter to "Resume".

Within the IF Condition move the Web Activity Suspend AAS from False to True and rename to Resume AAS

Debug to see if everything is working fine:

You have now learned how to Suspend and Resume your Azure Analysis Services Dynamically with the use of Global Parameters. Both Pipelines can be easily transferred to different customers.

Please feel free to download the Pipeline Templates here

If you're already using a database where you store your Meta Data, then you have also the possibility to store the necessary parameters in the database. The only thing you need to do is to add a Lookup Activity to get the parameters from your database(and replace the global parameters with the output from the lookup activity)

Hopefully this article has helped you a step further. As always, if you have any questions, leave them in the comments.

Based on above article you should now also be able to build a Pipeline to Process your Analysis Services Model with some help from this blog or you download the Pipeline Template from here.

Speaking(Virtual) at SQL Saturday #963 Denmark

by Erwin | Sep 10, 2020 | Events, SQL Saturday

SQL Saturday 963 Denmark

PASS SQLSaturday is a free training event for professionals who use the Microsoft data platform. These community events offer content across data management, cloud and hybrid architecture, analytics, business intelligence, AI, and more.

My first virtual event

I like to interact during my session, so I'm curious if that will work. Last week I recorded my session for SQL Bits and that is quite strange when you look back. I am really looking forward to it, my session starts at 14:30.

The complete schedule can be found here. Are there is still time to register!!

My Session

Session Title:

Azure Key Vault, Azure Dev Ops and Data Factory how do these Azure Services work perfectly together!

Session Details

Can we store our Connectionstrings or BlobStorageKeys or other Secretvalues somewhere else then in Azure Data Factory(ADF)? Yes you can! You can store these valuable secrets in Azure Key Vault(AKV). But how can we achieve this in ADF? And finally how do we deploy our DataFactories in Azure Dev Ops to Test, Acceptance and Production environments with these Secrets ? Can this be setup dynamically? During this session I will give answers on all of these questions. You will learn how to setup your Azure Key Vault, connect these secrets in ADF and finally deploy these secrets dynamically in Azure Dev Ops. As you can see a lot to talk about during this session.

Do I see you on Saturday 26th of September?

SQL BITS 2022 Session recordings

Recordings SQL Bits 2022 All sessions of SQLBits 2022 have been made available to everyone and can now be viewed via their Youtube channel. Microsoft asked me to present me this session during SQL Bits in the Cloud Scale Analytics solution area. Session Title: Lake...

DataGrillen 2022

DataGrillen 2022 Microsoft Purview When we say: Data, bratwurst and beer, we are of course talking about DataGrillen. After more than 2 years of absence, it was time again in recent days, with speakers from all over the world with almost 50 sessions, good weather and...

My Session at DataMindsConnect 2019

DataMindsConnect 2019 Date: 7 and 8 th October Location: Lamot Mechelen Conference number 3 within 7 days and this time in Mechelen.The location for this conference is in an old beer brewery in the center of Mechelen. Datamindsconnect is the largest Dataplatform event...

Datagrillen: Data, Bratwurst und Beer

Data, Bratwurst and Beer An event that started 5 years ago as a small event, that has grown into an event with 200 participants, 50+ sessions, 5 tracks, 2 days and a BBQ in a small place in Germany, called Lingen. William Durkin and Ben Weissman are the main...

Datatoboggan 2021(Video)

Recording of my session during Data Toboggan 2021 Azure Purview Microsoft's answer to Data Governance and Data Lineage

Data Saturday Stockholm

Data Saturday Stockholm Bra så alla tillbaka personligen This means great so everyone back in person. Last weekend I had the honor to speak at Data Saturday in Stockholm. About one of my favourite topics Azure Synapse Analytics. It was nice to see everyone in person...

My virtual session at Data Toboggan

Data Toboggan This Saturday I've joined the Data Toboggan to talk about Azure Synapse Analytics. Azure Synapse Analytics Today I've been talking on how to deal with all the different roles in Azure Synapse Analytics during Data Toboggan. An event 100% focussed...

Speaking at SQL BITS 2022

SQL BITS 2022 We’re Hitting the Arcade SQL Bits is back this year in London from March 8-12 2022. SQLBits is the largest data conference in the world and this year's theme is to bring us back to our incandescent youth, so prepare to level up your data skills and reach...

Data Toboggan January 2022 (Video)

Recording of my session during the Data Toboggan January 2022 conference. Dealing with different roles in Azure Synapse Analytics

Data Saturday #1 Pordenone(Video)

Recording of my session during Data Saturday #1 Pordenone Azure Purview Microsoft's answer to Data Governance and Data Lineage

Parameterize Linked Services in ADF

by Erwin | Jul 9, 2020 | Azure, Azure Data Factory, Azure Synapse Analytics

Erwin

by Erwin | Jul 9, 2020

Parameterize Linked Services

For my Azure Data Factory solution I wanted to Parameterize properties in my Linked Services. Not all properties are Parameterized by default through the UI. But there's another way to achieve this.

Linked Service

Open your existing Linked Services.

In this situation I want to Parameterize my FTP connection so that I can change the Host name based on a Azure Key Vault Secret.

By default is this not possible through the UI but in the Bottom of your Linked Service there is a Advanced box

If you enable this box you can start building your own connection, but also create your own Parameters for this connection.

How to start:

As a base we will use the default code or our connection

{
    "name": "LS_FTP_SOURCE",
    "properties": {
        "annotations": [
            "stage: none",
            "scenario: demo",
            "environments: dvlm"
        ],
        "type": "FtpServer",
        "typeProperties": {
            "host": "ftp.erwindekreuk.com",
            "port": 21,
            "enableSsl": true,
            "enableServerCertificateValidation": true,
            "authenticationType": "Basic",
            "userName": "ftp_down",
            "password": {
                "type": "AzureKeyVaultSecret",
                "store": {
                    "referenceName": "LS_AKV_OXGN",
                    "type": "LinkedServiceReference"
                },
                "secretName": "secretname"
            }
        }
    },
    "type": "Microsoft.DataFactory/factories/linkedservices"
}

Copy this code to your advanced box and enable the option Specify dynamic contents in JSON format/

Now you can start adding new parameters.

If you want to Parameterize your HOST name connection you have to add in the top of the code a new Parameter, under the type of your connection

    "properties": {
        "type": "FtpServer",
        "parameters": {
            "ConnectionKeyvaultSecret": {
                "type": "string"
            }

After you have done this, you need to specify for which properties you want to use this parameter. In my case I want to read the parameter form my Azure Key Vault for my HOST propertie.

The JSON code below will now use above parameter as an input.

        "typeProperties": {
            "host": {
                "type": "AzureKeyVaultSecret",
                "store": {
                    "referenceName": "LS_AKV_OXGN",
                    "type": "LinkedServiceReference"
                },
                "secretName": {
                    "value": "@linkedService().ConnectionKeyvaultSecret",
                    "type": "Expression"
                }
            }

Save your connection and you will see that your UI is changed and that you have to define all your setting through the Advanced Editor.

If you test your connection you will now see that you have to fill in a parameter.

And now you can create parameters of every TypeProperties within your connection.

The code below will create Parameters for your Host, Username and Password entries with Azure Key Vault enabled. For the authenticationType you have to choose between Basic and Anonymous. But can also at this to your Azure Key Vault.

{
    "name": "LS_FTP_SOURCE",
    "properties": {
        "type": "FtpServer",
        "parameters": {
            "ConnectionKeyvaultSecret": {
                "type": "string"
            },
            "UsernameKeyvaultSecret": {
                "type": "string"
            },
            "PasswordKeyvaultSecret": {
                "type": "string"
            },
            "authenticationType": {
                "type": "string"
            }
        },
        "annotations": [ ],
        "typeProperties": {
            "host": {
                "type": "AzureKeyVaultSecret",
                "store": {
                    "referenceName": "LS_AKV_OXGN",
                    "type": "LinkedServiceReference"
                },
                "secretName": {
                    "value": "@linkedService().ConnectionKeyvaultSecret",
                    "type": "Expression"
                }
            },
            "port": 21,
            "enableSsl": false,
            "enableServerCertificateValidation": false,
            "authenticationType": "@linkedService().authenticationType",
            "userName": {
                "type": "AzureKeyVaultSecret",
                "store": {
                    "referenceName": "LS_AKV_OXGN",
                    "type": "LinkedServiceReference"
                },
                "secretName": {
                    "value": "@linkedService().UsernameKeyvaultSecret",
                    "type": "Expression"
                }
            },
            "password": {
                "type": "AzureKeyVaultSecret",
                "store": {
                    "referenceName": "LS_AKV_OXGN",
                    "type": "LinkedServiceReference"
                },
                "secretName": {
                    "value": "@linkedService().PasswordKeyvaultSecret",
                    "type": "Expression"
                }
            }
        }
    }
}

Thanks for reading my blog post and have fun with Parameterization of your Linked Services in ADF.

Latest FMD Releases

Feel free to leave a comment

« Older Entries

Next Entries »

SQL BITS 2020, the greatest data show

Last week was SQL BITS week.

Recording

My Session

Conclusion

MS Ignite Sessions

MS Ignite 2020 was this year a virtual event.

Suspend or Resume your Azure Analysis Services in Azure Data Factory

Global Parameters

Build Pipeline

Add Azure Data Factory as Contributor to Azure Analysis Services

Error

Check Analysis Services Status

Create Pipeline to Resume your Analysis Services

SQL Saturday 963 Denmark

PASS SQLSaturday is a free training event for professionals who use the Microsoft data platform. These community events offer content across data management, cloud and hybrid architecture, analytics, business intelligence, AI, and more.

My first virtual event

My Session

Erwin

Parameterize Linked Services

For my Azure Data Factory solution I wanted to Parameterize properties in my Linked Services. Not all properties are Parameterized by default through the UI. But there's another way to achieve this.

Linked Service

Latest FMD Releases

Feel free to leave a comment

Categories