This Thursday May 13th 2021 I've been speaking during Cloud Lunch and Learn Marathon 2021. It was the first Cloud Lunch and Learn Marathon conference, more then 1200 registered attendees, 24hours Live and pre recorded sessions. Amazing. A big compliment to the organizers and thank you for having me.
This Saturday I've been speaking during Scottisch Summit 2021. It was my first Summit, but is was a great event, with more than 400 sessions covering the full Microsoft Stack in 7 different language English, Spanish, German, French, Italian, Portuguese and Polish. Proud that I was to able to join and to present.
Azure Data Factory
I presented a session on if there is a way that we can build our Azure Data Factory all with parameters based on MetaData?
In the beginning of my sessions the audio wasn't that well. I just double checked my uploaded recording and in there audio was fine.
Suspend or Resume your Azure Analysis Services in Azure Data Factory
Last week one of my customer asked me if they could start or stop his Azure Analysis Services within Azure Data Factory. After a search on the internet I came across a blogfrom Joost, I’m using that blog as input for this post. Most of the credits goes to him. For me the focus was more on making it parameterized so that I can reuse these Pipelines for all of my customers. A couple of weeks ago the ADF team released the Global Parameters and in this post I’m going to use these parameters.
Global Parameters
Global parameters are constants across a data factory that can be consumed by a pipeline in any expression. They are useful when you have multiple pipelines with identical parameter names and values.
Creation and management of global parameters is done in the management hub.
Create above Global Parameters to build the Pipeline.
The following parameters can now be used across all your Data Factory Activities:
Add IF Condition Activity (Check if AAS is Running)
Add an Expression on the If Condition Activity@bool(startswith(activity(‘Check_AAS_Status’).output.properties.state,’Paused’))
This expression will check if the Analysis Services is Running or not. If we want to to Suspend our Analysis Services we have to add to the Web Activity Suspend_AAS to False (Cut from the main frame and Paste in the False Activity). In case the Analysis Services is already Suspended we do nothing(True).
Debug your Pipeline, to see what is happening
Analysis Services was running, Web Activity Suspend AAS is called:
Analysis Services was already Paused/Suspended, no action required:
Create Pipeline to Resume your Analysis Services
Clone your PL_ACT_AAS_SUSPEND_GP and rename it to PL_ACT_AAS_RESUME_GP. Change your action Parameter to “Resume”.
Within the IF Condition move the Web Activity Suspend AAS from False to True and rename to Resume AAS
Debug to see if everything is working fine:
You have now learned how to Suspend and Resume your Azure Analysis Services Dynamically with the use of Global Parameters. Both Pipelines can be easily transferred to different customers.
Please feel free to download the Pipeline Templates here
If you’re already using a database where you store your Meta Data, then you have also the possibility to store the necessary parameters in the database. The only thing you need to do is to add a Lookup Activity to get the parameters from your database(and replace the global parameters with the output from the lookup activity)
Hopefully this article has helped you a step further. As always, if you have any questions, leave them in the comments.
Based on above article you should now also be able to build a Pipeline to Process your Analysis Services Model with some help from this blog or you download the Pipeline Template from here.
For my Azure Data Factory solution I wanted to Parameterize properties in my Linked Services. Not all properties are Parameterized by default through the UI. But there's another way to achieve this.
Linked Service
Open your existing Linked Services.
In this situation I want to Parameterize my FTP connection so that I can change the Host name based on a Azure Key Vault Secret.
By default is this not possible through the UI but in the Bottom of your Linked Service there is a Advanced box
If you enable this box you can start building your own connection, but also create your own Parameters for this connection.
How to start:
As a base we will use the default code or our connection
After you have done this, you need to specify for which properties you want to use this parameter. In my case I want to read the parameter form my Azure Key Vault for my HOST propertie.
The JSON code below will now use above parameter as an input.
Save your connection and you will see that your UI is changed and that you have to define all your setting through the Advanced Editor.
If you test your connection you will now see that you have to fill in a parameter.
And now you can create parameters of every TypeProperties within your connection.
The code below will create Parameters for your Host, Username and Password entries with Azure Key Vault enabled. For the authenticationType you have to choose between Basic and Anonymous. But can also at this to your Azure Key Vault.