My sessions at Pass Data Community Summit

My sessions at Pass Data Community Summit

A hybrid conference in Seattle and online

This year's PASS Data Community Summit is more than a conference – it's a homecoming. Reconnect with old friends, build new relationships, gain new skills, and get the world-class training you need to take that next step in your data career. With 3 different themes, 6 different tracks and 9 Learning Path ways it promises to be a great Summit.

InSpark

When the Call for Speakers on March 10 was announced, I immediately called my CTO of InSpark, my employer, whether he thought it was okay to send in a number of sessions and whether InSpark could pay for the travel and accommodation costs. The answer was immediatly, awesome. great, do it. It's very nice to have an employer who facilitates you in this. And the good news is that my colleague Marco is joining me to Seattle.

InSpark_Logo_FC

My Sessions?

First of all, I was pleasantly surprised when I received an email stating that my session had been selected by an independent volunteer Program Committee. But that 3 of my sessions were directly selected is of course absolutely great. I've always wanted to speak at the Pass Data Community Summit and it's a dream come true and definitely one of my bucket list items.

How to use Data Lineage in Azure Purview?

Category: General Session

Location: In-Person

Type: Live Stream

Length: 60-minutes + 15-minute Q&A

Abstract:

The use of data Lineage is a hot topic for many organizations who struggle with answers to the following questions:

  • I want to adjust a measure, but where do I have to adjust it and where does the data come from?
  • What will be the effect on my data if I rename this column in the source?
  • Can I visually overview my Data Estate including how the data has been transformed?

As you can see, data lineage is used for different kinds of backward-looking scenarios, such as troubleshooting, root cause discovery in data pipelines, and debugging. Lineage is also used for data quality analysis, compliance, and 'what if' scenarios, often referred to as impact analysis. How can Azure Purview helps us to create these visual overviews to better understand our Data Estate? During this session, I will show you how to enable Data Lineage with Azure Purview, Azure Synapse Analytics, and how to use Custom Lineage components for unsupported data sources.

DataLineage-Purview

How to Integrate Azure Purview in Azure Synapse Analytics

Category: Lightning Talk

Location: Online

Type: On-Demand

Length: 10-minutes

Abstract:

In this short talk I'll show you how to integrate Azure Purview with Azure Synapse Analytics and what extra possibilities you will have of using both Azure Data Services

Data Governance with Azure Purview - Ask the Experts

Category: Panel Session

Location: In-Person

Length: 60-minutes + 15-minute Q&A

Abstract:

This is going to be another very nice session, together with Victoria, Wolfgang and Richard. The first time we did this session was during SQL Bits, we got a lot of interesting and diverse types. Ask the question live during the session or submit your questions in advance here https://forms.office.com/r/dTP38LnmsJ

Pass Data Community Summit

You can still register for the In-person or Online event, just cilck here.

And have a look into all the sessions and pre cons this year

Session Catalog

Pre Cons

In the meantime my flights and hotels are booked and I started the preparation of my sessions. But first things first, it is first time to celebrate holidays with family and friends.

Hopefully I will see you in Seatlle and otherwise online.

Azure Data Factory updates June

Azure Data Factory updates June

Synapse

by Erwin | Jun 30, 2022

Azure Data Factory updates

There have been quite a few updates in Azure Data Factory and Azure Synapse Analytics in the last few days.
Below is a summary of these updates:

 

Time-To-Live (TTL) on Integration Runtime with managed virtual network enabled

The new TTL functionality, currently in Preview, will save you a lot of time. Instead of provision compute for every copy activity execution, with this new option the provisioned compute will be reserved. You can set this new option on every Integration Runtime with Managed Virtual Network enabled. Very handy and will advise you to do this direclty. This option is also available in Azure Synapse Analytics.

adf-ttl

More can be found on the blog below:

Time-To-Live (TTL)

CI/CD improvement using Global Parameters in Azure Data Factory

We used to have a PowerShell script to add the Global parameters to our CI/CD pipelines. Now we can include these as Override Parameters in our Deployment/Release Pipeline. Very handy and easy.

Deprecated:

adf-gloabal-parameters-depr

New:

adf-gloabal-parameters-new

If you are using the arm-template-parameters-definition.json file to customize your parameters make sure you add the global parameters to this file. Otherwise you cannot override the parameters in the Deployment/Release Pipeline.

adf-gloabal-parameters-template

When we have created a Global Parameters Environment, we have now have the option to override this parameter option during the release:

-default_properties_Environment_value "Development"

More can be found on the blog below:

CI/CD improvement using Global Parameters in Azure Data Factory

Public Preview of the SAP CDC solution in Azure Data Factory

The new SAP ODP connector leverages SAP Operational Data Provisioning (ODP) framework, which is an established best practice for data integration within SAP landscapes. ODP provides access to a wide range of sources across all major SAP applications and comes with built-in CDC capabilities. In combination with the predefined data flow templates to process and update the changed records to any sink, this makes SAP data integration into Azure very much straight forward.

More can be found on the blog below:

Public Preview of the SAP CDC solution in Azure Data Factory

Azure Data Factory studio preview experience

You can now choose whether to enable preview experiences/functionality in your Azure Data Factory.

There are 2 ways how you can enable this preview experience:

When opening the portal you will see banner, just click on the Open settings to learn more and opt in. link

adf-preview-update-portal

Or open the setting pane on the right top and enable the toggle Azure Data Factory Studio preview update.

Current Preview Updates

Currenlty the following options are available as preview features:

Dataflow data experimental view

Configuration panel

The configuration panel will only have Data Preview that will automatically refresh when changes are made to transformations.

Before:

adf-preview-before

After:

adf-preview-after
Transformation settings

A pop-up will now appear when you want to change your settings

adf-preview-transformation-settings
Data preview

Data preview now includes Elapsed time (seconds) to show how long your data preview took to load.

adf-preview-update-data-preview

Pipeline experimental view

Adding activities

You now have the option to add an activity using the Add button in the bottom right corner of an activity in the pipeline editor canvas. Clicking the button will open a drop-down list of all activities that you can add.

 adf-preview-adding-activitities
Iteration & conditionals container view

And the last one is really awesome, you now view the activities in the contained iteration and conditional activities like For each, Switch, Until and Switch

adf-preview-activitities

More can be found on the detailed page below:

Azure Data Factory studio preview experience

Please visit the blog below for all the latest update on Synapse Analytics in June 2022:

Azure Synapse Analytics June Update 2022

 

If you have any questions left, please leave them in the comments below.

Feel free to leave a comment

DataGrillen 2022

DataGrillen 2022

DataGrillen 2022

Microsoft Purview

When we say: Data, bratwurst and beer, we are of course talking about DataGrillen. After more than 2 years of absence, it was time again in recent days, with speakers from all over the world with almost 50 sessions, good weather and a large group of participants, quite a bit of knowledge has been shared again.
And as is traditional with this event, the first day will be finalized with a barbecue for all participants.
The organization is in the hands of Ben and WIlliam and you can leave that up to them. Everything was well organized again.
My session on Microsoft Purview was well attended, the slides can be found in the link below.

Datagrillen_4

Azure Purview March Updates

Azure Purview March Updates

Azure Purview updates

Announcements

Last week during SQLBITS, quite a few new updates were announced. I would like to include you in these announcements.

March updates

Support for SAP Business Warehouse (Preview)

Blogpost:

https://techcommunity.microsoft.com/t5/azure-purview-blog/azure-purview-adds-support-for-sap-business-warehouse/ba-p/3253404

Documentation:

https://docs.microsoft.com/en-us/azure/purview/register-scan-sap-bw

Azure Purview SAP BW

Dynamic lineage extraction from Azure SQL Databases (Preview)

Documentation:

https://docs.microsoft.com/en-us/azure/purview/register-scan-azure-sql-database?tabs=sql-authentication#lineagepreview

Video:

 

Certify assets in the Azure Purview data catalog

Blogpost:

https://techcommunity.microsoft.com/t5/azure-purview-blog/certify-assets-in-the-azure-purview-data-catalog/ba-p/3249460

Documentation:

https://docs.microsoft.com/en-us/azure/purview/how-to-certify-assets

Purview_Certified_Datasets

Ability to delete child terms when parent term is deleted

Documentation:

https://docs.microsoft.com/en-us/azure/purview/how-to-create-import-export-glossary

Connect to and manage an on-premises SQL server instance in Azure Purview

Documentation:

https://docs.microsoft.com/en-us/azure/purview/register-scan-on-premises-sql-server

Approval workflow for business terms (Preview)

Before you can start Authoring your workflows make sure you the correct user to the role assignment Workflow administrators, if you haven't done that correctly the option will be greyed out.

Purview_Workflow_Admin

workflow-authoring-experience

Blogpost:

Approval workflow for business glossary

Documentation:

https://docs.microsoft.com/en-us/azure/purview/how-to-workflow-business-terms-approval

Self-service data access workflows for hybrid data estates (Preview)

Purview-data-access-request

Documentation:

https://docs.microsoft.com/en-us/azure/purview/how-to-workflow-self-service-data-access-hybrid

Azure integration runtime supports scanning more source types

Azure Purview now supports scanning Snowflake, Salesforce, PostgreSQL, MySQL, Cassandra and Looker using managed Azure integration runtime.

Blogpost:

https://techcommunity.microsoft.com/t5/azure-purview-blog/azure-integration-runtime-supports-scanning-more-source-types/ba-p/3254148

Documentation:

https://docs.microsoft.com/en-us/azure/purview/manage-integration-runtimes

Localization

Azure Purview is localized in 18 languages. To change the language used, go to the Settings from the top bar and select the desired language from the dropdown.

Purview-Localization

Blogpost:

https://techcommunity.microsoft.com/t5/azure-purview-blog/localization-generally-available-in-azure-purview-studio/ba-p/3249453

Documentation:

https://docs.microsoft.com/en-us/azure/purview/use-azure-purview-studio#localization