Goodbye 2022, Hello 2023

Goodbye 2022, Hello 2023

Goodbye 2022

​Recap

It's that time of year again to reflect on the past year. Also think it's really good, to see what you've done in the past year. It is also the time again to traditionally bake Oliebollen on this day, a Dutch Tradition that we do on New Year's Eve.

Oliebollen

This year we celebrate New Year's Eve with the family, we were supposed to go to relatives, but unfortunately these have all been felled by the flu. We will enjoy an evening of games, oliebollen and some drinks.

Looking back at 2022, I can say that I personally had a great year. Finally we are released from all COVID restrictions, we can go to physical events again, back to our customers, back to the office and see many people in real life again.
It was a very busy year in terms of work, this is also one of the main reasons that I have blogged much less than other years. The inspiration and energy is slowly coming back for this, so with some hope I can change this soon.

And as many would say, I should exercise more in the evenings and not always sit behind my laptop. My work is my hobby, so that will be difficult, but it will be one of my New Year's Good intentions for 2023. A better work life balance would be better for my health and maybe I should seek help in the form of a coach for that. If you have suggestions for me, let me know.

Speaking/Volunteering

This year I have spoken at several events on the topics of Azure Synapse Analytics and Microsoft Purview.

DataMinds (Virtual)

Did a virtual session for the DataMinds UG on Data Governance with Azure Purview( Yes is was on that moment still Azure Purview and not Microsoft Purview)

Data Minutes (Virtual)

The second event of the year, within my team I proposed to attend this event together. During the event there was a Last-Minute cancellation, at the request of Ben Weissman I gave a short session about Access Control in Azure Synapse. It's a fun event where you have a bunch of 10 minute blocks. Works very inspiring.

Data Toboggan (virtual)

Talked about access control in Azure Synapse Analytics. Data Toboggan is one of the events that is 100% focused on Azure Synapse Analytics.

SQL BITS

This year I volunteered at SQLBits for the first time and became part of the Orange family and as you can see in the picture it is a very big family. In addition to volunteering, I was also asked by Microsoft to present a session during SQL Bits on Cloud Scale Analytics solutions. Thanks again Tony and Wee Hyong for the invite.

SQLBITS Orange Team 2022 SQLBITS Orange Team 2022

Data Saturday Stockholm

My first time in Stockholm, great event. Bra så alla tillbaka personligen This means great, so everyone back personally.

DataGrillen

When we say: Data, bratwurst and beer, we are of course talking about DataGrillen. After more than 2 years of absence, it was that time again in recent days, with speakers from all over the world, beautiful weather and a large group of participants.
And as usual with this event, the first day ends with a barbecue for all participants.
By the way, I can't imagine a better event to celebrate my 50th birthday. It is therefore a great pity that there will be no DataGrillen next year. Hopefully again soon, in any form.

Datagrillen 2022 Datagrillen 2022

Scottish Summit

First time in Glasgow, second time speaking at the Scottish Summit on one of my favorite topics Data Governance and Purview.

DATA Scotland

Second time in 3 months coming to Glasgow and this time it was sunny, over 400 people in attendance and over 50 sessions.

DATA SCOTLAND 2022 DATA SCOTLAND 2022

DataMindsConnect

This year I volunteered for 2 days, the event was sold out with over 650 attendees. For us as Dutch people, the event is easily accessible by car, which means that many Dutch people are also present as participants.

Experts Living in the Netherlands

This was a very big IT event in the Netherlands, my employer InSpark was one of the sponsors. But even better to mention that during this event I was allowed to give a session with my colleague Albert, this was the first time that I presented together with a colleague. And this is certainly worth repeating.

Pass Data Community Summit

I've always wanted to speak at the Pass Data Community Summit and it's a dream come true and definitely one of my bucket list items. I had 3 sessions on Microsoft Purview. The event was great again, had great conversations and met so many old and new people again. Nice thing about this event was that a MVP pre day was also organized on the Microsoft Campus, thanks again to Rie Merritt for the entire organization and putting together the agenda. It always remains a wonderful feeling to be on the Campus. Hopefully we can go to Seattle again next year. And let's not forget the day that as a community we showed our support to Hugo Kornelis #teamhugo. This was a very and beautifully moving moment, all messages on the socials, all blue shirts. Hugo is fighting Acute Myeloid Leukemia (AML.)

Pass Community Summit 2022 Pass Community Summit 2022

The above was not possible with the support from my wife and kids, you are also quite often on the road during the weekends and I would like to thank my employer InSpark very much as well, they are the ones who encourage and financially support all our MVPs, but also other colleagues, to speak at international events.

InSpark

At my employer InSpark, we have made great progress with the Data and AI team this year with Managed Oyxgen, our Data Platform as a Service. The 2023 roadmap for our solution looks impressive and I'm already looking forward to working on it with everyone.
👉 Within our team we can be proud that we:
✅ Having challenging and varied work.
✅ Added even more gears to our accelerators.
✅ Creating lots of space for your own input, innovation and creativity.
✅ Working and contributing to many interesting projects.
✅ Having a good working atmosphere.
✅ Even took yoga classes.
✅ Attend the coolest events and even speak there.

I am therefore very proud of the team that we have all achieved together this year, on to even more beautiful and innovative projects

And the above certainly contributes to the fact that I can close 2022 as a great year and I look forward to 2023 with great pleasure and enthusiasm.

I wish everyone a great 2023, on to a successful year

Rotterdam Vuurwerk Erasmusbrug.

SQL BITS 2022 Session recordings

SQL BITS 2022 Session recordings

Recordings SQL Bits 2022

All sessions of SQLBits 2022 have been made available to everyone and can now be viewed via their Youtube channel. Microsoft asked me to present me this session during SQL Bits in the Cloud Scale Analytics solution area.

Session Title:

Lake Database with Database Template and Mapping Data with Azure Synapse Analytics

Description:

Database templates in Azure Synapse Analytics are blueprints which can be used by organizations to plan, architect and design solutions.

How can we use these Database Templates in a day-to-day business, in order to speed up to automate this process? Map data tool can help us with that. The map data tool can generate a mapping data flow without having to start from a blank canvas. In this presentation, you will see how this all works in a step-by-step demo-based session.

During SQL Bits the Mapping Data tool was still in Preview, the great news is that this functionality is now GA.

SAVE THE DATE

SQLBits 2023 will back next year 14 – 18 March 2023, so mark you calendars.

Data:Scotland

Data:Scotland

Data: Scotland 2022

Microsoft Purview

Scotland’s Data Community Conference happened this year again in Glasgow. This years event was happening in  a sunny Glasgow, more then 400 attendees and more then 50 sessions.

It was great to see so many people live again. I presented a session on one of my favorite subjects Microsoft Purview. My session was well attended, the slides can be found in the link below.

My sessions at Pass Data Community Summit

My sessions at Pass Data Community Summit

A hybrid conference in Seattle and online

This year's PASS Data Community Summit is more than a conference – it's a homecoming. Reconnect with old friends, build new relationships, gain new skills, and get the world-class training you need to take that next step in your data career. With 3 different themes, 6 different tracks and 9 Learning Path ways it promises to be a great Summit.

InSpark

When the Call for Speakers on March 10 was announced, I immediately called my CTO of InSpark, my employer, whether he thought it was okay to send in a number of sessions and whether InSpark could pay for the travel and accommodation costs. The answer was immediatly, awesome. great, do it. It's very nice to have an employer who facilitates you in this. And the good news is that my colleague Marco is joining me to Seattle.

InSpark_Logo_FC

My Sessions?

First of all, I was pleasantly surprised when I received an email stating that my session had been selected by an independent volunteer Program Committee. But that 3 of my sessions were directly selected is of course absolutely great. I've always wanted to speak at the Pass Data Community Summit and it's a dream come true and definitely one of my bucket list items.

How to use Data Lineage in Azure Purview?

Category: General Session

Location: In-Person

Type: Live Stream

Length: 60-minutes + 15-minute Q&A

Abstract:

The use of data Lineage is a hot topic for many organizations who struggle with answers to the following questions:

  • I want to adjust a measure, but where do I have to adjust it and where does the data come from?
  • What will be the effect on my data if I rename this column in the source?
  • Can I visually overview my Data Estate including how the data has been transformed?

As you can see, data lineage is used for different kinds of backward-looking scenarios, such as troubleshooting, root cause discovery in data pipelines, and debugging. Lineage is also used for data quality analysis, compliance, and 'what if' scenarios, often referred to as impact analysis. How can Azure Purview helps us to create these visual overviews to better understand our Data Estate? During this session, I will show you how to enable Data Lineage with Azure Purview, Azure Synapse Analytics, and how to use Custom Lineage components for unsupported data sources.

DataLineage-Purview

How to Integrate Azure Purview in Azure Synapse Analytics

Category: Lightning Talk

Location: Online

Type: On-Demand

Length: 10-minutes

Abstract:

In this short talk I'll show you how to integrate Azure Purview with Azure Synapse Analytics and what extra possibilities you will have of using both Azure Data Services

Data Governance with Azure Purview - Ask the Experts

Category: Panel Session

Location: In-Person

Length: 60-minutes + 15-minute Q&A

Abstract:

This is going to be another very nice session, together with Victoria, Wolfgang and Richard. The first time we did this session was during SQL Bits, we got a lot of interesting and diverse types. Ask the question live during the session or submit your questions in advance here https://forms.office.com/r/dTP38LnmsJ

Pass Data Community Summit

You can still register for the In-person or Online event, just cilck here.

And have a look into all the sessions and pre cons this year

Session Catalog

Pre Cons

In the meantime my flights and hotels are booked and I started the preparation of my sessions. But first things first, it is first time to celebrate holidays with family and friends.

Hopefully I will see you in Seatlle and otherwise online.

Azure Data Factory updates June

Azure Data Factory updates June

Erwin

by Erwin | Jun 30, 2022

Azure Data Factory updates

There have been quite a few updates in Azure Data Factory and Azure Synapse Analytics in the last few days.
Below is a summary of these updates:

 

Time-To-Live (TTL) on Integration Runtime with managed virtual network enabled

The new TTL functionality, currently in Preview, will save you a lot of time. Instead of provision compute for every copy activity execution, with this new option the provisioned compute will be reserved. You can set this new option on every Integration Runtime with Managed Virtual Network enabled. Very handy and will advise you to do this direclty. This option is also available in Azure Synapse Analytics.

adf-ttl

More can be found on the blog below:

Time-To-Live (TTL)

CI/CD improvement using Global Parameters in Azure Data Factory

We used to have a PowerShell script to add the Global parameters to our CI/CD pipelines. Now we can include these as Override Parameters in our Deployment/Release Pipeline. Very handy and easy.

Deprecated:

adf-gloabal-parameters-depr

New:

adf-gloabal-parameters-new

If you are using the arm-template-parameters-definition.json file to customize your parameters make sure you add the global parameters to this file. Otherwise you cannot override the parameters in the Deployment/Release Pipeline.

adf-gloabal-parameters-template

When we have created a Global Parameters Environment, we have now have the option to override this parameter option during the release:

-default_properties_Environment_value "Development"

More can be found on the blog below:

CI/CD improvement using Global Parameters in Azure Data Factory

Public Preview of the SAP CDC solution in Azure Data Factory

The new SAP ODP connector leverages SAP Operational Data Provisioning (ODP) framework, which is an established best practice for data integration within SAP landscapes. ODP provides access to a wide range of sources across all major SAP applications and comes with built-in CDC capabilities. In combination with the predefined data flow templates to process and update the changed records to any sink, this makes SAP data integration into Azure very much straight forward.

More can be found on the blog below:

Public Preview of the SAP CDC solution in Azure Data Factory

Azure Data Factory studio preview experience

You can now choose whether to enable preview experiences/functionality in your Azure Data Factory.

There are 2 ways how you can enable this preview experience:

When opening the portal you will see banner, just click on the Open settings to learn more and opt in. link

adf-preview-update-portal

Or open the setting pane on the right top and enable the toggle Azure Data Factory Studio preview update.

Current Preview Updates

Currenlty the following options are available as preview features:

Dataflow data experimental view

Configuration panel

The configuration panel will only have Data Preview that will automatically refresh when changes are made to transformations.

Before:

adf-preview-before

After:

adf-preview-after
Transformation settings

A pop-up will now appear when you want to change your settings

adf-preview-transformation-settings
Data preview

Data preview now includes Elapsed time (seconds) to show how long your data preview took to load.

adf-preview-update-data-preview

Pipeline experimental view

Adding activities

You now have the option to add an activity using the Add button in the bottom right corner of an activity in the pipeline editor canvas. Clicking the button will open a drop-down list of all activities that you can add.

 adf-preview-adding-activitities
Iteration & conditionals container view

And the last one is really awesome, you now view the activities in the contained iteration and conditional activities like For each, Switch, Until and Switch

adf-preview-activitities

More can be found on the detailed page below:

Azure Data Factory studio preview experience

Please visit the blog below for all the latest update on Synapse Analytics in June 2022:

Azure Synapse Analytics June Update 2022

 

If you have any questions left, please leave them in the comments below.

Feel free to leave a comment