Collection of all ADF Mapping Data Flow videos

Collection of all ADF Mapping Data Flow videos

by | Feb 19, 2019 | Azure Data Factory

Erwin

by Erwin | Feb 19, 2019

ADF Mapping Flow

Did you use the Dataflow preview functionality in Azure Data Factory? This has recently be renamed to Mapping Data Flows.

 

All video's which the ADF team has created, are collected.

Start Here:

Debug and Prep:

Transformations:

Optimize:

Patterns:

Expressions:

 

Do you want to know more or you want to try out some examples, you can find them here

Feel free to leave a comment

Azure Data Factory: Generate Pipeline from the new Template Gallery

by | Feb 12, 2019 | Azure, Azure Data Factory

Erwin

Last week I mentioned that we could save a Pipeline to GIT. But today I found out that you can also create a Pipeline from a predefined Solution Template.

Template Gallery

These template will make it easier to start with Azure Data Factory and it will reduce development time when you start a new project.

Currently Microsoft has released the following templates:

Copy templates:

  • Bulk copy from Database
  • Copy multiple file containers between file-based stores
  • Delta copy from Database

Copy from <source> to <destination>

  • From Amazon S3 to Azure Data Lake Store Gen 2
  • From Google Big Query to Azure Data Lake Store Gen 2
  • From HDF to Azure Data Lake Store Gen 2
  • From Netezza to Azure Data Lake Store Gen 1
  • From SQL Server on premises to Azure SQL Database
  • From SQL Server on premises to Azure SQL Data Warehouse
  • From Oracle on premises to Azure SQL Data Warehouse

SSIS templates

  • Schedule Azure-SSIS Integration Runtime to execute SSIS packages

Transform templates

  • ETL with Azure Databricks

These templates can be found directly in the Azure Data Factory Portal:

Now you can select the option Create Pipeline from Template.

After selecting this option, all templates from the gallery but also the templates you saved yourselves, are visible.

Create a Pipeline from a Template

To start the creating of the template, click on the template you want to create. For this example, I have chosen for the template Bulk Copy from Database. A wizard will open which you have to follow.

The only thing you need to do right know is selecting the correct inputs. You can also create a new input from the Template wizard.

After selecting all the correct inputs you can finalize the template and the template will be added to your Factory.

Do you want to follow the detailed steps of creating this pipeline? The details can be found here.

Thanks so much for reading through this article today, and I hope you all take some time to try it out. It’s will make your life easier.

Azure Data Factory: Save a pipeline as a Template

by | Feb 8, 2019 | Azure Data Factory

Erwin

by Erwin | Feb 8, 2019

Saving a Pipeline

Another great improvement in Azure Data Factory. Saving you template!

How can you save your template?

First of all you need to connect your Azure Data Factory to  a GIT integration. Both Azure DevOps GIT and GitHub are supported. Please follow this link if you don’t have a Git Integration.

Select the template you want to save:

Name your template and save it to a Git Location of your choice:

The template you have saved can be found in Factory Resources.

With this new option it will  be easier to share Pipeline Activities with other Azure Data Factory instances.

And most important of saving your template it will save a lot of development time in your new instance.

 

Happy Saving

Feel free to leave a comment

Yeah My website is Finally Live

Yeah My website is Finally Live

by | Feb 1, 2019 | Divers

Erwin

by Erwin | Feb 1, 2019

Yeah, my first built website is LIVE

The last couple of weeks I've been working hard to design my website and finally the moment is there. My website live and I'm really proud.

 

Building my own website

The last couple of weeks/months I've been working hard... to design my own website and finally the moment is there! My website is live and I'm really proud that I can show it to you.

WordPress

I have never built a website with WordPress.
The last time I built a website was 10 years ago, with Joomla. I needed to learn everything from scratch and as everybody know, learning from scratch takes a while.
But with some help here and there, I made the building of the website my own. Thanks in part to all the advice from my colleague Joke, thanks for this.

Divi Builder

The website is build with Divi Builder, the Divi Builder Plugin is a stand-alone Drag & Drop page builder plugin which can be downloaded from https://www.elegantthemes.com/. This builder is really ease to use and I can certainly recommend it to everyone. You better invest time in writing content than in building pages for the website.

In case you have some comments or advice about the website, just let me know.

First articles and blogs

Also hope you are going to enjoy my blogs and articles. When you like to receive the latest news, please subscribe on the website or send me a message.

Erwin

Feel free to leave a comment

Calculate the Last Day of the Month using SQL

Calculate the Last Day of the Month using SQL

by | Jan 29, 2019 | Azure, Azure SQL Database

Erwin

by Erwin | Jan 29, 2019

Calculate the Last Day

Today I needed to calculate the last day of the previous month for a Customer.

Ever heard from the function EOMONTH?

Searching on the web I came across a function I never heard from before EOMONTH, this function can be used as of SQL Server 2012.

With this function you never have to worry anymore about how many days(28,29,30,31) your month has.

The function works as follows:

Defined date:

declare @datum datetime = '2018.11.01'

SELECT EOMONTH(DATEADD(mm, -1, @datum)) as LastDayPreviousMonth

Getdate()

SELECT EOMONTH(DATEADD(mm, -1, getdate())) as LastDayPreviousMonth

Current Month

SELECT EOMONTH(getdate()) as LastDayCurrentMonth

 

 

 

Feel free to leave a comment