Rerun Pipeline activities in Azure Data Factory
Rerun Pipeline activities in ADF!
As of today you can rerun or partially, yes you’re reading it correct partially, rerun you Azure Data Factory pipeline.
Where you previously had to run the entire Pipeline again, you can now run a part of the Pipeline. This can save a lot of time if many different activities are created within one pipeline. Another nice step forward, I'm curious what else is coming in the next months.
Visualized
Besides that you can rerun your Pipeline in Azure Data Factory in a easy way, you also have the possibilities to see your run, visualized in the Azure Data Factory Monitoring. This is a big improvement in my opinion.
Rerun a Pipeline
If you want to partially rerun a Pipeline, follow the steps below:
Select the Pipeline which has failed, go to the view activity runs and select the activity which failed.
Click on the Rerun Icon
You need to confirm that you want to rerun this activity.
The Pipeline will start and will first skip all the activities(the grey new icons in the upper right corner of each activity) in the Pipeline before your selected Activity.
Your Pipeline will now finalize all the activities from your newly defined starting point.
What else is new?
Monitor Rerun History
You can now view all the history reruns by clicking on the toggle to ‘View All Rerun History’.
By clicking on the red marked action, you can see all the History from an particular Pipeline run.
Thanks for reading.
Updated 10th of March:
Found a video on Channel9 which explains how to "Rerun activities inside your Azure Data Factory pipelines"
Feel free to leave a comment
1 Comment
Leave a Reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
Discover more from Erwin | Data & Intelligence
Subscribe to get the latest posts sent to your email.
How can we automate this rerun activity process. If an activity fails and the pipeline is retriggered, it should start from the fail activity automatically.
Looking for help on how to implement this using Python