Migrate Azure Storage to Azure Data Lake Gen2
Migrate Azure Storage to Storage Account with Azure Data Lake Gen2 capabilities
Does it sometimes happen that you come across a Storage Account where the Hierarchical namespace is not enabled or that you still have a Storage Account V1? In the tutorial below I describe the different steps that have recently become possible to perform this migration.
Azure Storage V1
The first step is to check what Account kind is currently deployed. If this is Storage (general purpose v1), we first need to Migrate the Storage account to V2, if this is already V2 then go to the next step.
You can click on change and a new window will pop-up.
Note: Choosing a storage access tier during account upgrade is free. Changing the storage access tier after the upgrade operation may result in changes to your bill.
Select the Tier you want to Migrate to, once you have done that start the Upgrade.
When the upgrade is successful, you will see that the Account kind is now StorageV2. We can now continue to the next step.
Azure Storage V2
To start the Migration click in the Taskbar on Data Lake Gen2 upgrade or click in the blob service properties on ‘Disabled’ for the Hierarchical namespace property.
The Migration window will open and we can start with step 1.
Take notice of the unsupported features/functionalities.
Agree with implications of Upgrading your Azure Data Lake Storage. Once this step is done we can continue with step 2, the validation.
If everything runs fine, you can start the upgrade step 3. If it fails check the errors. You need to download the error.json file to check which blobs are failing, mostly this are the unsupported functionalities or incompatible features.
{
“startTime”: “2021-08-04T18:40:31.8465320Z”,
“id”: “45c84a6d-6746-4142-8130-5ae9cfe013a0”,
“incompatibleFeatures”: [
“Blob Delete Retention Enabled”
],
“blobValidationErrors”: [],
“scannedBlobCount”: 0,
“invalidBlobCount”: 0,
“endTime”: “2021-08-04T18:40:34.9371480Z”
}
The upgrade will take a while, this mostly depends on how much data needs to be migrated.
At the end of the process you notice that the Hierarchical namespace is now enabled and can not be changed anymore.
Post Migration
Create new linked services in Azure Data Factory and Azure Synapse Analytics to make sure that you will use the DFS file system.
Change any other application to the correct End Point.
Test, test and Test all your workloads to make sure everything is working like expected.
Start migrating your Development Storage Account, test all the workloads, before you start Migrating your Production Storage account.
Like always, in case you have questions, leave them in the comments or send me a message.
Useful links
Upgrade to a general-purpose v2 storage account
Upgrade Azure Blob Storage with Azure Data Lake Storage Gen2 capabilities
