How to Publish Azure Data Factory Artifacts

No comments

Azure Data Factory artifacts can be edited and deployed using the Azure portal. However, as an enterprise solution, one would want the capability to edit and publish these artifacts using Visual Studio. Let’s check are options available to publish using Visual Studio.

Assuming you have the created a Data Factory project in Visual Studio and extracted the artifacts to this project:

Right click on the Project Name in Solution Explorer and select Publish.

Sign in using your Azure credentials.

If you have Admin privileges on the Data Factory resource, you can select an existing Data Factory and proceed to Publish.

If you do not have Admin privileges, you cannot publish using Visual Studio

In an enterprise environment, you may not want all the developers to have admin privileges. But developers need the capability to publish changes. This can be achieved using PowerShell scripts.

Here are the steps:

  • Open Azure Powershell
  • Enter Add-AzureAccount. This will open a window to sign into Azure. Use your credentials.
  • Enter Get-AzureSubscription to get a list of subscriptions
  • Select-AzureSubscription <SubscriptionName> to select the subscription
  • Navigate to the project folder where the Azure Data Factory artifacts are located using cd command
  • Enter Login-AzureRmAccount and login
  • Let’s create a variable so store the Resource Name and Azure Data Factory name

$var=Get-AzureRmDataFactory -ResourceGroupName <ResourceName> -Name <AzureDataFactoryName>

  • To publish a dataset enter: New-AzureRmDataFactoryDataset $df -File .\<DatasetName>.json

If a dataset exists, a warning message appears. Select Y to continue to publish

  • Similarly, New-AzureRmDataFactoryPipeline and New-AzureRmDataFactoryLinkedService can be used to publish a pipeline and linked service respectively

 

Here a few helpful resources:

Install Azure Powershell: https://azure.microsoft.com/en-us/documentation/articles/powershell-install-configure/

Azure Data Factory Cmdlets: https://msdn.microsoft.com/en-us/library/dn820234.aspx

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s