Azure Data Factory Pipe line run error - Pipeline execution throttled. If the concurrency value is set for a pipeline, a maximum of 100 runs will be queued for execution.

(0) Share

Report

Posted on by Mythe

One of the ADF pipe lines failed and the error message is "Pipeline execution throttled. If the concurrency value is set for a pipeline, a maximum of 100 runs will be queued for execution."

The pipe line when run in the debug mode runs but when run through trigger , it fails with this error.

I have the same question (0)

All responses (2)

Answers (1)

Suggested answer

Mythe 77 on at

Like (0)

Report

RE: Azure Data Factory Pipe line run error - Pipeline execution throttled. If the concurrency value is set for a pipeline, a maximum of 100 runs will be queued for execution.

Hi Steve,

Thanks for the suggestion , I checked the concurrency value and it is set to 1 and pipeline is set to run every 15 minutes. One of the executions was running for a long time and the further triggers exceeded the 100 limit and causing the error.

After cancelling the long running process , the queue freed up. Also we might have to set the trigger to run every 30 minutes because the pipeline is taking longer time to complete.

Was this reply helpful? Yes No
Verified answer

Community Member on at

Like (0)

Report

RE: Azure Data Factory Pipe line run error - Pipeline execution throttled. If the concurrency value is set for a pipeline, a maximum of 100 runs will be queued for execution.

Hi Mythe,

According to the error message, you set the concurrency value, right? For example, you set concurrency value to 1and your pipeline has to run for 10 minutes . Then you set your pipeline triggered every 5 minutes, the second task would be added to queue until your first task is completed(10 minutes). If the number of task in the queue is over 100, you would get the error. Please try to reduce the pipeline run time or add the pipeline triggered time to have a try.

Reference:

Azure subscription limits and quotas - Azure Resource Manager | Microsoft Docs

Data factory Run Fails - Microsoft Q&A

Was this reply helpful? Yes No