Views:

Applies to Product - Dynamics 365 for Finance and Operations (on-premises)


What’s happening?
Customers are experiencing errors related to the Spark compute pool in Azure Synapse workspace, specifically receiving messages indicating that the compute pool is insufficient or unavailable, leading to failures in updating tables.
 

Reason:
The underlying causes for this may include: - Insufficient compute capacity in the Spark pool to handle the workload. - The Spark compute pool being paused or deleted, which prevents table updates. - Configuration issues within the Synapse workspace that restrict scaling or access.
 

Resolution:
To resolve this related to the Spark compute pool, follow these steps: 1. Verify Spark Pool Availability: - Ensure that the Spark pool with the specified name exists. If it has been deleted, recreate the Spark pool with the same name.

  1. Request Capacity Increase:
  2. Go to the Azure portal and create a new support ticket.
  3. Select the issue type as "Service and subscription limits (quotas)" and the quota type as "Azure Synapse Analytics".
  4. In the Details tab, choose the quota type as "Apache Spark (vCore) per workspace", select your workspace, and request the desired quota increase.
  5. Adjust Auto-Scaling Settings:
  6. In Synapse Studio, navigate to the "Manage" hub.
  7. Under "Apache Spark pools", select the Spark pool you want to configure.
  8. Adjust the auto-scaling settings by specifying the minimum and maximum number of nodes to allow the pool to scale based on workload demand.
  9. Monitor Resource Utilization:
  10. Continuously monitor metrics such as Total Pending CPU, Total Pending Memory, Total Free CPU, and Total Free Memory to ensure that autoscaling is functioning as expected.
  11. Adjust the minimum and maximum number of nodes or dynamic allocation settings based on observed workload and performance.
  12. Check Network Configuration:
  13. Review the IP whitelist and DEP settings in the network configuration to ensure there are no restrictions that could affect the Spark pool's operation.
  14. Run Test Jobs:
  15. Attempt to run other Spark pool jobs to confirm that the Spark pool is operational and can handle tasks as expected.
If the issue persists after following these steps, further investigation may be required, and it is recommended to engage with the engineering team for additional support.