Hi, good question. Simple rule of thumb
Use OData when you need standard CRUD operations (read, create, update, delete) on existing D365FO entities. Example: external system reads customer list, creates a sales order, or updates an item price. No custom logic needed, just data in/out.
Use Custom Service when you need complex business logic, multiple operations in one call, or the data doesn't map cleanly to a single entity. Example: external system sends an order, and D365FO needs to validate stock, calculate price, apply trade agreements and confirm all in one API call. OData can't handle that cleanly.
In most cases OData is enough only go for custom service when OData's standard entity methods can't cover the requirement. Thanks,