Sometimes when you try to process big XML messages using File adapter you can get error:

The request failed with the following error: The formatter threw an exception while trying to deserialize the message: There was an error while trying to deserialize parameter http://schemas.microsoft.com/dynamics/2011/01/services:serviceName. The InnerException message was 'Maximum number of items that can be serialized or deserialized in an object graph is '65536'. Change the object graph or increase the MaxItemsInObjectGraph quota. '. Please see InnerException for more details. 

 The issue happens because we are hitting the configured WCF settings. To change the settings we need to:

1. Unfortunately with Dynamics AX 2012 RTM version there is no way to set up WCF settings (like timeouts, MaxIremsInObjectGraph, etc.) To achieve it we need to install application hotfix delivered with KB 2708687. Please contact Microsoft Dynamics Ax Support if hotfix is still not available on PartnerSource. After installing hotfix on "Inbound ports" form for File Adapter you should find new button "Configure AOS" which will allows you to open WCF editor.

2. Go to System administration > Setup > Services and Application integration framework > Inbound ports

3. Disable your file adapter port (if it is enabled)

4. Click on button “Configure AOS” (this button has been added together with hotfix)

-          In WCF editor go to: Advanced > Endpoint behaviors

-          Click on "New Endpoint behavior Configuration"

-          Change name to FileSystemBehaviour

-          Click on "Add" button

-          Select dataContractSerialiazer

-          Double click on dataContracSerializer

-          Change property value MaxItemsInObjectGraph from 65536 to 534288:

-          Select Client > Endpoints > DefaultServiceGroupEndpoint and change property: BehaviorConfiguration to FileSystemBehaviour:

-          go to: Advanced > Service behaviors > DefaultServiceGroupBehavior

-          Right click and select "Add Service behavior Element Extension"

-          Select dataContractSerialiazer

-          Double click on dataContracSerializer

-          Change property value MaxItemsInObjectGraph from 65536 to 534288

After applying above changes and enabling the port again you should not get the issue any more.

To learn more about MaxItemsInObjectGraph please follow: http://msdn.microsoft.com/en-us/library/system.runtime.serialization.datacontractserializer.maxitemsinobjectgraph.aspx