Lets say you want to filter Grid with "Datasource" by "Field" using custom filter
1. Add new control to custom filter group in UI designer for extension object of your form
2. Create FormName_Handler class and put onModified event handler for you new control
3. Use FindOrCreate() method to add range to "Datasource" by its "Field", provide value for the range from your custom control
4. ExecuteQuery for the DataSource
This article should give you an idea: dynamics-resources.com/exploring-sysquery-class