I have found a resolution to this issue if you are willing are able to overwrite the OOB chart Cases by Queues.
Cases by Queues does default to the 'All Queues' view.
In my case, I needed to simply show the same chart in a side bar chart, rather than pie, and also needed to be able to drill down into it.
First I created the chart that I wanted to display by using the OOB chart creator. I exported that chart, and made a couple of tweaks to its XML that I needed(that's irreleveant thought).
So I then went to my dashboard and popped it out to view the records used to create the OOB chart Cases by Queues on my dashboard. This allowed my to change charts Im using with the All Items view and All Queues view. Once I change the chart im viewing the data with to the chart I created earlier(the one I need to use in place of the OOB cases by queues chart) I can then export the chart again.
Now this exported chart is the one I want to import and overwrite the OOB Cases by Queues chart with.
The key to doing this is making sure the visualization ID is the same as the OOB chart.
SO, I exported the OOB Cases by Queues chart
copied its visualization id
<visualizationid>{475E6F02-32AE-4081-8CB8-E3E24A9FBC0F}</visualizationid>
into my exported chart I want to overwrite it with.
When I now attempt to import this chart I want to keep CRM prompts me it may be a duplicate and asks if I want to replace or keep both.
You choose 'replace', and the OOB chart is now overwritten with my chart's XML, but is using the same Visualization ID that the original Cases by Queues chart was using. CRM treats this visualization id different, it is seems to be hard coded in CRM somehow that this visualization ID will default to ALL Queues for its subview, however any other charts you try to create that are not using this visualization id will default to Queues im a member of.
This seemed to be my only option in figuring this out.