Hi Jeremiah,
Actually I agree with Deepesh, he mentioned the integral part, is it based on concept "To recipient" is aligned or not with the Regarding. Is yes, then it will be easier.
Jeremiah,
Now, hopefully this can help and refer to your exact question, let's see from other way, let's do query from another way.
Since you want to find all contact that being recipients after you send email but always "Pending Send" or "Failed", so let's start with finding all Contacts, not by Email, and match it with Activity Party, remember, all of the contacts/accounts if linked to the Activity (email is one of the activity) will become the Activity Party, so I will find filter the Contact by Participation Type and then link to the email and finally only filter the email status to Pending Send, failed, or any status you want.
So here is the final advanced find.

You can also select the Participation Type to "To Recipient" or even CC Recipient (if any)
And the Status reason to the status you want.
Then, it will show all of the contacts that you sent email to them but still pending Send or Failed, no matter your regarding is themselves (contact record itself), their Accounts, or even as you mentioned before, regarding as Case.

Remember, we didn't link to any Regarding, so it can be anything as long as the recipients are Contacts, if you want to get the Account, then change the entity from Contact to Account.
Then just insert the Contact email address to display column to get your recipient email result.
Once you see the result, you can see all the contacts that becoming your recipients but still failed to deliver and you can see the email address, either blank, valid, or just like dummy account, abc@example.com or firstname.lastname@domain.com, a common sense that usually if you put mandatory, the user can key in anything for the sake of saving the record.
So this is the example of the result

As we can see, Nancy Anderson is there (eventhough the email is regarding to Case, not herself record), maybe with fake email and another example, why it was failed? Emily Lee, for example, doesn't have valid email address (blank)
Hope this can help you.
If not, then we all know we can do query :
select ToRecipients, subject from FilteredEmail
Or you can link to the activity party if still can't get the data.
Thank you.