Hi Niels,
Could you share size of your segment in customer journey? A possible reason which cause delivery delay would be that there were large amount of recipients.
You could build a small size segment to test whether delivered time could be within 1 or 2 minutes.(or just use new Send now feature in marketing email)
In my trial envrionment, my test segment is always around 5 members and they can receive email in about 1 minute.
Also, check whether Time zone field is set to your timezone.
In addition, as alternative for send email with download link, you could instead set your file link as marketing form redirect URL, after people have submitted form successfully,
then a browser download dialog will immediately open.(Or redirect them to a custom page in Portal, and attach the link on the page)

Regards,
Clofly