Hi Brad,
Yes, a setting for your Event Manager could be below, if you want to restrict user's right to delete, just set it to None.(No access is allowed)

In addition, here are my suggestions:
1. Security role privileges are cumulative,
which means if you assign another security role to your user(it already has Event Manager security role),
you should ensure that it hasn't delete privilege on contact entity.
2. There is an existing security role called Event Planner, you could take it as reference while setting up Event Manager.

Regards,
Clofly