The short answer is 'Yes'.
The longer answer is 'It depends'.
Yes, if your Audit log is huge compared to the rest of your database, then doing creates and queries can cause the system to slow down. It really depends on what kind of resources your SQL server has, and also if you've added any additional indexes to that table. SQL Server will ALWAYS want as much ram as you can possibly give it, and the server should be dedicated to CRM if possible.
Review this document from Microsoft about best practices with regards to performance
www.microsoft.com/.../details.aspx
Good luck!
