Is it a good idea to remove un-wanted out-of-box fields from entities or let them be there and just remove them from forms and views?
You cannot remove/delete out of box fields from entity. you can remove those from forms or views.
From CRM you can only remove /delete custom fields.
My Blog: MS CRM Customization
As Mahadeo mentioned, you can't remove them entirely, but you can hide them.
If you want to make built-in fields hidden, you can remove them from the entity form, , remove them from any views they are on, and then set the "Searchable" field to "No". This will hide them from anywhere a user might try to find them,.
For entities, the simplest thing to do is remove "Read" permission for that entity to all user roles. Then the system will hide it entirely from users. Just make sure to do that for all user roles.
In order to crm system work properly system won't let you delete any of the system fields from the entity. If you dont want to use any of the system field just remove them from form and publish.