Text area boxes in CRM can be configured on the form to either be plain/standard, or to use the Rich Text field control. If a value is entered into a text area using the Rich Text field control, it will have HTML tags added to it (which is what allows the rich text). I agree this can be frustrating, especially when reporting or exporting from the system. You have two choices to fix this, depending on what behavior you are looking for,
1) Update your Case Quick Create form so that the Description field does not use the Rich Text Control OR
2) Update your Case Main form to use Rich Text
Which approach you take depends on whether you want rich text in your case description. There are pros and cons to having it or not.
You can update the control by opening the form editor and selecting the Description field. When selected, the settings on the right include a portion called "Components". In here is where you want to either add the Rich Text Control, or disable it depending on how you want to do this.
Hope this helps, and reduces the frustration a bit. If you run into issues updating this or have more questions let me know and I'll keep an eye out for responses!
Edit: One more quick note. The Rich Text control is added to specific forms, not to the field in general. This means some other forms for the Case table may have the Rich Text control already enabled for the Description. You can switch to other forms by clicking the dropdown near the record name at the top of the screen. Maybe explore some of those other OOB forms as well!