Hi Emma,
Besides the performance, I would like to raise another concern, that maybe is even more relevant, you should take in consideration the cost of storage in Dynamics CE as well.
I would proceed with this data importing only if it is really required to be storage in Dynamics, it means, if there is business requirements to manipulate such data inside Dynamics.
If the requirement is only to visualize this data inside Dynamics, you can find other implementations to visualize the data from your ERP in Dynamics Eg.: Virtual Entities
Regarding the performance, in the backend there is a MS SQL server running all the optimization algorithms and techniques of a regular SQL server to improve performance, as cache, index, statistics to optimize execution plans etc. In CRM application side, there is techniques to avoid impact as well, as the pagination of the records in the views, for example. I would not expect a huge impact for the end users, but as any SQL server, more records can always impact the execution time of select and updates operations.
Eventually you can perform tests and open a Support ticket to discuss with Microsoft if you verify a poor performance.