The issue you are facing is a serious one. You can upgrade GP 10.0 to GP 2013, but GP 2013 Utilities is not compatible with SQL Server 2016. The only way around this is to take the GP 10.0 databases and restore them on SQL Server 2008 R2 - You will need to first change the database compatibility level for all GP databases to 100 before backing them up and restoring them onto SQL Server 2008 R2. You will also need to apply GP 10.0 Service Pack 5 in order for GP 10.0 Utilities to be able to run and also recommended prior to upgrading to GP 2013.
Now that you have your GP 10.0 databases on SQL Server 2008 R2, you can run the upgrade to GP 2013 R2 (12.00.1745). Once you upgrade to GP 2013 R2, you can then, once again, back up and restore these databases onto SQL Server 2016 again, then run the GP 2016 upgrade.
In order to complete the transfer of databases between servers (in both cases), I suggest you take a look at KB article 878449.