In answer to my question it seems that Azure requires both a recovery phone number and external email. I only had a phone number and this was forcing the behavior it seems?
So adding both a recovery email and phone number to the user seems to have stopped this behavior.
Either that or I tried enabling and then disabling the multifactor on the user, but either way seems back to normal now.
