Inisghtful question!
The Qur'an itself does not specify which prayer is the "middle prayer" (salat al-wusta) in Qur'an 2:238.
According to Shi'i hadith (via Imam al-Baqir and Imam al-Sadiq), the "middle prayer" is the noon (zuhr) prayer (and possibly specifically the Friday prayer).
This corresponds with the idea that the sun is in the middle of the sky at this time, and so it is a middle prayer.
Sunni scholars mostly say that it is the 'asr prayer, although some say it is the zuhr prayer, and a few say it is the fajr prayer (as you have suggested). These views are also based on hadith/interpretation of hadith and an interpretation of what is meant by 'middle'.
The Qur'an speaks to us all individually, and I don't think there is any problem if this ayah speaks to different people differently since it is reminding us to focus on our prayers. Some people may need more focus on their fajr prayer (possibly because they are sleepy!), while others may need more focus on their noon/afternoon prayers, because they are busy.
Allah knows best!