If you turn away towards the right or left side - regardless of whether your back is towards Qibla or not - due to forgetting, you should start a new prayer as soon as you remembers, if there is time left even for one unit (rakat). If there isn't time enough to pray a new prayer, or at least one rakat, then you should continue with the same prayer towards Qibla, and the prayer will not be qaẓa. Similar rule applies to the one who has deviated because of the external force.
If there is an excuse, like, forgetting or an external force, like a strong wind blowing, which turns him away from Qibla, the prayer will be valid if one has not deviated towards the right or the left. It is necessary that one returns to the direction of Qibla as soon as the excuse disappears.