Imam Ali (AS) is the Imam of all Muslims by order of Allah (SWT) declared by the Prophet Muhammad (SAWA) in hundreds of occasions including Ghadeer Khum in which the Proohet ordered all Muslims who were present to give allegiance to Ali (AS) as the leader of all who believed in the Prophet ( SAWA). The political plot of Saqeefa did deprive Imam Ali (AS) from his right in the government, but did not change the fact the he is the real Imam of Muslims. Being the real Imam, Imam Ali (AS) never dealt with the three Khalifs as real successors of the Prophet (SAWA) but just worldly rulers. He never gave them his Zakat nor prayed behind any of them. ( Al-Thahabi; Seyar A'lam Al-Nubalaa' , Vol 9, Page 284) stated: Ali did not attend with the three Khalifs any Jum'ah Prayer nor Jama'ah prayers.
Wassalam.