Among the calamities that have befell the Muslim Ummah is that oppressors have usurped positions and taken control of holy places. This dates all the way back to the Umayyad Dynasty.
As you have read, during a certain time in the Abassid era, Ziyarah of Imam Husain (a.s.) was restricted, and if one was to visit they would have to offer their hand to be cut off. People still went. They sacrificed everything to keep holy places full of the true followers of Islam.
This is how we need to think of visiting the holy land of Meccah and Medinah. If we vacate it from our presence, it will further strengthen the enemies of Islam. It is for this reason that all our Jurists, and throughout history, have always said Hajj is wajib should one meet the fiqhi requirements of 'Istita'ah.
I believe it is a wrong misconception that performing Hajj pilgrimage validates the Saudi regime, or any other fallacy arguments that would deprive one from performing their Wajib obligation.
Please re-think your decision, and refer to the juristic views of our Fuqaha' in regards to the obligation of Hajj, and how we will be able to contribute so much by our presence in the holy land of Meccah and Medina.
And Allah knows best.