As such building a (mathematically) perfect cube is not difficult. So, my inkling is that that 'imperfection' has a special meaning. I have searched many websites but found no answer.
There is a book entitled Sustainable Vernacular Architecture edited by Ali Sayigh that has a few lines on how the Kaaba is very like particular houses in Portugal called 'saloio' houses (believed to have been influenced by Middle Eastern architecture). Maybe Ali Sayigh could have more to say on this. I will try to find his contact iA.