They base their claims on saying that companions such as Sulayman b. Surad and Rifa'a b. Shaddad wrote to Imam Hussain (as) inviting him to Kufa to be their Imam, but then deserted him at the last minute. They also cite the Tawwabun uprising as "proof" that the Shi'a deserted Imam Hussain (as).
First, I would question why someone is bringing this up. Are they asking because they have a genuine historical inquiry? Or is it just an attempt to prove "Sunnis are right, Shi'is are wrong" [or vice versa]? If it is the latter, people should be called on this at the outset.
Assuming it is a genuine question... First, the primary blame for a crime goes on the perpetrators. In this case, that is those who ordered the killing of Imam Husain (A) and those who struck blows. Yes, the people of Kufa had a secondary responsibility for not supporting the Imam, but it is rather revisionist to blame a bystander for a crime. (How often does that happen today!)
Second, the line between "Shi'i" and "Sunni" was not as firm then as it is today. While, early on, the concept of "Shi'a of 'Ali" was known, the concept of "Sunni" took time to develop. Many Muslims - in Kufa and otherwise - were favorable to Imam 'Ali (A) and his descendants because they had had a good experience with him as caliph, or because the Prophet (S) had respected Imam Husain (A), or because of a sort of tribal loyalty to the Prophet (S). However, they can't all be said to have been Shi'is, in the sense of having an unwavering, lifelong dedication to the Imam as the imam, or advocating Shi'i fiqh or theology. They were simply inclined towards him.
Basically, there was a big middle ground between what we would today consider "Shi'i" and "non-Shi'i" or "Sunni". There were some dedicated Shi'is, such as al-Mukhtar al-Thaqafi, who were genuinely unable to be present in Karbala for reasons such as imprisonment. Conversely, what happened in Karbala upset many Muslims, not just Shi'is. While the people of Kufa at that time, overall, had human failings (and exceptions apply), I don't think there is any text in which the Imams say that "our followers killed us".