Just having an attraction toward something can never justify doing it when it contradicts with the discipline set by The Creator who distinguishes between good attraction and bad attraction. Many persons have sexual attraction towards incest or children or married women and men etc. It must be clear that just feeling the sexual attraction does not legitimize what you are attracted to. Many people are attracted to drugs and billions of dollars are spent in today's world on drugs, but can such attraction legitimize the drugs? Desire is the originator of attraction and it can be be pure desire which initiates attraction towards good things, and can be bad desire which initiates bad attractions.

Allah (SWT) who created us and every thing and knows about every thing has told us to avoid impure desires and never go after evil attractions. In fact it is one of the best forms of piousness is to go against the impure desires (And as for him who feared standing in front of his Lord (Allah), and restrained himself from impure desires, then the Paradise will be his abode. (Sura 79, Verse 40 and 41).

Allah (SWT) did not forbid any act but only because it is harmful )ويحرم عليهم الخبائث (Sura 7, Verse 157)

Allah (SWT) mentioned in Quran (And remember (prophet) Lut, We granted him prophet hood and knowledge and we saved him from the people who used to do evil acts. (21:74) ونجيناه من لقرية التي كانت تعمل الخبائث

Every evil act is forbidden whether some people are attracted to it or no. Millions are attracted to alcohol, drugs, pork, and fornication etc but it remain evil acts in the reality, so it remains forbidden (Haraam). Forbidding evil acts and impure desires is essential to  save life and human beings.

Wassalam.