Yes even if the others were non Muslims as far as you do not know that the cloth is Najis. If it was Najis, we need to wash it to make it Taahir if we want to wear it during Prayers.
Hygiene needs to be observed especially when there is possibility of infection.
Clothes worn by pious people can have their blessings.
Our Infallible Imams used to wear their clothes then donate it at the end of the season, and people used to get lot of blessings from wearing such worn clothes.