Touching something will not break one's wudhu`, unless it is the touching of a dead human body after it gets cold, and before it is washed.
For anything else, najis or not, just touching it will not break wudhu`. Of course, if it is wet, and your hand gets contaminated with najasah, all you need to do is wash it.
As for touching something dirty, the same thing, and no, from a shar'i perspective it is not necessary to 'wash' your hands if it is just dirty and not najis.
And Allah knows best.