That's not so simple to answer because clothes generally don't fossilize. But evolutionary biologists wouldn't let something like that deter them - to estimate an answer they turned to lice.
I'm glad to say that I don't have much firsthand experience with lice, whether they be head lice, body lice or crab lice ("the crabs"). They all live on human beings, with preferences for the head, body, or genitalia respectively. (There are other kinds, but they live on other critters).
One major differentiator between head and body lice is that head lice lay eggs in your hair. Body lice attack hairless parts and lay eggs in clothing. So if you assume that at some point head lice and body lice were the same critter, it makes sense that they would have started differentiating about the same time humans started wearing clothes.
Voila - clothing must be about 40,000 to 72,000 years old.