I would say that you should bring sweaters with you in case he needs them but don't try to get him to put them on if he doesn't want to. People have vastly different perceptions of temperature; I've been to houses where I had to wear a coat just because the air conditioning was on so high.

If he says he can only get warm by cuddling, this could just be his way of saying that he wants to cuddle. So I would say cuddle with him. If this is the case then a sweater wouldn't help because he's not actually cold.

Your concern about him getting a cold is definitely admirable but not really warranted. Unless there's an underlying medical problem (in which case a doctor should be seen), he should be able to sense when he is at a healthy temperature for him, however warm or cold that may be.

Good luck!