We examined 200 college students, at rest in a sitting position in an air-conditioned classroom, to investigate the differences in physiological and psychological responses between men and women in the thermoneutral zone of SET*. The percentage of women who reported suffering from the cold more often in an air-conditioned environment was higher than that of men. The number of clothing items and the clothing insulation value, estimated using International Organization for Standardization (ISO) 9920 were higher for the women than for the men, based on the students’ own everyday clothing choices as worn during the experiments. However, the percentage of body surface area covered by clothing worn by the women was significantly lower than that for the men. Although oral and skin temperature decreased in both men and women after 60 min of exposure to a temperature of 25 °C, the oral and skin temperatures of the women’s forehead and palms were significantly lower than those in the men. Moreover, the decrease in the women’s oral temperature was greater than that of the men. Women felt cooler and more uncomfortable than men did in the ASHRAE comfort zone, with SET* of 22.2–25.6 °C. Gender differences in physiological responses, and differences in the clothing typically worn by men and women, might affect psychological responses to the moderate thermal environment encountered in daily life.