Sweatpants with non-elastic ankles