Pollutants come from four main sources:
1- Your body's metabolic activities (breathing and the like): carbon dioxide, ammonia, organics, and odours.
2- Your activities within the house:
-- Tobacco smoke: carbon monoxide, nitrogen dioxide, organics, odours.
-- Aerosol sprays: fluorocarbons, vinyl chloride.
-- Cleaning and cooking products: Hydrocarbons, odours, ammonia.
-- Hobbies and crafts: organics.
3- The contents of your house:
-- Combustion heating and cooking appliances: carbon monoxide, sulfur dioxide, nitrogen monoxide, nitrogen dioxide, particulates.
-- Furniture, drapes, and rugs: formaldehyde, organics, odours.
-- Water and natural gas service: radioactive radon gas.
4- Your house itself:
-- Concrete, stone and drywall plasterboard: radioactive radon gas.
-- Particleboard: formaldehyde.
-- Insulation: formaldehyde, fiberglass fibers.
-- Fire retardant: asbestos fibers.
-- Adhesives: organics, formaldehyde.
-- Paint: mercury, organics.
You probably don't have all of those pollutants in your house, but the facts boil down to this: if your house is well sealed and continuously ventilated enough to avoid moisture accumulation on a double-glazed window, you have probably diluted all of the pollutants to acceptable concentrations with a wide margin of error. The important point to remember is that most of these pollutants have been in our houses for decades -- they only bother us when they manage to become concentrated. Proper ventilation can prevent that.
Exhaust fans controlled by humidistats as the only ventilation system in the house can cause pollution problems because they ventilate thoroughly and then stop. Some of the pollutants can build up to dangerous levels before the humidity does, and leave you with a headache or worse even after the fan comes back on. Minimum ventilation in a well-sealed house should be continuous, or at least on a timer that brings in fresh air 20 minutes out of every hour as most HRVs are programmed to do, with an override higher fan speed for periods of heavy activity. Check out WON'T A VENT HOLE WITH NO DAMPER COST ME A LOT IN HEATING DOLLARS?