Gerry has a bathroom exhaust fan damper that keeps him awake at night.
Well, Gerry, we can't always stop it, but we certainly can tone it down.
The first and cheapest step is to get some of that soft foam self adhesive weather-stripping tape. Tape it to the exhaust hood where the flapper hits. This may just damper down the problem to not have to really solve it.
This flapping noise comes from the wind blowing from the other side of the house. This pressurises the house a bit on one side, and depressurises it on the side of the exhaust hood. With gusts of wind, that causes air to actually move from the house out through this exhaust duct. There are a couple of things that can help to reduce the problem.
The easiest is to change the exhaust hood for one that has a wind baffle, like in the photo above. If you can't find one like this, just make up a sheet metal shield that will cover the exhaust hood and extend down about two feet on the wall.
The second is to put in a better damper, like a butterfly damper as you see in the second photo. The real solution is to air seal your house so that there are no cold air drafts blowing in one side that want to blow out the other side, but that can be a big task in an old house.