Ah... engine blowby (combustion fumes) is a different problem than gas fumes. You really want to track down and eliminate any fuel leaks.
If it's combustion fumes - one thing to check is that the little vacuum line from the manifold is intact. Vacuum lines aren't usually very tight, but they shouldn't leak. The A4 recycles some of it's blowby through the little rubber hose that recycles gases back to the intake, through the flame arrestor. It seems like this thing inherently leaks, especially if vacuum is low.
Indigo Electronics sells an after-market
crankcase ventilation kit that is supposed to eliminate this leakage. I haven't tried it yet, but it seems plausible.
As Kapnkd mentions, exhaust leaks will make the cabin nasty in a hurry. They'll also set off your CO detector in a heartbeat. You do have a CO detector, don't you?