I never shut it off while swimming, the way i see it is swimming moves your water around, stirs up all of the debris (even if it's just a little amount), so your filter will actually catch some of this stuff and filter it out instead of it all settling to the bottom again after the swimmers are out of the pool.
Yes, older bottom drains have had some freak incidents associated with them, but today's drains are perfectly safe to be used while swimming - and on most older pools, the diverter valve can be used to bypass the drain while swimming.