Sounds exactly like what I have seen on both my system and a friends when trying to vacuum a dirty pool. In my cases dust and probably pollen that settled on the bottom. There was another similar thread recently and what I recommended there was to use skimmer socks to catch most of the crud before it gets to the filter. You will be surprised what they can catch, I was. A pain swapping and washing them so often but beats wasting water on a backwash. Your sand is probably perfectly good and doing it's job.
Al