Likewise it sounds like you just have to keep whacking away. One thing that may be a problem to remove is the pollen. Here is a link to one of my posts about adding a little DE to sand filters.
http://www.poolforum.com/pf2/showthr...ht=%28ajs-1%29
Another suggestion that aids the brushing process, which I think is a PITA especially trying to get to all the nooks and crannys. If you can do it, jury rig the vacuum hose and brush to the pressure side after the filter. Once you vacuum the best you can then use this to blow dust, algae, and whatever else you didn't get with the brush. Puts out quite a stream of water and stirs up stuff so it gets in suspension where the skimmer and filter can catch it and chlorine has a better chance of doing its job. I have the valving to let me shut off the returns and feed a pipe to which the hose is attached. You can probably find the right PVC sizes to mate with the hose end. You'll be surprised what this rig stirs up.
Al