Definitely something in the starter circuit for the low speed. On any of these type motors, when you lose the starter, you can get it going in either direction by spinning it manually. That's what's happening when you flip from hi to lo. To be honest I'm not sure how the starter circuits are configured for two speed motors. Never had to look into it. I'll see what I can dig up.
Al