Sure sounds like the starter circuit switch. Even though it looks like it's closing, could be pitted contacts. Being an intermittant problem points to that too. Use some 400 or 600 grit wet-or-dry sandpaper (auto stores) or a small flat file (preferred) to dress the contacts. Usually they are silver and soft so you don't need to be agressive. See if that solves the problem.
Al