Yes, there is a difference. How big a difference depends on the grade of Stainless Steel. When I was pricing vinyl pools, I noticed that the Stainless Steel every one was quoting was in the general type 304 (Some were 302, 304L etc.-all in the 18-8 series) which is moderately resistant to corrosion yet the price was almost double the price of the galvanized. Type 304 can and will rust over a period of time if exposed to chlorine, salt and/or water. The price increase may have been reasonable if it was Type 316 stainless (aka marine stainless) however it may be a little overkill. Galvanized panels will most definitely rust over a period of time, especially if exposed to chlorine, salt and/or water.
The question you must answer is "How long will I live here?" IMHO, I would go with the Galvanized since I don't think they would never rust completely through jeopardizing the structural integrity of the panels especially since the grade of stainless is not that much more resistant to corrosion compared to the increase in price. I'd bet that in twenty three years under normal circumstances, the galvanized panels would be fine.

Dave