Hi, no, not crazy. Not even insane. And you've done some good thinking about it. I BELIEVE the general rule is when you build up, you must build out. I think if you go up 5' with your retaining, your pool must be at least 10' from the edge, or you need to get it engineered. Anything you build up should be filled with QP ("quarry product"), essentially dirty gravel. But QP has a very high friction coefficient, packs really well, and drains almost as well as gravel. It's usually cheaper than gravel, too, and you can put sand on it. But you still need pea gravel for the French Drains.

But, unless you do all the work yourself or with buddies who work for beer, it's going to be very expensive. I went through this. My platform goes from level ground to 2 1/2 feet of rise at the lowest point. Retaining wall block, QP, and an engineer (couldn't get the 5' around the edges), it cost $5,000 for a base to support a 16'x40' AG structure (rectangular, not oval or round.) I laid out all the lines myself, too. (I have an old yard-sale transit that still works).

Carl