Well, I have the top and back to spec (or, as close as I'm going to get). I took to the back with a bit more assertiveness, and did my best to clean up the tear out with a scraper and RO sander. There are still a few little scars on the inside surface, but they're pretty minor.
Was having a bit of a hard time cutting the curves with the bandsaw, need to just take my time and make multiple cut, or pick up a narrower blade.
Also, on further inspection, the blade on my No 5 plane was pretty far off of square, that likely wasn't helping. Put some time into fixing that
Next up I'll set in on the sides.