Nice review MJ. Sorry you had so much trouble with MESS. Not sure if you are using an older version. Current version is 0.144u6. In the last couple of years MESS has really improved with both its 7800 color palette as well as incorporating the HLSL options found in MAME. You have so many great tweak settings to make your display, even on a modern LCD, look like an old CRT TV including brightness, RGB values, contrast, scanlines, and a slew of other setttings.
Sound emulation wise, especially POKEY, I found to always be better under MESS than ProSystem (At least respecting all the currently publicly released official and unofficial verisons). Graphically it has some small corruption in a couple of games (Top of Centipede), to really bad (Kung-Fu Master). Additionally, unfortunately, a few of the homebrews either keep cycling at the Atari logo screen or don't function correctly. Thankfully, the vast majority of both original release and self published titles run great.
I'm attaching a few quick screen caputres with some HLSL settings enabled, including screen curve. I'm hoping for both the few bugs found in MESS 7800 emulation to be ironed out eventually, and for better graphic options and sound under ProSystem.