I bought a a Canon EOS 60D DSLR early last year, and at the same time as a bit of a punt on a compact system I bought an Olympus Pen EP1 with a Voigtlander Nokton 40mm 1.4 lens. I'd been impressed with Flickr images using this combo and I'd never before heard of Micro 4/3 so was curious.
A year on and I've found that I've used the bulky DSLR on only a few occasions whilst the little Pen has acquired a Lumix 20mm 1.7 lens and become my go-to camera for almost every practical use. I've used it constantly and always been really impressed with the results with those lenses.
Much like yourself I've had reservations about selling the DSLR kit; I work as a graphic designer and animator, and have always had that nagging doubt that at least semi-professionally, I *should* own this kit..
However with sensible appraisal (and reviews like your own) I'm convinced that the OM-D can replace both the EP1 and the DSLR, genuinely give me the best of both worlds. I don't even think it's a compromise on APS-C crop sensor DSLRs and it's a fraction of the bulk. As you say - only full frame now offers much advantage.
My previous reservations about the M 4/3 systems have been the lack of in-built viewfinders, limited DR and poor high ISO performance. Even with the new EP3 I wasn't tempted to upgrade, it's a better camera with the same older sensor. The OM-D has changed that though, all of my DSLR kit and the Pen are now up for sale and I'm buying an OM-D this month with luck.
My only concern is that - contrary to consolidating a growing lens collection - I'll only now be tempted to acquire more!!