The phone argument is a good one. Unfortunately, it's often presented as the pretense for the wrong case, i.e., "Camera technology is improving in such a way that in X year's time, the cameras in our phones are going to be as good as professional DSLRs." Obviously not true, but it doesn't change the functional outcome of the situation, which is that more people are going to be satisfied with what their phones offer relative to the costs of picking up something like a PEN or a GM1.
The problem that phones represent to the mirrorless/prosumer market is that they're good enough for a lot of the people who want to take photographs. Reasonable megapixel count, reasonable dynamic range, reasonable functionality, a host of plug-and-play post processing apps; couple all that with the fact that they're already carrying it around on their person and it means these cameras are good enough for a lot of people. I think these people see moving beyond their phone cameras as an exercise in diminishing returns: there's a technical learning curve, a more difficult post process, a more complicated pipeline, and an added bulk that diminishes the relative benefits of better image quality and increased creative control. Many people simply don't care enough about those things to move up from an iPhone.
Clearly the mirrorless/M43 fills a need in the market, but is it big enough and sustainable enough to warrant the R&D that goes in to advancing the format? I don't know. That's the question. My honest guess would be that it does, but ultimately at the expense of DSLR sales (for all the reasons Andrew stated, and because, anecdotally, it seems people are becoming more "aware" or mirrorless as a viable alternative, even if they aren't yet spending in such a way that reflects this).
I certainly hope it plays out this way, at least. I love my GH3 and think M43 is a great format. I'll be eager to see what the coming year brings!