In the New Inquiry (which is a wonderful online periodical), Rob Horning has an article that pretty much sums up the problem of "literary" fiction:

I don’t like the word literary. It seems to imply some particular formal characteristics, but that implication only allows the term to serve as an alibi for the status aspirations of the people who use it, who want to control its meaning. It’s a sort of social tautology that way. The literary is what literary people say it is, which is what makes them literary people.

What’s at stake in claiming something is literary is different from claiming that some book is good. What counts as literary is a moving target, but it’s not always moving in one direction. We are not making collective critical progress toward a better, more comprehensive understanding of what the literary sometimes pretends to be: an expression of the perfectibility of prose, of the ability to capture the truth in words and well-formed sentences. Instead, literariness is ruled by the laws of fashion; it changes merely to replenish its potency for those empowered to declare what is and isn’t literary. Fixating on what is literary actually denies for books the possibility of transcending fashion. To evoke a book’s literariness is to evoke its transience, not its staying power. It says, this book’s only relevant and lasting meaning rests in its capacity to signify the literary.

In other words, the term "literary fiction" is about status and not quality.

Read the whole thing, it's worth it if only for poking fun at John Updike's descriptions of sex.