What is a Convention?
Allow me to describe a conception of art based around the twin poles of convention and novelty (which I will resist calling Convention and Novelty, because I am not French). A convention is simply a norm or collection of norms, and all art exists within certain conventions. In the visual arts, applying paint with a brush is a convention of method, and a landscape is a convention of genre, containing its own, respective conventions that can differ from time to time and place to place, as illustrated by the clear differences between traditional East Asian landscape paintings and traditional European ones. (European landscapes tend to be wider than they are high and emphasize the horizon, while East Asian landscapes tend to be higher than they are wide and emphasize scale. Each convention produces a remarkably different effect.) There is no art, or even expression, without conventions of some sort; conventions are the means by which things are expressed, the (sometimes literal, sometimes figurative) vocabulary and grammar we use to convey things. In this sense, conventions are a type of language.
Conventions are inherently subjective and the conventional expectations of a person in one place and time and background may be wildly different from one in another. Conventions are also inherently fluid, constantly changing, transforming into new conventions, with new forms overthrowing old ones.
Conventions and Society
We'll get back to art and specifically writing in just a moment, but bear with me for a moment while I digress into a little existentialism. Jean-Paul Sartre, drawing from ideas of Hegel and Kierkegaard [but mostly Heidegger, ~JFQ][and Nietzche, points out my wife -ER], proposed that society alienated the individual. It does this through the conventions it imposes; we all dress, behave, and think in certain ways to a greater or lesser extent because of the society that we live in and the conventions that are imposed on us from birth. When we start to realize that our behavior is dictated by society, we feel alienated from it— meaning, we feel that it is not ours, it belongs to someone else, it is alien, the Other. This is the famous "existential crisis", ie. the crisis of one's existence as an individual. The desire to define oneself as a unique individual in the face of social conventions is what Sartre called the need for authenticity, and becoming an "authentic" individual is one of the chief difficulties of being-in-the-world. But one can never escape completely from conventions, one is always already in society and the only way to not be a part of it is not to be at all— or, one might say, the quest for individuation is one that, at its farthest end, leads to death. However, total conformity to convention, eliminating all traces of the individual, is also a kind of death, a ceasing to exist. In order to live, to be in this society and a person unto oneself, one must find a balance between convention and individuation, and what that balance is is a question everyone must answer for themselves. [The responsibility for making this determination itself gives rise to existential Anguish, itself a precondition for freedom. ~JFQ]
Conventions and Writing
Likewise, a fiction writer cannot escape completely from convention, except through silence. You are always already dealing with conventions of language, story structure, portrayal of character, etc. Even the tense and syntax that one uses, exhibited by a phrase like "She went to the store," is a distinctive narrative convention; it is very rare to hear someone in casual speech say anything like that, but in writing it's perfectly ordinary. Genres are collections of conventions, tropes that one may deploy to certain effect, or overthrow in the quest for individuation, for novelty. And each writer must struggle to create their own authentic writing under the weight of the conventional past.
Conventions in writing are not bad things. They are entirely necessary, and indeed, simply employing a convention can create a certain kind of pleasure in the reader, a cozy, comfortable pleasure. Mystery readers will often happily go through the same plot over and over again, as much for the minor differences that make up the novelty of each work as for the comfortable enjoyment of experiencing something familiar, like seeing an old friend.
It is even true that, generally speaking, the more conventional the work is, the wider the audience; the new and authentic is frightening and dangerous and risky. However, you, like me, might find the highly conventional is also profoundly boring, cf. Avatar, Twilight, The DaVinci Code, all of which follow conventions that have long since ossified. And there is definitely an audience of literary thrill-seekers who intentionally pursue only the most experimental and uncompromising work, and those may be just the people you, as writer, want to communicate with.
The question of how conventional and how unconventional a writer's work should be is a question of identity, and if you are honest with yourself you can discover the kind of writing you want to make, and then you can worry about finding the audience who appreciates it. When writers try to work the other way, and tailor their writing to a perceived audience, the result is what Sartre called 'bad faith', meaning intentional phoniness— you are not writing work you really believe in, you're writing work you think will sell, and that can only lead to something inherently inauthentic.
Novelty and the Unconventional
It's very easy to create something unconventional. If I wrote a story where a single character screamed "OYSTERS! SUPERMAN! BANGLADESH!" for ten pages, that would be rather unconventional. But I don't know that it would be interesting to anybody. Wildly and aimlessly breaking with conventions doesn't accomplish very much, and this understanding is what separates great experimentalists from inferior ones. So what makes a particular break with conventions valuable and another simply random? The key is the elements of recognition and revelation.
To illustrate by example: One way to create novelty is simply to combine existing tropes in unexpected ways: Quentin Tarantino likes to take tropes of exploitation films and add to them the psychological depth of modern drama. (And if I use examples from film and television here rather than books, it's only because of the greater likelihood of familiarity in the reader.) Both of these sets of tropes are immediately recognizable, but combining them adds new dimensions for both; suddenly we see the Bride's quest for revenge not merely as a one-note action driven by anger, but as a complex story about her thwarted desire for redemption, which she comes to realize can only happen by first becoming everything she wants to escape and be redeemed from, that is, an instrument of extreme violence and death. By doing this the film asks pivotal questions about our own desire for violence (in the form of exploitation films, video games etc), and whether we can ever really escape from it (embodied by what is for me the key scene in the films, when the Bride tells the daughter of the woman she just killed that, if she ever feels the need for revenge herself, the Bride will be waiting for her). In other words, by combining tropes of radically different genres, Tarantino creates recognitions that lead to revelations about both the conventions he employs and of ourselves.
Likewise, Scott Pilgrim (both the comic and movie) mixes tropes of teen drama with the storytelling conventions of videogames to alarming affect, giving us startling new metaphors for the very old tropes of love, jealousy, dealing with your past, etc. However, this is also an example of how one has to find something recognizable in the conventions in order for reinventing those conventions to have any meaning. Scott Pilgrim is necessarily generationally polarizing; older people who did not grow up with videogames do not understand it because the conventions of videogames have no meaning for them, and so mucking around with them has no metaphorical resonance; they do not recognize them, so they do not gain new understanding from them. For those of us who did grow up with videogames, however, Scott Pilgrim is pure, delirious joy, Barthes' orgasmic joussance.
Mixing conventions together is, of course, far from the only way to create novelty. In psychological realism, it is created by revealing or exploring emotions and character in ways that haven't been explored before, that reveal things in ways that haven't been done before, that are not yet part of psychological realisms conventions (or which show those conventions, in new, expected lights) but that we, as people, recognize in ourselves anyway.
In the shock of recognition buried in the experience of novelty, the experiencer is suddenly brought into the work of art, comes to meet it, participates in its creation on the canvas of their own mind. We are shocked out of the rote acknowledgement of convention and forced us to face ourselves authentically.
The mistake of those who promote psychological realism over other modes of storytelling is in thinking that it is the only way to create novelty, the only way that stories tell us things about ourselves, a stance which is plainly naive. It's also the mistake made by people who think authenticity and honesty mean autobiography, when often autobiography can be the most banal, conventional thing imaginable. There is seemingly an endless font of writers who tell a semi-autobiographical story about some rather ordinary, suburban life, or worse, the life of an academic, in fanciful prose and are surprised when they are met with a chorus of yawns.
Another easy example of authenticity having no relation to realism, and one that leads me into the next point, is in the television series Buffy the Vampire Slayer (and if you're not at least open to the idea that Buffy is an important work worthy of study, then you should probably just go to a different website right now). The show's creator, Joss Whedon, made a conscious decision to apply feminist ideas to common horror conventions; the blond-haired cheerleader-type who is usually the damsel in distress instead becomes the empowered hero, the one who is chosen to fight evil, and who must often save the male characters. This forced those immersed in the tropes of horror and cliche to reconsider those conventions they had taken for granted and see the sexist elements, the helpless female victims, the masculine power trips, layered into them. (Buffy also creates novelty in other ways, but the feminist angle is the first and primary one.)
The Conventional Appetite
Ideas naturally move from novelty into conventionality and frequently then proceed into full exploitation, wherein the original meaning of the novelty is itself undermined. For Buffy the trope of the empowered female heroine was rapidly copied again and again until all the newness of its feminism had become old hat, and it was finally degraded completely into lascivious girls prancing around with little clothing on, sticking out their rears and bosoms, suggestively wielding a phallic gun or sword. Buffy also became exploited in another direction, by people with the best intentions, who created female heroines so perfect in every way that it denied their humanity, so that they became idols on pedestals and not people at all. In both cases the formerly feminist heroine was reduced to mere object.
This pattern is all too typical. Create a black protagonist, and he will turn into a blaxploitation protagonist. Create a more realistic take on superheroes, and it will turn into over a decade of "grim and gritty" superhero comics peddling mindless violence. Create a plotless story that emphasizes character depth, and it will turn into MFA fiction. Philip K. Dick leads inexorably to Total Recall. Convention by its nature devours ideas and shits out their twisted, self-parodic homunculum.
And then, as Tarantino shows (and this might be seen as the whole Tarantino project), the exploited convention can be reexamined on its own terms, exploded and transformed into new novelty, new convention and ultimately, new exploitation.
Genre, Medium and "Mainstream"
The definition of "mainstream" changes from medium to medium. For example, in film "mainstream" refers to popular hollywood fare, and is dominated by genres like science fiction-fantasy-action/adventure ("tentpole" blockbusters), romance, comedy and so on. In comic books, mainstream refers to superheroes, and the strangeness of this one, odd little genre having so dominated an entire medium has been commented on in many quarters. In prose fiction, mainstream refers to literary fiction. (And yes, various thrillers and young adult novels and so forth may sell very well, but they are still categorized as genre fiction.) Indeed, no other medium I can think of is so preoccupied with the status of "literary"-ness than prose fiction (though there have been some attempts in other media— for example, the critics who try to separate "film" from "movies"). This has, of course, everything to do with prose fiction's particular history and audience, but it doesn't make the attempt to separate "literature" from not-"literature" any less pernicious, causing, as it does, great work in "genre" fiction to often be sidelined by both large portions of the reading public and the academy (and thus often future generations as well as the present ones). The notion that the particular set of conventions that we call "literary" is somehow inherently better than those we call "genre" is absurd on the face of it, but it is so intricately wrapped up in signifiers of class and social status that we have trouble making out the absurdity. In fact, the very act of reading itself is a signifier of class and social status, and many people come to reading fiction for pleasure, often at an early age (when most of their peers are only reading what's assigned to them), specifically because they want to see themselves—to individuate themselves—as more refined and sophisticated than other people—in other words, of a higher status. I will be the first to admit that I am as guilty of this as anyone else, and one can see it in the derogations of genre—a lashing out and defensiveness borne, I think, of my own guilt in enjoying the stuff—that made up many of the posts in the early years of this blog. I can't think of any other whole medium so wound up in status.
And so we arrive at my personal stake in all this, as a writer. I started reading literary fiction in my teens, literally throwing away the dozens of science fiction and fantasy books that had gotten me through my pubescence, as a way of establishing myself as an "adult". But, though I read a lot of Philip Roth, attracted by his humor, treatment of sex and not least by his status as the widely declaimed "greatest living American writer", most of the authors I found myself gravitating towards were those who brought fantastic elements into their work, like Murakami, Garcia Marquez, Borges, Kafka, David Foster Wallace and José Saramago. In other words, what I really still liked was speculative fiction, even though I only read the speculative fiction that was categorized as literary fiction. Eventually, I started reading sf qua sf as a guilty pleasure (and I did feel guilty about it, what was I doing slumming with this crap?) and rediscovering everything I loved about it, and all the authors, especially from the New Wave (70s) forward, who had gone out of their way to subvert, reinvent, reanimate, and inject novelty into the tropes that make up the comfortable corpus of the field. This precipitated my own, personal literary existential crisis, played out in public, in which I realized that I'd been championing only literary fiction in bad faith. At some fundamental level, the conventions of sf speak to me in a way that the conventions created by Flaubert and passed down through early Joyce and Hemingway and so on, the conventions of literary realism, never could. And it's telling that in the list of literary authors above, only one of them, Wallace (who comes out of a postmodern tradition inspired by French critical thinking), is Anglo-American. Because in other countries, "literary" does not automatically mean realism.
Of course, this is slowly changing, and writers who poked the boundaries like Vonnegut ultimately gave rise to our current writers of sf who masquerade as literary fiction, like Nick Harkaway, Susanna Clarke, Lev Grossman, and so on. And there's also a number of sf writers who have become cross-over sellers, starting with the rehabilitation of Philip K. Dick and the mainstreaming of JG Ballard, and moving to writers like China Mieville, Neil Gaiman, William Gibson, Neal Stephenson and Kelly Link. Meanwhile, sf packaged as "Young Adult" fiction, like Harry Potter and Twilight, rockets to the heights of success, showcasing the popular appetite for sf that makes movies in the genre among the top grossing of all time.
Which is all just to say, that I came to realize that my prejudices against the sf genre had no basis in anything other then a false conception of myself as too good for that shit, a false conception based on signifiers of status I had absorbed rather than any honest appraisal of the genre in itself. Because the fact is, as I found out, there's just as much daring, startling, unconventional work in sf as outside of it.
The important thing to understand here is that the more meaningful a convention is to you, the more powerful it is when it's exploded or reinvented, because the conventions you enjoy represent your own personal mythological landscape, a worldview intricately linked to your identity. And just I cannot pretend that Rabbit, Run is meaningful to me, personally, as much as I might appreciate it on a theoretical level, I also cannot pretend that a book like Ted Chiang's Stories of Your Life and Others doesn't blow my mind several times over.
That doesn't mean I don't appreciate anything in other genres, or that realist books are never meaningful to me. But I'm done pretending that the latest literary fiction notable books are more interesting to me than the latest sf ones. For me, anyway, China Mieville's Kraken is more vital and valuable and compelling than Jonathan Franzen's Freedom could ever be, no matter who makes it to the cover of Time Magazine.
So what conventions do you as a reader and/or as a writer feel most engaged with, most excited to grapple with or rediscover? It's a question you'll have to answer for yourself. All I can ask is that you be honest.