This is a general discussion of the concept of genre and how it applies to RPGs. At its most general, a "genre" is simply a set of stories which are similar in some ways. A "genre convention" is anything which that set of stories have in common. These can be any sort of stories -- even non-fiction ones. For example, the "true crime" book is a non-fictional genre which has distinct conventions.
My overall recommendation is that GM and players should agree about what the genre of a campaign is -- and note how the game's genre differs from the literary sources for the game as well as from reality.
In the broad use of the term, all games have a genre -- even if they are pure tactical exercises or realistic simulations of daily life. Some genres are realistic. Other genres, that I call "stylized genres", are intentionally unrealistic. Stylized genres include comic-book superheroes, high fantasy, and space opera -- works which are often called "genre books" (or "genre films") as opposed to mainstream fiction. A simple distinction would be that in resolving a tricky case, a writer in realistic genre will turn to non-fiction sources to determine the answer. A writer in a stylized genre will consider what happens in fictional sources. For example, do you look up lasers in a real-world textbook, or do they work like they do in the comics?
As an example of a realistic genre, I ran a near-future cyberpunk campaign. As GM, I defined an NPC opponent who had fast-acting nerve gas as a last-ditch weapon. However, when it came up in-game, a player chimed in who had studied real-world nerve gas. He explained to me that even with reasonable advances, there were physical limits to how quickly the toxin's effects could set in (on the order of 30 seconds, as I recall). Because I had defined this as a realistic game, I took this into account and changed how the gas worked.
On the other hand, a realistic game will still have a genre and genre conventions. For example, many realistic games borrow conventions from the "naturalist" genre, a genre of novels which focus on everyday life of average people. The equivalent in RPGs is that "realistic" campaigns are not about extraordinary characters, even if such characters exist in the game-world. For example, GURPS Martial Arts explains "realistic" campaigns by saying,
In a truly realistic campaign, all PCs would be built on 25 points or so, and any hazardous situation would wipe out half an adventuring party immediately [...] and realistic mages will not be able to shatter armies with a spell.Of course, what a mage can or cannot do with a spell has nothing to do with realism. Rather, this is a convention of naturalism, that the protagonists are close to average people.
This might not be intended as a genre convention, but it is one nevertheless. Essentially, there are genre conventions which are conscious choices of the gamers, and also genre conventions that are subconscious tendencies. I would encourage reflecting on all of your tendencies and choices as being part of genre -- regardless of whether they are realistic or not. A conscious desire to emulate reality in some field is just as significant as a desire to break from reality.
An essential difference between role-playing games and fiction is that RPGs are created interactively. This means that it is possible for participants to have different ideas about what the genre is, and what that means. Because they work together to make the story, the GM and the players need to discuss the genre which they are trying for. Lack of communication or miscommunication about genre can cause arguments and bad feelings.
Some genre conventions are inherent in the choice of rules system and/or setting. i.e. Players expect a game of Steve Jackson Games' Toon to have the genre conventions of Warner Brothers cartoons. Players expect that a game set in Tolkien's Middle Earth will have the other genre conventions of Tolkien's stories.
However, this implication can lead to confusion. For example, a player who joins in a game set in Middle Earth might expect on an emotional level that the game's stories hold true to the spirit and morals of Tolkien. When the game diverges from that, he feels that the game has gone "wrong" and becomes unhappy. Conversely, a GM might run a game set in the Star Trek universe, and had pictured the PCs as having high moral standards. When the PCs instead do something underhanded and nasty to get what they want, he feels upset and would like to "punish" them.
These is nothing inherently wrong with stories set in an author's universe which don't follow all of the conventions. However, miscommunication over this can interfere with players' enjoyment of the game.
In discussing genre conventions with players, it may be helpful to group them into three basic categories:
World conventions are usually covered in the background for the game. These are generally the province of the GM, although some groups have a shared-world arrangement among players. Obviously the GM has a responsibility to explain the world background to the players. The GM should be careful if the background is not as it seems to be -- this is a matter of knowing the players.
Character conventions are usually covered in character creation. The players are nominally responsible for them, but often within limits set by the GM. Standard character creation methods are fairly thorough, but there are some topics which are often skipped. For example, the morality of the PCs is very important: not just whether they are good or evil, but the details of how they face moral issues. Another topic is how they behave as a group -- do they always act as a team, for example, or perhaps do they help each other but eventually find their own ways? It is important to consider not just how the characters are at the start, but to consider the arc which they are going through over time.
Story conventions are the most difficult, because given the dynamic story that RPGs produce, the GM and the players usually need to cooperate for story conventions to work. RPGs are different from other media in how the story is written. In film, the writer controls both the protagonists and the plot in broad terms. The director and actors implement and adjust that vision but keep the basics. In RPGs, the players are responsible for the choices of the protagonists, which are usually the core of the plot. The GM controls background and foils for the protagonists.
For example, I GMed a long campaign set in the universe of the original Star Trek series. One thing which I did fairly well, I think, was that each episode involved a science-fiction allegory for moral or social issue. However, episodes varied in how well they worked. In general, episodes worked well when the PCs took a bold stance on the issue. Regardless of what the stance was, this would lead to a strong ending which fit the genre. However, if they waffled on the issue, the episode would fizzle. The lesson that I somewhat belatedly learned was that genre was more in the hands of the players more than mine.