Naively, the current RPG community breaks down into three basic camps. New school, old school, and indie or story game. While this is an oversimplification, a perusal of a number of games on offer lends some substance to the categories. Fourth Edition D&D and Pathfinder: new school. B/X D&D and Lamentations of the Flame Princess: old school. Apocalypse World and Torchbearer: indie. This division exists based on a number of factors, including marketing reach, the fuzzy borders created by online forum participation, player expectations about game objectives, and differences between the actual texts of written rules.
Many rules properties have been highlighted as potential differences. For just a few examples: lethality, quantity of character options, thematic coherence, rules coverage comprehensiveness, friendliness to ad hoc rulings, loci of narrative control, amount of prep required, rules heaviness, the number of resolution mechanics used. None of these factors is necessarily primary, as the old/new/indie categories are vague, though many of them are important. A dimension which I have rarely seen discussed, however, and which seems core to the difference in the approach many of these games take is the idea of proceduralism, by which I mean the degree to which a game directs your actions as a player or referee.
Many old school games do not provide direct procedures. Instead, they give examples of the kind of things participants might do, often with a short script-like example of play. This is both a weakness and a strength. It is a weakness because it is notoriously hard to learn how to play an RPG (which involves conversational form, conflict resolution, rules math, and many other components) from a text alone. It is a strength because it leaves the borders of potential wide open, assuming that you want to use the rules more like a toolkit than a how-to manual.
Of the old school games that I have read, OD&D and the Basic/Expert series have the most direct procedures. The first most likely because Gygax was essentially just telling you to do what he had done (“Before it is possible to conduct a campaign of adventures in the mazey dungeons, it is necessary for the referee to sit down with pencil in hand and draw these labyrinths on graph paper”) and the second because Moldvay was trying to create a self-consciously pedagogical text (more than a game with focused design, I would argue). Both of these works are only peripherally (or unintentionally) procedural; certainly nobody reasonable would claim that a referee that did not “sit down with pencil in hand and draw these labyrinths on graph paper” was breaking the rules of OD&D.
Skipping forward a bit over the “story focused” 90s, we arrive at the Forge, the post-Forge indie scene, and the games that arose from those seedbeds. For some concrete examples, consider Apocalypse World, Dogs in the Vineyard, Torchbearer. Apocalypse World puts forth the rules as conversation mediators; they are things that kick in “when someone says particular things” and that “impose constraints.” Dogs in the Vineyard, Chapter VI: The Structure of the Game begins: “If Dogs in the Vineyard were a board game, this would be the board” and then presents an outline in the form of alternating directions to player and GM. And later in the book: “Every moment of play, roll dice or say yes.” That is a pretty strong directive.
Torchbearer moves play through a series of different phases in a predefined manner, sometimes requiring character tests for transitions. Town phase leads to adventure phase, which may lead to camp before adventure again, or back to town. To someone used to D&D, this may just sound like shorthand for referee narrative, but it is structurally different. After three adventures, there is a winter phase. Wait, what? But what if we adventured three weekends in a row? No, that is not how Torchbearer works; you can’t do that. It breaks the rules, which abstract time in a certain way. D&D flirts occasionally with turn structure at different time scales, moving from the dungeon, to the wilderness, to the domain (“wargame”) turn as needed, but in a less defined, sometimes confusing, and certainly often overlooked manner.
A game may be procedural in one domain but not others. For example, combat in all editions of D&D is more procedural than many other fictional activities. The game grabs hold of you and does not let you go until you have performed the necessary steps. Though there are little islands of such proceduralism, no edition of D&D really tells players what to do when moving between these islands. This is not necessarily a flaw, but rather a different rules property that creates its own set of consequences.
It seems to be an implicit article of faith among many game designers that proceduralism has pedagogical advantages, and that games written with strict procedures are easier to pick up. This is possible, but far from established, and any such claim must also take into consideration extratextual resources such as the oral knowledge of communities, both distributed on the Internet and as passed between groups of friends in person. The game is more than the text.
At a first glance, this may seem like a critical evaluation of the procedural tendency in many of these more recent games, but that is not the intent. In fact, the rules project that I am working on right now (which grew mostly out of my “JRPG Basic” Gravity Sinister experiments) is highly procedural. The many procedural additions I have made to my ongoing OD&D game have mostly resulted in gameplay improvements. You can see further gestures toward this direction in A method of play and Gravity Sinister gameplay. However, all that said, it seems to me that there are some real trade-offs involved in going in either direction along the proceduralism axis. This is not at all a case of more procedural games being more advanced or more evolved than less procedural games.