It struck me when reading Warhammer Fantasy Roleplay that the Cool stat really has a very similar role to morale in D&D. Description of cool (WFRP page 13):
This characteristic represents a creature’s ability to remain calm, collected – even sane – under severe psychological stress. Some of the creatures that inhabit the Old World are truly frightening, and may be confronted safely only by characters with a high Cl value. Cl is expressed as a percentage of 01-100%.
This stat is used when monsters can cause fear or terror. A failed check against fear basically means that a character may not take actions until they overcome the fear (one check possible per round, like a Fourth Edition saving throw). A failed check against terror sends the character to the fetal position for the rest of the encounter and grants an insanity point, which can lead to other bad things (like various types of madness). See pages 68 and 72 of WFRP for more details if you have access to the first edition.
Here is the morale text from Men & Magic (page 13):
Non-player characters and men-at-arms will have to make morale checks (using the above reaction table or “Chainmail”) whenever a highly dangerous or un-nerving situation arises. Poor morale will mean that those in question will not perform as expected.
One method to work a sanity system into D&D while cleaving to the traditional mechanics would be to start all PCs off with a morale of 12. This would represent naive young adventurers brimming with confidence and perhaps in some cases an iron will. Every time a character witnesses a sanity-threatening event (this would be campaign dependent, but could include encounters with undead, certain kinds of black magic, watching a companion die, etc) a fear saving throw would be required (probably a save versus spells). Failure would indicate loosing a point of morale, and open up the possibility of failing a morale check. I like this idea because it doesn’t require any new rules.
There have been several other recent D&D approaches to sanity. In Barrowmaze there is an optional fear rule where PCs accumulate points when they encounter undead and go insane when their total equals or exceeds their wisdom score. These points can be removed by spending time in civilization. Akrasia also has a wisdom-based sanity system. If you haven’t read his Swords & Sorcery house rules, get to it. It’s one of the best free OSR supplements out there (a free PDF is available). The free TOTGAD Compendium (now available in hard copy too) has terror, horror, and madness rules. The TOTGAD systems also rely on saving throws and have tables of possible outcomes for failed saves. I recently used his madness table for Death Frost Doom and it worked very nicely.