Almost all of the Dark Souls rules are somewhat applicable to the tabletop context, but the magic systems seem especially so suited. There are three magic systems: pyromancy, miracle, and sorcery. There are “classes” associated with each, but any character can level into the various kinds of magic by increasing the appropriate stats. Each magic requires a characteristic implement be equipped in order to cast spells of the given type. Each interacts with PC stats in a slightly different way, and the number of overall spell slots, which must be divided between all types of magic, is controlled by the attunement stat. Spells must be prepared (“attuned”) at bonfires, which is the equivalent of downtime or recovery in D&D, and may only be used a limited number of times before resting again.
Pyromancy is the simplest type of magic. There is some intimation that it is more primal and less sophisticated than sorcery. It is often associated with swamp dwellers, symbolic of exclusion and the primitive. There is a similarity here to the distinction WotC D&D makes between wizards (pseudo-academics) and sorcerers (wild talents, magic by lineage). Pyromancy power is not affected by any stats, other than the slots from attunement needed for spell preparation, and their power is dependent only upon the strength of the pyromancy flame used, which is the characteristic implement.
A pyromancy flame can be upgraded independently by spending souls, which, remember, function like both GP and XP. Most pyromancy is simple attack magic, fireballs and so forth, though there are also a few defensive spells, such as iron flesh and flash sweat (which increases defense against fire damage). From a game perspective, pyromancy gives players a way to get access to magic damage by dumping souls into an upgraded pyromancy flame and a few pyromancies without needing to increase level at all, assuming a PC meets the (very low) attunement requirement to be able to prepare any spells at all.
Sorceries are more academic, and higher precision. They often have intelligence prerequisites, and spell power is affected by intelligence. A character that wishes to be a competent sorcerer must dedicate a number of levels to the sorcery-oriented stats. The characteristic implement is the catalyst, often depicted as a wand or staff. Unlike pyromancy flames, or weapons, catalysts cannot be upgraded. You must find better ones.
Though there are several attack sorceries (such as soul arrows, which are basic “magic blasts” that also have the useful function of tracking enemy movement to some degree), there are in addition many utility and misdirection spells, such as aural decoy (which lures enemies away by creating a sound elsewhere), fall control (as feather fall), and hidden body (basically, invisibility). I have been playing a warrior and have only dipped lightly into sorcery so far, so I do not have much direct experience with these spells beyond soul arrows, but for a combat-oriented action RPG, there are a surprisingly large number of spells that seem to enable non-combat creativity.
Miracles are the province of the cleric, and are mostly, by default, as in D&D, defensive or restorative. However, the miracles a character has access to depends on which covenants are formed. For example, if you join the gravelord servant covenant, there are miracles that call giant phantom blades to attack your enemies. This is a fascinating system, reminiscent of “clerics of specific mythos” in 2E D&D, but much more dependent upon action during play. Further, a covenant comes with clear, objective factional duties and restrictions. I do not fully understand exactly how this affects gameplay, but what I have seen of the periphery makes the covenant system one of the most interesting aspects of Dark Souls design, and one that has been highly influential over my conception of clerics as servants of immortals in the world of The Final Castle. Beyond this contextual aspect, miracles work similarly to sorceries, requiring attunement at bonfires, and dependent upon the faith stat for power. The characteristic implement, which must be equipped to call a miracle, is the talisman.
This system design engenders several different kinds of trade-off. First, there are the advancement decisions about which stats are increased during level up. While strictly speaking it is possible to grind souls and increase everything, in practice this is tedious, and further unnecessary to be successful*. If you are just playing the game to explore and overcome challenges, you will naturally need to choose between physical capability and the various kinds of magic. This mode is also more applicable to the tabletop context, where grinding dynamics are minimized or nonexistent. Second, there is the cost and availability of various spells. Third, when you set out from a bonfire, you must divide your available attunement slots between spells. If you have four slots, for example, two could be miracles and two could be sorceries.
Fourth, and most importantly in terms of the gameplay experience, you must wield the appropriate implement to cast a given spell. While encumbrance rules do not prevent you from carrying everything with you (an aspect of the game I find somewhat strange), they do prevent you from equipping more than a few items, and switching between items that are not equipped during combat is asking for a quick death. In practice this means that you have a primary and secondary equipped item in each hand that is east to switch between. The left hand is usually occupied by a shield and either ranged weapon or tool (such as the skull lantern). This leaves the right hand for (likely) a melee weapon and magic implement. The final result of all of this design is that it is impractical to ready more than one kind of magic on a given excursion. Fifth, and finally, casting a spell has a more or less lengthy animation and thus requires a trade-off consideration in terms of when you start to cast a spell, as enemies may take advantage of that time or your vulnerability.
* At least in single-player mode. If you are into PvP it is likely different. The fact that grinding souls might make a big difference in the viability of character power is part of the reason I have little interest in PvP.