Q u o t e: Well basically it should work similar to armor vs attack rating (without the 5%/95% caps, and as % damage instead of % chance to hit) but I'll elaborate for you.
I think it's better to look at a real rating-based system to see how it works instead of pulling wierd numbers out of the air.
T = D * F^((R-P)/N)
T = damage taken/dealt (by target/caster)
D = spell damage (of caster's spell)
F = fraction of damage reduced per N rating (universal constant)
R = resistance rating (of target)
P = penetration rating (of caster)
N = amount of rating needed to take F (fraction) as much damage (universal constant)
Let us say for example that the constant F = 0.5, and constant N = 100. This means that every 100 Resistance gives half as much damage taken (multiplicative). It also means that every 100 Penetration grants twice as much damage dealt (multiplicative). Thus you get constant returns (at ANY point an extra 100 resistance makes you take 1/2 as much damage as you currently take, making you live twice as long), making it a much better game mechanic than the increasing returns (your first 25 resistance makes you take 3/4 as much damage, the next gives 2/3 that damage, and the next 1/2 as much damage) provided by a flat % resistance system such as Diablo 2's, which basically !@#$s you if you don't cap your resistances.
This. Although I would think that F and N are level dependent, correct? At least N would have to be level dependent. And by level dependent, I mean weighted based on the difference of the character and the monster.
Earth or somewhere like it.