I made a spell that uses the rigidbodymass control to enact a petrify effect. The spell works fine until dissipation, at which time a large area of terrain becomes semi-solid, causing huge problems. I'm using checks like IsActor and Dead to dispel the effect on non-NPC/creature targets.

Any suggestions/ answers?

Only suggestion is don't.

Using setrigidbodymass can have a wide range of undesired effects, like you mention, and can occasionally cause problems with the havok engine.

Do IsActor == 0 and Dead == 1 always prevent a script from running on an inanimate object?

Technically, No scripting is being run on things like rocks, buildings, or the terrain... So what you are experiencing is a glitch in the havok system. Additionally, setrigidbodymass will only affect things which have collision that is movable (clutter) or actors which are in ragdoll mode. It should have no effect of statics, or actors which are able to move around.

Sucks, because Unconscious and Restrained don't provide the effect of a character being turned to stone. The spell is awesome except for the Havok glitch.

Paralysis doesn't do it?

Oddly enough, it doesn't completely stop the "rag doll" effect. I'm setting paralysis to 1 (on) and it stops animation, but the bodies still bend at the joints when they fall; not much of a stone effect.