Remember, in Morrowind "Unarmored" was bugged.
You could increase it, but it would never 'reduce' damage UNLESS you had at least ONE piece of armor equipped as well.
to get around that for pure mages, I would usually wear leather bracers or neetch leather bracers, figuring mages would wear those at least. Or maybe a single pauldron *shrugs* something simple.
there's a mod for this out there, that equippis an invisible and very weak light armor helm on your character (so you still increase light armor albiet very slowly) and still lets you role -play out your mage.
I prefer just wearing gloves or bracers. And you can always 'shield' yourself, or just use some exquisit outfits (Rollie the gaur!!

) and give yourself "constant effect: restore health 1 - "x"" points and equip / reequip it until you get a number closer to "x"

(Fortify luck to help

:D )