Okay so back in the day (like a year ago) I could run Oblivion at max settings at 1600x1200 at about 15 frames.

A year later after a few mods (like 30 or 40 of them) Im down to 1400x1200 (windowed) and every other settings halfed. Medium textures....

Now I was wondering...is it just me or has my performance really gone down because of mods or is my PC just dying lol?

Do mods affect performance?
YES, even a basic script mod can reduce performance.
I am not an expert like some of the guys here but I do know that mods can put a strain on your game.
If a script tells the game to keep checking something (insert common mod stuff like factions) than you have slowed the performance down & if you add NPCs or even items.
Not to mention all those great looking re-textured higher, better looking mods.

Your operating system (registry) does get blowded over time also, this is normal for Microsoft products unfortunately.

As hoots7 pointed correctly most mods adds to the performance loss, but a few are designed with the purpose of enhancing it. There are others factors not concerning the mods and even the game themselves. Common troubles source is highly fragmented HD; should not be surprise than files that forces the HD arms going wild are more than a concern. This does more than just impact performance, it stress the mechanical parts over time.

Defragmentation should be performed with regularity and with reasonable times in between, but to some it is never done. Indeed defragmentation should be done whenever larger files are placed/changed into a partition.

Edit: larger applications with hundreds of files ranging from small to huge ones makes up for the worse scenery.

Framerate is only one part of the equation, and is mostly concerned with graphics cards. Something as simple as a body mod can occasionally kill a frame or two because all the bodies have more polygons. But most of the framerate loss comes from either textures or havok. If you have custom textures, often they are unaffected by changes to graphic detail because they are made without mip maps. It could also be the number of things within a single scene. Mods which add more NPCs or add more variety to an area mean more files need to be loaded when you are in that area, so you tend to have lower framerates.