GTX 970 - Weak Performance

I never went over 2.5gb though

No, the 970 is not your problem.. I can say this with 100% certainty.

I run a [email protected] and [email protected],5ghz, what you want to do is run AB and set framerate on, then look at the point of the min framerate graph and pick an fps cap value so framerate will be 90% flat line.
That will fix it I bet. I do run high-med setting and my usage % are avg CPU 75%, GPU 60% if your getting 90+ on CPU that will be bad for smooth play.
I run an fps cap of 63 with Vsync off on a 60hz monitor.

Yeah, the 4790k was beginning to show its age.