I noticed that the interface was extremely laggy no matter what I did. Maxed out settings, installed guest additions ect. Nothing helped. I then found this article:
https://askubuntu.com/questions/651013/can-i-remove-unity-now-and-how/651088#651088
Apparently the issue is Unity, and there is no fix. After rolling back to Gnome following the instructions there, performance was fantastic.
Hopefully this helps someone else.