I plan on buying the Lenovo Ideapad Y580 20994KU?

Is the Lenovo Ideapad Y580 20994KU a good laptop?

Link: http://www.amazon.com/...0084C2ZR2/

What can it run while using Dxtory to record? At how FPS? What settings? How could I upgrade it?

Unless you can get the Y580 below 1k, it isn't worth it.
The processor is a 3rd generation whereas the latest intel processor 4th generation is based on the Haswell architecture which uses significantly less power; ideal for laptops where temperature and battery life are major constraints.

The 660m gtx was a brilliant card in its day however Nvidia recently released the 860m gtx which is a LOT more efficient and powerful.
Here is a performance graph between the 760m and the 860m to give you an idea:
http://www.geforce.com/...erformance

I'm not too familiar with Dxtory but Nvidia recently released ShadowPlay that lets you capture gameplay.
Some info: http://www.geforce.com/...ng-laptops

You can download it here:
http://www.geforce.com/...shadowplay

Get a 4th generation intel (starts with a 4, e.g 4xxx) and a 850m gtx or 860m gtx (both are great cards).

Here are some examples:
If you are willing to get used to Windows 8, I would recommend that you look at this 17 inch laptop:
http://www.amazon.com/...00IKF2H12/

Intel Core i7-4700HQ 2.4Ghz (Turbo 3.4 GHz)
12 GB DDR3
1 TB 7200 rpm Hard Drive
17.3-inch screen, GeForce GTX 860M 2GB
Windows 8.1, 3.5-hour battery life

It is cheaper than the Y580 and has more harddrive space, more ram, much faster graphic card and a more power efficient CPU.

Here is another link to a 15 inch MSI laptop for 1.2k with windows 8.1:
http://www.amazon.com/...00IMTQ3UW/

If you can get the Y580 at around $800, get it but if not, it is way overpriced and there are a lot of better laptops out there.