Should I compile my application to the latest .NET Framework even if I'm not using any new features?

I've developed an application using Visual Studio 2010, which by default compiles to the .NET Framework 4. However, as far as I'm aware, my application is not using any .NET 4 specific features and would probably work fine compiled to 3/3.5 or even 2. The main problem I see with compiling to v4 is that many users won't have the v4 framework installed and so will need to go through the process of downloading and installing it. Is there any performance/security/etc benefits to running the same code compiled to v4 rather than previous versions which would justify using v4, or should I just use the older versions until I need the new features found in 4?

Amr