Must I still keep my 3rd party library binaries in source control?

Now that I'm using NuGet to add/update my required references, is it possible to only add packages.config to source control and have the .dlls added at build time?

Normally, like described in this related question, I store third party dependencies in a folder under the solution root and check them in to source control.

We're currently not using any custom build tools or scripts: just VisualStudio's default build with the .sln file.