Grizzly + Jersey Listening ONLY on Localhost

I'm using Jersey with an embedded version of Grizzly and I'd like to bind/listen on localhost ONLY. I'm creating the ThreadSelector using the GrizzlyWebContainerFactory with the create call:

threadSelector = GrizzlyWebContainerFactory.create("http://127.0.0.1:8080/", initParams);

This works, but I'm still able to hit the server from an external machine. How can I get it to bind to/listen on ONLY localhost?

This is for configuration stuff, so I don't want anything off box to be able to connect to this server.