Emacs Org-Mode: Turn off line numbers

I've recently moved from vim to Emacs because I want to use org-mode. I opened a ~10000 line, 50kb file in Emacs23 Org-mode and proceeded to add about 10 first-level headings. Performance on a quad-core with 3GB RAM in Emacs23 under Ubuntu 10.04/32bit was so slow that it was unusable. I found two threads on the Org-mode email list discussing this. It seems that enabling linum causes the slow performance. I can live without line numbers in .org files if I have to, but I don't want to disable line numbers for all files I edit. If I'm going to "live" in `Emacs', I'll want line numbers for all other files.

How can I disable linum for some or all .org files only? Is it possible to do this if I have several files open in Emacs and switch between them? I found some discussion about disabling line numbers for major modes here, but there was nothing that I could implement (although the linum-off.el script mentioned on the page looks promising, I don't (yet) know (E)Lisp, so I can't change it as I would need).

I updated Org-mode from version 6.21b which came with Emacs23 to version 7.5, but it made no difference. Performance in Emacs GUI is so bad that the application fails to respond at all. Performance with -nw is "better", but still unusable.