The interesting thing about Vim and Emacs is not simply how quickly you can work. It's the simple fact that Vim and Emacs let you edit your text at nearly the speed with which you think about making your changes. Since you have no need to make selections, to switch to the mouse, or to hunt through menus, your mind is free to focus on what's important while you edit: Your code.
This is to say that when you're writing code, there is only one thing you want to think about: What your code must do and how it must do it. Whenever you stop to make changes to code you've written, it's an interruption; you're forced to think not about what your code is doing but how to make the changes that are necessary. So, instead of stepping back and make a visual selection and changing my mode of thought from code-writing to text-editing, Vim and Emacs allow me to perform the edit and be done with it, without having to expend more mental energy on the task of making the change.
And when you take your hand off the keyboard to grab the mouse, it's another kind of context switch: You're no longer thinking about the code, but about the text you can see on the screen and the visual elements that surround your editing area. Then, you must make the switch back to writing code, as your hand hunts for the home keys. With Vim or Emacs, this context switch never needs to be done, and your mind never needs to leave the task at hand: that of writing code.
A lot of people, myself included, use Vim or Emacs as their preferred tools because they're liberating in the sense that you spend less time thinking about editing and more about what changes you're making. Because, really, what is programming but juggling lines of text?
Plugins and extensions? I bet nothing comes close to R# or IDEA in terms of refactoring support [...]
These kinds of tools exist. Tips and methods abound. The only limit to Vim and Emacs is that their primary purpose is editing text. But that still doesn't stop anyone from making it the umbrella under which their entire IDE is built.
I think the source of your consternation is that you choose not to see value in Vim and Emacs. That's fine. I'm not going to try to convert you.
But I will say that having tried every text editor and IDE I've come across, I still go back to Vim. I'm comfortable with it, I'm happy with the tools it gives me, and my Linux desktop meets my every need as an IDE, all in the terminal window. It has made me dependent on Vim's peculiar command structure, but because Vim runs everywhere, that has never been a hindrance; only a boon.