I have the armrest and I can't really add too much more to what other people have said. The thing that sticks out in my mind is that the catch for securing the top of the armrest is pretty weak, and I manage to open the blasted thing more often than not. It's gotten to the point that I'm unconsciously closing the top of the armrest now.

For the manual shifter, I do get annoyed with it because it does intrude a bit but even still, I think I'd keep it because it is certainly nice being able to let the right arm rest a bit. People are pretty adaptable, and good at ignoring minor annoyances.