Do you have the Google toolbar installed?
If so, then it may be that you have "Use Gmail for mailto links" ticked under the toolbar settings (seperate menu from Firefox preferences - little spanner at the end of the bar). This overides everything else and will always result in gmail being opened.
