As a solution you can try to bridge Magento and Joomla using jFusion and then set a forum software for Joomla. In this way the applications will share the same users’ database.
Also, I have searched the Magento online resources and it seems that there is a vBulletin integration option, which is a provided as a paid module:
You can contact the developers of this product and check whether it will meet your needs.
The third solution which I have found is regarding a SMF - Magento bridge:
It is a free one. You can test it for your software installation.