In general Magento handles great amounts of products and I came across the stores with 100 000 products and more. I have to mention that the speed of your admin side greatly depends upon your server environment. You also need to take into consideration that the public side of your web-site needs to have an appropriate load speed also.
In case your store frontend with caching options is working fast enough, and you would like to speed-up the work with your inventory only, so I can offer you to check a PC application called Store Manager for Magento - http://www.magentocommerce.com/magento-connect/store-manager-for-magento.html
Using it you can upload or change via import thousands of product in bulk locally with no delays at all (using so called Bridge connection) and synchronize data once bulk changes are made.
In any case, you can give it a try, since there is free trial version available for download.