Not sure if this is the pipeline.
If you assign a custom template to a product or category at the moment it will override the template you have set in the website template section.
What would be great is if the hierarchy went like this.
Default < Website < Store view < Category < Sub Categories < Product
Or something along those lines. So if i have a product that requires a slight layout change it can inherit all the other template files from the next template down in the hierarchy.