Admin->System->Configuration > developer, Set current confguration scope to Main Website
Template PathHints YES, Add Block Names to Hints YES and save config.
The above will show you the path and block details.
You will notice they are seperate blocks.
The Layout .xml file is layout/catalog.xml
The default category block is leftnav.
The post above shows it commented out.
it appears in the following page layout modules.
<default>, <catalog_category_default>, <catalog_category_layered>
From your post you mention colors ect. which leads me to assume you are using layered navigation.
page module <catalog_category_layered> which includes in the <reference name="left">
<block type="catalog/navigation" name="catalog.leftnav" after="currency" template="catalog/navigation/left.phtml"/>
<block type="catalog/layer_view" name="catalog.leftnav" after="currency" template="catalog/layer/view.phtml"/>
The navigation left found in template/catalog/navigation/left.phtml is responsible for ‘Browse By’ (categories)
The layer_view found in template/catalog/layer/view.phtml responsible for ‘Shop by’ (Attributes)
To remove left.phtml comment out the block in the layout file. or remove it using ‘unsetChild’ custom design>layout update on CMS page
So the key to finding anything in magento is the path hints.
Once you get the hang of xml contains block contains .phtml can contain block can contain phtml you should be right ; )