This is an interesting concept, but unfortunately one that Magento, by default, isn’t setup to handle. The problem is the way the category is built to show products on the grid view.
Near the bottom of the xhtml stack, you have list items <li> that contain each product. This works well for the normal layout but makes what you want to do impossible without modifications.
The problem is that each list item floats left which stacks it against the previous list item and when there is no more space they break and start a new line. What you need to do is have each one float left AND clear left. But this is where it gets tricky.
If you try to clear left and float left on a stack of list items, they will form one column and will never spill over into the second or third columns.
What you need to do is change the way the products are spit out so that a separate div contains each column. Each div will float left (no clear) and inside of the divs, each li will float left and clear left.
You will need to modify app/design/frontend/PACKAGE/THEME/template/catalog/product/list.phtml
The general outline of the new logic should be, add a new div, after every 3 (or however many products to a column) close the div and repeat.
Then add CSS that has each div float left and each li in the div float and clear left.
Make sure you add <div style="clear:both;"></div> after the loop so that the parent container doesn’t collapse if it itself does not float.