Ok, fixed this myself and this should be DEFAULT Magento behaviour with a setting in the backend when underlying condition is true..
Condition: this will only work when you have the same product-per-page numbers for list and grid view
In (famous) toolbar.phtml
<?php $currentpage = $this->getCollection()->getCurPage();?>
<?php if($this->getCollection()->getSize()): ?>
<?php echo $this->getModeUrl($_code) ?>
<?php echo $this->getModeUrl($_code) . "&p=" . $currentpage; ?>
Et Voila, a customer is not kicked back to category page 1 when they change grid-list view mode on page 12.
When somebody has a better way to make this also work with different product-per-page numbers, please reply to this topic.