OK I had a second look at this.
Your primary problem is your home page payload ~ of 1.8Mb - your machine will struggle to serve this to every visitor as you do not have Apache’s expires enabled which compounds your problems.
Suggest you resize these images, save as jpg’s and optimize for the web first, consider adding a CDN like Amazon’s Cloudfront and serve your static content from that which will take a lot of load off your VPS.
Also, depending on ping times, consider moving your hosting from Germany to Netherlands ( would need to test latency to see if worthwhile) however there is some SEO benefit from hosting in same country as your TLD.
The other delays you are experiencing I would suspect are mysql based at a guess
This may help others with performance issues. Running Nginx increases performance multiple times, using a web cluster such as three AWS micro instances plus an admin micro instance outperforms a single small instance due to parallel processing, obviously scaling up will help.
A CDN is also beneficial to offload the image loads to the users region such as CloudFront, however there are issues with Magento and the way it caches images if using a web cluster, you need NFS for the media and skins. APC needs to be activated for php caching along with compiled php in Magento. With this you should have sub-second page loads, you can also add memcache to enhance the performance further along with a read only database and a separate write database.
If the issue is the actual loading of the html page (not downloading the content), then you really need a Magento Full Page Cache Solution. This will get your load time down to about 80ms (that is 0.08s or less).
Based on how much slower your site runs when it is busy, it appears that your server is overloaded. The full page cache will reduce server load, so that more requests can be served to more users (and faster to boot).
If HTML performance is what concerns you most, here are some tips to help you boost the page loading speed.
- make sure you’re saving your images in the right format (jpeg and png) - it’ll lessen loading times
- reduce the number of HTTP requests by combining images into CSS sprites.
- avoid inline Java and CSS, it makes your HTML size much larger and prevents code caching.
- enable compression on your server
You can use Shopping Cart Diagnostics for measuring your website performance and optimising the necessary elements (the ones stated above and more).