Your shopping cart is empty. Browse our Store

Magento

Open Source eCommerce Evolved

Magento Forum

   
Performance issues and database growth
 
Pisang
Jr. Member
 
Total Posts:  5
Joined:  2008-05-01
 

After having added about 200 items in the store it seems the size is getting to about 85 MBs now.
With adding 1 single product, 1.0-1.2 MB is added now and the database is using like 800,000 rows.

Also, the backend is getting noticably slower and slower (the frontend is doing ok).
At some point in time, the following message popped up:

Fatal error: Maximum execution time of 30 seconds exceeded in /home/user/domainame.tld/html/shop/lib/Zend/Db/Select.php on line 1101

The thing is, it takes about 30 seconds now to duplicate an item. There’s not much to it. Just some text and an attribute set of course.

Anyone with the same issue(s) and/or a solution to this?

Your feedback is appreciated.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Michael
Magento Team
 
Total Posts:  573
Joined:  2007-08-31
 

The database containing 5 000 products in 80 categories and 3 store views is 151.8 MB and ~ 660 000 rows.

The database containing 80 000 products in 1 500 categories and 1 store view is 470.3 MB and ~ 3 470 000 rows.

What are the largest tables (by size, not by rowcount) in your database ?

 
Magento Community Magento Community
Magento Community
Magento Community
 
srinigenie
Guru
 
Avatar
Total Posts:  392
Joined:  2008-02-04
 

I have a catalog of 4000 odd products and the total space shown by PhpMyAdmin for all tables is 189 MB of which it was interesting to note that
sto_dataflow_batch_import alone took up 141 MB.

But I see another problem when my catalog was of this size ... the search performance decreases exponentially .... particularly if the search results hits are more ...if a search keyword has a 100 hits, the search takes close to 30-40 sec !!!! that was pretty discouraging ... was wondering if attributes of all the 100 hits are being queried up and that is the reason this appears to degrade the performance ...but not sure .....discussing this in a separate thread anyway

http://www.magentocommerce.com/group/blog/action/viewpost/384/group/168/

 
Magento Community Magento Community
Magento Community
Magento Community
 
dan_w
Sr. Member
 
Total Posts:  225
Joined:  2008-01-04
 

I’ve been noticing the same thing. We have a filtered layer nav attribute called item_classification, which is based on an ever growing classification guide that can have a limitless number of classifications. As we test the item_classification with larger sets, it certainly does seem to draw alot of resources in various scenarios such as the search functionality. I do think that, like you suggested, the search query is loading all of the attributes, which is consuming alot of resources. Right now I think we’re trying our item_classification attribute with upwards of ~400 entries, on ~ 4200 products. I’ll have to double check though as we are no longer able to access that attribute via the admin - any attempts just stall. I understand that attributes can be excluded from search results, but since our item_classification attribute is a dynamically evolving attribute based on keyword research which is updated from search engine queries, we really want to see it included in the search results. Hopefully we can make progress in this area.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Crucial
Guru
 
Avatar
Total Posts:  408
Joined:  2007-11-07
Scottsdale, AZ
 
Pisang - 09 May 2008 04:26 PM

Fatal error: Maximum execution time of 30 seconds exceeded in /home/user/domainame.tld/html/shop/lib/Zend/Db/Select.php on line 1101

That’s telling me there’s memory issues. What kind of environment are you running in? If it’s a VPS or dedicated, how much RAM is allocated to either the VPS or server? Have you tried increasing the max_execution_time and/or memory_limit for PHP?

 Signature 

Crucial Web Hosting
Are you ready to Split-Shared™ hosting?
Magento Containers lets you run your business, we’ll do the rest.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Pisang
Jr. Member
 
Total Posts:  5
Joined:  2008-05-01
 

Yes, the limit has been set to128MB.
It is run in a VPS with about 512MB dedicated memory, only about 60 MB is used currently. All is maintained by ourselves.

max_execution_time limit is set to 30

php-cgi process is quite busy during duplication.

After about 40 seconds, the process fails. Displaying a different message (line differs).

Fatal error: Maximum execution time of 30 seconds exceeded in /home/user/domainname.tld/html/shop/lib/Zend/Db/Adapter/Abstract.php on line 887

 
Magento Community Magento Community
Magento Community
Magento Community
 
Pisang
Jr. Member
 
Total Posts:  5
Joined:  2008-05-01
 

catalogindex_eav :: 1264745 rows :: 123,5 MB
catalogindex_price :: 400590 rows :: 68,7 MB
core_url_rewrite :: 4733 rows :: 2,9 MB
catalogindex_minimal_price :: 7246 rows :: 1,9 MB

These are the largest files. Only the first and second are bizar.

Actually, the number of products is about 160. The last 20 products added (without illustrations) have made the database catalogindex_eav grow from 85 to 123,5 MB.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Crucial
Guru
 
Avatar
Total Posts:  408
Joined:  2007-11-07
Scottsdale, AZ
 

I figured you were on a VPS. This is a recurring issue for a lot of people, they’re either getting blank pages or timeouts, all stemming from an environment with too little memory allocated.

What control panel are you using? If it’s cPanel, then already a lot of memory is being used by that, since it requires 256MB to even be installed on a system.

Have you tried disabling any unnecessary services to free up some room?

 Signature 

Crucial Web Hosting
Are you ready to Split-Shared™ hosting?
Magento Containers lets you run your business, we’ll do the rest.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Pisang
Jr. Member
 
Total Posts:  5
Joined:  2008-05-01
 

Yes, blank pages and timeouts indeed.
We use Parallels Containers. No Plesk, just plain Fedora as clean as possible, no unneeded packages.

There should be nothing wrong with a VPS, also the memory is more than sufficient since it can boost to over 4 GB.
But, what I have learned today is that the duplicate feature is causing all these troubles.
I have started using adding products without using the duplicate feature and this not only goes well, it is quicker than using the duplicate feature with the waiting period (2x ±30 seconds).
Also, after having added 30 products, the database growth was 1,0 MB and the rows (_eav) had grown from 10001912 to 1170459, this is quite different then before. Strangely enough this turned into 770.000 rows after adding a few more products.

So far it seems to be going ok, But the duplicate feature seems to be the cause of these troubles.

 
Magento Community Magento Community
Magento Community
Magento Community
 
listerine
Jr. Member
 
Total Posts:  11
Joined:  2007-11-21
 

performance is the issue that will always be there due to the design of this shopping cart

i happened to worked on a site 5 months ago with over 2000 sku.  we have to ditch the whole front end, there’s NO way this CART run fast . period .

the best you can do is to add APC, use lighthttpd/fastCGI , mysql query cache a must.

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
    Back to top
 
Sales: Call 877.832.5289 (North America) 310.295.4144 (International) to request a call-back.
© Copyright 2008 Varien. Magento is a trademark of Irubin Consulting Inc. DBA Varien
Privacy Policy|Terms of Service
Magento Community Count
32115 users|308 users currently online|72010 forum posts