Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Cut memory use by XX%, increase requests / sec, etc—Let’s get started! 
 
ShopGuy
Guru
 
Total Posts:  462
Joined:  2008-09-07
 

Whenever a collection is loaded (such as every time a list of products is displayed) all product data / models are loaded in memory at once. I wrote a script to try to calculate the memory cost of using this method to load a collection of data versus loading a product one at a time (so that memory can be reclaimed).

require 'app/Mage.php';
Mage::app();

// CONFIGURATION
$useDefaultMethod true;


// BEGIN SCRIPT
$m Mage::getModel('catalog/product');
$mc $m->getCollection();

$count 0;
$startMem $endMem 0;
$startMem memory_get_usage();
    if (
$useDefaultMethod{
        $mc
->addAttributeToSelect('*');
        foreach (
$mc as $model{
            
break;
        
}
    } 
else {
        $ids 
$mc->getAllIds();
        foreach (
$ids as $id{
            $model 
Mage::getModel('catalog/product')->load($id);
            break;
        
}
    }
$endMem 
memory_get_usage();
$count $mc->getSize();

$diffMem = ($endMem $startMem);

echo 
$count " products in the collection\n";
echo 
round($diffMem 1024) . "KB difference between start and end memory\n";
echo 
round($diffMem $count) . ' bytes of memory used per product';

$useDefaultMethod = true;

Running the script with the default method produces the following output:

120 products in the collection
2211KB difference between start and end memory
18870 bytes of memory used per product

$useDefaultMethod = false;

Running the script with a method that allows memory to be reclaimed outputs:

120 products in the collection
849KB difference between start and end memory
7241 bytes of memory used per product

Results of using the new method

A memory saving of 1,362KB or over one megabyte for a mere 120 products.

Using the old method for 10,000 products would use an additional 179 MEGABYTES while using the modified method will still use the same amount of ram.

Imagine the memory savings per time collections are used by Magento per PHP instance running on the server.

The Challenge

Now, I am wondering if there is any way we can seemlessly make app/code/core/Mage/Adminhtml/Block/Width/Grid.php not pre-load the collection, but instead use the modified method so that we can take advantage of huge RAM savings.

 
Magento Community Magento Community
Magento Community
Magento Community
 
crius
Guru
 
Avatar
Total Posts:  623
Joined:  2007-10-16
Denmark
 

You should probably consider execution time as well. Making 120 database queries takes a lot more time than one big query.

 
Magento Community Magento Community
Magento Community
Magento Community
 
ShopGuy
Guru
 
Total Posts:  462
Joined:  2008-09-07
 

That is true, but it is easily solved so they use the same number of queries. The above code is just displaying the memory overhead of loading all the classes at once. I am thinking of a method that utilizes the on-demand convenience of the modified method and the ability to utilize preloaded data.

This is how a collection is loaded:

public function load($printQuery false$logQuery false)
    
{
        
if ($this->isLoaded()) {
            
return $this;
        
}

        $this
->_renderFilters()
             ->
_renderOrders()
             ->
_renderLimit();

        
$this->printLogQuery($printQuery$logQuery);

        
$data $this->_fetchAll($this->_select);
        if (
is_array($data)) {
            
foreach ($data as $row{
                $item 
$this->getNewEmptyItem();
                if (
$this->getIdFieldName()) {
                    $item
->setIdFieldName($this->getIdFieldName());
                
}
                $item
->addData($row);
                
$this->addItem($item);
            
}
        }

        $this
->_setIsLoaded();
        
$this->_afterLoad();
        return 
$this;
    
}

As you can see models are loaded one by one and data is added from the row returned to the model.

I envision a method that does not require any additional queries AND uses memory efficiently. So, what would happen is the collection stores the query results in memory just like it does now. Then, you could interface with it like this:

while (($model $collection->getNextModel()) !== false{
 
// code here
}

Each time getNextModel() is found it would grab the next row, get an empty model, add the data from the row to the model, etc (just like it does not, except it will be done on an as-need basis instead of at the beginning).

Benefits of this method
Just as fast
Way, way, way less memory

 
Magento Community Magento Community
Magento Community
Magento Community
 
ttwhy
Member
 
Avatar
Total Posts:  70
Joined:  2008-01-27
 

sounds a bit “fakey” for me.

getting the ID’s of a Model is for sure less memory intensive than getting the full data (didn’t test the script, but i think a print_r will show the trues).

 
Magento Community Magento Community
Magento Community
Magento Community
 
ShopGuy
Guru
 
Total Posts:  462
Joined:  2008-09-07
 

No, not fakey at all. It operates the same way the collection class does. The second post of mine will operate EXACTLY as the collection class.

The ONLY thing we are doing is changing WHEN a model is loaded. If we load them all in the beginning we have to keep them in memory for a long time. If we load them one-by-one we can reclaim memory.

Btw, we did not just load the IDs. All the models are loaded here:

$model Mage::getModel('catalog/product')->load($id);

Again, we are only trying to accomplish one goal: Can we choose a more optimal time to load models to save memory?

I have already demonstrated this is possible. The next step to implement it in a way that requires as few changes as necessary and allows for the old behavior when needed.

 
Magento Community Magento Community
Magento Community
Magento Community
 
ShopGuy
Guru
 
Total Posts:  462
Joined:  2008-09-07
 

Bug report submitted: http://www.magentocommerce.com/bug-tracking/issue?issue=4512

Hopefully in the next release or two we can get more optimal collection management. Share hosts would benefit greatly.

 
Magento Community Magento Community
Magento Community
Magento Community
 
joyously
Guru
 
Total Posts:  447
Joined:  2008-08-21
 

Thank you for investigating this and writing the bug report.  I always thought it was crazy how much memory Magento uses, but hadn’t looked into it.

Is it just when a model is loaded? or any collection? or do the two go together?
It seems like you could change the base class so it gets handled transparently and you don’t have to go adding while loops in a lot of places.

 
Magento Community Magento Community
Magento Community
Magento Community
 
ShopGuy
Guru
 
Total Posts:  462
Joined:  2008-09-07
 

Joyously,

It is when a collection is loaded. The problem with collections is that ALL models are pre-loaded. Very inefficient from a memory stand-point. Model’s themselves are bulky, but there really is no good way to trim that fat.

Reason one to do the while() loop method is because of _afterLoads (or whatever they are called)

The only reason why I thought of the while loop implementation is because what happens is that XYZ’s custom collection class that extends the base class may do some _afterLoad() _beforeLoad() processing. With my method there would be no _afterLoad() because a collection is not actually loaded all at once. This could cause some problems with some collections that need the _afterLoad() to be called. So, the while() implementation makes it so that programmers can choose to use it when they know it is safe.

Most _afterLoads() do post-processing stuff. For example, an _afterLoad() may add attributes onto the model since that is stored in a seperate table it would not of been supplied by the original collection query. In these cases Magento could define an _afterItemLoad() and in 99.9% of cases it could just call _afterLoad() and make sure only one item is being acted on and not trying to act on the entire collection as a whole. So, in all, very few code changes would take place. It would have to be done by Varien though because it would touch a lot of core classes.

Reason two to do the while() method

The second reason is because of how the current implementation works. Lets look at this:

foreach ($collection as $item{
$item
->setName([name]);
}

foreach ($collection as $item{
$item
->setDescription([description]);
$item->save()
}

The current implementation will make changes to both the name and the description because the second loop is calling the same models as the first loop.

If we were to do my proposed change so it can be called as in the above it might give unexpected results for people used to the old way. The reason being is because in my implementation the second loop would re-create brand new models. Therefore, only the description would be updated.

I have never ran across an example like the above because most of the time you would do it all in one loop. I cannot think of any piece of code that would be affected, but just in case, you know. The last thing Magento needs is more bugs.

 
Magento Community Magento Community
Magento Community
Magento Community
 
kmikz
Jr. Member
 
Total Posts:  1
Joined:  2009-02-03
 

The biggest improvement I’ve had concerning requests/sec is using Zend Server http://www.zend.com/en/products/server/

Using wamp on a test machine I had 0,91 request/sec. After I installed APC I had 1,35 requests/sec. After moving the same site to Zend Server I had 2,63 requests / sec. Keep in mind that Zend Server is still in beta and I couldn’t install APC on it, but it definitely doing something right.

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top