Magento

I’m importing 5000+ simple products from an Excel XML. I have htaccess timeout set to 1800, but its timing out prematurely after 20 minutes, and only importing 100 records (93 1st attempt, 117 2nd attempt). Then when I go to delete the 100 simple products to try again, it takes very long. I’m waiting for it to delete right now, and I’m wondering if its going to timeout again.

I might try a CSV instead, but it just seems like these operations are taking an exceptionally long time to do very little. Last time I had the import working properly was three releases ago, and it only took around 10 minutes to import all 5000 simple products.

Any thoughts?

Update 1: I checked the list of of products on another browser and there was only 1 left while my other browser is still “loading” to complete the delete operation. It seemed to delete 99/100 products, but then it just hung there. Perhaps there was something about that product that was bugged… maybe when the import timed out. I’m only speculating though.

Update 2: Great. Now the dashboard page generates a fatal error after I deleted the remaining product on a 2nd browser, while the 1st browser timed out. I obviously messed something up - hope I don’t have to reinstall.

Update 3: I’m attempting to import a CSV, and it looks like I’m going to have similar results. After I initiated the import and while my 1st browser is in a loading state, I immediately checked the list of products under “Manage Products” in a 2nd browser, and there were 50 products right away, but no more. It looks like it stopped importing, but my 1st browser stays in a “loading” state. The dashboard page is still broken.

Update 4: Products seem to import in bursts. Some records might pause for minutes before getting to the next one, and then suddenly there are several. Eventually, the browser (Firefox) seems to just give up on waiting. I’m guessing there is something in (or not in) my data that’s causing a problem, but the import isn’t cutting out at any one particular product.

Update 5: Could be host? I’m looking in phpMyAdmin, and the created dates are all within a second or two - and then the next one is a whole 7 minutes later - the next few are all within seconds, and then minutes later again. Ultimately, it doesn’t get very far. Oh yeah, the dashboard fixed as I tried to update products, which subsequently timed out.

Update 6: Doesn’t look like host performance. Successfully imported 2000 sku’s, but when I tried to delete ‘em, my browser timed out and the database corrupted, so I had to reinstall. I am certainly having trouble moving data around. I could, and probably will, import in more bite-sized chunks, but what bothers me is that I did successfully import 5000 products in 10-15 minutes a few releases ago. I don’t think it was my imagination.

any updates on your dataflow import?
i want to try to import about 90,000 products

Increased php_value memory_limit to 512M
able to upload 1306 records.

Any ideas on tweaking to import 90,000 at once ?

though not an ideal solution, can you try in bursts of 500? run multiple imports of 500 records each and see if it helps…

we made improvements to the import feature and from our tests it shows that it can handle many more products. We have not tried 90k products yet

thanks

yoav

thanks Yoav ...two things that were of prime concerns for me
1. Was ability of dataflow to handle large quantities to the tune of tens of thousands (would be better if you can give us the number of products imported in your tests)
2. Efficiency of search when there is a huge catalog. Hope the search is also being worked upon for the first version.

Thanks.

Haha i know 90k is alittle crazy but would be nice
or tips on auto batching process

No luck here yet with my 5000. I may attempt to import programatically using PHP. Magento, correct me if I’m wrong, doesn’t seem to poll the browser in any way during a large import. Hence when 20 minutes passes without server response, the browser assumes something is amiss and then terminates the connection. I believe that if progress is echoed to the browser as the import loops (ie: echo “Record 4225 saved!<br>”;), it’d stay alive for much longer.

In any case, I have images that need relocated and reprocessed and everything. I don’t think Dataflow in its current form will be adequate for this massive, one-time import. Don’t get me wrong - its good, and I think you guys are doing a fantastic job. I just want Dataflow to be able to wash my car, and make me breakfast too.

good suggestion ..and more importantly it will keep us posted that records are being saved every now and then

And noticed one more thing,
http://www.magentocommerce.com/blog/comments/magento-0917740-available-for-download/#comments
seems to point that

Using 128MB: for customers the maximum is 406 items, for products the maximum is 12927 items from our tests

12 thousand products is good ...but from this thread and from my own experience I seem to face problems for smaller quantities (around 5000)...wonder why or are we missing something that Magento team tested with?

With version 1.0 out, I am quickly out to test dataflow performance on the production version

Keep me posted. I’m going to give it another stab, myself.

yeah me too,
update us on the amount of product you manage to import

I’m curious about importing too, because i ge this error:

Starting profile execution, please wait…
Warning: Please don’t close window during importing/exporting data
Warning: is_dir() [function.is-dir]: open_basedir restriction in effect. File(/home) is not within the allowed path(s): (/home/trace:/usr/lib/php:/usr/local/lib/php:/tmp) in /home/trace/public_html/magento/lib/Varien/Io/File.php on line 443[0] in is_dir("/home") in /home/trace/public_html/magento/lib/Varien/Io/File.php on line 443[1] in Varien_Io_File->_createDestinationFolder("/home/trace/public_html/magento/var/tmp/") in /home/trace/public_html/magento/lib/Varien/Io/File.php on line 417[2] in Varien_Io_File->createDestinationDir("/home/trace/public_html/magento/var/tmp/") in /home/trace/public_html/magento/app/code/core/Mage/Dataflow/Model/Convert/Parser/Abstract.php on line 109[3] in Mage_Dataflow_Model_Convert_Parser_Abstract->_copy("/home/trace/public_html/magento/var/import/import-20080331185113-1_import.csv") in /home/trace/public_html/magento/app/code/core/Mage/Dataflow/Model/Convert/Parser/Csv.php on line 75[4] in Mage_Dataflow_Model_Convert_Parser_Csv->parse(Array[0]) in /home/trace/public_html/magento/app/code/core/Mage/Dataflow/Model/Convert/Action/Abstract.php on line 235[5] in Mage_Dataflow_Model_Convert_Action_Abstract->run() in /home/trace/public_html/magento/app/code/core/Mage/Dataflow/Model/Convert/Profile/Abstract.php on line 132[6] in Mage_Dataflow_Model_Convert_Profile_Abstract->run() in /home/trace/public_html/magento/app/code/core/Mage/Dataflow/Model/Profile.php on line 142[7] in Mage_Dataflow_Model_Profile->run() in /home/trace/public_html/magento/app/code/core/Mage/Adminhtml/Block/System/Convert/Profile/Run.php on line 73[8] in Mage_Adminhtml_Block_System_Convert_Profile_Run->_toHtml() in /home/trace/public_html/magento/app/code/core/Mage/Core/Block/Abstract.php on line 553[9] in Mage_Core_Block_Abstract->toHtml() in /home/trace/public_html/magento/app/code/core/Mage/Adminhtml/controllers/System/Convert/ProfileController.php on line 189[10] in Mage_Adminhtml_System_Convert_ProfileController->runAction() in /home/trace/public_html/magento/app/code/core/Mage/Core/Controller/Varien/Action.php on line 341[11] in Mage_Core_Controller_Varien_Action->dispatch("run") in /home/trace/public_html/magento/app/code/core/Mage/Core/Controller/Varien/Router/Admin.php on line 141[12] in Mage_Core_Controller_Varien_Router_Admin->match(Mage_Core_Controller_Request_Http) in /home/trace/public_html/magento/app/code/core/Mage/Core/Controller/Varien/Front.php on line 147[13] in Mage_Core_Controller_Varien_Front->dispatch() in /home/trace/public_html/magento/app/Mage.php on line 424[14] in Mage::run("default") in /home/trace/public_html/magento/index.php on line 40

Starting Mage_Dataflow_Model_Convert_Parser_Csv :: parse
Unable to create directory ‘/home’. Access forbidden.
Finished profile execution.

Still facing some errors with my customization, but the import process seems niftier and much much better ...but need to test with more data ...just my first impression ...will keep you posted on more…

Well ... I just checked my php_value memory_limit and it’s at 40M ... I’m just trying import 3500 and it stops adding after 19 or 20 records. It’s an edited CVS from an export with my data in place of the generic data.

So, any progress? Any ideas?