Magento Forum

   
Importing products - using a cron job to call Magento Dataflow import
 
EazyEggbert
Jr. Member
 
Total Posts:  18
Joined:  2009-02-13
 

Hm...it seems like our isp does not allow that expression. Should it not be “0 0/10 * * *” anyway? Thank you and best regards from Hamburg!

Felix

 
Magento Community Magento Community
Magento Community
Magento Community
 
jan212
Guru
 
Avatar
Total Posts:  407
Joined:  2008-01-03
 

don’t believe that but could be possible… No, you and your example are wrong, sorry smile

The cron syntax is

minute:hour:day:month:weekday

So my example should work, you could try this code which is also running every 10min. instead

0-59/10 * * * *

But i recommend to save every bit you can, the code i posted before is clean and working.

*/10 * * * *

 
Magento Community Magento Community
Magento Community
Magento Community
 
EazyEggbert
Jr. Member
 
Total Posts:  18
Joined:  2009-02-13
 

ok thanks, now I understand…

hm...for us it does not seem to work with anything but “* * * * *”...using this expression I notice that the profile is being ran 2-5x shortly within a couple of seconds...however the file is not being read until the end when using the cronjob...again: manually the import works perfectly…

I am confused and I dont quite understand the principle behind the cron functionality in this case.

1. My isp allows us to run cronjobs via script or http call...I use the http call and set it to “once an hour”. the address for the cron to run is http://magento.mydomain.com/cron.php
2. cron.php runs all functions that are defined as crons in config.xml and those that are defined with lilmuckers import/export extension
3. what happens now? does lilmuckers extension make my advanced import profile start itself repeatedly? and how does the cronjob expression affect the interval since I configured the interval with my isp…

Thanks a lot!

Felix

 
Magento Community Magento Community
Magento Community
Magento Community
 
jan212
Guru
 
Avatar
Total Posts:  407
Joined:  2008-01-03
 

Hi Felix,

that’s a problem you should tell your provider - to make it clear, if * * * * * is running, everthing else should be running 2 cause you are using the shortest period then.

If you call cron.php it startup the shedulded tasks, in- and externaly (means: core and extension related). Every Extension using cron has its own XML configuration including the cron setup (interval). You can view the tasks in your DB, Table cron_shedulded…

If i understand it right, it’s not a real difference in outcome, the main important thing is to call cron.php which is calling lilmuckers extension and the internaly setup.

Ob du jetzt die Cron.php aufrufst die dann wiederrum die Config der Extension einliest ist im Ergebnis insofern kein Unterschied als das in beiden Fällen das Script alle 5min ausgeführt wird. Du musst halt die cron.php ausführen weil die Architektur dies erfordert wie du richtig erkannt hast.

Im weiteren heißt das, es reicht die cron.php z.B. stündlich aufzurufen, wenn du die lilmuckers Ext. auf 5min gesetzt hat wird die auch weiterhin 5-minütlich ausgeführt.

 
Magento Community Magento Community
Magento Community
Magento Community
 
nafnaf1000
Sr. Member
 
Total Posts:  209
Joined:  2008-02-21
 

code to expand a file on import/export

http://www.magentocommerce.com/boards/viewthread/67223/

 
Magento Community Magento Community
Magento Community
Magento Community
 
bailey86
Member
 
Total Posts:  47
Joined:  2009-03-05
 

Hi,

The script from the page before is importing records fine - but I have to import approx 45,000 records and I’m running into memory problems.  See at bottom.

Now - I can increase the memory allocation with

ini_set('memory_limit''128M');

And because this test machine has only 750MB RAM I can go for a live machine with more RAM again.

But I was wondering if there was something else which could be done.  Maybe something could be changed so that the script does not keep consuming memory.

Or maybe there’s nothing to be done - and this is caused by some sort of memory leak RE PHP’s database connection.  In the meantime, I’m going to see how far I can push memory_limit up to - and how many records can be brought in in one go - if necessary I can break the import files to smaller sizes.

moss1@cotton:~/public_htmltime php import.php 3 data.csv
Array
(   
    
[profile_id] => 3
    [store_id] 
=> 0
    [created_at] 
=> 2009-11-27 17:48:35
    [batch_id] 
=> 41
    [params] 
=> a:8:{s:9:"delimiter";s:1:",";s:7:"enclose";s:1:""";s:10:"fieldnames";s:4:"true";s:5:"store";s:1:"0";s:17:"number_of_records";s:1:"1";s:17:"decimal_separator";s:1:".";s:7:"adapter";s:31:"catalog/convert_adapter_product";s:6:"method";s:5:"parse";}
    [adapter] => catalog/convert_adapter_product
)
Processed: 2040
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 40180 bytes) in /home/moss1/public_html/lib/Zend/Date.php on line 1174

real    18m32.535s
user    12m15.110s
sys     3m25.553s

 
Magento Community Magento Community
Magento Community
Magento Community
 
kjyothis
Jr. Member
 
Total Posts:  9
Joined:  2009-11-24
 

It was a very valid post from srinigenie

Thanks a lot.

Krishna

 
Magento Community Magento Community
Magento Community
Magento Community
 
mysthr
Jr. Member
 
Total Posts:  5
Joined:  2009-09-18
 

Hi,
Thank you for posting this usefull script, but I got an issue.
I got a .csv file that contains all the products I want to import.

When I use the profile import directly in the Magento’s Back Office, it works pretty well.
But when I use your script I got this type of error for the same .csv file and the same profile :

2009-12-10T17:06:53+00:00 DEBUG (7): SQLSTATE[21S01]: Insert value list does not match column list: 1136 Column count doesn’t match value count at row 1

I really don’t understand what ‘s happen as the Magento controller works and not the script ?
I run magento ver. 1.3.2.4 is there some changes ?

Thank you for your help.

Erwan.

 
Magento Community Magento Community
Magento Community
Magento Community
 
DavorNo
Jr. Member
 
Avatar
Total Posts:  21
Joined:  2008-07-26
 

Hi,

my advanced profile for export runs manually. How to setup to work with cron?

My Action XML:

<action type="catalog/convert_adapter_product" method="load">
<var name="store"><![CDATA[0]]></var>
</action>

<action type="catalog/convert_parser_product" method="unparse">
<var name="store"><![CDATA[0]]></var>
<var name="url_field"><![CDATA[0]]></var>
</action>

<action type="dataflow/convert_mapper_column" method="map">
<var name="map">
<map name="sku"><![CDATA[sku]]></map>
<map name="name"><![CDATA[name]]></map>
<map name="qty"><![CDATA[qty]]></map>
<map name="price"><![CDATA[price]]></map>
</var>
<var name="_only_specified">true</var>
</action>

<action type="dataflow/convert_parser_xml_excel" method="unparse">
<var name="single_sheet"><![CDATA[]]></var>
<var name="fieldnames">true</var>
</action>

<action type="dataflow/convert_adapter_io" method="save">
<var name="type">file</var>
<var name="path">var/export</var>
<var name="filename"><![CDATA[export_all_products.xml]]></var>
</action>

Cron Expression is: 0 8 * * *

I need this configured export every day.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Sebastian Caffarelli
Jr. Member
 
Total Posts:  7
Joined:  2009-05-15
 

Hi I’m having the same issue right now. Any idea how to fix it?

Thanks
Sebas

mysthr - 11 December 2009 12:35 AM

Hi,
Thank you for posting this usefull script, but I got an issue.
I got a .csv file that contains all the products I want to import.

When I use the profile import directly in the Magento’s Back Office, it works pretty well.
But when I use your script I got this type of error for the same .csv file and the same profile :

2009-12-10T17:06:53+00:00 DEBUG (7): SQLSTATE[21S01]: Insert value list does not match column list: 1136 Column count doesn’t match value count at row 1

I really don’t understand what ‘s happen as the Magento controller works and not the script ?
I run magento ver. 1.3.2.4 is there some changes ?

Thank you for your help.

Erwan.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Sebastian Caffarelli
Jr. Member
 
Total Posts:  7
Joined:  2009-05-15
 

Hi All so far I could find something about it but I’ll need some kind of help from any of you.

The error below seems to be related to the setlocale option in the PHP. I saw that magento uses the following statement:

setlocale(LC_ALL, Mage::app()->getLocale()->getLocaleCode().’.UTF-8’);

well I’m pretty sure that the issue in somehow is related to that.

The field to be imported that is causing the issue is the “price”. This is due to my decimal pointer was set with the dot insted of comma. If I remove the dot and replace with the comma the process work fine with of course the result is not the correct due to the price is not correctly.

Any idea how I can resolve this big issue for me?

Thanks a lot
Sebastian

Sebastian Caffarelli - 07 January 2010 08:55 AM

Hi I’m having the same issue right now. Any idea how to fix it?

Thanks
Sebas

mysthr - 11 December 2009 12:35 AM
Hi,
Thank you for posting this usefull script, but I got an issue.
I got a .csv file that contains all the products I want to import.

When I use the profile import directly in the Magento’s Back Office, it works pretty well.
But when I use your script I got this type of error for the same .csv file and the same profile :

2009-12-10T17:06:53+00:00 DEBUG (7): SQLSTATE[21S01]: Insert value list does not match column list: 1136 Column count doesn’t match value count at row 1

I really don’t understand what ‘s happen as the Magento controller works and not the script ?
I run magento ver. 1.3.2.4 is there some changes ?

Thank you for your help.

Erwan.

 
Magento Community Magento Community
Magento Community
Magento Community
 
pastanislas
Member
 
Total Posts:  46
Joined:  2008-07-29
 

Hello Guys , I need your help !

I’ve tried both method in this thread and they both .... almost work.

Let me explain ,

- I only got one store with one website and one store view
- I use a csv file with 2 products for testing, values inside are admin values, original magento attributes name are in first row (store= admin, website = base),
- If I try importing this file via advanced profile or profile, It works like a charm : 2 products with all filterable attributes filled in , everything is ok.
- If I ‘cronify’ the import, whether using the magento connect module or the php script,
ONLY FIRST PRODUCT OF FILE HAS FILTERABLE ATTRIBUTES FILLED IN
, the second one only has text attribute filled in (price, name, color and other of my text attributes, filterable attributes are empty)
I’m sure it’s not coming from my file (tried other product line).
I think that the first row is well imported using ‘admin’ values and the second one is imported using ‘store view specific’ values (whicch are not in my file)..
I’ ve tried using this line in the code without success (Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID);)

Anybody has a solution ?

 
Magento Community Magento Community
Magento Community
Magento Community
 
monkey5255
Jr. Member
 
Total Posts:  4
Joined:  2009-08-24
 

Hi all,

seems I have a problem with this extension, too. I set all up as described here and with the extension but all in all the cron won’t run. Database (cron_schdule) and backend states the import was a success but no products are imported nor any data has changed.

I use the importscript made by CommerceThemes and manually the import runs smooth as can be. Only automated import won’t work.
What did I wrong, does anybody have a clue?

Here’s my setup for the profile:

<action type="dataflow/convert_adapter_io" method="load">
    <var 
name="type">file</var>
    <var 
name="path">var/import</var>
    <var 
name="filename"><![CDATA[products.csv]]></var>
    <var 
name="format"><![CDATA[csv]]></var>
</
action>

<
action type="dataflow/convert_parser_csv" method="parse">
    <var 
name="delimiter"><![CDATA[,]]></var>
    <var 
name="enclose"><![CDATA["]]></var>
    <var name="
fieldnames">true</var>
    <var name="
store"><![CDATA[0]]></var>
    <var name="
number_of_records">1</var>
    <var name="
decimal_separator"><![CDATA[.]]></var>
    <var name="
adapter">catalog/convert_adapter_productimport</var>
   <var name="
method">parse</var>
</action> 
<action type="
catalog/convert_adapter_product" method="parse">
</action>
<action type="
catalog/convert_adapter_product" method="finish">
</action>

Greetz
Monkey

 
Magento Community Magento Community
Magento Community
Magento Community
 
Vaidas_G
Jr. Member
 
Total Posts:  3
Joined:  2009-01-13
 
Sebastian Caffarelli - 07 January 2010 11:45 AM

Hi All so far I could find something about it but I’ll need some kind of help from any of you.
The error below seems to be related to the setlocale option in the PHP. I saw that magento uses the following statement:

setlocale(LC_ALL, Mage::app()->getLocale()->getLocaleCode().’.UTF-8’);

well I’m pretty sure that the issue in somehow is related to that.

mysthr - 11 December 2009 12:35 AM
Hi,
2009-12-10T17:06:53+00:00 DEBUG (7): SQLSTATE[21S01]: Insert value list does not match column list: 1136 Column count doesn’t match value count at row 1

Erwan.

Sebastian, maybe this will help: http://www.magentocommerce.com/boards/viewreply/207173/

 
Magento Community Magento Community
Magento Community
Magento Community
 
eysikal
Jr. Member
 
Total Posts:  2
Joined:  2009-11-23
 

I haven’t read this whole thread, but check .htaccess files and you may find a “php_value_memory_limit 128M” that is overriding any value set for “memory_limit” in php.ini.

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
    Back to top