Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Importing products - using a cron job to call Magento Dataflow import
 
jan212
Guru
 
Avatar
Total Posts:  407
Joined:  2008-01-03
 

Hey lilmuckers,

thanks for the fast reply! I setup the standard Cronjob to run cron.php every minute cause i need to setup this to get the amazon thing and standard magento shedules running.

In profile history there are only appearing the manual started imports i think (not sure, could be one entry from a cron run). Thanks for the logfile info! I ran the profile manually without errors and correct updated stock levels.

I checked the cron_shedule table in DB, there are 6 entries for the Ext.:

1080     cronprofiles_run_profiles     success     NULL     2009-11-12 18:00:21     2009-11-12 18:00:00     2009-11-12 18:00:26     2009-11-12 18:00:43
1081     cronprofiles_run_profiles     success     NULL     2009
-11-12 18:00:21     2009-11-12 18:05:00     2009-11-12 18:07:08     2009-11-12 18:07:22
1082     cronprofiles_run_profiles     pending     NULL     2009
-11-12 18:00:21     2009-11-12 18:10:00     0000-00-00 00:00:00     0000-00-00 00:00:00
1083     cronprofiles_run_profiles     pending     NULL     2009
-11-12 18:00:21     2009-11-12 18:15:00     0000-00-00 00:00:00     0000-00-00 00:00:00
1111     cronprofiles_run_profiles     pending     NULL     2009
-11-12 18:07:22     2009-11-12 18:20:00     0000-00-00 00:00:00      0000-00-00 00:00:00
1112     cronprofiles_run_profiles     pending     NULL     2009
-11-12 18:07:22     2009-11-12 18:25:00     0000-00-00 00:00:00     0000-00-00 00:00:00

One of them (18:07:08 ) is matching the execution time in profiles history, but that could be hap…

 
Magento Community Magento Community
Magento Community
Magento Community
 
jan212
Guru
 
Avatar
Total Posts:  407
Joined:  2008-01-03
 

I noticed some errors in my cron system so the script couldn’t run properly, this problem is solved.

Actually i got it running automaticly as i can see in profiles history, but i doesn’t update the stocklevels correctly. If i am running it manualy it does… Any points to start debugging this?

Regards,
Jan

 
Magento Community Magento Community
Magento Community
Magento Community
 
lilmuckers
Jr. Member
 
Total Posts:  15
Joined:  2008-06-18
 

Post a sample of your stock file and import XML and i’ll take a look.

the only thing i can think off is a particular quirk in the import/export system in magento.
It calls a couple of functions that aren’t explicitly stated in the XML.

The first is one that’s in the XML but not built in a consistent way.
If you look in the XML, if there’s a bit similar to the following

<var name="adapter">catalog/convert_adapter_product</var>
    <var 
name="method">parse</var>

this needs removed from the block it’s in, and this added underneath the block similar to this - as you can see, i’ve copied the ‘adapter’ and ‘method’ tag values to the attributes in the ‘action’ tag:

<action type="catalog/convert_adapter_product" method="parse">
</
action>

And the second one that’s called silently in the manual run process. This is particularly important for product and stock imports. It calls the “finish” method on the adaptor specified in that main XML (the one i mentioned above).
To call this for the product import for example i would add the following to the bottom of the XML

<action type="catalog/convert_adapter_product" method="finish">
</
action>

I was going to add this standard magento behavior into cronprofiles but i didn’t for one reason - I forsee this kind of module being used in situations where regular high load imports are done. And certainly with the imports this module was conceived to deal with the finisher action was just too heavy.

On the product import it’s really very heavy, since it refreshes all the product caching and flat catalogs, etc. And the way i have these big ones set up is that it splits the import file into 500 line pieces which it imports every 5 minutes. and having such a heavy action running every 5 minutes was a sure fire way to overload the server.
So what i do now is i call that finish action on its own every 2 hours

 
Magento Community Magento Community
Magento Community
Magento Community
 
jan212
Guru
 
Avatar
Total Posts:  407
Joined:  2008-01-03
 

Hey lilmuckers,

thanks for the detailed reply - i think my file causes exactly the trouble you write about cause it has about 10k lines in it.
I will try to import a smaller file in the next 10minutes, after that i will try to add the finish action to my xml.

Subsequent you’ll find my actual xml:

<action type="dataflow/convert_adapter_io" method="load">
    <var 
name="type">file</var>
    <var 
name="path">var/import</var>
    <var 
name="filename"><![CDATA[import_wawi_stocks.csv]]></var>
    <var 
name="format"><![CDATA[csv]]></var>
</
action>

<
action type="dataflow/convert_parser_csv" method="parse">
    <var 
name="delimiter"><![CDATA[,]]></var>
    <var 
name="enclose"><![CDATA["]]></var>
    <var name="
fieldnames">true</var>
    <var name="
store"><![CDATA[0]]></var>
    <var name="
number_of_records">1</var>
    <var name="
decimal_separator"><![CDATA[.]]></var>
    <var name="
adapter">catalog/convert_adapter_product</var>
    <var name="
method">parse</var>
</action>
 
Magento Community Magento Community
Magento Community
Magento Community
 
lilmuckers
Jr. Member
 
Total Posts:  15
Joined:  2008-06-18
 

give this profile XML a try

<action type="dataflow/convert_adapter_io" method="load">
    <var 
name="type">file</var>
    <var 
name="path">var/import</var>
    <var 
name="filename"><![CDATA[import_wawi_stocks.csv]]></var>
    <var 
name="format"><![CDATA[csv]]></var>
</
action>

<
action type="dataflow/convert_parser_csv" method="parse">
    <var 
name="delimiter"><![CDATA[,]]></var>
    <var 
name="enclose"><![CDATA["]]></var>
    <var name="
fieldnames">true</var>
    <var name="
store"><![CDATA[0]]></var>
    <var name="
number_of_records">1</var>
    <var name="
decimal_separator"><![CDATA[.]]></var>
</action> 
<action type="
catalog/convert_adapter_product" method="parse">
</action>
<action type="
catalog/convert_adapter_product" method="finish">
</action>

the frequency you want to use may effect your decisions on this front.
But stock imports are a lot less intensive than product imports (obviously)

The setup i’m working with is 4 feeds from different suppliers, each in the region of 25,000 products
So i have quite a complicated system for dealing with them, along with a bunch of customised adaptors and parsers since the feeds are not the same as what magento expects

but basically, i download the file and split it up on one profile, that runs at say, 2300 every night. and ever 5 minutes for the next hour or two another profile picks up the split files and imports them.

but every hour between 0600 and 2000 we do the same import, only instead of importing all the product data, we just import the stock data.
And we don’t need to split that up since each file only takes in the region of 10-20 minutes.

in that instance if it’s a one-off then it doesn’t really need to be split up, even with the finish action - so give that XML a try.

 
Magento Community Magento Community
Magento Community
Magento Community
 
jan212
Guru
 
Avatar
Total Posts:  407
Joined:  2008-01-03
 

Let me say thanks again at first. I just tested around with reducing the csv file to 10 lines, but that wasn’t the point. So i just added your xml and looking forward to the next test.

It’s not clear to me why i have to call

<action type="catalog/convert_adapter_product" method="parse">
</
action>
<
action type="catalog/convert_adapter_product" method="finish">
</
action>

and not

<action type="dataflow/convert_parser_csv" method="parse">
</
action>
<
action type="dataflow/convert_parser_csv" method="finish">
</
action>

Maybe you can give me a hint…

 
Magento Community Magento Community
Magento Community
Magento Community
 
jan212
Guru
 
Avatar
Total Posts:  407
Joined:  2008-01-03
 

GREAT - you got it working man!

The finish action made it!

I’ll keep you informed about my performance with 10k products, maybe you’ll have to give me some advices again how you split up your files and coordinate everything, i noticed some interessting details in your posts above… But it seems to work fine atm.

THANKS!

 
Magento Community Magento Community
Magento Community
Magento Community
 
lilmuckers
Jr. Member
 
Total Posts:  15
Joined:  2008-06-18
 

Glad i could help!
and i hope my module is useful for you

quick explanation on the extra actions i put in there for you

If you look at your profiles original XML:

<action type="dataflow/convert_parser_csv" method="parse">
    <var 
name="delimiter"><![CDATA[,]]></var>
    <var 
name="enclose"><![CDATA["]]></var>
    <var name="
fieldnames">true</var>
    <var name="
store"><![CDATA[0]]></var>
    <var name="
number_of_records">1</var>
    <var name="
decimal_separator"><![CDATA[.]]></var>
    <var name="
adapter">catalog/convert_adapter_product</var>
    <var name="
method">parse</var>
</action>

If you notice those last two “var” tags. They’re called “Adapter” and “Method”
These are the variables that the core magento run script use to run the actual product import.
Effectively all it ends up doing is calling another action block through an external method.

This needs to be translated into an action block to be run as a raw method.
this is why those are the values used.

And the Finish action block is laid out in that way because magentos core code calls the “finish” method on the same adapter as specified in the

<var name="adapter">
tag above.

These all need to be explicitly stated in the XML for cronprofiles.

 
Magento Community Magento Community
Magento Community
Magento Community
 
jan212
Guru
 
Avatar
Total Posts:  407
Joined:  2008-01-03
 

You could help a lot, a big stone has fallen from my heart that i could get it running cause christmas biz is near.

I just noticed that piece

<var name="adapter">catalog/convert_adapter_product</var>
3min ago after posting a link to your addon in the german part of the magento community. I think some people will be glad to read this cause they are searching for a running solution to automate profiles.

Your description makes it clear to me and is written in a comprehendible way, once again i want to say thanks a lot for all!

 
Magento Community Magento Community
Magento Community
Magento Community
 
lucywoodson
Jr. Member
 
Total Posts:  1
Joined:  2009-11-15
 

How to automate our profiles? I am finding it hard to automate them...please help.
regards,
lucy
Simulation pret

 
Magento Community Magento Community
Magento Community
Magento Community
 
jan212
Guru
 
Avatar
Total Posts:  407
Joined:  2008-01-03
 

Hey Lucy,

you’ll have to install lilmuckers extension you can find here http://www.magentocommerce.com/extension/2268/cron-profiles.

After that you can create a simple profile and copy the xml from it to an advanced profile. Then you’ll have to test the profile manually if it’s running without any problems and products/customers/… are getting updated correctly.

You can automize the profile by activating the cron and adding the regular syntax to it like i did it before as you can see in my posts before!

 
Magento Community Magento Community
Magento Community
Magento Community
 
EazyEggbert
Jr. Member
 
Total Posts:  18
Joined:  2009-02-13
 

Hey guys,

I am currently trying lilmuckers extension...looks very good but does not seem to work correctly on my machine:

I copied and the XML from lilmuckers in post #95 and modified it to my parameters. Otherwise my stocks are not being updated either using the default XML generated from the “simple import export profiles” using the cronjob. Running the profile manually works perfectly.

However this is my XML:

<action type="dataflow/convert_adapter_io" method="load">
    <var 
name="type">file</var>
    <var 
name="path">var/import</var>
    <var 
name="filename"><![CDATA[webshop-bestaende.csv]]></var>
    <var 
name="format"><![CDATA[csv]]></var>
</
action>

<
action type="dataflow/convert_parser_csv" method="parse">
    <var 
name="delimiter"><![CDATA[;]]></var>
    <var 
name="enclose"><![CDATA[]]></var>
    <var 
name="fieldnames">true</var>
    <var 
name="store"><![CDATA[0]]></var>
    <var 
name="number_of_records">1</var>
    <var 
name="decimal_separator"><![CDATA[.]]></var>
</
action
<
action type="catalog/convert_adapter_product" method="parse">
</
action>
<
action type="catalog/convert_adapter_product" method="finish">
</
action>

But this does not seem to work for my system. Even when I run the script manually I get this output:

* Starting profile execution, please wait…
* Warning: Please don’t close window during importing/exporting data

* Starting Mage_Dataflow_Model_Convert_Adapter_Io :: load
* Loaded successfully: “/kunden/eickhorst.com/eickhorst/magento/var/import/webshop-bestaende.csv”
* Starting Mage_Dataflow_Model_Convert_Parser_Csv :: parse

* Please declare “adapter” and “method” node first
* Starting Mage_Catalog_Model_Convert_Adapter_Product :: parse
* Starting Mage_Catalog_Model_Convert_Adapter_Product :: finish
* Finished profile execution.

What do I have to do? Did I miss anything? A step by step tutorial would be great!

Any suggestions?

Thank you guys!

Felix

 
Magento Community Magento Community
Magento Community
Magento Community
 
jan212
Guru
 
Avatar
Total Posts:  407
Joined:  2008-01-03
 

Hey,
you have forgotten to declare

<var name="adapter">catalog/convert_adapter_product</var>
<var 
name="method">parse</var>
within
<action type="dataflow/convert_parser_csv" method="parse"></action>

Try

<action type="dataflow/convert_adapter_io" method="load">
    <var 
name="type">file</var>
    <var 
name="path">var/import</var>
    <var 
name="filename"><![CDATA[webshop-bestaende.csv]]></var>
    <var 
name="format"><![CDATA[csv]]></var>
</
action>

<
action type="dataflow/convert_parser_csv" method="parse">
    <var 
name="delimiter"><![CDATA[;]]></var>
    <var 
name="enclose"><![CDATA[]]></var>
    <var 
name="fieldnames">true</var>
    <var 
name="store"><![CDATA[0]]></var>
    <var 
name="number_of_records">1</var>
    <var 
name="decimal_separator"><![CDATA[.]]></var>
    <var 
name="adapter">catalog/convert_adapter_product</var>
   <var 
name="method">parse</var>
</
action
<
action type="catalog/convert_adapter_product" method="parse">
</
action>
<
action type="catalog/convert_adapter_product" method="finish">
</
action>

Sollte funzen…

 
Magento Community Magento Community
Magento Community
Magento Community
 
EazyEggbert
Jr. Member
 
Total Posts:  18
Joined:  2009-02-13
 

Ok great now it works! thanks a lot!

FYI: I noticed that the cronjob expression needs to be set. I used “* * * * *” for “execute process when cronjob is started” (http://www.magentocommerce.com/wiki/how_to/how_to_setup_a_cron_job#magento_s_cronjobs)

I have a file of 195 records only. However the import seems to only process the first 30 or so records when it is started via cronjob. I suppose this is due to the limited power and script execution time on our vserver. Manually it does work though. It processes all 195 sets correctly.

I read above that you have separated the parse and finish processes in different cronjobs. Would this help me with our limited vserver capacities? And if yes what XML do I need?

Would my guess work like this? :

Profile 1 (executed often, i.e. interval of 1hr):

<action type="dataflow/convert_adapter_io" method="load">
    <var 
name="type">file</var>
    <var 
name="path">var/import</var>
    <var 
name="filename"><![CDATA[webshop-bestaende.csv]]></var>
    <var 
name="format"><![CDATA[csv]]></var>
</
action>

<
action type="dataflow/convert_parser_csv" method="parse">
    <var 
name="delimiter"><![CDATA[;]]></var>
    <var 
name="enclose"><![CDATA[]]></var>
    <var 
name="fieldnames">true</var>
    <var 
name="store"><![CDATA[0]]></var>
    <var 
name="number_of_records">1</var>
    <var 
name="decimal_separator"><![CDATA[.]]></var>
    <var 
name="adapter">catalog/convert_adapter_product</var>
   <var 
name="method">parse</var>
</
action
<
action type="catalog/convert_adapter_product" method="parse">
</
action>

Profile 2 (executed not so often, ie interval of 3hrs):

<action type="catalog/convert_adapter_product" method="finish">
</
action>

Is this the right way to do it?

Thank you!

Felix

 
Magento Community Magento Community
Magento Community
Magento Community
 
jan212
Guru
 
Avatar
Total Posts:  407
Joined:  2008-01-03
 

Hey Felix,

don’t split it, reduce the frequenzy the cronjob is running, actually it’s running every minute!!!
That means the process is starting again but hasn’t finished the run before.

Try to import products every 10min at first and play with the value!

*/10 * * * *

You could also run the script manually and measure execution time with YSlow.

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top