Magento Forum

   
Importing products - using a cron job to call Magento Dataflow import
 
barrmy
Sr. Member
 
Avatar
Total Posts:  102
Joined:  2008-06-03
 

Hi

This is working well for me for small runs, but my csv has upwards of 3000 items in it.

Is there a way for the log description not to start over after 100 rows?  Right now it looks like this so it’s hard to find out exactly where I have a problem:

Import Started 2008-11-03T13:13:27+00:00 DEBUG (7): Import Started 2008-11-03T13:14:05+00:00 DEBUG (7): 20 Completed!! 2008-11-03T13:14:33+00:00 DEBUG (7): 40 Completed!! 2008-11-03T13:15:00+00:00 DEBUG (7): 60 Completed!! 2008-11-03T13:15:28+00:00 DEBUG (7): 80 Completed!! 2008-11-03T13:15:39+00:00 DEBUG (7): Import Started 2008-11-03T13:16:17+00:00 DEBUG (7): 20 Completed!! 2008-11-03T13:16:44+00:00 DEBUG (7): 40 Completed!! 2008-11-03T13:17:10+00:00 DEBUG (7): 60 Completed!! 2008-11-03T13:17:38+00:00 DEBUG (7): 80 Completed!! 2008-11-03T13:17:41+00:00 DEBUG (7): Import Started 2008-11-03T13:18:22+00:00 DEBUG (7): 20 Completed!! 2008-11-03T13:18:51+00:00 DEBUG (7): 40 Completed!! 2008-11-03T13:19:19+00:00 DEBUG (7): 60 Completed!! 2008-11-03T13:19:43+00:00 DEBUG (7): Import Started 2008-11-03T13:20:22+00:00 DEBUG (7): 20 Completed!! 2008-11-03T13:20:51+00:00 DEBUG (7): 40 Completed!! 2008-11-03T13:21:18+00:00 DEBUG (7): 60 Completed!! 2008-11-03T13:21:45+00:00 DEBUG (7): Import Started 2008-11-03T13:22:07+00:00 DEBUG (7): Import Started 2008-11-03T13:22:45+00:00 DEBUG (7): 20 Completed!! 2008-11-03T13:23:11+00:00 DEBUG (7): 40 Completed!! 2008-11-03T13:23:37+00:00 DEBUG (7): 60 Completed!! 2008-11-03T13:24:03+00:00 DEBUG (7): 80 Completed!! 2008-11-03T13:24:19+00:00 DEBUG (7): Import Started 2008-11-03T13:24:57+00:00 DEBUG (7): 20 Completed!! 2008-11-03T13:25:24+00:00 DEBUG (7): 40 Completed!! 2008-11-03T13:25:51+00:00 DEBUG (7): 60 Completed!! 2008-11-03T13:26:18+00:00 DEBUG (7): 80 Completed!! 2008-11-03T13:26:21+00:00 DEBUG (7): Import Started 2008-11-03T13:26:57+00:00 DEBUG (7): 20 Completed!! 2008-11-03T13:27:23+00:00 DEBUG (7): 40 Completed!! 2008-11-03T13:27:50+00:00 DEBUG (7): 60 Completed!! 2008-11-03T13:28:16+00:00 DEBUG (7): 80 Completed!! 2008-11-03T13:28:23+00:00 DEBUG (7): Import Started 2008-11-03T13:29:02+00:00 DEBUG (7): 20 Completed!! 2008-11-03T13:29:28+00:00 DEBUG (7): 40 Completed!! 2008-11-03T13:29:54+00:00 DEBUG (7): 60 Completed!! 2008-11-03T13:30:19+00:00 DEBUG (7): 80 Completed!! 2008-11-03T13:30:25+00:00 DEBUG (7): Import Started 2008-11-03T13:31:03+00:00 DEBUG (7): 20 Completed!! 2008-11-03T13:31:29+00:00 DEBUG (7): 40 Completed!! 2008-11-03T13:31:57+00:00 DEBUG (7): 60 Completed!! 2008-11-03T13:32:24+00:00 DEBUG (7): 80 Completed!!

Also, I’m getting a “Connection Interrupted:The connection to the server was reset while the page was loading.” error when running via HTTP, and the cron job also never completes (gets to a a little over 100).  I tried to add “set_time_limit(0);” but its still timing out.

 
Magento Community Magento Community
Magento Community
Magento Community
 
barrmy
Sr. Member
 
Avatar
Total Posts:  102
Joined:  2008-06-03
 

*edit*

my error above was due to running the cron as a curl method, my host informed me that due to the size i had to use cli to avoid timeout

ben’s changes worked like a charm once i figured out the settings...49min for a full import of 3207 items

 
Magento Community Magento Community
Magento Community
Magento Community
 
zadpro
Sr. Member
 
Avatar
Total Posts:  247
Joined:  2007-12-10
FL, USA
 

Is anybody using this code in version 1.1.8 that works?
I tried this code several times and I can’t figure it out how to make it work.
I really appreciate if anybody has a functional code, that works with 1.1.8, to post the complete code here.
I know that many people like me are looking for this.

Thanks in advance.

 
Magento Community Magento Community
Magento Community
Magento Community
 
ChipH
Jr. Member
 
Total Posts:  4
Joined:  2008-05-02
Cincinnati, OH
 

This actually works perfectly for me on 1.1.8. I’ve got about 2500 items in the system that were imported manually, and I use this to suck in new items, changes, and inventory deductions from in-store sales every 2 hours.

 
Magento Community Magento Community
Magento Community
Magento Community
 
goldfish22
Jr. Member
 
Total Posts:  26
Joined:  2009-02-05
 

I’m trying to get this to work but having trouble.
I have set up a cron job in Cpanel, looks a bit like this
I use the following in my browser
www.mydomain.com/cron_import.php/?files=export_product.csv
and get, result Import completed, and it updates as I would expect.

However I have trouble when try to use a Cron
Using the following in cpanel
php -f /home/directory/structure/public_html/mydomain/cron_import.php/?files=export_product.csv

I get an error in email
No input file specified.

from what I can understand the cron is being run, but looks like the export_products.csv insn’t being passed correctly.

any help appreciated.

Also rather than setting up a new cron, shouldn’t it be possible to change the config.xml crontab section as mentioned in
http://www.magentocommerce.com/wiki/how_to/how_to_setup_a_cron_job?do=show
?

 
Magento Community Magento Community
Magento Community
Magento Community
 
zadpro
Sr. Member
 
Avatar
Total Posts:  247
Joined:  2007-12-10
FL, USA
 

In cron the command is a little different.
Try this:
Php -f path/to/directory/cron-import.php export_file.csv 4

Note there is a space between the cron file and the csv file
Replace number 4 with your profile ID number

 
Magento Community Magento Community
Magento Community
Magento Community
 
goldfish22
Jr. Member
 
Total Posts:  26
Joined:  2009-02-05
 

Thanks,
have tried that, however got the same problem, No input file specified.

will continue playing with the cron command
any other ideas appreciated.

 
Magento Community Magento Community
Magento Community
Magento Community
 
goldfish22
Jr. Member
 
Total Posts:  26
Joined:  2009-02-05
 

this may be a silly question, but if you set an import up to accept automatically a file already on the server, would it be possible to just use

http://www.domain.comu/index.php/system_convert_gui/run/id/9/

to run the import,

 
Magento Community Magento Community
Magento Community
Magento Community
 
justbeez
Jr. Member
 
Total Posts:  7
Joined:  2008-04-09
 
goldfish22 - 10 March 2009 04:43 PM

this may be a silly question, but if you set an import up to accept automatically a file already on the server, would it be possible to just use

http://www.domain.comu/index.php/system_convert_gui/run/id/9/

to run the import,

The URL you entered is incorrect.  The correct URL is:
http://www.domain.com/index.php/admin/system_convert_gui/run/id/9/

The only problem is that this would still require you to be logged-in to your Magento admin.  While you could probably hack together a way to log in automatically, I’m not sure that this is the ideal solution.

What really needs to happen is that the Magento team needs to add the ability to schedule imports and exports, and also add the ability to trigger existing profiles from the core API.  This would take the guess work out of data synchronization for all of us.

 
Magento Community Magento Community
Magento Community
Magento Community
 
goldfish22
Jr. Member
 
Total Posts:  26
Joined:  2009-02-05
 

your right made a mistake on the path, good spotting.

I agree, automatic import and export would be a nice feature.

Have setup a cron to do the importing for me from CSV file.

If anyone is interested I found a program called FLING which syncs a folder on my local PC with the import folder on the website.
http://www.nchsoftware.com/fling/index.html

And I use a third data base manipulation program to manipulate my data base from my physical shop to be acceptable for magento.

complex to setup but now it is going everything is automated.

 
Magento Community Magento Community
Magento Community
Magento Community
 
brianpat
Member
 
Total Posts:  44
Joined:  2008-04-05
 

Hi goldfish22,

Thanks for your suggestion. Can you please tell me what do u mean “third data base manipulation program” ? What type of software/program do u use for data manupulation? I think the biggest task is how to automate/translate the data according to magento format. It is actually a tedious task and each time there is a change in the data, you need to manually change.

To automate the task, I think the best thing is to follow the following steps, but I don’t know how to implement them automatically:
1) Get the csv file and then convert/migrate it to any sort of the database
2) Get the updated new csv file and compare the new csv with the old csv file
3) Update the database from the new csv file
4) create the new csv file according to magento fromat from the database

Does anybody know how to get them automatic?

Thanks
Brian

 
Magento Community Magento Community
Magento Community
Magento Community
 
goldfish22
Jr. Member
 
Total Posts:  26
Joined:  2009-02-05
 

Brian,
no probs.  this is exactly what we do.

I use cobian backup to copy my data base files from my physical shop.  (MDF, MDX and DBF files) to a folder where we can manipulate them.  (I do this as am terrified that something might go wrong and destroy the physical shop data base.  at least we are now working on a backup of the originals).

Then we use an Access script program that takes the inputs from the Physical shop, and an exported file from Magento (downloaded via fling).

The scripted access program then compares a few things.
1. checks if the item from physical shop is enabled for publish on web.
2. checks to make sure the item is configured in Magento correctly (images, titles etc..)
3. updates the current stock levels, pricing etc.
4. creates an uploadable file in the magento output file style (using the headder info from the magento downloaded file)
5. creates an error file, with a list of all items that are enabled to be published, but not yet setup correctly.

the access program imports a CSV and the MDF files and does conversions, and mapping and error checking etc. and creates it’s output based on the csv headder lines from the downloaded magento file.

I use a cron the create the download file (which is active products + some extras to check they are setup correctly)
and another cron to import the uploaded file.
Fling is used to get the files from the www to my pc and back.
this is run on the physical shop server, so only updates when the server is run.

There are some issues with this.

1. a www sale is not directly imported into the physical shop data base.  to date our online volumes are low, so we have a copy of the magento invoice sent to us and then enter it in to the physical shop directly.  will look at automating this in the future if the volumes deem it necessary.
2. the prices/stock are not automatically updated every time there is a change on the physical shop. so it is possible that we could take an order for something that we have run out of.  not happened yet.
3. there is a delay, depending on crons/fling upload/ and data base conversion and it can be hours.  in our case it is hours but that is OK in our case.
4.  just thought of this one, the import cron automatically imports the CSV uploaded file every time, even if there is no change to it.  so it is possible over the weekend to sell out of stock, then the cron will import the csv again and show stock, will have to change the cron so there is a check on the file size or delete the import csv after a successful import is done.

the next thing we will do is use cobian to back up the files from the main data base, and cobian has a neat feature that allows you to run a command before and after the backup.  so we will modify cobian to
1. backup the data base files.
2. run command line fling to download files
3. run access script to convert
4. run command line fling to upload files.
this will sync all the download and upload.  just then need a way to make it automatically run the cron.

cobian url http://www.cobian.se/

If you are interested we can develop an access script for you and help get it setup, but this would be a paid service.  contact me via pm if you are interested.

 
Magento Community Magento Community
Magento Community
Magento Community
 
goldfish22
Jr. Member
 
Total Posts:  26
Joined:  2009-02-05
 

I have noticed some unexpected things with the cron_import function.
noticed that I must have the fields
“store”,"websites”,"category_ids"
in the uploaded CSV file. using the cron.

I can import a file with just
“sku”,"price”,"special_price”,"qty”,"is_in_stock”,"status"
via the control pannel, using import profile 3 (and 7 one of my own)

however when I try to use the same import profile using a cron, and the csv has only the “sku”,"price”,"special_price”,"qty”,"is_in_stock”,"status" fields, it does not import.

However add the “store”,"websites”,"category_ids" to it and will import via the cron fine.

Has anyone else noticed this?

 
Magento Community Magento Community
Magento Community
Magento Community
 
aiva
Jr. Member
 
Total Posts:  17
Joined:  2009-02-04
 

Thanks for the awesome script!!!

I tried it out on 1.3.0 - got it to work with new products, imports them well as long as the required fields are defined in the csv.

However, i can’t get it to update the products that are already there. I am trying to update stock quantities and when i run the script it only says “Import Completed” and writes the following in the log file:

2009-03-31T20:39:56+00:00 DEBUG (7): Import Started
2009-03-31T20:39:56+00:00 DEBUG (7): Warning: Invalid argument supplied for foreach() in /app/code/core/Mage/Eav/Model/Entity/Abstract.php on line 937
2009-03-31T20:39:57+00:00 DEBUG (7): Warning: Invalid argument supplied for foreach() in /app/code/core/Mage/Eav/Model/Entity/Abstract.php on line 937
2009-03-31T20:39:57+00:00 DEBUG (7): Starting Mage_Dataflow_Model_Convert_Parser_Csv :: parse
2009-03-31T20:39:57+00:00 DEBUG (7): Found 4 rows
2009-03-31T20:39:57+00:00 DEBUG (7): Starting catalog/convert_adapter_product :: parse
2009-03-31T20:39:57+00:00 DEBUG (7): Import Completed

So it throws the warning for each row that has an sku that already exists in the db. New products get imported fine though

So i can’t figure out what to do to make it update the existing products. So far i’ve been running the script through browser but the aim is to make it a scheduled job that updates all the products’ stock quantities automatically. Need help!!! Cheers

 
Magento Community Magento Community
Magento Community
Magento Community
 
goldfish22
Jr. Member
 
Total Posts:  26
Joined:  2009-02-05
 

All I can suggest is that you define an EXPORT with the fields that you need to import,
export a file then alter a couple of the stock qtys (use open office or a text editor rather than excel).
try importing that and see how it goes.

Also try importing the file that you are having trouble with via the import function in the backend, this will help determine if it is the file or something else.

We use this option to successfully import new qtys and prices from our physical store data base, no trouble so far.
We do not use it to import new products, only update the qtys, enable and price.

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
    Back to top