Magento Forum

Page 1 of 11
Importing products - using a cron job to call Magento Dataflow import
 
srinigenie
Guru
 
Avatar
Total Posts:  539
Joined:  2008-02-04
 

Hi all,

Am trying to import all products using Cron. Referred to thread
http://www.magentocommerce.com/boards/viewthread/1982/#

But that doesnt work on the latest version. So am still scratching my head and rewriting. Would appreciate if anyone can share their experience on using Cron for dataflow module.

Thanks!!

 
Magento Community Magento Community
Magento Community
Magento Community
 
srinigenie
Guru
 
Avatar
Total Posts:  539
Joined:  2008-02-04
 

Pasting the code that helped me to import products using a cron-

STEP 1: Save the below file as Cron_Import.php on the Magento base folder.

<?php
//uncomment when moved to server - to ensure this page is not accessed from anywhere else
//if ($_SERVER['REMOTE_ADDR'] !== '<your server ip address') {
//  die("You are not a cron job!");
//}

require_once 'app/Mage.php';
// wget -O - http://<www.example.com>/Cron_Import.php/?files=3XSEEEE.csv
  
umask(0);

  
//$_SERVER['SERVER_PORT']='443';
  
Mage::app();

  
$profileId 3//put your profile id here
  
$filename Mage::app()->getRequest()->getParam('files'); // set the filename that is to be imported - file needs to be present in var/import directory  
  
if (!isset($filename))  {
 
die("No file has been set!");
  
}
  $logFileName
$filename.'.log';  
  
$recordCount 0;

  
Mage::log("Import Started",null,$logFileName);  
 
  
$profile Mage::getModel('dataflow/profile');
  
  
$userModel Mage::getModel('admin/user');
  
$userModel->setUserId(0);
  
Mage::getSingleton('admin/session')->setUser($userModel);
  
  if (
$profileId{
    $profile
->load($profileId);
    if (!
$profile->getId()) {
      Mage
::getSingleton('adminhtml/session')->addError('The profile you are trying to save no longer exists');
    
}
  }

  Mage
::register('current_convert_profile'$profile);

  
$profile->run();
  
  
$batchModel Mage::getSingleton('dataflow/batch');
  if (
$batchModel->getId()) {
    
if ($batchModel->getAdapter()) {
      $batchId 
$batchModel->getId(); 
      
$batchImportModel $batchModel->getBatchImportModel();
      
$importIds $batchImportModel->getIdCollection();  

      
$batchModel Mage::getModel('dataflow/batch')->load($batchId);      
      
$adapter Mage::getModel($batchModel->getAdapter());
      foreach (
$importIds as $importId{
        $recordCount
++;
        
try{
          $batchImportModel
->load($importId);
          if (!
$batchImportModel->getId()) {
             $errors[] 
Mage::helper('dataflow')->__('Skip undefined row');
             continue;
          
}

          $importData 
$batchImportModel->getBatchData();
          try 
{
            $adapter
->saveRow($importData);
          
catch (Exception $e{
            Mage
::log($e->getMessage(),null,$logFileName);          
            continue;
          
}
        
          
if ($recordCount%20 == 0{
            Mage
::log($recordCount ' - Completed!!',null,$logFileName);
          
}
        } 
catch(Exception $ex{
          Mage
::log('Record# ' $recordCount ' - SKU = ' $importData['sku']' - Error - ' $ex->getMessage(),null,$logFileName);        
        
}
      }
      
foreach ($profile->getExceptions() as $e{
        Mage
::log($e->getMessage(),null,$logFileName);          
      
}
      
    }
  }
  
echo 'Import Completed';
  
Mage::log("Import Completed",null,$logFileName);
 
?>

STEP 2: Next step would be to turn on the logging in Admin so that the log file is created.  (Navigation: System->Configuration->Developer->Log Settings)

STEP 3: Test this on your SSH login to check if this works on command line. Use the below command

wget -T0 -t1 -http://<www.example.com>/Cron_Import.php/?files=3XSEEEE.csv

Note here that the file to be imported is passed as parameter in the URL. Susbtitute ‘3XSEEEE.csv’ with your CSV name.

The completion of the above command should create a log file with name <your CSV file name>.log in the var/log directory. Open this file to see the log messages.

STEP 4: Now setup your Cron with the command mentioned in STEP 4. See if the log file gets created once the cron is activated and runs to completion.

Note: You can make it better by ensuring MAIL TO attribute is set for the Cron so that a mail is sent to inform the completion of the cron.

 
Magento Community Magento Community
Magento Community
Magento Community
 
srinigenie
Guru
 
Avatar
Total Posts:  539
Joined:  2008-02-04
 

The only other thing for which I am working on, is to secure this script from external hackers. I think I need to use $_SERVER for that. But am not able to figure how to use this for allowing only the server access to this particular script. Any help is appreciated.

Thanks!!

 
Magento Community Magento Community
Magento Community
Magento Community
 
srinigenie
Guru
 
Avatar
Total Posts:  539
Joined:  2008-02-04
 

Used ‘$_SERVER[’REMOTE_ADDR’]’ to secure the Cron page. Have also modified logic to accept the CSV filename as part of URL parameter so that multiple crons can be triggered at the same time.

 
Magento Community Magento Community
Magento Community
Magento Community
 
cibernoid
Sr. Member
 
Avatar
Total Posts:  208
Joined:  2008-02-12
 

Hello, thanks for sharing you cronjob script. That was exactly what I was looking for.
Seems like I Cannot get it working though smile

The problem is row 16 does not recognize the file name so the script returns the error message “No file has been set!”.
If I hardcode the file name, the script says “Import Completed”, but when I check it the Dashboard I notice the file hasn’t be imported.

Any idea, please?

 
Magento Community Magento Community
Magento Community
Magento Community
 
srinigenie
Guru
 
Avatar
Total Posts:  539
Joined:  2008-02-04
 

@cibernoid —try to get this working from the browser first and then try for the cron ...
i.paste the URL you are using to access the script
ii.what is the name of CSV file being imported

Your filename is somehow not getting set on the request :(

 
Magento Community Magento Community
Magento Community
Magento Community
 
cibernoid
Sr. Member
 
Avatar
Total Posts:  208
Joined:  2008-02-12
 

Sorry, It was my faulty: I have created a simple csv with 1 new product and only few fields.
Since the field “Description” was missing, Magento did not run the profile and stopped to execute the script. Looks like the Dataflow needs all fields when you import a new product.

An improvement for the script would be to make a deeper check so the log can give more details. It is a great script anyway, thank you very much.

 
Magento Community Magento Community
Magento Community
Magento Community
 
jack001
Jr. Member
 
Avatar
Total Posts:  20
Joined:  2007-11-29
France
 

thank you for your script job…
i will test it. But it was what i was looking for.
regards
Ps: how can i do it work by itself
every 2 hours or twice a day?
because i think i will use your script with my erp

 
Magento Community Magento Community
Magento Community
Magento Community
 
srinigenie
Guru
 
Avatar
Total Posts:  539
Joined:  2008-02-04
 

@jack001- you do that with the cron where you can setup to run every two hours ...check the web on how to setup a cron job

 
Magento Community Magento Community
Magento Community
Magento Community
 
alkarim
Guru
 
Total Posts:  368
Joined:  2008-04-10
 

hi is anyone try this before ??

I need some help

 
Magento Community Magento Community
Magento Community
Magento Community
 
Martijn Boeve
Jr. Member
 
Total Posts:  4
Joined:  2008-02-11
 

Hi all,

I’m using this script to import a large number of products. And it works pretty good! Except for two things! It ignores the fields for visibility and status.

Anyone knows why??

here’s the first line of my csv file:

"store","attribute_set","type","sku","category_ids","name","url_key","gift_message_available","meta_title","meta_description","url_path","isbn","auteur","uitgeverij","bindwijze","paginas","image","small_image","thumbnail","verwijder_indicatie","nur_code","onder_titel","reeks_nr","reeks_naam","aantal_delen","stadium_levencyclus","sbo_code","redacteur","co_auteur","vertaler","illustrator","custom_design","status","visibility","tax_class_id","weight","price","cost","news_from_date","news_to_date","description","short_description","meta_keyword","custom_layout_update","qty","min_qty","use_config_min_qty","is_qty_decimal","backorders","use_config_backorders","min_sale_qty","use_config_min_sale_qty","max_sale_qty","use_config_max_sale_qty","is_in_stock","low_stock_date","notify_stock_qty","use_config_notify_stock_qty","in_cb"
"default"
"Default""Simple""9789020966183""3""Kippenvel op je huid en vlinders in je buik""kippenvel_op_je_huid_en_vlinders_in_je_buik""Nee""Kippenvel op je huid en vlinders in je buik 9789020966183 Vereecken, K.""Kippenvel op je huid en vlinders in je buik 9789020966183 Vereecken, K. - In samenwerking met de K.U. Leuven, professor Vera Hoorens en professor Jan Eggermont - Prachtig geïllustreerd door Judith Vanistendael - Hoe komt het dat elk huis anders ruikt? - Bestaat hartzeer Ã©cht? - En vele andere boeiende vragen""kippenvel_op_je_huid_en_vlinders_in_je_buik.html""9789020966183""Vereecken, K.""Terra - Lannoo, Uitgeverij""GEB""140""/product/f4db4ca6b140f49c5efa4f38b4af3e9e.jpg""/product/bd7ce6e473f316791020719561a1b48d.jpg""/product/bd7ce6e473f316791020719561a1b48d.jpg""N""213""kinderuniversiteit""""""""V""05""""""""Istendael, J. Van""""Ingeschakeld""Catalogus, zoeken""BTW 2""456""19.9500""0.0000""2008-07-23 00:00:00""2008-08-23 00:00:00""Waarom krijg je water in je mond als je iets lekkers ruikt?
Waarom huil je als je verdriet hebt? 
Waarom lusten mijn ouders witlof en ik niet? 
Waarom voel je vlinders in je buik als je verliefd bent? 
Waarom kijken veel mensen zo graag naar enge films? 

In het najaar van 2005 stelde de K.U.Leuven haar deuren open voor leergierige kinderen. Ze bestookten de professoren met honderden vragen over wetenschap, geschiedenis, psychologie en nog zoveel meer. 
Kathleen Vereecken bundelde de interessantste, leukste en meest originele vragen over zintuigen en gevoelens. Ontdek waarom knuffelen gezond is, hoe bijen zich gedragen als ze kou hebben en waarom we pijn voelen. 

"
"- In samenwerking met de K.U. Leuven, professor Vera Hoorens en professor Jan Eggermont - Prachtig geïllustreerd door Judith Vanistendael - Hoe komt het dat elk huis anders ruikt? - Bestaat hartzeer Ã©cht? - En vele andere boeiende vragen""Kippenvel op je huid en vlinders in je buik 9789020966183 Vereecken, K.""""9.0000""0.0000""1""0""0""1""1.0000""1""0.0000""1""1""0000-00-00 00:00:00""0.0000""1""ja"
 
Magento Community Magento Community
Magento Community
Magento Community
 
srinigenie
Guru
 
Avatar
Total Posts:  539
Joined:  2008-02-04
 

hmmm ... check your data ... magento as of the version I use is having a bug with double quotes, so if you repeat your delimiter (in your case comma) within a data, I think it doesn’t work ...so choose a different delimiter and retry

 
Magento Community Magento Community
Magento Community
Magento Community
 
Martijn Boeve
Jr. Member
 
Total Posts:  4
Joined:  2008-02-11
 

I’m going to try, but with the import function in the admin I had no problems so I don’t think it’s going to solve it.

I’ll get back to you shortly!

Thanks for your reply!

 
Magento Community Magento Community
Magento Community
Magento Community
 
Martijn Boeve
Jr. Member
 
Total Posts:  4
Joined:  2008-02-11
 

No that wasn’t it…

I’m using version 1.0.19870.4 by the way.

Do you know if it’s possible to set the visibility and status through phpmyadmin? That would be a solution for me also…

edit: I’ve checked the imported products, but everything else gets imported. Also the fields behind te status and visibility fields. So it’s just not accepting them. I’m using the dutch values (like I got in the export) could that be it? And in that case what should I use?

 
Magento Community Magento Community
Magento Community
Magento Community
 
Flashnet
Jr. Member
 
Total Posts:  1
Joined:  2008-07-25
 

I used this cron job for importing products and have same issue, that visibility status is inactive. Did you find alredy a solution for this?
I use german version, but it seams, that tis not matters.

 
Magento Community Magento Community
Magento Community
Magento Community
 
finch8243
Sr. Member
 
Total Posts:  79
Joined:  2007-08-31
 

This would be a great feature for export too—I submit several shopping feeds (Google, Shopping.com, Yahoo shopping) which are a great way to garner traffic, and Magento’s export feature makes it easy to pull the feeds together.  The issue is I have to run them manually.  Would be nice to get them working on a cron.

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top
Page 1 of 11