Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

the quest for an efficient product import script
 
craigcriv
Sr. Member
 
Total Posts:  85
Joined:  2009-08-05
 

Just ran a small test

initializing…
Import Started
imported 8 items (100 %) in 0.0625 secs (last 8 in 0.0625 secs) - avg speed : 7681 rec/min
No decimal Attributes created for sku ACCURITE00334
No decimal Attributes created for sku ACCURITE00465
No decimal Attributes created for sku ACCURITE00614
No decimal Attributes created for sku ACCURITE00786
No decimal Attributes created for sku ALLIEDPR18M
No decimal Attributes created for sku ALLIEDPRH50
Magento 1.4 indexer not found, you should reindex manually using magento admin
Back to Configuration PageBack to Import

Web version working great on magento 1.3.23

Thanks for fixing the 0 qty issue. Also the status option is working great also.

This will be a lifesaver for me when updating inventory.

Thanks a lot

Craig

 
Magento Community Magento Community
Magento Community
Magento Community
 
markw87
Jr. Member
 
Total Posts:  27
Joined:  2010-06-28
 

Hey,

Just trying to run the script on my development machine which is running Apache, PHP, MYSQL under windows and all i am getting is a white page.

Any Ideas ?

 
Magento Community Magento Community
Magento Community
Magento Community
 
ctnm_sru
Jr. Member
 
Total Posts:  3
Joined:  2010-08-02
 

Hello dweeves,

first of all thank you for your excellent tool and exemplary support.

We tested your tool with magento 1.4.0.1 and it seems to work for us with one exception:

The entity_type_id is hard coded with value \\\\\\\’4\\\\\\\’ in \\\\\\\’magento_mass_importer.class.php\\\\\\\’,
but for some reason in our environment \\\\\\\’10\\\\\\\’ is the appropriate value.

I\\\\\\\’am not shure, if this behaviour is 1.4.0.1 specific. This might also occur with 1.4.1.

Would it be possible, to define \\\\\\\’entity_type_id\\\\\\\’ as a variable, as proposed below,
or even better define it in \\\\\\\’magento_mass_importer.ini\\\\\\\’ ?

Thanks,
Stephan

35c35
< protected $entity_type_id = 10; // 2010-08-20, Stephan
---
>
181c181
< WHERE ($tname.attribute_code IN ($qcolstr)) AND (entity_type_id=$this->entity_type_id)\\\\\\\”;
---
> WHERE ($tname.attribute_code IN ($qcolstr)) AND (entity_type_id=4)\\\\\\\”;
233c233
< \\\\\\\"SELECT attribute_set_id FROM $tname WHERE attribute_set_name=? AND entity_type_id=$this->entity_type_id\\\\\\\”,
---
> \\\\\\\"SELECT attribute_set_id FROM $tname WHERE attribute_set_name=? AND entity_type_id=4\\\\\\\”,
260c260
< $values=array($item[\\\\\\\’type\\\\\\\’],$asid,$item[\\\\\\\’sku\\\\\\\’],$this->entity_type_id,null);
---
> $values=array($item[\\\\\\\’type\\\\\\\’],$asid,$item[\\\\\\\’sku\\\\\\\’],4,null);
719c719
< $data[]=$this->entity_type_id;
---
> $data[]=4;

 
Magento Community Magento Community
Magento Community
Magento Community
 
dweeves
Enthusiast
 
Total Posts:  877
Joined:  2010-06-26
FRANCE
 

hi stefan, i’ll post a 0.4.2 with your fix suggestion within a few minutes to adress the case, thanks for spotting the bug.

 
Magento Community Magento Community
Magento Community
Magento Community
 
dweeves
Enthusiast
 
Total Posts:  877
Joined:  2010-06-26
FRANCE
 

updated version 0.4.2

- added support for non standard product entity_type_id

File Attachments
magento_mass_importer.zip  (File Size: 20KB - Downloads: 110)
 
Magento Community Magento Community
Magento Community
Magento Community
 
ayasoftware
Jr. Member
 
Avatar
Total Posts:  21
Joined:  2009-06-02
 

to markw87

White Page = PHP error when the PHP Error Reporting is disabled.  You can add this:

ini_set("display_errors"1);
on the top of the file you are running!

To Dweeves: You can add this to your code. Developer mode or not (enabling php errors or not).

Regards,

 
Magento Community Magento Community
Magento Community
Magento Community
 
markw87
Jr. Member
 
Total Posts:  27
Joined:  2010-06-28
 

Hi There,

Thanks for that ayasoftware, I completely overlooked that That has given me the following errors,

Warning: require_once(magento_mass_importer.class.php[function.require-once]failed to open streamNo such file or directory in D:\WebServer\httpdocs\magento\mmi\mmiweb\mmi.php on line 3

Fatal error
: require_once() [function.require]Failed opening required 'magento_mass_importer.class.php' (include_path='.;C:\php5\pear:..'in D:\WebServer\httpdocs\magento\mmi\mmiweb\mmi.php on line 3

Im guessing its got something to do with

] ini_set('include_path',ini_get('include_path').":..");
On line 1.
 
Magento Community Magento Community
Magento Community
Magento Community
 
markw87
Jr. Member
 
Total Posts:  27
Joined:  2010-06-28
 

The latest release is still causing the script to overflow and prevent any more connections to SQL on my production machine.

 
Magento Community Magento Community
Magento Community
Magento Community
 
dweeves
Enthusiast
 
Total Posts:  877
Joined:  2010-06-26
FRANCE
 

To markw87

my guess is that it seems that you are hosting your magento database with a restrictive hosting plan.
what provider (ISP) do you use for your hosting solution, what plan did you take ,do you know your request number limit ?

the script issues around 8-10 sql requests per product. it would be difficult to optimize more (need at least one request per attribute_type per product + categories assignment (if necessary) + stock adjustment (if qty is managed) )

the script is optimized for speed,so it will issue several tens of thousands of requests per minute to achieve that speed.
MySQL server can achieve that but you ISP may have put very severe (or somehow restrictive) restrictions on your account:

for example,your account may have been configured by your ISP:
GRANT USAGE ON *.* TO ‘markw87’@’mysql_isp’
WITH MAX_QUERIES_PER_HOUR 200; //severe
GRANT USAGE ON *.* TO ‘markw87’@’mysql_isp’
WITH MAX_QUERIES_PER_HOUR 75000; //standard but still very insufficient if you are importing 1000’s of products.

so,check with your ISP if you can get another login with less restrictive rights or to update your plan.

alas, having a limited SQL access to the server will just prohibit usage of the script ,and even if the script could adapt, it would loose the whole benefit of its speed to adapt to max queries constraint.

if you’ve got access to a phpmyadmin or equivalent sql console, try this:

SHOW GRANTS FOR CURRENT_USER;

this will show you your account restrictions.

i’ve updated head post limitations to list the MAX_QUERIES_PER_HOUR use case as restriction, will test it in the future release to prevent script execution.

 
Magento Community Magento Community
Magento Community
Magento Community
 
ctnm_sru
Jr. Member
 
Total Posts:  3
Joined:  2010-08-02
 

Hello dweeves,

thank you for support of non standard product entity_type_id.

There seems to be an error in magento_mass_importer.class.php :

“function nullifempty($inval,$outval)”
should be replaced by
“function nullifempty($val)”

Greetings,
Stephan

 
Magento Community Magento Community
Magento Community
Magento Community
 
markw87
Jr. Member
 
Total Posts:  27
Joined:  2010-06-28
 

Hi There Dweeves,

The problem seems to occur even if I try update 1 entry.

With my large file, It seems to execute the 2500 entries, Because I can see all the information entered into the backend, It then starts again, And it loops a few times and then throws a Max Connections SQL error and keeps refreshing the page.

My SQL Privaledges are as follows,

GRANT USAGE ON *.* TO 'nurdnet'@'localhost'
GRANT ALL PRIVILEGES ON `nurdnet\_magento`.* TO 'nurdnet'@'localhost'
GRANT ALL PRIVILEGES ON `nurdnet\_webmail`.* TO 'nurdnet'@'localhost'
GRANT ALL PRIVILEGES ON `nurdnet`.* TO 'nurdnet'@'localhost'
GRANT ALL PRIVILEGES ON `nurdnet\_%`.* TO 'nurdnet'@'localhost'

Update: I checked the log file this might help.

[25-Aug-2010 08:52:00] PHP Warning:  Missing argument 2 for nullifempty(), called in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 820 and defined in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 16
[25
-Aug-2010 08:52:01] PHP Warning:  Missing argument 2 for nullifempty(), called in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 820 and defined in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 16
[25
-Aug-2010 08:52:02] PHP Warning:  Missing argument 2 for nullifempty(), called in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 820 and defined in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 16
[25
-Aug-2010 08:52:03] PHP Warning:  Missing argument 2 for nullifempty(), called in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 820 and defined in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 16
[25
-Aug-2010 08:52:04] PHP Warning:  Missing argument 2 for nullifempty(), called in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 820 and defined in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 16
[25
-Aug-2010 08:52:05] PHP Warning:  Missing argument 2 for nullifempty(), called in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 820 and defined in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 16
[25
-Aug-2010 08:52:05] PHP Warning:  Missing argument 2 for nullifempty(), called in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 820 and defined in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 16
[25
-Aug-2010 08:52:06] PHP Warning:  Missing argument 2 for nullifempty(), called in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 820 and defined in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 16
[25
-Aug-2010 08:52:07] PHP Warning:  Missing argument 2 for nullifempty(), called in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 820 and defined in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 16
[25
-Aug-2010 08:52:08] PHP Warning:  Missing argument 2 for nullifempty(), called in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 820 and defined in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 16
[25
-Aug-2010 08:52:09] PHP Warning:  Missing argument 2 for nullifempty(), called in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 820 and defined in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 16
[25
-Aug-2010 08:52:10] PHP Warning:  Missing argument 2 for nullifempty(), called in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 820 and defined in /home/nurdnet/public_html/magento/mmi/magento_mass_importer.class.php on line 16
 
Magento Community Magento Community
Magento Community
Magento Community
 
dweeves
Enthusiast
 
Total Posts:  877
Joined:  2010-06-26
FRANCE
 

new version 0.4.2a

Hi all,

really sorry for 0.4.2 syntax error (this should not have happened, but it seems that my tests were not covering fields that triggered that call)
so i’ll repost a 0.4.2a that fixes the error on nullifempty.

thanks for spotting guys.

File Attachments
magento_mass_importer.zip  (File Size: 20KB - Downloads: 75)
 
Magento Community Magento Community
Magento Community
Magento Community
 
mennos
Member
 
Total Posts:  47
Joined:  2010-04-11
 

hi dweeves,

found a bug in createOptionValue

for multiselect value:

if you have a single or multiple attribute option(s) that do not exist in the option list, the option value won’t get inserted.

However:
If one of the option vales DOES exist, the new option value will be inserted.

example:

field country
value Japan (does not exist as option value)

-> option value not inserted

field country
values Japan, China (both do not exist as option value)

-> option value not inserted

field country
values USA, China (USA exists, China does not exist as option value)

-> opion value China is inserted

cheers,
Menno

 
Magento Community Magento Community
Magento Community
Magento Community
 
dweeves
Enthusiast
 
Total Posts:  877
Joined:  2010-06-26
FRANCE
 

i’ll post a 0.4.3 tonight (france time) with:
- multiselect fix
- better management of page reload while importing (currently relaunches an import even if an import is already running)
- some code enhancements

 
Magento Community Magento Community
Magento Community
Magento Community
 
dweeves
Enthusiast
 
Total Posts:  877
Joined:  2010-06-26
FRANCE
 

new version 0.4.3

- fix for multiple select
- enhancement on web interface :
- multiple browsers/tabs can be opened while importing (fun to see)
- cancel button (however, the base will not be reverted to initial state)

File Attachments
magento_mass_importer.zip  (File Size: 21KB - Downloads: 127)
 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top