Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Page 1 of 2
My Import Experiences (22k+ products)…
 
MarkoPoloUSC
Jr. Member
 
Avatar
Total Posts:  1
Joined:  2008-04-22
Columbia, SC
 

I figured I’d chime in on the topic, and maybe help some people out who are going through the same issues, and ask some questions while I’m at it!

I’m currently migrating a CRE Loaded (osCommerce) store with over 22,000 items - all with pictures, and manufacturers associated with them.  All of these items were entered in manually over the course of about 2 years with the help of one full-time person and about 3 other people when we “had time” (yeah right!).

First, I exported all my products using Easy Populate that was included with CRE Loaded.  Then, imported that text file into FileMaker Pro to edit and clean up the files.  Excel can’t handle a “cell” with over so many characters, and often my descriptions go way above the limit, so I’m left with a bunch of descriptions that read and export as “#####.”

I uploaded one product to my Magento installation, then exported it.  I imported that csv file into FileMaker Pro.

Then using my magento.csv file, I imported my edited CRE Loaded database, used FileMaker’s import tools to align the old tables with the new ones, and volia!  I’ve got a nice file to import into Magento.  Or so you would think....

For some reason, I had to export the file as an Excel XML file before I could get ANYWHERE at all.  I don’t know why.  I’m just satisfied I got something to work.

I split the file up by manufacturers, so the import would actually work.  The problem with 22,000 items means I have about 1,000+ categories.  Until the powers that be make the Category system more user friendly, I have to manually create the categories, and add the products.  This is very undesirable, as we have over 100 manufactures.

Once imported, I had the same problem as everyone else.  The products didn’t show up until they were opened from the backend and then saved again.  I did however figure out that all you have to do is…
1) go to products > manage products
2) click “select all”
3) next to “Actions” pull down the box and select “Change Status”
4) pull down the new box that pops up, select “Enable”
5) click “Submit”
That’s it!  It takes care of all your products, so for those if you in my situation with tons of products, it’s a huge time saver.

Next, I created the Category and was able to mass assign the imported products to that category.  Wohooo I’m lovin this stuff.

Ok, so here’s my problems.  Images don’t work!  I zipped all the images from CRE Loaded, dropped them in the /media/import folder in magento and made sure that i had /image.jpg for all three images in my XML file.  I learned this was how to get images to work from Anna’s post (http://www.magentocommerce.com/boards/viewthread/7018/).  Anyone else find this dosen’t work for them?  When I click on the products that have been imported through the product manager, then click “Images” I find that there are no images assigned.  For some reason the Import is ignoring my instructions.. *grrrr*

Next thing I have issue with is Manufacturers. From what I understand I’m going to have to insert my manufacturers manually, then import the products.  Talk about nightmare!  I’ll need a serious supply of Starbucks and about 36 hours to get this done.  I know it’s possible, but it somehow doesn’t seem worth it.  Manufacturers is a pretty important deal to retailers.. I’m surprised its not an easier feature to deal with.  Maybe in the next release… unless anyone has figured out a workaround.

So I’ve sort of halted my imports until I can get the issues I have fixed.  It seems I may be putting in maaaaaany hours to get things up and going, and I’d rather just do it once!  Any input on my issues are much appreciated!!

For now, I’m stuck with the unreliable, customers-get-pissed-off-and-call-freaking-out CRE Loaded!

Thanks!
Mark

 
Magento Community Magento Community
Magento Community
Magento Community
 
Michae1
Enthusiast
 
Total Posts:  826
Joined:  2007-08-31
 

Hi Mark,

Can you please PM me a sample file containing one product of those that you tried to import ?

Thank you,
Michael

 
Magento Community Magento Community
Magento Community
Magento Community
 
Appmerce
Guru
 
Avatar
Total Posts:  376
Joined:  2008-05-13
Amsterdam, The Netherlands
 

So in conclusion, the Magento import function are not as good as osCommerce’s Easypopulate yet. Two major differences:

on product import:

- you cannot specify (a tree of) categories to be created on-the-fly; *
- on product import, you cannot specify a manufacturer (or other attribute) to be create on-the-fly; **

You basically have to create all categories and manufacturers BEFORE an import. Then you need to place the correct category_id in the CSV MANUALLY.

That’s not the way to go for live production stores. Im working with 100+ categories, like this:

Honda -> 2006 -> CRF250 -> AIR CLEANERS
Honda -> 2005 -> CRF200 -> CRANKSHAFTS

Etc. Its just impossible to do by hand.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Chris Williams
Member
 
Avatar
Total Posts:  67
Joined:  2007-09-02
Dallas, TX
 

I’ve got a lot more products that 22k to import.  I don’t think DataFlow is built for that.

What I really need is a way to send batch updates from another source (retail store).  I think we’re looking at something custom outside of DataFlow at this point.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Bonaventure
Jr. Member
 
Avatar
Total Posts:  29
Joined:  2007-09-26
 

@Chris. Please share your “custom” experience as it comes available. We have over 100,000 products and I can’t get the dataflow to work--even when I break it into small chunks.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Chris Williams
Member
 
Avatar
Total Posts:  67
Joined:  2007-09-02
Dallas, TX
 

I’d be very interested in what you come up with.

 
Magento Community Magento Community
Magento Community
Magento Community
 
blakew
Sr. Member
 
Total Posts:  107
Joined:  2008-06-20
 

I in interested in this too, as I am importing 60k with 1000+ categories. The Magento database is so confusing that I don’t know that I could write a manual script to import all the categories. Perhaps someone has a script already that populates categories?

 
Magento Community Magento Community
Magento Community
Magento Community
 
Brandlive_br
Guru
 
Avatar
Total Posts:  653
Joined:  2008-05-19
Buenos Aires, Argentina
 

I was trying to use a cli process, described into this forum. I’m playing with 2 updates every day, with 26k products.
At first time, I have to reconfigure php to use unlimited execution time.
I just don’t remember how much time toke me that import. After that, I try to make a really really heavy test, but it was impossible.

My little solution, was use a Procedure (Mysql 5) that make all the dataflow steps (most of them).
Using an external php, call to that procedure, I can import 220k prodcuts in about 2 hours 20 minutes (using a conventional pc).

I know taht 220k it’s an important number, but that was my stress test.

Using the same steps, import 26k products, it’s about 20 minutes. The procedure makes the inserts, updates o logical deletes (that’s it’s a custom functionality of my procedure).

 
Magento Community Magento Community
Magento Community
Magento Community
 
Brandlive_br
Guru
 
Avatar
Total Posts:  653
Joined:  2008-05-19
Buenos Aires, Argentina
 

I forgot, using DataFlow and trying to import 220k products, take me about 19 hours.

 
Magento Community Magento Community
Magento Community
Magento Community
 
blakew
Sr. Member
 
Total Posts:  107
Joined:  2008-06-20
 

I may be interested in discussing how you did this. I have spent the past two days learning the database and now I have written a custom php script to parse data from our AS-400 db2 database and it exports a csv… I take that csv and import it. I plan on automating this who process once I get it working manually. I am not sure how mysql 5 will run Procedures with an AS-400. We are in quite an akward situation with our makeshift nightly refresh.

 
Magento Community Magento Community
Magento Community
Magento Community
 
JikeSpingleton
Jr. Member
 
Total Posts:  30
Joined:  2008-05-01
 

its funny that dataflow is so unusable that people have started writing their own import tools. really goes to show how bad it really is, its a shame that the magento team has done nothing to improve it.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Brandlive_br
Guru
 
Avatar
Total Posts:  653
Joined:  2008-05-19
Buenos Aires, Argentina
 

I’m not agree.
It’s true, the Dataflow it’s not the option for a hugh quantity of products, but I don’t think that, at this moment, this be a shame.
Also, Magento Open Source application give Us the code, they improve it, and we improve it too.
That’s antoher interesting part of Open Source philosophy.
I’m not part of Magento, and I need a better DataFlow option, but I don’t think that We can say that everything into the application it’s perfect, or that everything into the application it’s a shame.
I hope that the community and Varien can build a better application.

 
Magento Community Magento Community
Magento Community
Magento Community
 
blakew
Sr. Member
 
Total Posts:  107
Joined:  2008-06-20
 

I agree with Barbanet. I have wrote my own import tool and it actually works now.

I have 935 categories online, but I haven’t put the products online yet. I will be doing that in a few hours. I have noticed that each category takes about 4 seconds to come up. This is a server with a local connection so bandwidth is not an issue. I assume when we go live with our site, we will have a much faster server, and we will seperate the database and the webserver. I am afraid to see how slow magento runs when I add 20k - 40k products. I am also researching the cache management stuff they recommended installing. I will try to post my results, perhaps I will start my own thread with my problem, steps taken to fix it, and current status.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Chantal
Member
 
Avatar
Total Posts:  68
Joined:  2008-05-11
Netherlands, Hellevoetsluis
 
blakew - 11 July 2008 07:05 AM

I have wrote my own import tool and it actually works now.

Is it ready to share with the community? wink

 
Magento Community Magento Community
Magento Community
Magento Community
 
blakew
Sr. Member
 
Total Posts:  107
Joined:  2008-06-20
 

I would be more than glad to share my import process for categories, I will post it in another thread and post a link in here...perhaps later today! smile

I am running into a serious problem though. I have 900+ categories and I have bogged the software down so much that memory is being exhausted at the 128mb level. I can’t even go into “Manage Categories”. This is a very serious problem and is bringing me right back to ground zero, and looking at alternative solution, even though I have put weeks into implementing Magento into our test server. It seems Magento isn’t ready for a company our size, 900+ categories and 20k - 40k products. I hope someone from Magento comments in here or the new beta fixes this.

 
Magento Community Magento Community
Magento Community
Magento Community
 
SimpleHelixcom
Enthusiast
 
Avatar
Total Posts:  906
Joined:  2007-08-31
Huntsville, AL
 

Hi Blakew,

You might want to up your memory limit to atleast 512mb.
Most of our hosted clients who have over 20k products are running fine with a 512mb setup.

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top
Page 1 of 2