Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Page 1 of 3
Importing Categories - Step-by-Step - Part 1
 
sean.harrison
Jr. Member
 
Avatar
Total Posts:  7
Joined:  2009-04-21
 

Santhosh Kumar did an excellent job writing an DataFlow importer for Categories in this thread: http://www.magentocommerce.com/boards/viewthread/26022/

But, as they say, the devil is in the details. Some of the code snippets were missing minor but essential details and the instructions assumed knowledge of how to override with custom models as explained here:
http://www.magentocommerce.com/wiki/how-to/customize_part_of_configuration

Being brand new to Magento, I floundered as I simultaneously attempted to learn the concepts surrounding customizations and apply all the changes I needed to in just the right way. So, for others brand new to Magento, here is what I did to successfully batch import categories in excruciating step-by-step detail:

#### Create an Import/Export Advanced Profile ####

Went to Admin page > System > Import/Export > Advanced Profiles > Add New Profile
Put in the XML Santhosh provided at http://www.magentocommerce.com/boards/viewthread/26022/#t110067
Saved.

#### Make File System Changes ###
I logged in to my Magento server and navigated to the installation folder I have entitled “magento”;

[root@localhost magento]# ls -la app/code/
total 12
drwxrwxrwx 3 user user 4096 Apr 16 14
:02 .
drwxrwxrwx 6 user user 4096 Apr 16 14:02 ..
drwxrwxrwx 4 user user 4096 Apr 16 14:02 core
[root
@localhost magento]# mkdir app/code/local
[root@localhost magento]# mkdir app/code/local/Mycomp/
[root@localhost magento]# mkdir app/code/local/Mycomp/Catalog
[root@localhost magento]# mkdir app/code/local/Mycomp/Catalog/Model
[root@localhost magento]# mkdir app/code/local/Mycomp/Catalog/Model/Convert
[root@localhost magento]# mkdir app/code/local/Mycomp/Catalog/Model/Convert/Adapter
[root@localhost magento]# mv /root/magento/Category.php app/code/local/Mycomp/Catalog/Model/Convert/Adapter/
What we’re doing above is adding a folder structure and custom code. The names will match some xml config files below. First, here’s Santhosh’s custom class…

continued in Part II...

 
Magento Community Magento Community
Magento Community
Magento Community
 
sean.harrison
Jr. Member
 
Avatar
Total Posts:  7
Joined:  2009-04-21
 

Importing Categories - Step-by-Step - Part 2

In Part 1 we added a folder structure and custom code. The names of which will match some xml config files later. First, here’s Santhosh’s custom class…

[root@localhost magento]# cat app/code/local/Mycomp/Catalog/Model/Convert/Adapter/Category.php

<?php

class Mycomp_Catalog_Model_Convert_Adapter_Category
    
extends Mage_Eav_Model_Convert_Adapter_Entity
{
    
protected $_categoryCache = array();

    protected 
$_stores;

    
/**
     * Category display modes
     */
    
protected $_displayModes = array( 'PRODUCTS''PAGE''PRODUCTS_AND_PAGE');

    public function 
parse()
    
{
        $batchModel 
Mage::getSingleton('dataflow/batch');
        
/* @var $batchModel Mage_Dataflow_Model_Batch */

        
$batchImportModel $batchModel->getBatchImportModel();
        
$importIds $batchImportModel->getIdCollection();

        foreach (
$importIds as $importId{
            
//print '<pre>'.memory_get_usage().'</pre>';
            
$batchImportModel->load($importId);
            
$importData $batchImportModel->getBatchData();

            
$this->saveRow($importData);
        
}
    }

    
/**
     * Save category (import)
     *
     * @param array $importData
     * @throws Mage_Core_Exception
     * @return bool
     */
    
public function saveRow(array $importData)
    
{
        
if (empty($importData['store'])) {
            
if (!is_null($this->getBatchParams('store'))) {
                $store 
$this->getStoreById($this->getBatchParams('store'));
            
else {
                $message 
Mage::helper('catalog')->__('Skip import row, required field "%s" not defined''store');
                
Mage::throwException($message);
            
}
        } 
else {
            $store 
$this->getStoreByCode($importData['store']);
        
}

        
if ($store === false{
            $message 
Mage::helper('catalog')->__('Skip import row, store "%s" field not exists'$importData['store']);
            
Mage::throwException($message);
        
}

        $rootId 
$store->getRootCategoryId();
        if (!
$rootId{
            
return array();
        
}
        $rootPath 
'1/'.$rootId;
        if (empty(
$this->_categoryCache[$store->getId()])) {
            $collection 
Mage::getModel('catalog/category')->getCollection()
                ->
setStore($store)
                ->
addAttributeToSelect('name');
            
$collection->getSelect()->where("path like '".$rootPath."/%'");

            foreach (
$collection as $cat{
                $pathArr 
explode('/'$cat->getPath());
                
$namePath '';
                for (
$i=2$l=sizeof($pathArr); $i<$l$i++) {
                    $name 
$collection->getItemById($pathArr[$i])->getName();
                    
$namePath .= (empty($namePath) ? '' '/').trim($name);
                
}
                $cat
->setNamePath($namePath);
            
}

            $cache 
= array();
            foreach (
$collection as $cat{
                $cache[strtolower
($cat->getNamePath())$cat;
                
$cat->unsNamePath();
            
}
            $this
->_categoryCache[$store->getId()$cache;
        
}
        $cache 
=& $this->_categoryCache[$store->getId()];

        
$importData['categories'preg_replace('#\s*/\s*#''/'trim($importData['categories']));
        if (!empty(
$cache[$importData['categories']])) {
            
return true;
        
}

        $path 
$rootPath;
        
$namePath '';

        
$i 1;
        
$categories explode('/'$importData['categories']);
        foreach (
$categories as $catName{
            $namePath 
.= (empty($namePath) ? '' '/').strtolower($catName);
            if (empty(
$cache[$namePath])) {

                $dispMode 
$this->_displayModes[2];

                
$cat Mage::getModel('catalog/category')
                    ->
setStoreId($store->getId())
                    ->
setPath($path)
                    ->
setName($catName)
                    ->
setIsActive(1)
                    ->
setIsAnchor(1)
                    ->
setDisplayMode($dispMode)
                    ->
save();
                
$cache[$namePath] $cat;
            
}
            $catId 
$cache[$namePath]->getId();
            
$path .= '/'.$catId;
            
$i++;
        
}

        
return true;
    
}

    
/**
     * Retrieve store object by code
     *
     * @param string $store
     * @return Mage_Core_Model_Store
     */
    
public function getStoreByCode($store)
    
{
        $this
->_initStores();
        if (isset(
$this->_stores[$store])) {
            
return $this->_stores[$store];
        
}
        
return false;
    
}

    
/**
     *  Init stores
     *
     *  @param    none
     *  @return      void
     */
    
protected function _initStores ()
    
{
        
if (is_null($this->_stores)) {
            $this
->_stores Mage::app()->getStores(truetrue);
            foreach (
$this->_stores as $code => $store{
                $this
->_storesIdCode[$store->getId()$code;
            
}
        }
    }
}
Now add the configuration for this custom class…

[root@localhost magento]# mkdir app/code/local/Mycomp/Catalog/etc
[root@localhost magento]# mv /root/magento/config.xml app/code/local/Mycomp/Catalog/etc/
[root@localhost magento]# cat app/code/local/Mycomp/Catalog/etc/config.xml
<?xml version="1.0"?>
<config>
  <global>
        <
models>
            <
catalog>
                <
rewrite>
                    <
convert_adapter_category>Mycomp_Catalog_Model_Convert_Adapter_Category</convert_adapter_category>
                </
rewrite>
            </
catalog>
        </
models>
  </global>
</
config>

Now make Magento aware of this new addition…

continued in Part III...

 
Magento Community Magento Community
Magento Community
Magento Community
 
sean.harrison
Jr. Member
 
Avatar
Total Posts:  7
Joined:  2009-04-21
 

Importing Categories - Step-by-Step - Part 3

In Part II we added custom code and it’s custom configuration. Now make Magento aware of this new addition…

[root@localhost magento]# mv /root/magento/Mycomp_Catalog.xml app/etc/modules/
[root@localhost magento]# cat app/etc/modules/Mycomp_Catalog.xml
<?xml version="1.0"?>
<config>
  <
modules>
    <
Mycomp_Catalog>
      <
codePool>local</codePool>
      <
active>true</active>
    </
Mycomp_Catalog>
  </
modules>
</
config>

The cache needs to be cleared to load the new configurations. WARNING: Now I delete the cache in brute fashion cuz I don’t know any better and this is just a development box. There is almost certainly a better way that I’m not yet aware of…

[root@localhost magento]# ls var/cache/
mage--0  mage--2  mage--4  mage--6  mage--8  mage--a  mage--c  mage--e
mage
--1  mage--3  mage--5  mage--7  mage--9  mage--b  mage--d  mage--f
[root
@localhost magento]# rm -r var/cache/*

#### Add the data and run the Import ####

The Advanced Profile we added in Part I expects a certain file in a certain place…

[root@localhost magento]# mv /root/magento/Categories.csv var/import/
[root@localhost magento]# cat var/import/Categories.csv
"store","categories"
"default"
,"Nifty"
"default"
,"Nifty/Nifty Accessories"
"default"
,"Nifty/Nifty Software"
"default"
,"Nifty/Nifty Software/Windows"
"default"
,"Nifty/Nifty Software/Linux"
"default"
,"Nifty/Nifty Systems"
"default"
,"Nifty/Nifty Training and Support"

Now let’s see if it works…

From browser output:

* Starting profile execution, please wait…
* Warning: Please don’t close window during importing/exporting data

* Starting Mage_Dataflow_Model_Convert_Adapter_Io :: load
* Loaded successfully: “var/import/Categories.csv”
* Starting Mage_Dataflow_Model_Convert_Parser_Csv :: parse
* Found 7 rows
* Starting catalog/convert_adapter_category :: parse
* Processed 100% 7/7 records
* Imported 7 records
* Finished profile execution.

Yea!

For a MUCH better understanding of what is going on here you really need to go to the two links at the top of this article but sometimes it’s nice to learn from a recipe.

 
Magento Community Magento Community
Magento Community
Magento Community
 
sean.harrison
Jr. Member
 
Avatar
Total Posts:  7
Joined:  2009-04-21
 

Ha!

I made this post cuz I was stymied by details and then I see I dropped a detail. Categories.php should end with “?>” but doesn’t in Part II.

I tried to edit my post but the forum has a bug where it thinks my character count is -65 and won’t post any change. Bummer!

Here’s the code for Categories.php again in full:

[root@localhost magento]# cat app/code/local/Mycomp/Catalog/Model/Convert/Adapter/Category.php

<?php

class Mycomp_Catalog_Model_Convert_Adapter_Category
    
extends Mage_Eav_Model_Convert_Adapter_Entity
{
    
protected $_categoryCache = array();

    protected 
$_stores;

    
/**
     * Category display modes
     */
    
protected $_displayModes = array( 'PRODUCTS''PAGE''PRODUCTS_AND_PAGE');

    public function 
parse()
    
{
        $batchModel 
Mage::getSingleton('dataflow/batch');
        
/* @var $batchModel Mage_Dataflow_Model_Batch */

        
$batchImportModel $batchModel->getBatchImportModel();
        
$importIds $batchImportModel->getIdCollection();

        foreach (
$importIds as $importId{
            
//print '<pre>'.memory_get_usage().'</pre>';
            
$batchImportModel->load($importId);
            
$importData $batchImportModel->getBatchData();

            
$this->saveRow($importData);
        
}
    }

    
/**
     * Save category (import)
     *
     * @param array $importData
     * @throws Mage_Core_Exception
     * @return bool
     */
    
public function saveRow(array $importData)
    
{
        
if (empty($importData['store'])) {
            
if (!is_null($this->getBatchParams('store'))) {
                $store 
$this->getStoreById($this->getBatchParams('store'));
            
else {
                $message 
Mage::helper('catalog')->__('Skip import row, required field "%s" not defined''store');
                
Mage::throwException($message);
            
}
        } 
else {
            $store 
$this->getStoreByCode($importData['store']);
        
}

        
if ($store === false{
            $message 
Mage::helper('catalog')->__('Skip import row, store "%s" field not exists'$importData['store']);
            
Mage::throwException($message);
        
}

        $rootId 
$store->getRootCategoryId();
        if (!
$rootId{
            
return array();
        
}
        $rootPath 
'1/'.$rootId;
        if (empty(
$this->_categoryCache[$store->getId()])) {
            $collection 
Mage::getModel('catalog/category')->getCollection()
                ->
setStore($store)
                ->
addAttributeToSelect('name');
            
$collection->getSelect()->where("path like '".$rootPath."/%'");

            foreach (
$collection as $cat{
                $pathArr 
explode('/'$cat->getPath());
                
$namePath '';
                for (
$i=2$l=sizeof($pathArr); $i<$l$i++) {
                    $name 
$collection->getItemById($pathArr[$i])->getName();
                    
$namePath .= (empty($namePath) ? '' '/').trim($name);
                
}
                $cat
->setNamePath($namePath);
            
}

            $cache 
= array();
            foreach (
$collection as $cat{
                $cache[strtolower
($cat->getNamePath())$cat;
                
$cat->unsNamePath();
            
}
            $this
->_categoryCache[$store->getId()$cache;
        
}
        $cache 
=& $this->_categoryCache[$store->getId()];

        
$importData['categories'preg_replace('#\s*/\s*#''/'trim($importData['categories']));
        if (!empty(
$cache[$importData['categories']])) {
            
return true;
        
}

        $path 
$rootPath;
        
$namePath '';

        
$i 1;
        
$categories explode('/'$importData['categories']);
        foreach (
$categories as $catName{
            $namePath 
.= (empty($namePath) ? '' '/').strtolower($catName);
            if (empty(
$cache[$namePath])) {

                $dispMode 
$this->_displayModes[2];

                
$cat Mage::getModel('catalog/category')
                    ->
setStoreId($store->getId())
                    ->
setPath($path)
                    ->
setName($catName)
                    ->
setIsActive(1)
                    ->
setIsAnchor(1)
                    ->
setDisplayMode($dispMode)
                    ->
save();
                
$cache[$namePath] $cat;
            
}
            $catId 
$cache[$namePath]->getId();
            
$path .= '/'.$catId;
            
$i++;
        
}

        
return true;
    
}

    
/**
     * Retrieve store object by code
     *
     * @param string $store
     * @return Mage_Core_Model_Store
     */
    
public function getStoreByCode($store)
    
{
        $this
->_initStores();
        if (isset(
$this->_stores[$store])) {
            
return $this->_stores[$store];
        
}
        
return false;
    
}

    
/**
     *  Init stores
     *
     *  @param    none
     *  @return      void
     */
    
protected function _initStores ()
    
{
        
if (is_null($this->_stores)) {
            $this
->_stores Mage::app()->getStores(truetrue);
            foreach (
$this->_stores as $code => $store{
                $this
->_storesIdCode[$store->getId()$code;
            
}
        }
    }
}

?>
 
Magento Community Magento Community
Magento Community
Magento Community
 
Burkharts
Jr. Member
 
Total Posts:  4
Joined:  2008-08-10
 

What is the file layout for Categories.csv when using this import routine?

 
Magento Community Magento Community
Magento Community
Magento Community
 
sean.harrison
Jr. Member
 
Avatar
Total Posts:  7
Joined:  2009-04-21
 

Not sure what you mean, Burk, but I print out the Categories.csv file above. Is this what you’re looking for?

[root@localhost magento]# cat var/import/Categories.csv
"store","categories"
"default"
,"Nifty"
"default"
,"Nifty/Nifty Accessories"
"default"
,"Nifty/Nifty Software"
"default"
,"Nifty/Nifty Software/Windows"
"default"
,"Nifty/Nifty Software/Linux"
"default"
,"Nifty/Nifty Systems"
"default"
,"Nifty/Nifty Training and Support"
 
Magento Community Magento Community
Magento Community
Magento Community
 
ADLO Design
Jr. Member
 
Total Posts:  20
Joined:  2008-04-16
 

I am building an on-line store to sell around 2,000 products from a wholesaler who provides product & stock information in XML & CSV file formats. I am developing an Access database which will download the product & stock files in CSV format at the press of a button, link some queries to those files, then output CSV file(s) for importing into Magento.

As the wholesaler product files contain the name of the category & sub-category for each product, I have made the database output two files - one containing the categories & one containing the products. I have used the solution described in this & the other thread to import the categories into Magento, but I’ve no way of knowing which category id each category is assigned to, and the only way I can find to import category placement along with the product information is to include a column called ‘category_id’, as the columns ‘category’ & ‘categories’ seem to have no effect.

Would it be possible, or rather simple, to modify this import routine so that I can set the category_id within the Access database, then update the categories in Magento rather than add them all each time? Then I could control the category names & ids, and simply import the products & categories files each day.

I’m pretty familiar with databases in general, but my Magento knowledge and PHP is next to non-existent, so I’m not really confident enough to go poking around in the code of something that is working. But if someone could assist me in modifying the category import to work more like the product import, then it would make a fantastic solution from which I’m sure others would benefit too.

 
Magento Community Magento Community
Magento Community
Magento Community
 
ADLO Design
Jr. Member
 
Total Posts:  20
Joined:  2008-04-16
 

Maybe my Access database could simply output some SQL to update the categories - is there any way to run a SQL update from the Magento back-end interface?

 
Magento Community Magento Community
Magento Community
Magento Community
 
ADLO Design
Jr. Member
 
Total Posts:  20
Joined:  2008-04-16
 

Sorry this was a correction to my earlier post, then I realised I could just edit it - d’oh!

 
Magento Community Magento Community
Magento Community
Magento Community
 
sean.harrison
Jr. Member
 
Avatar
Total Posts:  7
Joined:  2009-04-21
 

Hi, ADLO,

If somebody could develop an import routine or Advanced Profile to fit your (and my) use case, that would indeed be cool.  I’m taking a more labor intensive approach. If someone knows a better way let us know!

1) Like ADLO, make two data stores: the categories and the products.
2) Import the categories as discussed above.

Now I’ve got a products table(s) with the old categories. To learn what Magento expects I ran an export to get all the possible columns, then reimported a couple new products to see what the minimum columns required were.(see other threads for details). During this process, then, I learned that a product import expects a column entitled, “category_ids” which will contain a comma-separated list of category ids.  I decided real quick to just keep it simple for myself and limit the categories to 1 id. So with this information I…

3) Make a copy of my products data and modify it so that my table has a category column named “category_ids”

At this point my “category_ids” column contains the source data, ie. the old database ids or textual names of the categories, and these need to be changed to the new category ids in Magento. I’m sure there’s a better way to get the ids but I noticed that when I’m logged in as an Administrator and am on the Manage Categories page I can then select a particular category and the id is displayed at the top of the page. Something like: “Nifty Category (ID: 6)”

4) Get the id of a particular category from the Magento admin page
5) Run an update on the products datastore to change the old id or name to Magento’s new category id
6) Repeat 4 and 5 until my “category_ids” column is fully updated.

7) Proceed with the details of getting the products imported

 
Magento Community Magento Community
Magento Community
Magento Community
 
ADLO Design
Jr. Member
 
Total Posts:  20
Joined:  2008-04-16
 

Hi sean.harrison

I’m glad I’m not the only one with this issue! I’m currently trying to avoid the manual updates & adjustments to the data before importing it, although I’m about to start looking at ways I can do what you’re doing in a semi-automated way. I guess this will involve creating a table in the database which contains the category names & ids, which I can get from the Magento Admin interface, then I can link the outgoing products information with this table so it contains the category ids.

I suspect our wholesaler will very rarely add new categories, so it shouldn’t be a problem, but as I’m hoping to hand over the day-to-day running of the site to a fairly non-skilled person, I would really like the update process to be fully automated & fool-proof.

Hopefully someone else has already solved this for themselves, & will feel like sharing!

 
Magento Community Magento Community
Magento Community
Magento Community
 
ADLO Design
Jr. Member
 
Total Posts:  20
Joined:  2008-04-16
 

Hi again sean.harrison

Do you, or could you have, your product information in some kind of database? If so, then you could probably populate a table with the category names & ids in, then link this table with your products table so you don’t have to keep updating the products table with the category ids.

I could probably help you to do that via this thread, if that would be useful? I’d need to know what tables you currently have, & what kind of database you’re using. Let me know.

 
Magento Community Magento Community
Magento Community
Magento Community
 
mffowler
Member
 
Total Posts:  31
Joined:  2008-05-16
 

Everything is working for me BUT I get the following error:

“Skip import row, shop “default” field not exists”

I’ve tried changing the store name but nothing ever seems to exist. I have multiple language store views setup and no “defailt” store. Does anyone know what I can put in the store column to be able to import categories?

Thanks,

Mike

<b>FIXED:</b> Just used the store code, in my case was “english” etc.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Frederik Krautwald
Member
 
Avatar
Total Posts:  58
Joined:  2007-09-25
Reykjavík, Iceland
 

Very nicely put. Unfortunately, this will not work if your categories have forward slashes in their names. Would have been nice if there was a variable in the Category.php code, where this could be changed.

Anyhow, thanks for the guidelines.

 
Magento Community Magento Community
Magento Community
Magento Community
 
EmmaWells
Jr. Member
 
Avatar
Total Posts:  8
Joined:  2009-04-05
Belfast City
 

Works like magic!
Thank you!

 
Magento Community Magento Community
Magento Community
Magento Community
 
Francisco Caamano
Jr. Member
 
Total Posts:  6
Joined:  2009-10-03
 

# Processed 0% 0/321 records

I get stock in this line how long should 321 rows take??

Image Attachments
Importcat.gif
 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top
Page 1 of 3