How Import/Export is Implemented in Magento
This is an old revision of the document!
This long anticipated feature is arguably one of the most important ones in ecommerce world. Flexibility of this interface defines capability of the application to communicate with the outside world, be it legacy systems, 3rd party applications, price aggregators, external order sources, fulfillment systems, accounting, etc.
Thus we took it very seriously and, as with everything else in Magento went one or more steps further than most systems. Effectively any import or export is a conversion. From one medium to another. From database to local file, from FTP server to database, from parsed HTML page to Excel spreadsheet, from Google Spreadsheet to catalog.
Basically, it boils down to loading data from one resource, applying unlimited alterations/reformating, and saving it to another resource.
Welcome to the world of conversion.
Let’s take an example of exporting a product catalog to excel spreadsheet file. The logical steps are:
- Load records from database
- Convert internal identificators into human understandable text
- Map database fields to our custom file headings (sku ⇒ Part Number)
- Format data array as Excel XML
- Save resulting text as local or remote file
This sequence of actions we call a Profile.
Here we have 3 categories of components:
- Adapter (database, input/output)
- Mapper (data value processing)
- Parser (data format conversion)
Adapters are anything that can plug in to external resource. This may include:
- Local files
- FTP, SFTP servers
- Database table
- Soap services
- Standard input/output
- HTTP interface
- HTML source
- Google Spreadsheets
- Custom resources (specialized db interface, cache repository, etc.)
Adapters can have 2 methods:
load- extract data from resource into internal container.
save- save data into resource.
Mappers are processing data without changing it’s format or structure. They’re useful for tasks such as mapping one field to another or apply formulas on values.
Mappers can run one method:
Parsers are changing format of data. Examples:
- CSV text ⇒ Grid array
- Excel XML text ⇒ Grid array
- Grid array ⇒ Product model collection
Parsers can have 2 methods:
parse- usually convert from more human readable data to machine friendly format
unparse- the opposite, into human readable format
You might notice that Parsers are separated from Adapters, when usually they are associated as one thing (export to CSV file).
Validators are created to make sure that conditions are met. The conditions can be applied to processed data or environment variables.
Validators can only
Let’s take an example of exporting catalog products into CSV file.
- <action type="catalog/convert_adapter_product" method="load">
- <var name="store">base</var>
- <action type="catalog/convert_parser_product" method="unparse">
- <action type="varien/convert_mapper_column" method="map">
- <var name="sku">Part#</var>
- <action type="core/convert_parser_csv" method="unparse">
- <var name="delimiter">,</var>
- <var name="enclose"><![CDATA["]]></var>
- <var name="fieldnames">true</var>
- <action type="core/convert_adapter_io" method="save">
- <var name="type">file</var>
- <var name="path"></var>
- <var name="filename">products.csv</var>
- <var name="link">/dev/moshe/magento/var/export/products.csv</var>