Announcing Magento DataFlow - A Flexible Data Exchange Engine

It's been a busy week for the Magento core development team and today we are announcing Magento DataFlow, a flexible and powerful data exchange engine.

How does it work?

Magento DataFlow is a framework for creating components that are used in data exchange processes. The system consists of 4 types of components; adapters, parsers, mappers, and validators. All components can be sequenced to transfer and transform data bidirectionally from one medium to another (eg. from database to local file, from FTP server to database, from SOAP/REST/XML-RPC to database, from Google Spreadsheet to database). If you are non-technical, please feel free to skip to the What Can Be Done section below.

• Adapters are anything that can plug into an external resource and fetch requested and filtered data. This may include:
  • Local files
  • FTP, SFTP servers
  • Database table
  • Web Services
  • HTTP interface
  • Custom resources (specialized db interface, cache repository, etc.)
  • Shell Pipes
• Parsers are components that transform data from one format to another. Examples include:
  • CSV text to 2D array
  • Excel XML text to 2D array
  • Grid array to product model collection
  • Database table to 2D array
• Mappers are components that alter data from one value to another. They're useful for tasks such as mapping one field to another or manipulating data values.
• Validators determine that certain conditions are met. The conditions can be applied to processed data or environment variables. (Currently there are no validators implemented in Magento and this is reserved for future use).

What can be done?

As a first implementation, Magento's next release will preview a product import/export interface that facilitates adding and/or editing batched product data via a file (MS-EXCEL, CSV and Tab-Separated values), and a customer batch export interface. Others uses for Magento DataFlow may include:

• Batch export of any type of data in any implemented format and the ability to import back after changes.
• Data export for pricing aggregators (eg. Shopping.com, PriceGrabber, Bizrate, etc.), including the ability to automatically FTP files.
• Data import and integration with 3rd party ERP / inventory management / accounting applications (eg. SAP, Great Plains, MAS 90/2000, Oracle, Netsuite, Quickbooks, Peachtree, etc.)
• Integration with 3rd party CRM solutions (eg. SugarCRM, SalesLogix, Salesforce.com, ACT, etc.)
• Integration with order management applications and EDI interfaces
• Fetching or publishing RSS/Atom feeds (eg. new products, special prices, product availability, order status, etc.)

It is important to note that Magento DataFlow is a framework and the specific implementations listed above should serve only as examples of the endless possibilities that can be developed. We hope the Magento Community will embrace this concept and utilize to contribute different implementations.

Below you'll find a preview of the product export interface implementation (view high resolution screencast):

A new forum and Magento Group have been started to discuss all Magento DataFlow related issues.