Announcing Magento DataFlow - A Flexible Data Exchange Engine
It's been a busy week for the Magento core development team and today we are announcing Magento DataFlow, a flexible and powerful data exchange engine.
How does it work?
Magento DataFlow is a framework for creating components that are used in data exchange processes. The system consists of 4 types of components; adapters, parsers, mappers, and validators. All components can be sequenced to transfer and transform data bidirectionally from one medium to another (eg. from database to local file, from FTP server to database, from SOAP/REST/XML-RPC to database, from Google Spreadsheet to database). If you are non-technical, please feel free to skip to the What Can Be Done section below.
- Adapters are anything that can plug into an external resource and fetch requested and filtered data. This may include:
- Local files
- FTP, SFTP servers
- Database table
- Web Services
- HTTP interface
- Custom resources (specialized db interface, cache repository, etc.)
- Shell Pipes
- Parsers are components that transform data from one format to another. Examples include:
- CSV text to 2D array
- Excel XML text to 2D array
- Grid array to product model collection
- Database table to 2D array
- Mappers are components that alter data from one value to another. They're useful for tasks such as mapping one field to another or manipulating data values.
- Validators determine that certain conditions are met. The conditions can be applied to processed data or environment variables. (Currently there are no validators implemented in Magento and this is reserved for future use).
What can be done?
As a first implementation, Magento's next release will preview a product import/export interface that facilitates adding and/or editing batched product data via a file (MS-EXCEL, CSV and Tab-Separated values), and a customer batch export interface. Others uses for Magento DataFlow may include:
- Batch export of any type of data in any implemented format and the ability to import back after changes.
- Data export for pricing aggregators (eg. Shopping.com, PriceGrabber, Bizrate, etc.), including the ability to automatically FTP files.
- Data import and integration with 3rd party ERP / inventory management / accounting applications (eg. SAP, Great Plains, MAS 90/2000, Oracle, Netsuite, Quickbooks, Peachtree, etc.)
- Integration with 3rd party CRM solutions (eg. SugarCRM, SalesLogix, Salesforce.com, ACT, etc.)
- Integration with order management applications and EDI interfaces
- Fetching or publishing RSS/Atom feeds (eg. new products, special prices, product availability, order status, etc.)
It is important to note that Magento DataFlow is a framework and the specific implementations listed above should serve only as examples of the endless possibilities that can be developed. We hope the Magento Community will embrace this concept and utilize to contribute different implementations.
Below you'll find a preview of the product export interface implementation (view high resolution screencast):
A new forum and Magento Group have been started to discuss all Magento DataFlow related issues.
1moonlord |posted November 10 2007
Wow, this exceeds any expectations I had concerning the Data Import/Export in Magento.
Thank you guys for all the hard work you put in this excellent product.
Compared to how much time and effort it took me to adjust the Import/Export of products in the osCommerce system this is a piece of cake!
2Crispy5449 |posted November 10 2007
I can only echo moonlords comment - much, much more than I was expecting from this, so thanks again guys.
3WisdOMbooks from Kolkata (Calcutta) - INDIA|posted November 10 2007
WOWonderful
This & the attributes feature
makes Magento unbeatable !
P.S.:
Will data flow imp/exp
work with/for images, too?
4willsabstract |posted November 10 2007
I’m really looking forward to this feature!! I love that you will support import and export. Also, I am elated about the proposed Peachtree and EDI support. I certainly believe many of us would pay extra for these features. You all do such great work. Thank you!
5phrench |posted November 10 2007
Congratulations and thank you! This engine looks great!
It’s an interesting and important question WisdOMbooks has postet: What about importing/exporting images? How will this work?
Besides that I’m wondering if one can use DataFlow in own scripts easily or if the GUI is the only way to go.
6DavePit.com |posted November 10 2007
Simply brilliant! Good question regarding images though
7Mootrealm from Bellingham, WA|posted November 10 2007
Sounds great. It brings the question to mind - how will Megento handle the printing of invoicing and shipping labels. This really concerns me because it has never been done right. Which is astonishing to me because it is one of the most important aspects of an e-commerce workflow. Who wants to print one label at a time. That’s nonsense. It’s all about doing things in batches.
Personally, I use ebay and will be using Endicia (http://endicia.com/) to handle shipping as do so many others on ebay and beyone. Ebays Blackthorne software is integrated nicely with it. I would love to be able to use Endica with Magento. I suppose that is what plugins will be for, but really there should be some sort of internal system for shipping and invoicing. It is just too overlooked in my opinion. At the very least I should be able to print in batches.
8Moshe from Los Angeles|posted November 10 2007
@phrench: Magento DataFlow sequences can be created either by using regular PHP class instantiations or by parsing XML scripts, which could be generated by other software or GUI pages, so there’s no limitation here.
@WisdOMbooks: great idea!
9Crucial Web Host from Phoenix, AZ|posted November 10 2007
This is very exciting! Great vision, Roy!
We are very much looking forward to this implementation. The import feature is something that sounds & looks like it could be extended to possible support the importing other xml data types such as content from a wp blog.
Very exciting stuff. Thank you!
Crucial Web Hosting
10npithia |posted November 14 2007
Well even without this Magento is unbeatable. If you are add this its going to be something that would help of Larger Businesses too…
Hmmm just wondering if anytime Magento will go commercial. Guys like Netsuite charges roof high to get their system running and on top charge thousands of pounds for custumisations. I hope together we can tackle this problem.
Very excited to see more..
Nishant Pithia
11egwada |posted November 14 2007
Hi all !
I think it is a very “sexy” feature for magento. In fact many companies already have a products repository that can be synchronized with magento.
Bravo that feature!
12Tillo |posted November 15 2007
Nice, but we need in europe other interfaces e.g. BMEcat (http://www.bmecat.org/English/index.asp) and Opentrans (for further informations about opentrans contact please ).
13jorge964 |posted December 7 2007
Can this be used to import a OSCommerce DB into Magento and thus facilitating the migration process?
14RoyRubin from Los Angeles, CA|posted December 7 2007
@jorge964 - This will be the platform the integration will be based upon.
15jorge964 |posted December 7 2007
Thanks Roy!
Could you give an approximate update of when the migration tool from OSCommerce could be ready?
Anything partially working of that migration that could be released to “selected” people for beta testing?
Thanks again.
16RoyRubin from Los Angeles, CA|posted December 7 2007
Unfortunately, we do not have a timeframe yet. We have to have it ready as soon as possible.
17Buck |posted January 21 2008
Roy, is there a “simple” CSV or XML format that we can use to upload our products.
I/m 77 yrs old, and I easily mastered OScommerce, and now I’ve moved into ZEN cart, which is much greater than OSC… however, I’m told by our ISP that your Magento is even better than ZEN cart....
All I’ve seen is a lot of videos that go on and on and on… leaving me overwhelmed, and unable to understand HOW....
Is there a formated XML or CSV file that will let me modify, and add products similar to the way we do in E Z Populate… OScommerce and/or ZEN cart.
I like the way EZ Populate 3.03 works in ZEN Cart. If we could get a small product base in CSV or your XML, and load it into Excel, then add our products, or transfer our ZEN card CSV files into, we could have a Magento store up in no time, but as it is now, I’m at a loss as to how we can manage our products in a Magento store… we are dead-N-the water… see store at :
http://www.buckscom.com (Magento)
http://www.buxcomm.com (ZEN Cart)
Is there some form of documentation that tells us how to upload and import the product files… and how to enter the images?
Thanks for any help… G. E. Rogers Sr, CEO BUXCOMM Corporation
18YoavKutner |posted January 23 2008
@Buck - Thanks for trying Magento. This feature exist as displayed above in the Data Flow. As Magento is in beta we are still working on improving it. We have plans to add templates of import/export using the Data Flow functionality that will be straight forward to use.
Thanks
yoav
19Buck |posted January 23 2008
Thanks anyway.... YoavKunter,
We found Magento to be more hype than reality.
Today we’re converting buckscom.com back to ZEN Cart
At least it offers REAL documentation and EZ Populate that really Imports & Exports, and performs flawlessly, and is much less complicated.
Good luck to you and the “team.”
de G E Rogers
CEO BUXCOMM Corp