Mapping different XML and CSV feeds

Question

Not sure if this is the right venue to be asking this but here goes.

A little background.

I'm trying to build an ecommerce app that would allow sellers from other venues--like, amazon and newegg--to import their products using their export/import files.

An example would be a user having an XML feed that looks somewhat like this:

<?xml version="1.0" encoding="UTF-8"?>
<products>
    <product>
        <sku>12345</sku>
        <name>Dell Computer</name>
        <description>This is a very nice Dell computer. Much quality. Very speed. Great wow.</description>
    </product>
    <product>
        <sku>123456</sku>
        <name>Dell Laptop</name>
        <description>This is a Dell laptop. Much quality as well.</description>
    </product>
</products>

I'd like to be able to map the headers to columns in my database.

ie. sku -> product_sku, name -> product_name, description -> description.

What I'm doing right now is since not all feeds have the same structure, I developed a "mapping tool" that basically just parses the file, takes the headers and shows it in a UI that allows the user to "map" these to the list of my database columns.

The problem I'm encountering now is that since the feeds are not the same, I have a hard time getting that list of headers.

My question is what's a good way to approach this problem? Maybe I'm doing this all wrong.

I'm doing this in nodejs/javascript if that's at all relevant.

Voting to close as "unclear" - the description above is not suitable to understand what the OPs problem is. — Doc Brown, Dec 04 '16 at 06:36

score 1 · Answer 1 · answered Jun 07 '16 at 03:43

If you mean that they should be able to import their existing files, then you will have to create a mapping to translate each of the existing Amazon, Newegg, ... formats into your database. For XML feeds that means mapping XPath expressions to your columns. For flat file feeds it means mapping the headers in the flat file to your database columns.

score 0 · Answer 2 · edited Sep 10 '15 at 20:39

Look for product not headers.

Product is the basic unit you are working with. First identify that. In your example, you can see that <product> is the only tag that has siblings of the same name. Although the same would be true for things like <category>, and <colorOption>.

Work with XML paths, not just tag names.

Let a human do the hard bit: get your sellers to pick the "headers" straight from the XML.

Display the XML (maybe as a menu tree) and ask the seller the selected the tag for a Product. Then highlight that fragment and ask the seller to select the tag for ProductName and so on.

Learner_101 · Answer 3 · 2016-04-08T02:14:05.797

You will need to map the all tags from all the feeds to the correct field.

For example in one feed if you have sku you map it to product_sku In other feed say you have an other tag sku_product and you need it mapped to product_sku. Now since only one tag will exist in one feed for a product ( hopefully) , you can have a config mapping the tags to a particular database fields

e.g of Config

sku=product_sku

sku_product=product_sku

sku3=product_sku

cat=product_category

category=product_category

category2=product_category

....

now when you parse your xml you just check if the tag exists in this config and get the field to map to . It will only yield one tag for a particular feed.

You will have to manually create and maintain this config in a file or a database and load it in to a hashmap prior to your parsing.

Mapping different XML and CSV feeds

3 Answers3