How do you deal with external sources that do not conform to service standards and protocols?

Question

I'm working on a product that requires a large amount of interactions with different external data providers. These data providers uses a number of standards (primarily xml-based) which are all governed and agreed upon. Some of these standards aren't that great and can be open for interpretation.

Now, once in a while, some data source decides to differ just a little bit from these standards, which means that my application shows an error when the user tries to connect to them - however - a lot of competing products seems to be less concerned about correctness, and simply goes in to quirks-mode to get something working. From my customers' perspective, my product sucks since it can't do X but the competing product can, and when I try to contact the data source to have them fix their issues, I either don't receive an answer, I get to talk with some non-technical person who has no clue about what I'm talking about, or the time-to-deploy on the fix is simply really long.

The way I see it, I got 2 options:

1. Stand firm and risk losing customers, or
2. Do ad-hoc quirks and hacks for semi-compatible sources (which would then need to be supported)

I've been looking for historical precedence for how others have handled these problems (for example, the Firefox vs. IE battle, where some sites simply didn't work in FF because the site contained errors or non-standard constructs), but without much luck about finding anything of substance.

Regarding option #1 I think it's clear that if that is the route to go, I somehow have to make the user understand that the problem is on the side of the source, without getting too technical, which can get tricky.

Regarding option #2 I also have to consider the huge upkeep cost it's gonna be we I have 50+ differently hacked and quirked sources, not to mention the overhead in the of the weird bug-reports that are bounds to come. Also, by embracing quirks I would myself be part of the problem of everyone not following the standards.

Does anyone have any experience with a situation like this? What are the major things to consider here? Have I overlooked something?

[Update] After reading the comments here I've decided to give some more context. I'm working for a small team (3 devs, ~10 people total) managing a product in a large coorporation (1k+ employees). Our product is a spatial suite (think google maps) with a lot of domain-specific workflows build in. My main issues are with WFS/WMS servers using the OGC standards (http://www.opengeospatial.org/standards/wfs and http://www.opengeospatial.org/standards/wms respectively). These standards are a maze of optional parameters, open-for-abuse, and open-for-interpretation definitions specifically regarding default values (well, that's my opinion anyway). Now, if you look past the standards, in the world of spatial data you also have a lot of other issues like wrong projections, self-intersecting geometry, and oh so much more :)

As I stated, we are a rather small team, but since our corporate name is quite large, our customers expect us to handle everything in quite a professional manner - well, some customers anyway.

Answer 1

This depends entirely on the role of your company and its software. In some models, you simply need to estimate the cost of making your integration layer flexible and adaptable and weigh it against the estimated cost of losing customer who either will not or cannot comply with the particular standards you've chosen to respect.

And, then there are cases wherein your business model may revolve more around adapting to customer needs.

A few examples to consider:

Browser vendors never fully implement the web standards. And, to the extent that a browser does implement the latest standard, it's also forgiving of bad markup. No one's going to use a browser that doesn't render 90% of the web because the vendor decided to "stand their ground."

The SQL Server development team authors and implements the T-SQL standard in their database engine. It's the paying customer's expectation that the database manages their data efficiently, safely, and predictably. It would be harmful to the customer and therefore to Microsoft to "forgive" mistakes in the customer's T-SQL and/or make assumptions about their intent!

Clever, a school system integration service provides a single API for curriculum publishers (the paying customers). But, it's their business model to write hundreds (or thousands?) of integrations specific to their customers' customers, who may or may not even be aware of any standards.

---

With respect to the update: The role and importance of the data you're managing is still unclear to P.SE. The criticality of your data could be anywhere from routing ambulances to pizza deliveries. We don't know. And that's OK -- let's keep it that way.

You have lots of technical context to bring to your manager. And, if the complexity of the software or the ecosystem is significant, which it appears to be, it's a complexity your upper-management team needs to understand. They need to answer these questions for you from a business perspective.

^{If you dare to hesitate in approaching them, thoughtfully consider how thrilled your President or CEO will be to learn that business decisions are being made by you, or svidgen ... or even Kilian Foth!}

Answer 2

You're taking yourself entirely too seriously.

The customer wants data aggregated from certain sources. You've agreed to make it happen in exchange for money. Therefore, it's your obligation to deliver on that promise. The fact that some sources don't conform strictly to a protocol that you thought they did is of no relevance to the customer; indeed, they probably wouldn't even understand the difference, which is why they hired you: you're the expert.

Therefore, the thing to do is to extract what data you can from the sources in the way the customer most likely would like (in this case, it's easy: extract at least as much as your competitors do). The fact that this requires hacks, ugly code or breaking standards is irrelevant - you've already agreed to shield the customer from these complications by taking their money.

You're only justified in rejecting a task if quirks in the input would make the entire project so expensive that you can't turn a profit at the agreed price. Then it's time to renegotiate (although in the real world, you'd often have to swallow the cost if the customer is otherwise profitable). Otherwise, it's sell-your-idealistic-soul-for money time.