My boss has a bad case of "Not Invented Here"

Question

My department specializes in converting customer data into our database schema so that they can use our software.

Right now, we have C# applications that take an IDataReader (99% of the time it is a SqlDataReader), perform some cleaning and mapping, insert it into a DataRow object, and then use a SqlBulkCopy to insert it in to our database.

Sometimes (especially when the source database contains images as varbinary objects), this process can really bog down with a SQL transfer from the server to the app to just then turn right around and go back to the server.

I feel that if we re-wrote some of the conversions as SSIS packages it could speed things up a lot. However the biggest stonewall I keep running into is when my boss, in Not Invented Here fashion, pushes back and says "What if Microsoft drops support for SSIS? We will have all this obsolete code and we will be screwed."

This is not the first time that I have hit a "What if they remove that feature...?" answer from my boss. I don't have the time to write the conversion the old way, self-teach myself SSIS, and also write it the new way to demonstrate/test the benefits (None of us have used SSIS so there would be a period where we would have to learn how to use it).

What should I do in this situation? Stop pushing the new technology? Wait till he leaves the department (I am the 2nd most Sr. person in the department after him, but it could be years before he quits/retires)? Find a new way of getting him to stop being afraid of 3rd party tools?

Answer 1

I'll take a crack at this from a managerial standpoint, but keep in mind that I'm aware I don't have all of the details. I'll summarize what I see:

1. Mid-level developer, we'll call him "Scott", recommends a rewrite of legacy code into SSIS to improve the performance of important process ProcessA.
2. ProcessA is currently behaving in a functioning state with no known major issues.
3. ProcessA is written with proprietary tools using common or potentially tribal knowledge to maintain.
4. Recommendation to rewrite will require new tools to support.
5. Current staff has no documented experience/knowledge with the new tools.
6. New tools are relatively recent replacements to older tools, and support for these tools may change reasonably within 4 business quarters.

From this perspective, all I see is a large outlay of money on the company's part to improve a process that isn't broken. From a technical standpoint I can see the appeal, but when you get right down to it, it just doesn't make business sense to make this change. If you had staff on hand with documented experience with SSIS and benchmarks to show massive improvement to this process (keep in mind, massive improvement MUST equate to $$$), the outcome might be a little different. As it stands now, though, I'd have to agree with management (somewhere a tree just died).

If you want to foster the adoption of SSIS and potentially lead into this refactor, you need to gain this experience and training with smaller, less important projects. Provide benchmarks and support for SSIS, and make sure that all of the infrastructure and support is in place before management will even consider making the change. Once you have the tool in use elsewhere, people on the team experienced with its usage, and a business "comfort" factor that support won't change and uproot everything, you will be more likely to sway someone to your viewpoint. Without those, you're barking up the wrong tree with that argument.

As stupid as it sounds, sometimes the "best" way isn't the best way.

Edit: In response to some updates to the question, I'll post a slight modification to my answer.

If the process is experiencing distress of some kind, rewriting it will still be a costly venture. You may want to consider what the cost of fine-tuning the existing code would be against rewriting the package. Consider the impacts not just to software but to any human interfacing processes. When attempting to get management buy-in to a rewrite, it still will always come down to the money. Unless you can show that the current distress is costing money now or will become large in the aggregate, management will still not see the benefit. This cost may not necessarily be financial in nature. If the slow down compromises a system causing downtime, introduces intrusion vectors, or some other "hard to quantify" symptom, you may need to find a way to translate that problem into a monetary risk equivalent. An intrusion vector, for instance, may lead to an intrusion which could result in lost, stolen, or corrupt data. The company could lose reputation or may fail a necessary security audit. The key is to get the manager in question to recognize the quantifiable benefits of the change.

Answer 2

Breaking Things Is A Skill

I've worked at far too many places that embraced the "if it isn't broken" argument to the point that they fail to innovate, and when they finally are forced to innovate they over react by changing everything. Simply because they lack the experience of breaking things.

It takes maturity, skill and experience to break things.

Development teams that always play it safe are the easiest for a competitor to surpass. Only teams that have failed, made mistakes, and broken things are able to truly do honest risk assessments.

Keeping The Status Quo

While it's true the current system meets the current business requirements. That isn't true for the future unforeseen changes to those requirements. As the old proverb says "fortune prefers the prepared".

This question has nothing to do with SQL or performance. It's about asking the question "is there a better way?" and not being afraid to try an alternative.

Your boss suffers from a case of Keeping The Status Quo.

The Mayan's

Nothing is truly working perfectly.

The Mayans kept growing their food on the side of mountains. Until all the nutrients were washed away, and they had no way to feed their people. They kept doing things the same way until it was too late.

Assuming you'll have time to fix a problem when the problem presents itself is a mistake.

What To Do?

Your boss suffers from a case of conditioning. People who accept the status quo often do so, because they lack the ability to make difficult decisions. When faced with a challenge they will tend to choose the option closest to what they are familiar with.

Fear for him is a big motivation. Fear of the unknown or changing conditions shakes his perspective of what is the status quo. He will tend to try and return the conditions back to normal as quickly as possible.

You need to present ideas in a non-conflicting way. Try to find common ground between what you would like to do with what is already the status quo. Present arguments that reduce his fear of change, and offer reassurances that you want to complete a task that won't cause significant change.

Take Baby Steps

It would be best to offer changes that move the project in the direction you want, but via small incremental projects. Rather then hit your boss with the big question about supporting SSIS. Offer to create a separation layer in the code that would allow you to plugin "alternative" methods for processing tables with large attachments. You can then progress to recommend SSIS with all the prerequisites have already been added to the project. This reduces the risk your boss envisions by accepting the change.

It Takes Time

My experience has been that risk takers keep a project moving, and status quo keepers are like a brick wall. Persistence is your only option to breaking down their barriers. Expect to keep hearing No to your inquiries.

When the time comes to innovate. Your boss will turn to you quickly, because you demonstrate fearlessness in the face of change. Something a status quo person will be looking for, and you will be rewarded for your efforts. Even if none of your previous inquiries have been accepted. You will quickly become an irreplaceable asset in a company faced with change that embraces no-change.

Answer 3

I feel that if we re-wrote some of the conversions as SSIS packages it could speed things up a lot. However the biggest stonewall I keep running into is when my boss, in Not Invented Here fashion, pushes back and says "What if Microsoft drops support for SSIS? We will have all this obsolete code and we will be screwed."

In my opinion, skepticism about SSIS is valid.

• Seasoned developers hate black boxes, and there's a lot of magic that happens inside of an SSIS package.
• Source control support for SSIS packages is sorely lacking. Diffing and merging SSIS dtsx files can be a nightmare if your dtsx packages aren't sufficiently modular.
  • By contrast, C# console applications are very transparent. You can always follow the code to figure out what's going wrong.

---

Also, consider that your boss is on the hook if anything goes wrong.

• Therefore, he's entitled to have the overriding/only opinion on the matter.

---

I don't have the time to write the conversion the old way, self-teach myself SSIS, and also write it the new way to demonstrate/test the benefits (None of us have used SSIS so there would be a period where we would have to learn how to use it).

What should I do in this situation? Stop pushing the new technology? Wait till he leaves the department (I am the 2nd most Sr. person in the department after him, but it could be years before he quits/retires)? Find a new way of getting him to stop being afraid of 3rd party tools?

I recommend that you learn SSIS well enough so that you can demonstrate its benefits.

And if, after your self-study, you find that SSIS offers significant advantages over the more "traditional" approach, and you still can't convince your boss to change course, then I recommend that you find a different job in which you can use SSIS.

Answer 4

The response you are almost always going from most manager-types is something like:

"It's not worth it, it works now, and it will cost time to change."

And in general, this is valid. However, this is not always a valid argument, because the core issue surrounding "Not Invented Here" syndrome is not whether things work or don't work, but that technology is being pointlessly re-written, wasting company hours, and detracting substantial value from the product being delivered.

You need to determine if Not Invented Here is actually taking place before deciding what to do. The internal tech may have been written for a reason.

Signs that the Tech is Rewritten for a Reason:

• The tech you are selling is also the product. If you are a database vendor, using MySQL code, no matter how much time it will save, is a dangerous idea for many reasons.
• The internal tech is well-maintained and effectively addresses shortcomings of the off-the-shelf solution, while still providing comparable base functionality.
• The tech improves the productivity of the entire development team, and new developers are easily sold on why it is in use.
• There are other examples where the company has successfully adopted other external technologies.
• Your business would be immediately and seriously impacted if the OTS solution were discontinued or broken.
• The business is so large that it has the resources to write high-quality tools at a low cost (for example, Google may be able to write a database that costs less than a $1000/seat MS SQL license)

In other words, the team understands the drawbacks of re-solving already-solved problems, but judiciously made exceptions where it made sense from a business perspective.

Signs of "Not Invented Here":

• Seems like everything is internal, and there are excuses for everything: "uh, it was too slow", "uh, it's Microsoft, we hate Microsoft", "uh, it looks really hard to use", etc.
• For cases where external tech is being used, you hear "yeah, well it stinks and we plan on writing our own eventually".
• The owners of those components don't have other jobs they are capable of working on.
• You see most new developers coming in with a widely-accepted skill-set, but it takes considerable time for them to ramp-up to the internal tooling
• After careful analysis, it becomes apparrent that switching from the OTS solution to a custom or other OTS solution in the "what if it's discontinued!" scenario would have minimal business impact. For example, if String.Join() were removed from the .NET Framework, re-implementation to a custom StringJoin() method would be trivial.

In other words, NIH is the pattern of failing to stay objective and realistic in cases where external tech is being used instead of one's own code.

When tech is rewritten for a reason, bringing up your concerns should be praised and appreciated by your superiors. It should have been a careful decision when the tech was implemented, and reviewing the decision occasionally is good for the business. Things do change over time and you may have a valid point. If you can provide numbers about time wasted in the past, projected costs to switch, and other information, you could (in theory) make a case for switching.

Understand that given your experience, they may not agree with your numbers, but regardless, they should at least hear you out. It may take time to gain respect.

If the conversation can't even be brought up, even if you're being polite, it might just be "Not Invented Here". In which case, it is most likely a personality or political issue that you probably cannot fix easily. Working in an environment that is heavily bogged down by constant reinvention is toxic to the developers and the business. Run.

(Side note: In most companies, there is always an element of NIH, but it's at a tolerable level, and as long as they regularly review their code and practices. In the long term we are all guilty of it to a degree, but it never becomes an issue.)

Answer 5

Well, its all about the cost/benefit analysis.

What do you gain by rewriting it to SSIS? Speed? Does it matter? If its a process that get executed in a GUI and waste everyone time, then yeah, its probably a good idea to speed it up, because the money spent changing it will be gained back by more happy customer or internal employee doing their job instead of waiting for the software. But if its a night batch that start at 12am and it will end at 1am instead of 6am... there isn't much point. Yeah its 6 time faster but no one will feel the difference.

And your boss does have a good point. Microsoft do tend to abandon some of their technologies. VB6, Linq-to-SQL, ASP classic, COM+... This is a risk with any non-open source software (and open-source software that would be too big for your organization to maintain in case lack of update). If its central to your application, you should have a tight control over it.

Software is all about adding value to the customer, and technical improvement that don't yield much benefit while introducing risk don't really fulfil that role.

Answer 6

Develop a POC (Proof of Concept) and then demo it to your superior. The POC should help you determine any real benefit with the technology you are proposing. Then you and your superior can talk about the technology and develop pros and cons to implement it.

You will probably have to spend some extra time outside of normal work time to develop the POC.

As a side note from an SSIS point of view, I have used it and have developed SSIS packages. We actually replaced a proprietary load process using SSIS packages. We only did this because actual customers complained about the load times. Load times decreased significantly with SSIS and everyone got happier.

SSIS is basically a drag and drop workflow with very little programming involved. It does take some time to learn how the black box works, so you will need to take that under consideration if your team starts using it.

Answer 7

Good answers. If it ain't broke, don't fix it. I would only add

• While the performance concerns might actually be true, that word "might" is a red flag. I would do performance diagnosis first, so I would have proof of what the performance problem was, before committing any resources to addressing it.

• When considering the latest "solution" from Microsoft, one should consider that plenty of people have been burned by what MS were once touting but subsequently deprecated and then de-supported. If you want software to run for 10 or 20 years without major recoding, you should be very cautious of that. Our company has been badly hurt that way.

Answer 8

Turnover is going to be a consideration for your boss. SSIS is getting into the DBA arena compared to a programmer having that skill set. If your application doesn't use SSIS beyond the initial conversion, I'm not sure it makes sense to learn it and get new C# programmers up to speed because like your team, most don't have any experience with it.

Answer 9

You could point out to your boss that SSIS is, in fact, an older technology layer than the .NET Framework, if you go back to its roots as the "Data Transformation Framework" of SQL 7.0.

Where your boss might have a point is in the fact that SSIS is only a part of the Standard and Enterprise versions of Microsoft Sql Server; that's a rather big chunk of change for your clients to pony up, when your application, in a small business scenario, may well be perfectly fine with Sql Express (or MySQL, for that matter, which works with ADO.NET but can't use SSIS integration).

Now, let me be perfectly clear; IMO, NIH is almost never a good thing for a software development house. It locks you out of a lot of very powerful tools and services. It's also hypocritical on its face; did your company write Visual Studio? How about the .NET Framework? Sql Server? Windows? No. You build your software on top of the tools and platforms you already have (and so do your clients). Therefore, if you see a tool that is gaining general acceptance, and that could be useful in performing your core business, you adopt it, and you learn to live with the risk that you will have to maintain your implementation to keep up with the latest versions of those tools, or even replace them. I bet your boss first developed the software to run on Windows 95/98 or thereabouts (if not long before that, like 3.0/3.1). If so, he's seen half a dozen versions of Windows workstation OSes come and go, and had to update his software to run on the newer platforms, and he's facing another one with XP officially EOLed. While he may complain, that's simply the cost of doing business.

However, a NIH attitude doesn't necessarily follow from a refusal to accept one, or even a number, of technologies that may be viewed as helpful. Those refusals could be based on perfectly valid cost-benefits analyses. I work for a video surveillance and alarm monitoring company, and we make decisions to support or not support various new technologies or products on a daily basis. Usually, the decision to accept a new technology, and the pain of bodging it together with our existing heap of supported viewer and alarm management software, is based primarily on what it's worth to our customers. I recently finished a big integration project with a new type of DVR, simply because one of our biggest existing customers said they were upgrading to that DVR from another big-name but technologically-lagging DVR product, and they would do it with or without our help. On the other hand we regularly reject new manufacturers, even major household names, simply because our customers don't use it and we see no value in starting to offer it, even if it loses us a few potential customers that have it and don't want to pay for our version of the same thing.

Answer 10

I don't have the time to write the conversion the old way, self-teach myself SSIS, and also write it the new way to demonstrate/test the benefits (None of us have used SSIS so there would be a period where we would have to learn how to use it).

That's kind of the problem, isn't it? You're asking other people to risk their productivity on your idea, and you're not willing to go out on a limb to show them it's worth it. Technical leadership requires either risking your reputation or your free time to show that your ideas are worth having.

Answer 11

Try to see things from your boss' perspective. It sounds like the functionality you're trying to replace is core to what your software does (cf. his "screwed" comment). A good manager knows that you outsource your core business at your own peril. He's rightly worried about betting the farm on some piece of tech that might disappear in the future. When core functions are involved, Not Invented Here is actually a good thing.

If speed is a feature, you're going to have to find another way to speed things up. Otherwise, if speed matters more to you than to your clients, I say drop it and forget about it.

Answer 12

While there is merit to “don’t fix what is not broken” I disagree with the accepted answer.

The accepted answer comes from the perspective that problem is not broken. From a mid-level management's perspective this is true. If a cost analysis were to be done on the time spent by developers to create and maintain an ETL solution in C# compared to buying an out-of-the-box solution, it would show the C# solution takes 3 to 4 times longer to create and maintain and cost as much as 10 times more (I looked for the reference on the numbers, but I could not find it).

Unless it’s the company’s core competency there is very little reason to write and maintain data transformations in C#. The homegrown solution will be slow, there will be mistakes (i.e. corrupt data), there will be edge cases, there will be little reuse between ETL classes, and it will be fragile. Honestly, what developer wants to spend his/her days writing and maintaining ETL in C#? I’ve done it. It’s a load of crap. It’s about as far away from creative work as one can get.

Use a tool like Informatica, a company whose business is ETL. They’ve been working this problem for over 15 years. They have it solved. Their tools are drag and drop, reusability is built-in, as is performance. Most anyone (i.e. developers don’t have too) can create ETL with Informatica tooling. I’m not trying to sell Informatica, use any tool. Just don’t re-invent the wheel.

In the long run buying a tool, such as Informatica or using SSIS will save the company potentially millions over the long-haul. And best of all you will not have to maintain the ETL or be responsible for it when it breaks.

Answer 13

I've tried SSIS to do simple tasks before.

It can be very annoying, since even simple tasks gives birth to low to mid-level complexity diagrams (no, there's no other way to "code" it except the diagrams). And the source control problems pointed by Jim's answer I wasn't aware.

Side problem: So, for speed up, I suggest reduce the size of the transactions for your image bulks. Say, every 800 instead of 5000 rocords. It can reduce the size of the I/O needed to support that transaction.