4

Everything I've read about Hadoop seems like exactly the technology we need to make our enterprise more scalable. We have terabytes of raw data that is in non-relational form (text files of some kind). We're quickly approaching the upper limits of what our centralized file server can handle and everyone is aware of this. Most people on the tech team, especially the more junior members of the tech team are all in favor of moving from the central file system to HDFS. The problem is, there is one key (most senior, etc.) member of the team who is resisting this change and every time Hadoop comes up, he tells us that we could simply add another file server and be in the clear.

So, my question (and yes, it's really subjective, but I need more help with this than any of my other questions) is what steps can we take to get upper management to move forward with Hadoop despite the hesitation of one member of the team?

Sorantis
  • 2,720
  • 17
  • 24
Ramy
  • 162
  • 1
  • 7
  • 1
    Give him objective/hours-money related reasons. If you can indeed when necessary, just add another server and be in the clear, then why change? It is a very reasonable way of thinking. Always pushing the latest is not very cost effective from the business point of view (nor from most others). – Rook Mar 15 '11 at 04:52

2 Answers2

8

Why are you in favor of this move?

  • Does it make life easier for the team?
  • Does it help users in managing, accessing files?

Unless you have a technical person who is a decision maker chances are they don't care about who favours what. They care about:

  • ROI (Return on Investment) and how long will it take to get there

Would moving to this new system require re-organization or restructuring things? This can be very expensive.

Being a programmer I understand that everyone (except one) may be in favor of something, but the business side of me is saying there needs to be justification for it.

Create a case for moving to the new system. Some good things to include would be:

  • What will the ROI be?
  • How long will it take to switch everything over? (take your guesstimate and multiple it by 3 to get a realistic number - seriously)
  • Provide real world cases and samples where the new system improved, helped, made life easier, etc for other companies.
  • What will it cost to implement the solution? (be sure to include costs for hardware, software, programming, project management, etc)
  • Compare the differences between your proposed solution and others (ie. just adding a new server). What are the cost differences? What are the long term effects? How long will an additional file server hold them over for?

Basically, to sum it up, take the time to do the research, show facts and figures, present your case.

N.B. Do not make one of your reasons "b/c the team is in favor of it", b/c it will not fly!

RDL
  • 1,069
  • 7
  • 10
  • +1 great answer. Icould use some help on actually calculating/researching the things you mention. How, for example, do I determine how long themigration might take? – Ramy Mar 15 '11 at 05:41
  • Thank you. I would recommend breaking the entire process down into chunks, such as: equipment ordering/setup/installation (don't forget about the purchase approval process if you company has one and time for getting it shipped to you), converting all existing data to new system, training, support existing functionality (programming). The list could go and on. Break it down and work on the timing figured for each one. Get your team involved (including the one not fully in favor yet - this is very important). The whole (team) is greater than the sum of it's parts. – RDL Mar 15 '11 at 05:52
  • For the senior programmer not yet in favor, present it as something you are looking into. Explain that there might be a solution that shows the potential of working for long down the road but you need to evaluate and come up with a case for it for whether it makes sense or not. Chances are they would be more than willing to help in this case as it's an opportunity for them to grow and learn something as well. It's much better than trying to force something on someone that really isn't interested. Plus, if he's a senior dev you will want them on your side. Hope that helps. – RDL Mar 15 '11 at 05:55
2

what steps can we take to get upper management to move forward with Hadoop despite the hesitation of one member of the team?

Solve the problem.

You can't "convince" people of anything with argumentation or science or even facts established outside the organization.

You need to actually solve an actual problem using Hadoop.

Once they have two side-by-side solutions (Hadoop and whatever you're using now) they can compare cost and complexity and performance.

It's not sufficient to say that Hadoop is faster or uses less storage. You must show how much faster and how much less storage.

Even then people will argue that you picked a particularly easy problem to solve.

Proof of concept is the only thing that works.

You can't often get permission to do a proof of concept. In a few companies that actually value innovation, you might get permission. In the rest of companies (which say they value innovation, but actually don't) you have to break the rules and do it on your own.

Even when you have a total solution in Hadoop and can show how much faster, simpler and how much less storage, you will still have to sell this to everyone. Prepare yourself for years of work. Patience is the most important part of making this kind of change.

If you're very, very lucky, others will see the advantages quickly.

If you're not lucky, others will resist actively.

[I was in a meeting just yesterday where the switch from Oracle to MySQL was actively resisted by some developers who were able to lift up one feature in Oracle that's missing in MySQL and claim that we could not work around that single feature. Sigh. Everything can be worked around. However, the resisters have raised a barrier in the form of another proof-of-concept on another application that uses that Oracle feature. What will be next? I'm sure there will be another barrier.]

S.Lott
  • 45,264
  • 6
  • 90
  • 154