4

I'm wondering if I wanted to implement a web service based on java that does web analytics, what sort of architecture should I use. The actualy processing of the Big Data would be done by Hadoop.

However I am not sure what I would need to do to make it asynchronous, or is Hadoop already asynchronous by nature? Could I do something with JMS? If so, how would that fit into the whole thing? Would it be something along the lines that upon receiving the web service request, I use JMS to send a message to Hadoop to handle a particular data, and then wait for it to come back to me? I am not too familiar with asynchronous java so I'm not sure where to start.

yannis
  • 39,547
  • 40
  • 183
  • 216
Jason Madux
  • 83
  • 4

2 Answers2

2

" I use JMS to send a message to Hadoop to handle a particular data, and then wait for it to come back to me" -- this is using an asynchronous protocol to implement a synchronous interaction.

Is there any need for your program to wait for the reply? If there is couldn't you have another process handle the replies?

There are several asynchronous patterns you could use depending on your requirements:-

  • "Fire and Forget" simply put the request in a queue and trust the receiving process to apply the changes.
  • "Send and Poll" - loop around put all your requests on the queue, process all the replies.
  • "sender and receiver processes" -- have different processes, programs threads or whatever, one or more to send requests, one or more to handle replies.
James Anderson
  • 18,049
  • 1
  • 42
  • 72
  • What do you mean by another process? You mean another thread in java? Or you mean an intermediary? Not really sure what you mean? As mentioned before, this area is not too familiar for me so I'm open to any suggestions, but please be specific :) – Jason Madux Feb 23 '12 at 03:34
1

You might want to consider using something like the Mule ESB: http://www.mulesoft.com/mule-esb-open-source-esb

It's open source, pretty lightweight, can host web services and supports full asynchronous processing of incoming events.

You could start with just creating a lightwieght Mule web service that fires off requests to Hadoop (perhaps via JMS, but you could also just call Hadoop directly). But you'd have the flexibility to add lots more services and capabilities later.

mikera
  • 20,617
  • 5
  • 75
  • 80
  • Interesting. So the concept is that the Mule becomes the web service, and within Mule I configure it to asynchronously call Hadoop, once result is returned send it back to the requestor? Or would Mule be the intermediate layer between some web J2EE framework and Hadoop? – Jason Madux Feb 23 '12 at 04:56
  • You can use Mule as a sort of lightweight alternative to a big J2EE framework, more focused on handling messages and events. So you probably wouldn't need anything other than Mule and Hadoop. Mule will handle the web service endpoints for you, take care of calling Hadoop for you and then returning the required result. If you're intereseted in the Mule option, it's probably worth coding up a couple of the tutorials to get a feel for how it works. – mikera Feb 23 '12 at 08:16