Currently we are working on some storage problems for log data from various servers and communication message logs (HTTP(S), XMPP). There will be many write operations and for read operations we will use search queries with filters.
Should we stick with the classic relation database schema solution or concentrate on Cassandra with ColumnFamily based indexes?
If all you're going to be doing is searching the records once they're stored, then I don't think there's going to be much difference. If you need to re-process them (or a large chunk thereof) they you might see some advantage from the NoSQL approach. If you know what fields you're going to be searching on, I'd recommend compressing the data into a BLOB and storing that along with the searchable fields in a relational table, in order to cut down on your storage requirements (I don't know if that's possible with Cassandra).
You can stay relational and be NoSQL at the same time, if you can partition data(perhaps by time) with PlayORM/Cassandra so that you can do "scalable JQL" like:
@NoSqlQuery(name="findJoinOnNullPartition", query="PARTITIONS t(:partId) select t FROM TABLE as t INNER JOIN t.security as s where s.securityType = :type and t.numShares = :shares")
It also supports of course the full ManyToOne, OneToMany, etc, but they work slightly different from Hibernate as this is NoSQL after all.
Hadoop, hbase, hive/pig is generally used for these kind of log analysis or to handle huge data produces logs & messages. This llnk can give give u more details http://www.cloudera.com/blog/2009/06/analyzing-apache-logs-with-pig/. But this has huge learning curve.
Hadoop plays nice with NoSQL stores like Cassandra. I'm doing development on a side project that uses Cassandra and Hadoop. It was a bit of a pain to get set up, as there are some extra jars needed in the Hadoop installation to support Cassandra and not much documentation about how to do it. The Thrift API is a little awkward but still manageable once you're writing map-reduce code.
I think the decision comes down to the queries you need to run and the volume of processed data you're looking to store. Complex queries on small volumes of data push more toward MySQL. Simple queries or larger volumes of data push more toward Cassandra or HBase.
If you are only going to store logs, you've strayed into one of the few places were relational databases do not make much sense.
SQL is surely not going to be the ideal medium to perform your work- no joins, not many different tables, etc. Transactions and UPDATEs probably won't be necessary.
I would choose something that let me run map-reduce jobs easily- it might well be the case that a single-threaded algorithm be all you need, but if it ever becomes too slow for your purposes, being able to throw cores at the problem will come in handy.
If you are going to be searching a lot (i.e. perform calculations on small subsets of data), indexing is going to be another defining factor- having a store which supports indexing which handles your searches well will save plenty of time.
On the other hand, depending on what you do, it might make sense to "summarize" your logs using a non-relational data store, but put in the massaged data in a RDBMS. If there's structured/relational data hidden in your logs, the ability to perform ad-hoc queries using SQL is invaluable- aggregates, window functions, etc. can be made to run pretty quickly in a decent RDBMS, perhaps even more efficiently than with map-reduce and highly-parallel algorithms; definitely, if you know your SQL, implementation is usually much quicker.
For instance, say the logs you are really interested all look like:
[timestamp] add student xxxxx
[timestamp] create class yyyy at [date-time] with professor zzzz
[timestamp] student xxxx books class yyyy
[timestamp] student xxxx cancels class yyyy
then dumping them into student, class, student_booking and performing aggregates on them makes plenty of sense.
(of course, I would contend that you are not parsing logs then...)
