4

I work in Healthcare and we use SAS to Extract and Transform medical and pharmacy claims data for use in downstream reporting applications. For a given Report Request(usually 40 are running at a time) we have to extract from various DBMS's and transform the data. Once completed with this ET step, the data is over 100 gig in aggregated form(SAS Tables). This data is indexed and optimized for report consumption. We don't have to go back to the database after we have extracted the data.

To me using ReST doesn't make much sense in this situation as XML is not very efficient for large data volumes. But I thought I would ask to see what the rest of the Programmers' world thinks.

Charlie Bastnagel
  • 41
  • 2

2 Answers2

5

The answer discusses in general the aspect of sending large data over HTTP using REST. Make sure you also read the answer by Robert Jack Will which is more specific and suggests a way which makes it possible to avoid sending large data in the first place.

REST has nothing to do with XML: the format of the data, would it be JSON, XML, plain text or binary stream, is irrelevant. For instance, you can serve images (JPEG, PNG...) through REST, which doesn't mean that you need to enclose them in XML.

On the other hand, REST is based on HTTP, and HTTP is indeed suitable for large data transfers. Of course, you have to be careful when configuring both your service and your client to use streams when downloading such large data, instead of loading it in memory.

As for the actual format of the data:

  • If it is imperative to keep self-documenting structure, use JSON, XML or plain text, but make sure you compress the content at server side before sending it. The choice of the compression algorithm will depend on your platform and the interoperability requirement, given that proprietary formats such as RAR may often achieve better compression ratio (still, make the test with your data to see how well different algorithms perform).

    Note that it may not be that easy to find a serializer/deserializer for XML or JSON which will produce/accept a stream instead of a string. In this case, plain text may be your best choice.

  • If self-documenting structure is not required, binary format may be a better alternative in terms of the size. Still, make an actual comparison between plain text compressed data and binary data to see which one produces smaller result; you may be surprised.

Community
  • 1
Arseni Mourzenko
  • 134,780
  • 31
  • 343
  • 513
2

As for if and how to use HTTP for the data transfer, I completely agree with MainMa's answer.

However, independently of the size of the data, the process which you describe doesn't sound a like a typical application for Rest. One of the main ideas of Rest is having named resources like mycompany.com/claims/customer/{number}/claim/{number} which can be individually queried and updated.

Your problem, on the other hand, sounds more like batch processing: You have a large bunch of data, even including indexes, and since you want to run reports over it later, why don't you store that data into a database or analytics engine like Elastic Search? You could then have the reporting logic efficiently query that database / search-engine and provide the report results via Rest: mycompany.com/healthreport/{year}/{month}/claimsdata/{region} or somesuch.

Depending how many entities and relationships you have, marshalling all this data on its way from the ET engine to the report generator can be a lot of work and is only worth the trouble if you need to decouple those two stages by a large extend. (For example, when they run in independent IT organisations / companies.) Using separate data bases can already provide some amount of decoupling with less effort.

Robert Jack Will
  • 511
  • 3
  • 8