4

Scenario

We are building a UI that allows users to query our data in bulk. The return format is CSV and there is a decent amount of processing that needs to happen, so this processing processing is done in parallel and then sets of rows are streamed back to the UI.

Current Implementation

An AJAX POST request is sent to the server, it waits for all of the data to come in and then uses createObjectURL to trigger a download. The issue is that it can take several minutes for the request to complete and the user sees a spinning indicator indicating that their request is in progress.

Problem

It's a bad experience that the user cannot see any other progress and has no idea whether their request is actually processing or how long it will take.

Goal

To have the browser download the file as it's being streamed back to the client.

Obstacle

It does not seem possible to enable this behavior in JavaScript. The closest that I found was a script StreamSaver.js but using an external MITM is not acceptable and I don't understand the code well enough to determine whether this is a viable (or even good approach).

Other Ideas

  • Use a GET request or a form. The issue here is two-fold: (1) The request data can be nested and of variable length. It doesn't seem right to have to be parsing a request string on the backend. (2) It seems like this causes a page reload, which kills the state of the React application.

  • Instead of downloading the file continuously on the client, we can add a progress bar instead. To be this, we could add a header to the response indicating how many chunks are expected and then increment when each chunk is received. The downside of this approach is that the user cannot cancel the request and they cannot recover partial results.

Help

It feels like I'm missing a much better solution to this issue. Please feel free to challenge any of the assumptions above. My main goal is to get a clean solution that provides a good user experience.

Notes

The backend is a C# web application, React is being used on the front-end, and it's possible to pipe through a PHP layer if somehow that would help.

Jared Goguen
  • 200
  • 1
  • 1
  • 8
  • Use Skip() and Take() in Linq to section off the data into smaller pieces, and send those pieces in individual requests. Alternatively, just provide a file download; websites do this all the time, for very large files. – Robert Harvey Mar 30 '19 at 01:23
  • @RobertHarvey The data is being sent back to the client in chunks -- it's the client download portion that's proving troublesome. Is there a way to provide a file download that can be done in unison with generating the file? We'd prefer not to have to maintain a directory with old downloads, but this can be done if all of the other needs are met. – Jared Goguen Mar 30 '19 at 01:31
  • I really think that's probably the easiest and best solution. https://stackoverflow.com/q/4668906 – Robert Harvey Mar 30 '19 at 01:32
  • I'm not sure if that solves the client issue, I'm using `PushStreamContent` to stream the file back to the client, but there is no way to download a stream from an AJAX request in JavaScript. – Jared Goguen Mar 30 '19 at 01:38
  • Well, if the problem is that the user gets a spinning indicator, why don't you load the data on a background thread and let the user do something else while it downloads? – Robert Harvey Mar 30 '19 at 01:57
  • That's the way it is, but there isn't anything else to do and it could take a long time -- the goal is to have them download the file as its being sent across the network. – Jared Goguen Mar 30 '19 at 02:11
  • And that's not what is happening already? – Robert Harvey Mar 30 '19 at 02:12
  • Are you asking if there's a way to give the user partial results, so that they can start analyzing the data while the rest of the download completes? – Robert Harvey Mar 30 '19 at 02:13
  • No, it seems like there isn't a way to download a stream like this with client-side Javascript short of navigating to a different page – Jared Goguen Mar 30 '19 at 02:14
  • That would be an acceptable solution. The stream is currently being accumulated in the client and then the download triggers when it's complete, but there isn't a logical reason why this has to be in memory the whole time – Jared Goguen Mar 30 '19 at 02:16
  • Perform a greater number of smaller downloads. – Robert Harvey Mar 30 '19 at 02:17
  • Note that, if you really want a data stream, you could always try opening a socket. – Robert Harvey Mar 30 '19 at 02:18
  • @JaredGoguen I have manage to download a stream from ajax (fetch) by using streamsaver [example](https://github.com/jimmywarting/StreamSaver.js/blob/master/examples/fetch.html). If you don't like the external MITM, you could always host the hole lib on your own, and assign `streamSaver.mitm = url` also know that you need to have https on your site for doing so. – Endless Jun 27 '19 at 08:05

1 Answers1

2

I think what you are looking for is ReadableStream

https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream/getReader

you can generate one from a fetch request and stream it as required

https://developer.mozilla.org/en-US/docs/Web/API/Streams_API/Using_readable_streams

Ewan
  • 70,664
  • 5
  • 76
  • 161
  • I'm getting a ReadableStream from the AJAX response but there does not appear to be a way to download it while it's still running. (Also, trying to use it to fuel a custom progress bar with download-at-the-end, but I'm having trouble transforming one chunk back into a string.) – Jared Goguen Mar 30 '19 at 16:57
  • the example seems to be doing it, you have to output the reads to another stream that you can listen to – Ewan Mar 30 '19 at 20:00