2

As part of my workflow, I need to do all these steps in one transaction - I need to ftp files to 2 different FTP servers. - There is also a spreadsheet that gets generated which needs to be FTP'ed. Can this be streamed, instead of downloading and then pushed to FTP server.

I am using Ruby Net::SFTP and Net::FTP libraries to send the files.

I would like it to be robust. I am not sure if I need to do anything else or of this is good enough.

Just to be clear, this is already working in production, I am not stuck, just looking to exchange design/architecture ideas on how to improve this.

aarti
  • 175
  • 10
  • why didn't you ask at Stack Overflow? http://meta.stackexchange.com/a/129632/165773 – gnat Mar 13 '14 at 18:05
  • But it says, if it is subjective, design related to ask in Programmers. What did I do wrong? – aarti Mar 13 '14 at 18:34
  • "This site is all [about] getting answers. **It's not a discussion forum.**" – gnat Mar 13 '14 at 18:50
  • I was trying to get answers. I work alone on a lot of projects and it is hard to meet experience people who have solved these problems. – aarti Mar 13 '14 at 18:53
  • How is it not a programming design question? A service that has multiple points of failure, and how to make it robust. – aarti Mar 13 '14 at 19:01
  • "looking to exchange design/architecture ideas" is not a question. It's an invitation to discussion. "This site is... not a discussion forum." http://meta.programmers.stackexchange.com/questions/6483/why-was-my-question-closed-or-down-voted/6491#6491 – gnat Mar 13 '14 at 19:05
  • Then why does this question have 5 upvotes, this is a design question. http://programmers.stackexchange.com/questions/230311/layers-and-layers-of-soap – aarti Mar 13 '14 at 19:09
  • "does this approach kind of blur the lines and make my design less modular?" - this _is_ a question – gnat Mar 13 '14 at 19:10
  • 1
    This is very subjective, based on one person's opinion. Also two people answered this question, with design strategies that are helpful. – aarti Mar 13 '14 at 19:13
  • "How do I verify ftp is successful?" is not a valid question? – JeffO Mar 13 '14 at 19:24
  • A better approach to engage new people would be to suggest how to improve the question. – aarti Mar 13 '14 at 20:17

2 Answers2

3

Ahh, with FTP, the simple answer is - you don't.

What you can do is to retrieve the file you sent to the FTP server, and check it is the same as the original file. If they match, all worked as well as you'd hoped.

This gets tricky if you're not allowed to read from the FTP server (as some credit card acquirers do), security is set up such that you're allowed to write to some directories but not read from them. In these cases, the server tends to have a service that generates a summary report of your upload that you can retrieve from a read-only directory. In the case of acquirers that do not do this, you just have to cross your fingers and wait for them to complain usually on the following day.

gbjbaanb
  • 48,354
  • 6
  • 102
  • 172
1

It depends what you mean by "robust" e.g. considering scenarios like:

  • PUT fails on remote host
  • PUT succeeds but file is corrupted somehow
  • Host is temporarily unavailable
  • Host is permanently unavailable (e.g. wrong config, or host is decommissioned)

You can code around this (or buy software that provides guarantees around delivery), but if you're in control of the whole workflow, it may be better to have clients "pull" the files rather than having the publishers "push" them. The system I work on creates and sends a huge amount of files, and we have no idea if the remote clients even need some of them. A "pull" model at least tells you that a system somewhere is actively trying to fetch a given file.

You could, for example, built a service that accepts file requests, and builds them if they are not available, or else uses an existing / cached version, perhaps from the filesystem. A REST API would be a natural fit for this, e.g.

http://server.example.com/myservice/mydata/file12345.csv

...would build file12345.csv on-demand and cache it for subsequent requests (if you needed that).

Rory Hunter
  • 1,737
  • 9
  • 15