0

I have a group of 100 clients around the globe, that need to communicate between then self, fast. Each message that each of the clients sends needs to be broadcasted to everyone. The messages will be between 1 to 16 MB. Each client is expected to send one message every half minute.

I was thinking of a tree structure and then using p2p connections, but then it slows down the system.

Another idea that I had is using email protocol(this is where the cheap part come into the picture). There are many email providers that already have a good distribution of servers and can help achieving this task. So it might be cheaper using those solutions.

What will be a good solution to my problem? A cheap fast and reliable way to communicate with 100 clients.

I guess that my biggest concern is that I don't want to build a system and later find out that it takes minutes to deliver a message. Performance is my most important parameter and I need a fest solution.

  • scale is not a consideration at the moment
Ilya Gazman
  • 285
  • 5
  • 14
  • Never ask for _"the best way"_ at any SE site, if you don't want to get subjective and opinion based answers. – πάντα ῥεῖ Mar 02 '18 at 19:14
  • @πάνταῥεῖ tnx, I edited. – Ilya Gazman Mar 02 '18 at 19:19
  • 2
    You already have a scaling problem. Using a worst case estimate, these 32MB per minute add up to 45 GB of traffic each client is _sending_ during a day (in addition to receiving all the other messages). Times 100 clients, your network has a monthly traffic of 131TB. For comparison, on Amazon AWS these 130TB outgoing traffic per month would set you back roughly $10k. Unless I've miscalculated and am off by an order of magnitude. – amon Mar 02 '18 at 19:28
  • @amon your calculations are correct, this is why the email idea sounds more attractive to me than Amazon AWS, it's cheaper. But it still not a scale problem as I don't anticipate a change in the near future. – Ilya Gazman Mar 02 '18 at 19:33
  • 1
    "people" or "software clients" ?? No way would humans be able to keep up with such rate of messages of such size. – Euphoric Mar 02 '18 at 19:34
  • @Euphoric tnx, edited – Ilya Gazman Mar 02 '18 at 19:35
  • Couldn't the data be compressed? Either within message itself, or temporarily, between each message? Eg. sending only diff from previous message/ – Euphoric Mar 02 '18 at 19:39
  • 1
    What is the nature of the data you are transmitting? Have you given any thought to finding ways of thinning this data? – Robert Harvey Mar 02 '18 at 19:39
  • @robert yeah, unfortunately it's not an option for me. One way or another I need a solution under those conditions – Ilya Gazman Mar 02 '18 at 20:02
  • Why is it not an option? So far, you are trying to solve impossible problem. You either find clever way to send much less data or you pay for bandwidth necessary to send all the data. I'm skeptical there is third option. – Euphoric Mar 02 '18 at 20:03
  • How often are these messages the maximum size of 16MB? One percent of the time? – Robert Harvey Mar 02 '18 at 20:04
  • 1
    As for your email idea. I'm sure all email providers that have bandwidth necessary for you have really good ways to detect this kind of "malevolent" behavior. So they would immediately flag you are spammer and either stop you outright or demand explanation. And they won't like your explanation of using their free service as data distribution platform. – Euphoric Mar 02 '18 at 20:11

1 Answers1

3

Its the classic triangle problem. you want:

cheap fast and reliable

But you have to sacrifice at least one.

reliability: You could sneakily use other peoples bandwidth by leveraging free tier services such as email distribution lists or dropbox/google drive/amazon free accounts

But you would soon hit the various controls to stop people abusing the services and suffer an outage until you had developed a new trick.

speed: You can use the clients connection via a p2p network, but unless they all have lots of bandwidth, speed will suffer.

cost: You can pay for the bandwidth on a centralised service and have both speed and reliability. But it will be expensive

Ewan
  • 70,664
  • 5
  • 76
  • 161