8

The application we have in mind is a relay server. The aim is to accept large numbers of incoming socket connections (at least thousands) that will stay open for lengthy periods (hours or maybe days). They will exchange modest quantities of data and they need reasonably low latency.

The technical design is straightfoward and we have a test implementation. Our preference is to use Windows hosting and .NET, because that's the technology we know. However, this kind of usage is well outside what we are familiar with.

The question is whether there are specific limits or constraints to be aware of that are inherent in or common to software that does this, and that we should allow for in our design and/or test for before a roll-out.

I found this question (Handling large amounts of sockets) and this link (http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-1), which tend to suggest that our solution should work.

--

Commenters have suggested opening and closing ports, or using some kind of protocol, without suggesting how. The problem is that at any time a message might be relayed to an individual known destination with the expectation that it be received quickly (say 1 second at most, preferably sooner). The destination is (in general) behind a firewall with unknown properties, so cannot (in general) run a server or accept incoming connections. It can only make outgoing connections. We think it needs a persistent outgoing connection in order to receive packets at any time without notice. Alternative suggestions would be of interest, although strictly off topic for this question.

Other commenters have suggested there are OS limits, but not specified any. Assuming that this is some version of Windows server and some (perhaps large) amount of memory, are the limits likely to be a problem? Roughly what will they be? Is Windows a bad idea?

zelusp
  • 103
  • 4
david.pfx
  • 8,105
  • 2
  • 21
  • 44
  • 1
    List of things to check: OS limits on number of open sockets total, OS limits on number of open sockets in a given process, whether there are any limitations on the routines that you'll use in .NET to read the sockets. – Michael Kohne Apr 03 '14 at 13:40
  • 1
    Thousands shouldnt be a problem, thats essentially what a lot of node.js apps and game servers do. – GrandmasterB Apr 03 '14 at 16:20
  • Windows will work fine for what you are trying to do; however, see my answer below if you are interested in firewall constraints. – miniscule Jul 01 '15 at 23:22
  • My answer and ThinkingMedia answer below are realistic for high-demand, persistent connections state-of-the-art. What ever happened with this project. Please give more details if this was ever a real-life project with a real budget and not a pipe dream. Consultants tend to ask open-ended questions like this without any experience. If you are still looking to move forward, we need to know what your constraints are in terms of dollars and dev time. A relay server scenario could be anything, and it sounds like you do not know your message size to begin with yet want an optimized system answer. – miniscule Jul 02 '15 at 07:00
  • @miniscule: Project is live and running fine with 50 or so connections, hosted on Azure. Call it a pilot, while we decide what to do next. Thanks for all the helpful comments. – david.pfx Jul 02 '15 at 11:20

5 Answers5

6

I think you're looking for a protocol: something that can handle errors, retransmissions, etc. For example, what happens if one of the socket is dropped because the underlying network had a problem? Or if your messages are received twice because of a faulty switch along the line? Or if they arrive in the wrong order?

Given that you're planning for a large number of connections, you'll also have to consider the increase in probability that something will malfunction.

Also, what happens if you want to scale your architecture horizontally, i.e. adding more servers? You can't load-balance open sockets and seamlessly transfer them across nodes.

In the end I'd recommend using a more robust message-passing protocol, designed expecting failures. In simpler terms, consider each communication atomic, where the worst case is a new connection every time. More or less along the lines of the C10K problem.

If you still need more convincing, try to test your architecture with a mock up: see how it reacts to ten thousands client connecting at the same time (it's easier in a LAN). Then imagine adding network latency, errors, etc.

lorenzog
  • 419
  • 2
  • 8
  • Yes, we plan to impose a protocol on top of this structure, to handle errors and reconnecctions. Message passing doesn't look like a solution. Yes, the mock up is an excellent suggestion. – david.pfx Apr 03 '14 at 22:01
  • And C10K is an excellent link. Thanks for that one. – david.pfx Apr 03 '14 at 22:01
4

computers have a hard limit of how many connections can be open at the time (decided by the OS). Programs see a subset of that.

each open socket requires some resources and a network-timeout heartbeat so disconnections can be detected. Having a lot of sockets sending those heartbeats will start to fill the bandwidth.

my suggestion is to close the connections as needed and just accept that you will need to reopen them.

ratchet freak
  • 25,706
  • 2
  • 62
  • 97
  • Can you put any numbers on those limits? Closing and reopening is not an option for a message receiver. – david.pfx Apr 03 '14 at 13:25
  • 2
    @david.pfx - Those limits are usually tuneable parameters. So there isn't one answer. You'll have to look at your proposed server and see what it can do. – Michael Kohne Apr 03 '14 at 13:38
  • @david:Each connection requires a port number. Port numbers range from 0...65535. Some number of those port numbers are reserved. If your computer can handle it then it should be able to use up nearly all of those unreserved ports. However, there may be some applications on your computer that need ports also. Anyways, bottom line is 65535 is the upper limit. However, unless data is being sent fairly constantly the best solution is to define your messaging protocol to close the connection when its transaction is done. Usually it is the sender who knows when a transaction is done. – Dunk Apr 03 '14 at 13:41
  • 3
    @dunk no each connection has a local port, remote IP and remote port, meaning that there are a theoretical `65535*4294967296*65535` connections a single IPv4 address can handle. Incoming connections to a single known local port will max out at `65535*4294967296` – ratchet freak Apr 03 '14 at 13:45
  • 1
    Besides using less system resources there are other benefits to opening and closing connections instead of just leaving them open. #1 - You get the built in error handling for free, because you have to handle the same type of processing even in the "keep it open" methodology. #2 - The sender learns much sooner that it lost its connection. – Dunk Apr 03 '14 at 13:46
  • @ratchetfreak: Obvious when you know how. +1 for that. – david.pfx Apr 03 '14 at 22:02
  • @Dunk "Each connection requires a port number" is a common misconception, but a misconception nevertheless. All sockets accepted from the same listening socket share the same local port as the listening socket. 65535 is therefore not the 'upper limit'. Closing connections is not preferable to keeping them open: that's why we have HTTP keep-alive. – user207421 Apr 10 '14 at 03:32
4

I worked on a relay server for stock market data in C# on a Windows Server. There was no way I could get thousands of simultaneous connections relayed by one machine. The specs for the relay were very simple 1 connection to stock market data provider, and unlimited outbound connects to SilverLight clients.

There are two basic approaches that I investigated.

  • Use a thread pool, each client gets a socket and worker thread.
  • Use a worker thread, worker thread pushes data by iterating over all open sockets.

Neither approach could exceed the performance limits of the CPU, and each approach had serious limitations and restrictions.

Using a thread pool.

Windows sucks at multi-thread handling. Once I hit around 250 threads things just started to go down hill. It isn't a memory or system resource problem. It's a quantity problem. While Windows has no problem managing 250 threads. It's another story asking Windows to keep those 250 threads busy relaying data. As performance lags a data backlog starts to happen.

Using a worker thread.

You can't use a worker thread to iterate sockets if those sockets are blocking. Each time the thread hits a socket that has to timeout all other sockets are left waiting. If you switch to asycn socket operations, then a huge backlog of callbacks are generated to quickly and everything breaks.

For me the results were.

100 clients everything is stable. 250 clients everything is working but limit reached. 1000 clients never achieved.

Conclusion.

C# on Windows is not the right tool for a socket relay server. Not for client connections ranging in the thousands.

The only alternative is to not use HTTP sockets, and switch to a broadcasting protocol like UDP or TCP. For me this was not an option as no data was allowed to be dropped. Most broadcasting protocols assume packet loss is acceptable.

Finally.

If you are able to create a C# relay that can handle thousands of clients. Please come back and let me know how you did it.

Reactgular
  • 13,040
  • 4
  • 48
  • 81
  • 6
    Windows itself doesn't have an issue with async callbacks. It won't create a backlog if your responses are quick enough. I'd blame your C# code. In native code, it would be quite efficient: copy async input to async output, big array to manage state, done. As for "broadcasting protocol ... TCP ... dropped", you'd better read up on protocols. TCP doesn't drop packets, HTTP on top of TCP doesn't, HTTP on top of UDP does. (which is why HTTP by defult runs on TCP port 80) – MSalters Apr 04 '14 at 16:12
  • Very helpful. Thank you. Likewise @MSalters. Sooner or later we'll need to build one and run it to destruction. Good to know some things to watch out for. – david.pfx Apr 05 '14 at 06:32
  • FYI the design we have uses async sockets and C# with a thread pool. CPU and memory usage on small scale tests are tiny. – david.pfx Apr 07 '14 at 11:08
  • @david.pfx good luck with it. Let us know how it turns out and what limits you were able to reach. – Reactgular Apr 07 '14 at 17:21
  • This is closest to an actual answer to the question asked. Thanks -- it will be helpful. – david.pfx Jul 02 '15 at 11:21
  • A thread pool with 250 threads <- that right there is your problem. Use as many threads as you have (hyperthreaded) cores and a proper async IO library and you can relay thousands of user. See this example of websocket servers, which is very similar the the issue at hand. C++ implementation can handle upwards of 30.000 users. https://hashrocket.com/blog/posts/websocket-shootout – mmlac Mar 17 '17 at 22:11
1

Experience suggests that there is a better and probably fairly different solution for your problem.

However, it is possible to create and maintain substantial numbers of socket connections for a long period of time with a less than 100% reliability.

A robust design would be done very differently.

The problem is what happens when your relay server crashes. If you only have one server, then your service is 100% lost. If you have multiple servers, then the clients can reconnect and be connected to a different server - but any messages being relayed back to them during the disconnect and reconnect will have been lost. Depending on your product, this may be important.

If your 'relay' was actually implemented as a distributed message queue service, then it provides a tool set that solves most of these issues for you and provides a mechanism for implementing the level of robustness and availability that suits your application.

This is not a recommendation, but RabbitMQ is one example of a product that may perform the relay function for you. There are others, and you will need to review a selection to decide which is most appropriate for your product.

Michael Shaw
  • 9,915
  • 1
  • 23
  • 36
  • See edits. More than happy to consider any alternative approaches. We just can't think of one. – david.pfx Apr 03 '14 at 21:59
  • answer edited to give hints towards an alternative approach – Michael Shaw Apr 04 '14 at 08:41
  • Thanks for the link, but I still don't see how message queuing helps. I'm not planning to write a client (they already exist), and the protocol already exists and is restartable. This is just about how to implement a relay server. I checked out RabbitMQ and I cannot how I could use it. – david.pfx Apr 05 '14 at 07:00
  • Imagine client connects and sends an action that needs relaying. During the relay connection, the client drops the connection, and then reconnects and posts sends the same action again. This creates a duplicate relay. Assuming both relay actions succeed, does this cause a problem? – Michael Shaw Apr 07 '14 at 08:08
  • The protocol is designed on ACID principles. The data being sent is idempotent. A client can repeat a message exchange ad lib. No, this should not cause a problem. – david.pfx Apr 07 '14 at 11:04
1

Having provided firewall and VPN support for a couple of years now, I can tell you with confidence that applications which keep ports open for an extended period of time are not stable from both the local server perspective and the network point of entry perspective. It can also be a security risk (if that is a consideration in your case).

Leaving a persistent connection open on a server port can potentially cause an issue on the firewall as the NAT table accumulates entries, for example.

I personally recommend keeping your connected client numbers in the 200 to 250 range per server for persistent connections via TCP/IP where the data being sent and received requires minimal latency.

If you have the hardware to move to ATM (Asynchronous Transfer Mode), you can reconfigure your clients to listen only to their specific address, which will take a significant burden off of your servers (at the expense of the clients). ATM should get you into the 300 to 350 active clients per server capacity.

To clarify, the solution to this issue is to put your server in a DMZ on the firewall to enhance access and use ATM connectivity with as many clients as possible. Otherwise you are at the mercy of the remote network firewall and other intermediate intelligent routers.

miniscule
  • 111
  • 5