4

I am looking to perform a large overhaul on a complex simulation system that simulates several instances of several vehicle models in a classroom training environment. For example, 24 students may be running simulations on three different vehicles for maintenance and operation. Instructors will be required to have tablets that can connect to any of the 24 active simulations to control the training scenario.

The primary system will be running on Linux but there are no other requirements for OS and machine specs can be built as needed. Performance of each simulated pass must be able to consistently run at ~10ms intervals with a +-2ms tolerance.

A primary goal is to make this system very modular so that it can be extended and reused by other training facilities with unique vehicles and needs.

My thought was to use a layered architecture (system, business, UI). The definition of each vehicle model can be stored in a database and therefore edited independently by a superuser (modularity/extensibility of the vehicles). Each layer would likely have to read this database to dynamically allocate the resources that particular layer will require.

Originally I planned to use shared memory for the system layer, setting up permissions and authentication for any business layer to attempt to login. The primary simulation business logic would then continually update the vehicle details according to the active data. The instructor interface would have a business layer server that connects to all 24 clients and also logs into the system layer to modify simulation parameters. All inputs from the student and visual outputs would each have a business layer that can login to the shared memory system layer as well. All of these acting as separate applications so that they can be removed/added/extended as needed.

The problem then came when I realized that classes do not work well with shared memory. I would need to serialize every get/set of the shared memory into a flat memory structure. Having not worked with this architecture before, I am unsure if this plan will create a large performance hit. Technically I can dedicate some cores to the primary business layer logic that performs the simulation.

Would using shared memory with a suite of applications be an appropriate way to resolve this system? Would another cross-process communication type such as pipes be more advisable than shared memory? Would it be better to maintain the system and business logic into a single application and simply use mutexes and cross-threading to ensure performance? Am I going about this all wrong?

Gren Meera
  • 69
  • 5
  • 2
    Such an architecture should be developed using a huge budget with experienced staff, not discussed on stackoverflow o_O – ooxi Feb 07 '18 at 17:23
  • 3
    @ooxi Agreed, but life is life and I don't get to hand off the task because I want to. Also, I'm often surprised how "experienced staff" and "experts" in my industry are often people just as clueless as I am. I'm also amazed at how often I know better than people who get paid double my salary. :) The grass is always greener I suppose. – Gren Meera Feb 07 '18 at 17:38
  • 1
    You said the simulated passes need to finish within ~10ms. Are these passes simulating physics (moving cars, running motors)? Why is there a 10ms realtime requirement? Sometimes a simulation will have a critical part that needs to update like clockwork and other parts that can be a little slower. If this is the case, the communications to the instructor's tablet may be able to take a bit longer (work split across frames or running in a thread). – Kyle A Feb 07 '18 at 19:58
  • @KyleA Allow me to clarify. The performance requirement is for the simulation business logic only and ONLY for those entities that require rapid response. We will use a dedicated core scheduler to process some entities much slower than others, but our requirements for primary physics characteristics, such as vehicle position/orientation must meet those specifications. Other data, such as the simulated ARINC devices, need only update as often as the fastest ARINC message (50 ms) for example. UI for the instructor can be much slower. – Gren Meera Feb 07 '18 at 20:37
  • 1
    @KyleA Refocusing on the question at hand, the system layer using shared memory for massive amounts of data serialization is my real concern. I am uncertain if this is an appropriate way to handle this when some of my data will indeed be checked every 10 ms. Side note: Our current model before this refactor can already meet the timing requirements, but it is a flat singular multi-threaded process. We were hoping to improve how extensible we are by layering our logic and separating our processes. – Gren Meera Feb 07 '18 at 20:41

1 Answers1

5

I've worked on exactly 1 simulation application, with looser requirements (i.e. we had to have everything update 1/s). Feel free to take this with a large grain of salt, but it is someplace to start. I think you are going to have difficulty realizing your timing goals with stock operating systems. I can only highlight some lessons learned:

  • Sometimes the requirements are over-stated
    • Example: you have 10ms intervals with activity done in +/- 2ms, it might be good enough to have that overall rate but the real version is more like 100ms with activity done in +/- 20ms. Same rate, but much more reasonable to design for.
    • Example:your external clients probably don't have the same near real time requirements as your application
  • You need to be able to stop processing early, either stop pulling from the queue if you are nearing the end of your window or actually cancel existing work
  • Depending on what is being simulated, the Actor Model or some similar approach might be the best bet for your overall design

In the similation I was working on, we were modeling communications with signal devices and towers and the effects of devices moving in near real time. Considering the number of devices we needed to simulate we couldn't devote a full thread to each actor (device and tower), but we could have a finite number of threads work against the corpus to process the messages up to the current simulation time-stamp.

The only thing I would recommend is to start with a stable foundation and stress test your simulation. It's those stress tests that will force you to make the hard decisions about where you can trade absolute fidelity with something that gives you the same overall effect. I guarantee you that no human user will notice the difference between 10ms and 100ms.

Any simulation is a trade-off between fidelity with your working model, and something that approximates it.

Also, don't make the decisions on shared memory vs. pipes or sockets initially. Focus on the core behavior.

Something I'm learning with some of my newer endeavors is that sometimes a shared nothing approach provides the best scalability overall. To take an idea from the microservices approach, you would have an architecture where one simulation service provides the entire environment for the high precision timing events. The trick is to partition the overall environment safely into the smaller environments. That can be done per vehicle or per air-space.

We have the following knowns:

  • Your critical code meets the requirements
  • Not everything has the strict timing requirements (based on your comments above and below)
  • Socket communications are slow, even on the same machine
  • Shared memory can be tricky and still has serialization costs, but limits you to one machine

Essentially, to have several simulation nodes work together you need to package and transmit state to affected nodes at regular intervals. You need that transmission to be fast, and dense. Options are:

  • named pipes, fairly high speed, high reliability
  • UDP packets, fairly high speed, packets can be split by routers with a maximum safe size of about 1134 bytes of payload.
  • TCP packets, slower, but much higher reliability

Communications can be done asynchronously invoking events in the receiving node. In my case we opted for UDP packets since the entire system was intended to be run in one subnet. We also used a binary transmission format like Google protocol buffers to maximize information per byte.

In my project, most simulations could be run on one machine, but the sensor visualization software was typically run remotely.

In this case it was a trade-off of how frequently we updated the external system and the quantity of data. Each node had a pair of sockets open, one for sending and one for receiving.

In most cases we could mathematically project our movement within enough accuracy that course corrections we received would go unnoticed and the simulation was still valid.

But again, an approach like this has to be tested against the known criteria for your system.

Berin Loritsch
  • 45,784
  • 7
  • 87
  • 160
  • Just as a comment on timing, 100ms or less is almost always good enough response time for a user interface (except with VR, AR, and some video games). Some people might be able to notice the lag, but they will almost never care on a standard user interface. – Kyle A Feb 07 '18 at 19:53
  • Actually the timing goals are already currently being met. I did say that this was a refactor of an existing code base. Our clients DO require these timing requirements because of FAA certification limits between control input, processing, motion-base response, and IGs. – Gren Meera Feb 07 '18 at 20:30
  • @Berin Loritsch "Also, don't make the decisions on shared memory vs. pipes or sockets initially. Focus on the core behavior." This is the crux of my question I suppose. Is the core behavior of a layered architecture a smart way to go when dealing with both performance and modularity? Sockets and pipes can be fairly slow simply due to synchronicity, and shared memory seems to require serialization for OO classes. What is the trade-off of these performance hits when compared to the benefits that a layered architectures give for maintainability, scalability, and testability? – Gren Meera Feb 07 '18 at 20:31
  • I could do a "try and see" method, but these decisions have impact that I was hoping other's experience may help with early on. – Gren Meera Feb 07 '18 at 20:31
  • 1
    Serialization can be done asynchronously so it doesn't affect the timing of your core logic too much--but I really don't know enough about the timing requirements of your communications between nodes. Shared memory can be quite finicky to work with, I don't have a lot of information about that. Like I said I've worked on only one simulation system in my career so far. What was the main reason for the redesign other than modularity? What was the original design _not_ doing that the new design does? I.e. are there scalability requirements that require multiple processing nodes? – Berin Loritsch Feb 07 '18 at 23:45
  • @BerinLoritsch The original design could only handle a single simulation at a time, with one vehicle, one instructor, and one student. We lost a proposal with a government agency due to not having the flexibility right out of the gate, so we're stretching our design goals to include multiple vehicles and multiple training sessions. Also, each project that we land is often a unique vehicle. It is cumbersome to refactor the code base on a per project basis continually. We are hoping to modularize and script the pieces for faster turn around. – Gren Meera Feb 08 '18 at 14:36
  • @BerinLoritsch (cont) Usually the differences between vehicles are about modular pieces. Two pieces of equipment instead of one, or the engine temperature is output to a new gauge. If we can modularize the vehicle definitions into a database and spawn the details dynamically we will greatly enhance our development time of new vehicles as our code reuse won't be a refactor of all connected dependencies from our former design. – Gren Meera Feb 08 '18 at 14:39