What stops an assembly program from crashing the operating system?

Question

First of all, I am a beginner, so if this question sounds silly, please point out the incorrect assumptions.

From what I understand, the job of an operating system is to manage hardware and the software that runs on the OS. Also, from what I understand, assembly programs allow one to control hardware almost directly. In an assembly program, one can read and write data into registers, and read and write data into RAM.

Given this freedom to mess with registers and RAM, would it not be possible for assembly programs to affect the operating system? Suppose an OS is using register A to store crucial information, and suppose I run an assembled program on that OS. If the program successfully writes junk into register A, the OS will surely be affected.

Questions:

1. Is it possible to mess with register A in the manner described above?

2. If not, what stops assembly programs from modifying the registers used by the OS?

Answer 1

In the end, all programs are machine code, regardless of whether the source language was assembler or a high-level language.

The important thing is that there are hardware mechanisms that limit what a given process can do, including "messing with" registers that could affect other programs or the operating system itself.

This started with a simple distinction between "user" and "supervisor" modes of operation, and has since evolved into a security architecture with multiple "rings" of privilege.

---

Here's a very generic example to make it a little more concrete:

• In "user mode", a process cannot access memory that hasn't been assigned to its process ID. Memory assigned to other processes and the operating system itself is blocked. This includes the values of registers used by those other processes. This is enforced by the MMU hardware.

• Therefore, in "user mode", a process cannot access the MMU control registers.

• And obviously, in "user mode", a process cannot change the mode to "supervisor mode" except through a very well-defined mechanism that involves calling an operating system function.

The operating system gets control if the process tries to break any of these rules. Once that happens, the OS can simply halt the offending process, never executing any more of its instructions.

Answer 2

Many multitasking operating systems use a data structure called a Process Control Block (PCB) to take care of the register overwrite problem. When you run your code, the OS creates a new process to keep track of it. The PCB contains information about your process and space allocated to hold register contents. Let's say process A is currently running on the processor and your code is in process B. What happens when you run your code goes something lke this:

1. Process A's state data (register contents, program counter, etc.) are copied into its PCB.

2. Process B's state data are copied from its PCB into the CPU registers

3. Process B runs on the processor until it finishes or is preempted

4. Process B's state data are copied back into its PCB

5. Process A's state data are copied back into the CPU and it continues running

If the operating system is running as process A, then you can see how saving its registers before your program runs then copying them back into the CPU once your program ends prevents user programs from messing with what's going in other processes.

To avoid user processes writing over OS data in memory, most platforms use memory segmentation. Basically, using virtual memory, the address space a process sees can be mapped to any arbitrary range of physical addresses. As long as the processes' physical memory spaces don't overlap, it's impossible for one process to overwrite another's data.

Of course, different OSes do things differently, so this won't apply in every case. Also, on most platforms, OS processes run in a different mode from user processes and there are some intricacies there.

Answer 3

Depends on what platform you're talking about.

• On a more basic processor, the processor just executes whatever instructions the program tells it to execute.

• On a more sophisticated processor, there's (at least) two modes. In one mode, the processor does whatever the program tells it to do. In the other mode, the processor itself refuses to execute certain instructions.

What stops a program crashing the whole system? With the first type of processor, the answer is "nothing". With a basic processor, a single rogue program can indeed crash the entire system. All the early 8-bit home computers, and many of the 16-bit ones, fall into this category.

On a modern PC, the processor has "protection" hardware. Basically the OS runs in a special mode that lets it do anything, whereas normal programs run in a mode where the processor will only allow certain actions (based on what settings the OS has configured on the processor). Stuff like only accessing certain registers, or only accessing particular memory ranges.

Obviously, allowing a single rogue program to crash the entire system is bad. (There are also severe security implications in letting a rogue program access whatever data it wants.) To avoid this, you need two things:

1. A processor that actually has protection hardware (i.e., it can be configured to refuse to execute certain instructions).

2. An OS that actually uses these facilities to protect itself. (It's no good having a chip with protection circuitry if the OS never uses it!)

Just about any modern desktop OS you can name (Windows, Linux, Mac OS, BSD...) it a protected-mode OS running on a processor with protection hardware. If you're doing embedded development on some 8-bit microcontroller, that probably doesn't have any protection hardware. (Or any OS, for that matter...)

Answer 4

Q. What stops an assembly program from crashing the operating system?

A. Nothing.

However, a lot of very clever programmers have tried very hard over the years to make it more and more difficult. Unfortunately, for every clever programmer, there are many, many others who between them are more creative, more ambitious, and sometimes just luckier then the clever ones. Every single time a clever programmer says that nobody should, would or could do something, someone out there will find a way to do it. Microsoft Windows (as an example) has been around for almost 35 years and we still have BSoD (Blue Screens of Death), which are just instructions that crashed the operating system.

Lets start with a little terminology. Everything that runs on a computer does so in machine code. The bit that reads the keystrokes or the movement of the mouse pointer, the bit that changes the colour of a pixel on the display or reads a byte from a file and the bit that calculates whether your bullet hit the bad guy or the bit that decides if your credit card application will be accepted are all executed as a sequence of machine code instructions. Some jobs are so common and are done so often that it is makes sense to assemble the instructions required to do them and have everybody use that assembly. The bunch of these jobs that allow or help others to use the computer tend to be called the operating system but there is nothing inherently different between them and any other programs. They are all just sequences of machine code instructions. You will hear about protected modes and privilege rings but these are just more sequences of the same machine code telling the computer how to behave.

What makes operating systems more complicated (and therefore prone to crashing) is that they have to account for things that you don't normally have to think about. Take the simplest of jobs as an example. I want to write a message to the end of a file. In a high level language you would write something like:

  with open("myFile.txt", "w+") as f:
      # do some really clever things
      f.write("Goodbye cruel world!")

Lets ignore all the details about how the physical states are accessed and changed or how they are interpreted as bits and bytes or how those bytes are transferred to and from the memory and CPU, and trust that all that is handled by programs that the OS provides behind the scenes. Lets just think about the how you append to the end of a file. 1) Find out where the end of the file is, 2) write something at that position. What could possibly go wrong? Actually, quite a lot. Think about what else is happening on the computer while you are doing clever stuff. If anything else being done by anybody else (including the operating system itself) changes the file you are working on in any way, then this really simple job suddenly gets a whole lot more complicated. The file is longer, the file is shorter. The file is not there anymore. The disk is full, the disk would be full if all the bytes are written, the disk is locked, the disk is faulty... There are now so many different conditions and exceptions that the operating system has to deal with that it is amazing it ever gets anything done at all.