10

Might be kind of an odd question.

A guy writing a C++ compiler (or whatever non-VM language): Does he need to be able to read/write raw machine language? How does that work?

EDIT: I am specifically referring to compilers that compile to machine code, not to some other programming language.

Aviv Cohn
  • 21,190
  • 31
  • 118
  • 178
  • http://meta.programmers.stackexchange.com/questions/6483/why-was-my-question-closed-or-down-voted/6491#6491 – gnat Apr 19 '14 at 08:20
  • 1
    No. You don't even need to know it, you can just blindly, mindlessly copy your target ISA specification: http://dl.acm.org/citation.cfm?id=1706346 – SK-logic Apr 19 '14 at 09:00
  • 1
    Coffescript **compiles** to javascript. – Kartik Apr 19 '14 at 15:25
  • @Kartik Does the CoffeeScript compiler that compiles to Javascript, also include a Javascript compiler that compiles to whatever Javascript is compiled to? Or does it only compile to Javascript source code and nothing more? – Aviv Cohn Apr 20 '14 at 00:10
  • The coffeescript compiler simply converts cofeescript to javascript. Javascript is not compiled, it is handled by the browser. I wanted to say that you can write a compiler which compiles one language to another, you don't have to know machine language for that. Another example is the 'SPL' compiler which compiles Shakespeare plays to C++. http://shakespearelang.sourceforge.net – Kartik Apr 20 '14 at 06:01
  • @Kartik Interesting. Question: How common is it for compilers of "professional", serious general-purpose languages to compile to a different programming language, such as C, instead of compiling to machine code, and then letting a C compiler someone else wrote finish the job? – Aviv Cohn Apr 20 '14 at 11:00
  • @Prog I don't know about that because I am not a programmer. I mean, I know how to program in Python but I am not a programmer. – Kartik Apr 20 '14 at 14:38
  • compilers generally compile to assembler not machine code. and it doesnt really matter whether or not it is assembly language or machine code, you do need to be able to program that processor using its instruction set...if you want to generate code using its instruction set. So in short, yes if you want to program in a language you must learn the language...C, Python, assembly language, machine code, etc. So whatever your compiler outputs you need to know that language or script or rules. – old_timer Apr 23 '14 at 18:40
  • @Kartik: That's not that uncommon. In embedded industry for example it's not uncommon to model in Simulink (TargetLink), and use Embedded Coder (or TargetLink) to generate C-Code from that. Point is, that for every embedded system a C-Compiler is at hand, but the implementation of an algorithm is more easy in a "domain specific" language – Lord_Gestalter Apr 24 '14 at 06:37

5 Answers5

15

No, not at all. It is perfectly possible (and often even preferrable) for your compiler to emit assembly code instead. The assembler then takes care of creating the actual machine code.

By the way, your distinguishing of non-VM implementation vs. VM implementation is not useful.

  • For starters, using a VM or precompilation to machine code are just different ways to implement a language; in most cases a language can be implemented using either strategy. I actually had to use an C++ interpreter once.

  • Also, many VMs like the JVM both have a binary machine code and some assembler, just like an ordinary architecture.

The LLVM (which is used by the Clang compilers) deserves special mention here: It defines a VM for which instructions can be represented as either byte code, textual assembly, or a data structure which makes it very easy to emit from a compiler. So although it would be useful for debugging (and to understand what you are doing), you wouldn't even have to know about the assembly language, only about the LLVM API.

The nice thing about the LLVM is that its VM is just an abstraction, and that the byte code isn't usually interpreted, but transparently JITted instead. So it's entirely possible to write a language that's effectively compiled, without ever having to know about your CPU's instruction set.

amon
  • 132,749
  • 27
  • 279
  • 375
  • And another nice property of LLVM is that one does not need to understand deeply the target ISA to implement an efficient backend. It's pretty declarative, so one can nearly copy-and-paste an ISA specification into .td files without even trying to understand it. – SK-logic Apr 19 '14 at 09:02
  • Thanks for answering. Question: I understand from your answer and other's answers that the compiler-writer doesn't have to understand machine code, he can use another tool that does the actual converting-to-machine-code for him. However, the guy who wrote that tool *did* have to understand machine language, right? The guy who wrote the software that does the actual conversion from some language to machine code has to actually understand machine language right? – Aviv Cohn Apr 19 '14 at 11:47
  • 5
    @Prog yes. If you create a layer of abstraction, you only have to understand the layer below you. While it is useful to have a basic understanding of all layers, this isn't really necessary. You don't need to understand quantum physics to use a transistor. You don't need to understand chip design to to use a CPU. You don't need to know machine code to write assembly. You don't need to know the platform's instruction set when using a VM, etc.. But someone had to build those abstraction levels below yours. – amon Apr 19 '14 at 11:56
  • What is more common when creating compilers from a high level language to machine language: programming the compiler to do the actual conversion, or programming the compiler to use some tool someone else wrote that does the conversion? Do compiler writers usually understand machine code or not? – Aviv Cohn Apr 19 '14 at 12:13
  • The last time I worked on a compiler it generated assembly language but I had to fully understand the machine code and the processor behavior, in order to generate correct and efficient code. – Patricia Shanahan Apr 19 '14 at 14:20
  • 1
    @SK-logic Not true (at least if you want good code) from what I've heard. The people who implemented the Aarch64 backend for llvm had quite a few challenges (relocations for one, load-store patterns horrible,..). And that's ignoring *the* biggest elephant in the room: The ISA's memory model and the memory model of the language you're interested in. You can work on a compiler, but you can't work on a backend without understanding the architecture.. – Voo Apr 19 '14 at 15:29
  • @Voo, true, but to take all this stuff into account you still don't need to understand the *machine code* well. You don't even need to know how the instruction encoding works. No deep knowledge is required. – SK-logic Apr 19 '14 at 19:12
  • 1
    I would add there is really no substantial difference between assembly and machine code, conceptually. You are not really going to have much benefit, once knowing how to use a particular instruction, what that instruction's opcode is. If you know how to use MOV, it doesn't really matter if you don't know that it is instruction 27, which I think is similar to what @SK-logic is describing. – whatsisname Apr 20 '14 at 04:25
  • @SK-logic Oh but "those things" are what I generally consider when saying "understanding an ISA". I wrote an assembler (and parts of a simulator) for Aarch64 and god I can't remember pretty much anything about that - but I'd still say I know the ISA itself pretty well. The encoding is the least interesting part of the ISA imo. – Voo Apr 28 '14 at 21:36
9

No. The key point of your question is that compilation is extremely broad term. Compilation can happen from any language to any language. And assembly/machine code is only one of the many languages for compilation target. For example Java and .NET languages like C#, F# and VB.NET all compile to some kind of intermediate code instead of machine-specific code. It doesn't matter if it then runs on VM, the language is still compiled. There is also option to compile to some other language, like C. C is actually quite popular compilation target and many tools do it. And finally, you could use some tool or library to do the hard work of producing machine code for you. there is for example LLVM which can reduce effort needed to create a standalone compiler.

Also, your edit doesn't make any sense. It is like asking "Does every engineer need to understand how engine works? And I'm asking about engineers working on engines." If you are working on a program or library that emits a machine code, then you have to understand it. The point is, you don't have to do such a thing when writing compiler. Many people did it before you, so you need to have serious reason to do it again.

Euphoric
  • 36,735
  • 6
  • 78
  • 110
  • And the person writing the tool or library that does the actual conversion to machine language, does have to understand machine language completely, right? – Aviv Cohn Apr 19 '14 at 11:50
  • 3
    @Prog Do you need to understand a programming language fully to program in it? No, but you'll possibly write sub-optimal code, and you can't do certain things others might be capable of doing. Do you need to understand machine language completely if you write a compiler that translates to that. No, but your compiler will be suboptimal, and incapable of doing certain things. – Sumurai8 Apr 19 '14 at 12:47
  • @Sumurai8: although to some extent you can "understand" the machine language piecewise in order to write a machine-code emitter that understands the whole thing better than you do. For example if you write a good framework you might configure the definition of each opcode together with its costs and pipelining considerations, and then your framework can write optimized machine code even though you have no expertise whatever at optimizing that particular machine. Being able to program that machine code competently yourself probably doesn't hurt. – Steve Jessop Apr 19 '14 at 16:37
  • @SteveJessop If you understand each opcode to a point that you can learn a machine how to chain that opcode together with other opcodes to express a higher level concept, you do understand machine language completely. You are then just too lazy to find the optimal solution for every problem out there ;-) – Sumurai8 Apr 19 '14 at 16:42
  • @Sumurai8: hmm, but at least in principle I might "understand" each opcode briefly for the the five minutes it takes me to configure it, and then have forgotten it by the time I "understand" the opcode after next. That's probably not what the questioner means by "be able to read/write raw machine language". Of course I'm assuming a pretty darn good framework here, that's configurable enough to define and use all the useful information about each opcode of the instruction set. LLVM somewhat aims for this but according to "Voo" (in a comment below) hasn't hit it. – Steve Jessop Apr 19 '14 at 16:45
  • The .NET example is missing the point. .NET includes an implementation of the back-end of a compiler. The programmers responsible for that part of the framework were presumably intimate not only with the relevant machine instructions, but also how to generate such instructions to create an optimised implementation of the programme. The questioner specifically references people creating compilers that emit machine code. – Dominic Cronin Apr 19 '14 at 18:36
3

Classically a compiler has three parts: lexical analysis, parsing, and code generation. Lexical analysis breaks up the text of the program into language keywords, names, and values. Parsing figures how the tokens that come from the lexical analysis are combined in syntactically correct statements for the language. Code generation takes the data structures produced by the parser, and translates them into machine code or some other representation. Nowadays the lexical analysis and parsing may be combined into a single step.

Clearly the person writing the code generator has to understand the target machine code at a very deep level, including instruction sets, processor pipelines and cache behavior. Otherwise the programs produced by the compiler would be slow and inefficient. They very well might be able to read and write machine code as represented by octal or hexadecimal numbers, but they'll generally write functions to generate the machine code, referring internally to tables of machine instructions. Theoretically the folks writing the lexer and the parser might not know anything about the generation of the machine code. In fact, some modern compilers let you plug in your own code generation routines which might emit machine code for some CPU the lexer and parser writers have never heard of.

However, in practice compiler writers at each step know a lot about different processor architectures, and that helps them design the data structures the code generation step will need.

Charles E. Grant
  • 16,612
  • 1
  • 46
  • 73
2

A long time ago I wrote a compiler that converted between two different shell scripts. It went no way near machine code.

A compiler write has to understand their output, but that is often not machine code.

Most programmers will never write a compiler that outputs machine code or assembly code, but custom compilers can be very useful on lots of projects to produce other outputs.

YACC is one such compiler that does not output machine code….

Ian
  • 4,594
  • 18
  • 28
0

You don't need to start with a detailed knowledge of the semantics of your input and output languages, but you better finish with an exquisitely detailed knowledge of both, otherwise your compiler will be unusably buggy. So if your input is C++ and your output is some specific machine language you will eventually need to know the semantics of both.

Here are some of the subtleties in compiling C++ to machine code: (just off the top of my head, I'm sure there are more I'm forgetting.)

  1. What size will int be? The "correct" choice here is an art, based on both the natural pointer size of the machine, the performance of the ALU for various sizes of arithmetic operations, and the choices made by existing compilers for the machine. Does the machine even have 64-bit arithmetic? If not then addition of 32-bit integers should translate to an instruction while addition of 64-bit integers should translate to a function call to do the 64-bit add. Does the machine have 8-bit and 16-bit add operations or do you have to simulate those with 32-bit ops and masking (e.g. the DEC Alpha 21064)?

  2. What is the calling convention used by other compilers, libraries and languages on the machine? Do the parameters get pushed on the stack right-to-left or left-to-right? Do some parameters go in registers while others go on the stack? Are ints and floats in different register spaces? Do the register allocated parameters need to get treated specially on varargs calls? Which registers are caller-saved and which are callee-saved? Can you perform leaf-call optimizations?

  3. What does each of the machine's shift instructions do? If you ask to shift a 64 bit integer by 65 bits what is the result? (On many machines the result is the same as shifting by 1 bit, on others the result is "0".)

  4. What are the memory consistency semantics of the machine? C++11 has a very well defined memory semantics that places restrictions on some optimizations in some cases, but permits optimizations in other cases. If you are compiling a language that does not have well defined memory semantics (like every version of C/C++ before C++11, and many other imperative languages) then you will have to invent the memory semantics as you go along, and usually you will want to invent the memory semantics that best matches your machine semantics.

Wandering Logic
  • 240
  • 1
  • 7