I'm interested in expanding my knowledge of security issues: things like buffer overflows, format string vulnerabilities, etc. I'd like to be able to go through a language and understand its security implications and pitfalls to get a better understanding of secure coding practices in general.
Are there certain languages that are more conducive to a learning experiment like this? Are there languages, due to abstractions or complexity, I should avoid? Where should I start?
Modern C++ style has a number of constructs that help mitigate security problems. std::vector and string in particular.
So, I would recommend C.
Or, for super-fun times, you could write some assembly programs.
To gain a better understanding of what languages like Java buy you in terms of security and memory management, work in C. The only thing that will drive the point home any deeper will be working in assembler¹. While it's nothing like what Java provides, C++ gives you too many tools to manage the kind of complexity that causes these problems, and the temptation to use them will be too great.
Memory management in C is very labor intensive compared to Java and C++. You don't have any sort of automatic garbage collection, you don't have explicit destructors to free up resources when an object goes out of scope, so you have to manually track every memory allocation and match it up with a corresponding deallocation. Arrays don't know how big they are and there's no bounds checking, so you have to track all those buffer sizes yourself and enforce those limits manually. C doesn't provide much in the way of exception handling (signals and setjmp/longjmp are the extent of it, and that's not very much at all), so you have to be careful about how you handle error conditions.
If you really want to appreciate what Java (or even C++) buy you, implement a C library that mimics the behavior of the Java String class.
¹ The best of both worlds would be to work in C and have your compiler write the generated assembly to an output file and study that.
The key thing to understand is that most security issues are inherently low-level. C++ is roughly a superset of C, but to really be forced to understand how low-level things work, you need to at least dabble in straight C, or write inherently low-level code (things like memory allocators or string classes) in C++. If you just learn C++ coming from a background of even higher level languages, chances are you'll try to program in a Java-like style that you're already comfortable with, using lots of objects and never touch the lower-level features inherited from C, which are the important ones if you're trying to learn about security.
There's no reason to pick C over C++. Pretty much anything C can do, C++ can do, but better. Things like automatic resource management and templates are irreplacable tools. Having to learn to free() is really irrelevant. If you write C code, you still don't have to write your own malloc() or anything like that.
There's nothing language-specific about security.
