I am trying to create an authentication client and server for my project, and I am trying to think of ways on how a potential hacker could modify my code and use it to hack the servers. Do people actually do this often and if so, how do programmers try to get around it in open source projects?
2 Answers
You cannot trust input from the outside world, whether it's messages in your authentication scheme or data files.
Attackers can and will feed you invalid messages to trigger bugs or information leaks in your public-facing endpoints. They will also happily spoof credentials if your (assumedly homebrew) authentication scheme blindly trusts the foreign party.
There is no universal security or trust, all established mechanisms rely on establishing trust relationships up-front (pre-sharing keys) or by deferring to existing trust vendors (certificate authorities for TLS).
The only thing that differs between attackers leveraging your source and attackers reverse-engineering protocols and binaries is that it's slightly easier for them to get a grip on what your program does from decently written source code.

- 2,106
- 1
- 17
- 14
People do this sometimes, yes. For example: http://m.slashdot.org/story/40281. One mitigation is to require cryptographic signing of commits from maintainers. But you still have to trust your maintainers.

- 1,513
- 10
- 13
-
Sorry, I’m new here and not sure why this is getting downvoted. – Davislor Sep 28 '15 at 09:00