The policy of deliberately refusing to branch is usually a mistake, but I don't think branching is feasible in your case. Not because you are using SVN - which is the usual reason for not using branches, since SVN's handle on branches is especially crappy, even compared to SVN's general crappiness - but because of this:
our product can't be compiled and run locally, so our work is actually
more like the whole team modifying a single test server.
This, IMO, is your main problem, and as long as you don't fix this - I don't think you can use branching even if you wanted to. Different developers can't work on separate branches since they are forced to share a working copy. If one developer checks out a branch, that branch will be checked out for the the developers, which means that you miss the main advantage of branching - allowing multiple developers to work on different features of the same project without getting between each other's legs.
So, you can't use branches to coordinate the collaborations, but solo developers(that use SCMs that are not total garbage - which is not your case here, but let's ignore it for the point of the debate) also use branching - e.g. to test new features that may or may not be merged or to explore the system in a scrapable branch.
Can you use branches in such a way? Well, even if you are OK with moving the whole team to a feature branch or to a scrapable branch - how do you plan to coordinate it? Will you use email, or install a PA system where you can announce WE ARE NOW MOVING TO FEATURE X'S BRANCH? And what will you do when some dev is in the middle of something in the current branch? Will you make them roll it back, or make the entire team wait for that developer to finish?
Do you really want that headache?
Another point to consider - branching usually needs some level of cleanness in the build process and project tree structure, so artifacts of one branch won't leak to other branches when you switch branches. SCMed projects usually have that - or else they wouldn't be able to build on all the machines of all the developers - but seeing that your project can only build on the test server I doubt you have that minimal level of cleanness. Oh, your branches will probably work - at least in the beginning, until a leaky artifact will surprise you with some weird, hard-to-detect bugs.