The worst thing, that could happen to you if go open source is, that someone forks your code, and creates a project that is more successful. This requires the two of you to have fundamentally different oppinions about where the project should go.
Firstly, this is not much of a risk, and secondly, I think not much is lost, if somebody successfully uses some of your code to achieve something else than you intended. It's what open source is about. Create free code, that somebody else can put to good use.
The worst thing, that could happen to you if you offer a proprietary software for free, is that it becomes popular. You will have a big user base with a lot of wishes and complaints. Even though they didn't pay for your software, they expect support, bug fixes and new features.
That being said, you can still open source it at that point. Or you can try putting a prize tag on it. However this will cause a lot of displeasure among your users and comes with a lot of overhead (accounting, legal stuff, etc.).
The best thing, that could happen to you if you offer a proprietary software for free, is that it becomes popular. Appreciation is basically the ideological reward you're looking for, if you give something away for free. However, it comes with its burdens, as stated in the paragraph above.
The best thing, that could happen to you if you go open source, is that it becomes popular. You have the same ideological reward as with proprietary software, all the while you get the advantages of a successful open source project: user contribution.
And yet, monetizing the project is still an option: support, training, getting paid for features, enterprise licences (open source doesn't need to be gratis).
A likely thing to happen, is that your project doesn't really take of, and only very few people use it. This is the case for the vast majority of software for free, because the least of them are reaching the critical user base for word-of-mouth-propaganda, and even less of them are advertised.