Ryan - You raise some good points, and normally I don't address commercial issues on the site, but I will give you a few of my thoughts on this subject.
Fees for usage rights on stock photography varies a lot on the type of photo and intended use. Magazine, advertising, editorial, and internet usage will all carry different usage fee rates. Image placement, print size, print volume, and duration of use also plays a large factor too.
Then there is microstock, CENSORED WORD free rates, and rights managed rates. The spread is so big it is hard to really give you a straight answer. If you wanted to know what microstock rates are you can check them out on
www.istockphoto.com and for rights managed rates you could find a similar photo to one of yours say on
www.gettyimages.com and run through their pricing structure on a specified type of use to see what they would charge.
As for photo contests in general, personally I never bother with them, partly because I am not sure there is much commercial recognition from it. I could be wrong, but I think they would propel you more in an artistic sense rather than a commercial sense and my focus is more on the commercial side of photography at this point.
There is also the issue of cost that you raise, and unless you are confident your photos will knock the judges over, then you could be spending a lot of money for nothing. I often see winning photos from photo contests and they are not ones I probably would have picked myself.
This makes me think that the judges make very subjective selections and have certain agendas which could be difficult to read into. Of course, if you enjoy the whole process, the competition, and the chance of winning something then go for it. But I guess what I am saying is that yes, you could spend a lot of money without any returns and I doubt the commercial returns will be there from it either. The free usage they will get from your photos is of course another issue you have to be comfortable with.
I hope that helps.