Would you enjoy a book without antagonists?

Man vs. man is not the only kind of conflict. There's also man vs. nature, man vs. self, and others. I think you have a fine idea.