Well yeah, because it really sucked. Early agrarians were much less healthy, suffering from malnutrition and diseases that hunter-gatherers did not.

People persisted though. And over ten thousand years they eventually won out. Turns out that being able to store enough food to last all winter is a huge long term advantage. Specialization was an even greater advantage (that also took millennia to develop).

And the issue with trying to put the genie back in the bottle is that if one group left that money on the table another group would come along and pick it up.