Then we have to define what body serves as a reference point. “Relative to the observer” doesn’t seem to work here, since we try to decide where should the observer themselves go.

If so, then why should it be Earth? Why not the Sun, or the center of a Milky Way, or literally anything else? As you said, it’s arbitrary. And how do we choose the reference frame?

Doesn’t make any sense outside spacetime as a whole.