This entirely depends on your point of view.

We also passively vote on the law by electing the people in charge of passing it. Why not also vote on the people interpreting it?

How else do you want to fill the position for a judge? By a panel of experts? Who decides who is part of this panel? Who decides what the selection process is?

Effectively there is a popular voting process somewhere. The debate is only where exactly in the chain.