I would imagine its more of a speech thing. If you just silently boycott they likely wont know or be able to prove it, but if you publicly say that you are boycotting Israel (which is something that makes such efforts more effective anyway since boycotts are a group action), or perhaps advocate for it in a way that makes it seem likely that you are doing it yourself, then they might be able to go after you for that (or at least the fear that they could so so, even if they never actually bother, might deter some people).