I think tool calling is just a form of structured output, where the LLM outputs something like json that describes the function to call and the arguments, then you parse that and run the function with the arguments, then feed the output back to the LLM in a new message, if you want. IDK the specific details, I’m guessing there are some special tokens some LLMs produce for tool calling, and I’m also guessing there is “controlled generation” (masking the logits/tokens, and only choosing to generate the tokens that would be valid). Ollama apparently doesn’t support the special tool-calling tokens or output structure that some models use.

This is how it looks when using OpenAI compatible APIs: https://platform.openai.com/docs/guides/function-calling?api-mode=responses.

I’ve never heard of TabbyAPI (I’m new to using local models, in general). I’m also not sure what TabbyAPI brings over Ollama, llama.cpp, or vLLM. So would be curious as well.

Edit: For my little toy project I’m working on, I’m switching to not using tool-calling at all, and building a LangGraph and using structured output, which should be more reliable with my use-case. I.e. just always call the tool manually, feed the output back to the LLM to have the LLM evaluate if the output was correct, retry calling the tool with different arguments if not, and just fail after 5 calls.

I like watching Yannic Kilcher, to keep up with some of the newer developments, and read various papers. I do have a background in ML (mostly the more traditional, non-generative, supervised learning and reinforcement learning) though.