How Language Models Use Tools With No Agency
A language model has no agency. The Llama 3.1 docs mention clearly:
Note that the model itself does not execute the code output; you need to use our llama-stack-apps, or other similar framework to leverage code interpreters. When using
llama-stack-apps, the results of the code are passed back to the model for further processing.
So how does a language model “use tools”?
It doesn’t! It generates executable Python code in response to user’s query and emits a special token to signal presence of code to the surrounding system. The surrounding system recognizes the signal. It knows to extract the code. It must furnish the executor to execute the code and return the result appropriately to the model.
Llama 3.1 uses special tokens and a special role for tool usage. The docs mention:
- The model’s response is wrapped in a <|python_tag|> and terminated with an <|eom_id|> tag.
- The <|eom_id|> indicates a continued multi-step reasoning. That is, the model is expecting a continuation message with the output of the tool call.
Sample input:
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
Environment: ipython<|eot_id|><|start_header_id|>user<|end_header_id|>
Write code to check if number is prime, use that to see if the number 7 is prime<|eot_id|><|start_header_id|>assistant<|end_header_id|>
Response:
<|python_tag|>def is_prime(n):
if n <= 1:
return False
for i in range(2, int(n**0.5) + 1):
if n % i == 0:
return False
return True
print(is_prime(7)) # Output: True<|eom_id|>
After the executor has run the code, it returns the result using a special role. As per docs:
There are 4 different roles that are supported by Llama text models:
- system: Sets the context in which to interact with the AI model. It typically includes rules, guidelines, or necessary information that help the model respond effectively.
- user: Represents the human interacting with the model. It includes the inputs, commands, and questions to the model.
- ipython: A new role introduced in Llama 3.1. Semantically, this role means “tool”. This role is used to mark messages with the output of a tool call when sent back to the model from the executor.
- assistant: Represents the response generated by the AI model based on the context provided in the system, ipython and user prompts.
So what does an executor look like? For Python code, we can construct an executor using Python’s built-in exec() function! This function allows access to the host program’s environment in a manner that can be regulated with args global and local that are dicts passed to control scope. Refer to RealPython’s blogpost on the exec() function.
Executive Decision Python's Built-in Function
The execute method of class CodeExecutor can be as simple as:
def execute(self, code: str) -> dict:
try:
exec(code, self._environment)
except Exception as e:
raise CodeExecutionError("Code execution failed") from e
return self._environment
The environment can be initialized to include any libraries that are required to run Python code in the expected context of the app design. The GitHub repo of Pandasai, a framework for tabular data analysis with LLM, illustrates one possible implementation of executor with environment suited to analyzing spreadsheets.
