170 lines
6.0 KiB
Plaintext
170 lines
6.0 KiB
Plaintext
<!--Copyright 2024 The HuggingFace Team. All rights reserved.
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
|
the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
|
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations under the License.
|
|
|
|
⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
|
|
rendered properly in your Markdown viewer.
|
|
|
|
-->
|
|
# Models
|
|
|
|
<Tip warning={true}>
|
|
|
|
Smolagents is an experimental API which is subject to change at any time. Results returned by the agents
|
|
can vary as the APIs or underlying models are prone to change.
|
|
|
|
</Tip>
|
|
|
|
To learn more about agents and tools make sure to read the [introductory guide](../index). This page
|
|
contains the API docs for the underlying classes.
|
|
|
|
## Models
|
|
|
|
You're free to create and use your own models to power your agent.
|
|
|
|
You could use any `model` callable for your agent, as long as:
|
|
1. It follows the [messages format](./chat_templating) (`List[Dict[str, str]]`) for its input `messages`, and it returns a `str`.
|
|
2. It stops generating outputs *before* the sequences passed in the argument `stop_sequences`
|
|
|
|
For defining your LLM, you can make a `custom_model` method which accepts a list of [messages](./chat_templating) and returns an object with a .content attribute containing the text. This callable also needs to accept a `stop_sequences` argument that indicates when to stop generating.
|
|
|
|
```python
|
|
from huggingface_hub import login, InferenceClient
|
|
|
|
login("<YOUR_HUGGINGFACEHUB_API_TOKEN>")
|
|
|
|
model_id = "meta-llama/Llama-3.3-70B-Instruct"
|
|
|
|
client = InferenceClient(model=model_id)
|
|
|
|
def custom_model(messages, stop_sequences=["Task"]):
|
|
response = client.chat_completion(messages, stop=stop_sequences, max_tokens=1000)
|
|
answer = response.choices[0].message
|
|
return answer
|
|
```
|
|
|
|
Additionally, `custom_model` can also take a `grammar` argument. In the case where you specify a `grammar` upon agent initialization, this argument will be passed to the calls to model, with the `grammar` that you defined upon initialization, to allow [constrained generation](https://huggingface.co/docs/text-generation-inference/conceptual/guidance) in order to force properly-formatted agent outputs.
|
|
|
|
### TransformersModel
|
|
|
|
For convenience, we have added a `TransformersModel` that implements the points above by building a local `transformers` pipeline for the model_id given at initialization.
|
|
|
|
```python
|
|
from smolagents import TransformersModel
|
|
|
|
model = TransformersModel(model_id="HuggingFaceTB/SmolLM-135M-Instruct")
|
|
|
|
print(model([{"role": "user", "content": [{"type": "text", "text": "Ok!"}]}], stop_sequences=["great"]))
|
|
```
|
|
```text
|
|
>>> What a
|
|
```
|
|
|
|
> [!TIP]
|
|
> You must have `transformers` and `torch` installed on your machine. Please run `pip install smolagents[transformers]` if it's not the case.
|
|
|
|
[[autodoc]] TransformersModel
|
|
|
|
### HfApiModel
|
|
|
|
The `HfApiModel` wraps huggingface_hub's [InferenceClient](https://huggingface.co/docs/huggingface_hub/main/en/guides/inference) for the execution of the LLM. It supports both HF's own [Inference API](https://huggingface.co/docs/api-inference/index) as well as all [Inference Providers](https://huggingface.co/blog/inference-providers) available on the Hub.
|
|
|
|
```python
|
|
from smolagents import HfApiModel
|
|
|
|
messages = [
|
|
{"role": "user", "content": [{"type": "text", "text": "Hello, how are you?"}]}
|
|
]
|
|
|
|
model = HfApiModel()
|
|
print(model(messages))
|
|
```
|
|
```text
|
|
>>> Of course! If you change your mind, feel free to reach out. Take care!
|
|
```
|
|
[[autodoc]] HfApiModel
|
|
|
|
### LiteLLMModel
|
|
|
|
The `LiteLLMModel` leverages [LiteLLM](https://www.litellm.ai/) to support 100+ LLMs from various providers.
|
|
You can pass kwargs upon model initialization that will then be used whenever using the model, for instance below we pass `temperature`.
|
|
|
|
```python
|
|
from smolagents import LiteLLMModel
|
|
|
|
messages = [
|
|
{"role": "user", "content": [{"type": "text", "text": "Hello, how are you?"}]}
|
|
]
|
|
|
|
model = LiteLLMModel("anthropic/claude-3-5-sonnet-latest", temperature=0.2, max_tokens=10)
|
|
print(model(messages))
|
|
```
|
|
|
|
[[autodoc]] LiteLLMModel
|
|
|
|
### OpenAIServerModel
|
|
|
|
This class lets you call any OpenAIServer compatible model.
|
|
Here's how you can set it (you can customise the `api_base` url to point to another server):
|
|
```py
|
|
import os
|
|
from smolagents import OpenAIServerModel
|
|
|
|
model = OpenAIServerModel(
|
|
model_id="gpt-4o",
|
|
api_base="https://api.openai.com/v1",
|
|
api_key=os.environ["OPENAI_API_KEY"],
|
|
)
|
|
```
|
|
|
|
[[autodoc]] OpenAIServerModel
|
|
|
|
### AzureOpenAIServerModel
|
|
|
|
`AzureOpenAIServerModel` allows you to connect to any Azure OpenAI deployment.
|
|
|
|
Below you can find an example of how to set it up, note that you can omit the `azure_endpoint`, `api_key`, and `api_version` arguments, provided you've set the corresponding environment variables -- `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_API_KEY`, and `OPENAI_API_VERSION`.
|
|
|
|
Pay attention to the lack of an `AZURE_` prefix for `OPENAI_API_VERSION`, this is due to the way the underlying [openai](https://github.com/openai/openai-python) package is designed.
|
|
|
|
```py
|
|
import os
|
|
|
|
from smolagents import AzureOpenAIServerModel
|
|
|
|
model = AzureOpenAIServerModel(
|
|
model_id = os.environ.get("AZURE_OPENAI_MODEL"),
|
|
azure_endpoint=os.environ.get("AZURE_OPENAI_ENDPOINT"),
|
|
api_key=os.environ.get("AZURE_OPENAI_API_KEY"),
|
|
api_version=os.environ.get("OPENAI_API_VERSION")
|
|
)
|
|
```
|
|
|
|
[[autodoc]] AzureOpenAIServerModel
|
|
|
|
### MLXModel
|
|
|
|
|
|
```python
|
|
from smolagents import MLXModel
|
|
|
|
model = MLXModel(model_id="HuggingFaceTB/SmolLM-135M-Instruct")
|
|
|
|
print(model([{"role": "user", "content": "Ok!"}], stop_sequences=["great"]))
|
|
```
|
|
```text
|
|
>>> What a
|
|
```
|
|
|
|
> [!TIP]
|
|
> You must have `mlx-lm` installed on your machine. Please run `pip install smolagents[mlx-lm]` if it's not the case.
|
|
|
|
[[autodoc]] MLXModel
|