Meta logoLlama 3.1 405B Instruct

The best open source large language model

Deploy Llama 3.1 405B Instruct behind an API endpoint in seconds.

Talk to sales

Example usage

Input
1import requests
2
3# Replace the empty string with your model id below
4model_id = ""
5baseten_api_key = os.environ["BASETEN_API_KEY"]
6
7messages = [
8    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
9    {"role": "user", "content": "Who are you?"},
10]
11data = {
12    "messages": messages,
13    "stream": True,
14    "max_new_tokens": 512,
15    "temperature": 0.9
16}
17
18# Call model endpoint
19res = requests.post(
20    f"https://model-{model_id}.api.baseten.co/production/predict",
21    headers={"Authorization": f"Api-Key {baseten_api_key}"},
22    json=data,
23    stream=True
24)
25
26# Print the generated tokens as they get streamed 
27for content in res.iter_content():
28    print(content.decode("utf-8"), end="", flush=True)
JSON output
1[
2    "arrrg",
3    "me hearty",
4    "I",
5    "be",
6    "doing",
7    "..."
8]

Deploy any model in just a few commands

Avoid getting tangled in complex deployment processes. Deploy best-in-class open-source models and take advantage of optimized serving for your own models.

$

truss init -- example stable-diffusion-2-1-base ./my-sd-truss

$

cd ./my-sd-truss

$

export BASETEN_API_KEY=MdNmOCXc.YBtEZD0WFOYKso2A6NEQkRqTe

$

truss push

INFO

Serializing Stable Diffusion 2.1 truss.

INFO

Making contact with Baseten 👋 👽

INFO

🚀 Uploading model to Baseten 🚀

Upload progress: 0% | | 0.00G/2.39G