# Bedrock

Source: /aws/services/bedrock/

## Introduction

Bedrock is a fully managed service provided by Amazon Web Services (AWS) that makes foundation models from various LLM providers accessible via an API.

LocalStack allows you to use the Bedrock APIs to test and develop AI-powered applications in your local environment.

The supported APIs are available on the API coverage section for [Bedrock](#api-coverage) and [Bedrock Runtime](#api-coverage-bedrock-runtime), which provides information on the extent of Bedrock's integration with LocalStack.

## Getting started

This guide is designed for users new to AWS Bedrock and assumes basic knowledge of the AWS CLI and our [`lstk aws`](/aws/developer-tools/running-localstack/lstk/cloud-and-iac-commands/#aws) command.

Start your LocalStack container using your preferred method with or without pre-warming the Bedrock engine.
We will demonstrate how to use Bedrock by following these steps:

1. Listing available foundation models
2. Invoking a model for inference
3. Using the conversation API
4. Using batch processing

### Pre-warming the Bedrock engine

The startup of the Bedrock engine can take some time.
Per default, we only start it once you send a request to one of the `bedrock-runtime` APIs.
However, if you want to start the engine when localstack starts to avoid long wait times on your first request you can set the flag `BEDROCK_PREWARM`.

On startup, the `DEFAULT_BEDROCK_MODEL` is pulled from the Ollama library and loaded into memory.
However, you can define an additional list of models in `BEDROCK_PULL_MODELS` to pull additional models when the Bedrock engine starts up.
This way you avoid long wait times when switching between models on demand with requests.

### List available foundation models

You can view all available foundation models using the [`ListFoundationModels`](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_ListFoundationModels.html) API.
This will show you which models are available on AWS Bedrock.

:::note
The actual model that will be used for emulation will differ from the ones defined in this list.
You can define the used model with `DEFAULT_BEDROCK_MODEL`
:::

Run the following command:

```bash
lstk aws bedrock list-foundation-models
```

### Invoke a model

You can use the [`InvokeModel`](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html) API to send requests to a specific model.
In this example, we selected the Llama 3 model to process a simple prompt.
However, the actual model will be defined by the `DEFAULT_BEDROCK_MODEL` environment variable.

Run the following command:

```bash
lstk aws bedrock-runtime invoke-model \
    --model-id "meta.llama3-8b-instruct-v1:0" \
    --body '{
        "prompt": "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nSay Hello!\n<|eot_id|>\n<|start_header_id|>assistant<|end_header_id|>",
        "max_gen_len": 2,
        "temperature": 0.9
    }' --cli-binary-format raw-in-base64-out outfile.txt
```

The output will be available in the `outfile.txt`.

### Use the conversation API

Bedrock provides a higher-level conversation API that makes it easier to maintain context in a chat-like interaction using the [`Converse`](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html) API.
You can specify both system prompts and user messages.

Run the following command:

```bash
lstk aws bedrock-runtime converse \
    --model-id "meta.llama3-8b-instruct-v1:0" \
    --messages '[{
        "role": "user",
        "content": [{
            "text": "Say Hello!"
        }]
    }]' \
    --system '[{
        "text": "You'\''re a chatbot that can only say '\''Hello!'\''"
    }]'
```

### Model Invocation Batch Processing

Bedrock offers the feature to handle large batches of model invocation requests defined in S3 buckets using the [`CreateModelInvocationJob`](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_CreateModelInvocationJob.html) API.

First, you need to create a `JSONL` file named `batch_input.jsonl` that contains all your prompts:

```json
{"prompt": "Tell me a quick fact about Vienna.", "max_tokens": 50, "temperature": 0.5}
{"prompt": "Tell me a quick fact about Zurich.", "max_tokens": 50, "temperature": 0.5}
{"prompt": "Tell me a quick fact about Las Vegas.", "max_tokens": 50, "temperature": 0.5}
```

Then, you need to define buckets for the input as well as the output and upload the file in the input bucket:

```bash
lstk aws s3 mb s3://in-bucket
lstk aws s3 cp batch_input.jsonl s3://in-bucket
lstk aws s3 mb s3://out-bucket
```

Afterwards you can run the invocation job like this:

```bash
lstk aws bedrock create-model-invocation-job \
  --job-name "my-batch-job" \
  --model-id "mistral.mistral-small-2402-v1:0" \
  --role-arn "arn:aws:iam::123456789012:role/MyBatchInferenceRole" \
  --input-data-config '{"s3InputDataConfig": {"s3Uri": "s3://in-bucket"}}' \
  --output-data-config '{"s3OutputDataConfig": {"s3Uri": "s3://out-bucket"}}'
```


```bash title="Output"
{
    "jobArn": "arn:aws:bedrock:us-east-1:000000000000:model-invocation-job/12345678"
}
```

The results will be at the S3 URL `s3://out-bucket/12345678/batch_input.jsonl.out`

## Available models

LocalStack's Bedrock emulation supports models from the [Ollama Models library](https://ollama.com/search).

To use a model, retrieve its ID from Ollama and set `DEFAULT_BEDROCK_MODEL` to that ID.
LocalStack will pull the model from Ollama and use it for emulation.

For example, to use the Mistral model, set the environment variable while starting LocalStack:

```bash
LOCALSTACK_DEFAULT_BEDROCK_MODEL=mistral lstk start
```

You can also define models directly in the request, by setting the `model-id` parameter to `ollama.<ollama-model-id>`.
For example, if you want to access `deepseek-r1`, you can do it like this:

```bash
lstk aws bedrock-runtime converse \
    --model-id "ollama.deepseek-r1" \
    --messages '[{
        "role": "user",
        "content": [{
            "text": "Say Hello!"
        }]
    }]'
```

## Troubleshooting

Users of Docker Desktop on macOS or Windows might run into the issue of Bedrock becoming unresponsive after some usage.
A common reason for that is insufficient storage or memory space in the Docker Desktop VM.
To resolve this issue you can increase those amounts directly in Docker Desktop or clean up unused artifacts with the Docker CLI like this

```bash
docker system prune
```

You could also try to use a model with lower requirements.
To achieve that you can search for models in the [Ollama Models library](https://ollama.com/search) with a low parameter count or smaller size.

## Limitations

* At this point, we have only tested text-based models in LocalStack.
Other models available with Ollama might also work, but are not officially supported by the Bedrock implementation.
* Currently, GPU models are not supported by the LocalStack Bedrock implementation.

## API Coverage


### bedrock API coverage

Source service: `bedrock`. 6 of 108 tracked operations are implemented.

Service documentation: /aws/services/bedrock/
License availability: available starting with the Ultimate plan. See /aws/licensing/ for current plan details.

| Operation | Status |
| --- | --- |
| BatchDeleteAdvancedPromptOptimizationJob | Not implemented |
| BatchDeleteEvaluationJob | Not implemented |
| CancelAutomatedReasoningPolicyBuildWorkflow | Not implemented |
| CreateAdvancedPromptOptimizationJob | Not implemented |
| CreateAutomatedReasoningPolicy | Not implemented |
| CreateAutomatedReasoningPolicyTestCase | Not implemented |
| CreateAutomatedReasoningPolicyVersion | Not implemented |
| CreateCustomModel | Not implemented |
| CreateCustomModelDeployment | Not implemented |
| CreateEvaluationJob | Not implemented |
| CreateFoundationModelAgreement | Not implemented |
| CreateGuardrail | Not implemented |
| CreateGuardrailVersion | Not implemented |
| CreateInferenceProfile | Not implemented |
| CreateMarketplaceModelEndpoint | Not implemented |
| CreateModelCopyJob | Not implemented |
| CreateModelCustomizationJob | Not implemented |
| CreateModelImportJob | Not implemented |
| CreateModelInvocationJob | Implemented |
| CreatePromptRouter | Not implemented |
| CreateProvisionedModelThroughput | Not implemented |
| DeleteAutomatedReasoningPolicy | Not implemented |
| DeleteAutomatedReasoningPolicyBuildWorkflow | Not implemented |
| DeleteAutomatedReasoningPolicyTestCase | Not implemented |
| DeleteCustomModel | Not implemented |
| DeleteCustomModelDeployment | Not implemented |
| DeleteEnforcedGuardrailConfiguration | Not implemented |
| DeleteFoundationModelAgreement | Not implemented |
| DeleteGuardrail | Not implemented |
| DeleteImportedModel | Not implemented |
| DeleteInferenceProfile | Not implemented |
| DeleteMarketplaceModelEndpoint | Not implemented |
| DeleteModelInvocationLoggingConfiguration | Not implemented |
| DeletePromptRouter | Not implemented |
| DeleteProvisionedModelThroughput | Not implemented |
| DeleteResourcePolicy | Not implemented |
| DeregisterMarketplaceModelEndpoint | Not implemented |
| ExportAutomatedReasoningPolicyVersion | Not implemented |
| GetAccountDataRetention | Not implemented |
| GetAdvancedPromptOptimizationJob | Not implemented |
| GetAutomatedReasoningPolicy | Not implemented |
| GetAutomatedReasoningPolicyAnnotations | Not implemented |
| GetAutomatedReasoningPolicyBuildWorkflow | Not implemented |
| GetAutomatedReasoningPolicyBuildWorkflowResultAssets | Not implemented |
| GetAutomatedReasoningPolicyNextScenario | Not implemented |
| GetAutomatedReasoningPolicyTestCase | Not implemented |
| GetAutomatedReasoningPolicyTestResult | Not implemented |
| GetCustomModel | Not implemented |
| GetCustomModelDeployment | Not implemented |
| GetEvaluationJob | Not implemented |
| GetFoundationModel | Implemented |
| GetFoundationModelAvailability | Not implemented |
| GetGuardrail | Not implemented |
| GetImportedModel | Not implemented |
| GetInferenceProfile | Not implemented |
| GetMarketplaceModelEndpoint | Not implemented |
| GetModelCopyJob | Not implemented |
| GetModelCustomizationJob | Not implemented |
| GetModelImportJob | Not implemented |
| GetModelInvocationJob | Implemented |
| GetModelInvocationLoggingConfiguration | Not implemented |
| GetPromptRouter | Not implemented |
| GetProvisionedModelThroughput | Not implemented |
| GetResourcePolicy | Not implemented |
| GetUseCaseForModelAccess | Not implemented |
| ListAdvancedPromptOptimizationJobs | Not implemented |
| ListAutomatedReasoningPolicies | Not implemented |
| ListAutomatedReasoningPolicyBuildWorkflows | Not implemented |
| ListAutomatedReasoningPolicyTestCases | Not implemented |
| ListAutomatedReasoningPolicyTestResults | Not implemented |
| ListCustomModelDeployments | Not implemented |
| ListCustomModels | Not implemented |
| ListEnforcedGuardrailsConfiguration | Not implemented |
| ListEvaluationJobs | Not implemented |
| ListFoundationModelAgreementOffers | Not implemented |
| ListFoundationModels | Implemented |
| ListGuardrails | Not implemented |
| ListImportedModels | Not implemented |
| ListInferenceProfiles | Not implemented |
| ListMarketplaceModelEndpoints | Not implemented |
| ListModelCopyJobs | Not implemented |
| ListModelCustomizationJobs | Not implemented |
| ListModelImportJobs | Not implemented |
| ListModelInvocationJobs | Implemented |
| ListPromptRouters | Not implemented |
| ListProvisionedModelThroughputs | Not implemented |
| ListTagsForResource | Not implemented |
| PutAccountDataRetention | Not implemented |
| PutEnforcedGuardrailConfiguration | Not implemented |
| PutModelInvocationLoggingConfiguration | Not implemented |
| PutResourcePolicy | Not implemented |
| PutUseCaseForModelAccess | Not implemented |
| RegisterMarketplaceModelEndpoint | Not implemented |
| StartAutomatedReasoningPolicyBuildWorkflow | Not implemented |
| StartAutomatedReasoningPolicyTestWorkflow | Not implemented |
| StopAdvancedPromptOptimizationJob | Not implemented |
| StopEvaluationJob | Not implemented |
| StopModelCustomizationJob | Not implemented |
| StopModelInvocationJob | Implemented |
| TagResource | Not implemented |
| UntagResource | Not implemented |
| UpdateAutomatedReasoningPolicy | Not implemented |
| UpdateAutomatedReasoningPolicyAnnotations | Not implemented |
| UpdateAutomatedReasoningPolicyTestCase | Not implemented |
| UpdateCustomModelDeployment | Not implemented |
| UpdateGuardrail | Not implemented |
| UpdateMarketplaceModelEndpoint | Not implemented |
| UpdateProvisionedModelThroughput | Not implemented |

## API Coverage (Bedrock Runtime)


### bedrock-runtime API coverage

Source service: `bedrock-runtime`. 3 of 11 tracked operations are implemented.

Service documentation: /aws/services/bedrock/
License availability: available starting with the Ultimate plan. See /aws/licensing/ for current plan details.

| Operation | Status |
| --- | --- |
| ApplyGuardrail | Not implemented |
| Converse | Implemented |
| ConverseStream | Not implemented |
| CountTokens | Not implemented |
| GetAsyncInvoke | Not implemented |
| InvokeGuardrailChecks | Not implemented |
| InvokeModel | Implemented |
| InvokeModelWithBidirectionalStream | Implemented |
| InvokeModelWithResponseStream | Not implemented |
| ListAsyncInvokes | Not implemented |
| StartAsyncInvoke | Not implemented |