# CloudWatch Logs

Source: /aws/services/logs/

## Introduction

[CloudWatch Logs](https://docs.aws.amazon.com/cloudwatch/index.html) allows to store and retrieve logs.
While some services automatically create and write logs (e.g. Lambda), logs can also be added manually.

:::note
We've introduced a **new CloudWatch Logs provider (`v2`)** backed by a persistent SQLite database.
It is an optional alternative to the default provider and is planned to become the default in an upcoming release.

The main benefit is a significant reduction in memory usage for long-running instances that generate high volumes of log events, which addresses a memory leak present in the default provider.

You can opt in to the new provider by setting:

```bash
PROVIDER_OVERRIDE_LOGS=v2
```

**Known limitation:** Switching to the `v2` provider on an existing instance results in an empty log event history, since log event data cannot be migrated from the default provider's in-memory store.
Metadata such as log groups, log streams, subscription filters, and tags is migrated correctly.
:::

## Subscription Filters

Subscription filters can be used to forward logs to certain services, e.g. Kinesis, Lambda, and Kinesis Data Firehose.
You can read upon details in the [official AWS docs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html).

### Subscription Filters with Kinesis Example

In the following we setup a little example on how to use subscription filters with kinesis.

First, we setup the required resources.
Therefore, we create a kinesis stream, a log group and log stream.
Then we can configure the subscription filter.
```bash
lstk aws kinesis create-stream --stream-name "logtest" --shard-count 1
kinesis_arn=$(lstk aws kinesis describe-stream --stream-name "logtest" | jq -r .StreamDescription.StreamARN)

lstk aws logs create-log-group --log-group-name test

lstk aws logs create-log-stream \
    --log-group-name test \
    --log-stream-name test

lstk aws logs put-subscription-filter \
    --log-group-name "test" \
    --filter-name "kinesis_test" \
    --filter-pattern "" \
    --destination-arn $kinesis_arn \
    --role-arn "arn:aws:iam::000000000000:role/kinesis_role"
```

Next, we can add a log event, that will be forwarded to Kinesis.

```bash
timestamp=$(($(date +'%s * 1000 + %-N / 1000000')))
lstk aws logs put-log-events --log-group-name test --log-stream-name test --log-events "[{\"timestamp\": ${timestamp} , \"message\": \"hello from cloudwatch\"}]"
```

Now we can retrieve the data.
In our example, there will only be one record.
The data record is base64 encoded and compressed in gzip format:

```bash
shard_iterator=$(lstk aws kinesis get-shard-iterator --stream-name logtest --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON | jq -r .ShardIterator)
record=$(lstk aws kinesis get-records --limit 10 --shard-iterator $shard_iterator | jq -r '.Records[0].Data')
echo $record | base64 -d | zcat
```

## Filter Pattern <Badge text="Pro" size="large" />

[Filter patterns](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html) can be used to select certain logs only.

LocalStack currently supports simple json-property filter.

### Metric Filter Example

Metric filters can be used to automatically create CloudWatch metrics.

In the following example we are interested in logs that include a key-value pair `"foo": "bar"` and create a metric filter.

```bash
lstk aws logs create-log-group --log-group-name test-filter

lstk aws logs create-log-stream \
--log-group-name test-filter \
--log-stream-name test-filter-stream

lstk aws logs put-metric-filter \
  --log-group-name test-filter \
  --filter-name my-filter \
  --filter-pattern "{$.foo = \"bar\"}" \
  --metric-transformations \
  metricName=MyMetric,metricNamespace=MyNamespace,metricValue=1,defaultValue=0
```

Next, we can insert some values:

```bash
timestamp=$(($(date +'%s * 1000 + %-N / 1000000')))
lstk aws logs put-log-events --log-group-name test-filter \
   --log-stream-name test-filter-stream \
   --log-events \
    timestamp=$timestamp,message='"{\"foo\":\"bar\", \"hello\": \"world\"}"' \
    timestamp=$timestamp,message="my test event" \
    timestamp=$timestamp,message='"{\"foo\":\"nomatch\"}"'
```

Now we can check that the metric was indeed created:

```bash
end=$(date +%s)
lstk aws cloudwatch get-metric-statistics --namespace MyNamespace \
    --metric-name MyMetric --statistics Sum  --period 3600 \
    --start-time 1659621274 --end-time $end
```

### Filter Log Events

Similarly, you can use filter-pattern to filter logs with different kinds of patterns as described by [AWS](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html).

#### JSON Filter Pattern

For purely JSON structured log messages, you can use JSON filter patterns to traverse the JSON object.
Enclose your pattern in curly braces, like this:

```bash
lstk aws logs filter-log-events --log-group-name test-filter --filter-pattern "{$.foo = \"bar\"}"
```

This returns all events whose top level "foo" key has the "bar" value.

#### Regular Expression Filter Pattern

You can use a simplified regex syntax for regular expression matching.
Enclose your pattern in percentage signs like this:

```bash
lstk aws logs filter-log-events --log-group-name test-filter --filter-pattern "\%[fF]oo\%"
```

This returns all events containing "Foo" or "foo".
For a complete set of the supported syntax, check [the official AWS documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html#regex-expressions)

#### Unstructured Filter Pattern

If not specified otherwise in the pattern, we look for a match in the whole event message:

```bash
lstk aws logs filter-log-events --log-group-name test-filter --filter-pattern "foo"
```

## Resource Browser

The LocalStack Web Application provides a Resource Browser for exploring CloudWatch Logs.
You can access the Resource Browser by opening the LocalStack Web Application in your browser, navigating to the **Resources** section, and then clicking on **CloudWatch Logs** under the **Management/Governance** section.

![CloudWatch Logs Resource Browser](/images/aws/logs-resource-browser.png)

The Resource Browser allows you to perform the following actions:

* **Create Log Group**: Create a new log group by clicking on the **Create Log Group** button followed by entering the details in the dialog box.
* **Create Log Stream**: Create a new log stream by clicking on the **Create Log Stream** button in the log group detail followed by entering the details in the dialog box.
* **Filter Log Events**: Filter log events by clicking the log stream name followed by entering the filter pattern and clicking **Apply**.
* **Delete Log Group**: Delete a log group by selecting the log group name and clicking on the **Actions** dropdown menu, then selecting **Remove Selected**.
* **Delete Log Stream**: Delete a log stream by selecting the log stream name and clicking on the **Actions** dropdown menu, then selecting **Remove Selected**.

## API Coverage


### CloudWatch Logs API coverage

Source service: `logs`. 57 of 118 tracked operations are implemented.

Service documentation: /aws/services/logs/
License availability: available starting with the Hobby plan. See /aws/licensing/ for current plan details.

| Operation | Status |
| --- | --- |
| AssociateKmsKey | Implemented |
| AssociateSourceToS3TableIntegration | Not implemented |
| CancelExportTask | Implemented |
| CancelImportTask | Not implemented |
| CreateDelivery | Implemented |
| CreateExportTask | Implemented |
| CreateImportTask | Not implemented |
| CreateLogAnomalyDetector | Not implemented |
| CreateLogGroup | Implemented |
| CreateLogStream | Implemented |
| CreateLookupTable | Not implemented |
| CreateScheduledQuery | Not implemented |
| DeleteAccountPolicy | Not implemented |
| DeleteDataProtectionPolicy | Implemented |
| DeleteDelivery | Implemented |
| DeleteDeliveryDestination | Implemented |
| DeleteDeliveryDestinationPolicy | Implemented |
| DeleteDeliverySource | Implemented |
| DeleteDestination | Implemented |
| DeleteIndexPolicy | Not implemented |
| DeleteIntegration | Not implemented |
| DeleteLogAnomalyDetector | Not implemented |
| DeleteLogGroup | Implemented |
| DeleteLogStream | Implemented |
| DeleteLookupTable | Not implemented |
| DeleteMetricFilter | Implemented |
| DeleteQueryDefinition | Not implemented |
| DeleteResourcePolicy | Implemented |
| DeleteRetentionPolicy | Implemented |
| DeleteScheduledQuery | Not implemented |
| DeleteSubscriptionFilter | Implemented |
| DeleteSyslogConfiguration | Not implemented |
| DeleteTransformer | Not implemented |
| DescribeAccountPolicies | Not implemented |
| DescribeConfigurationTemplates | Not implemented |
| DescribeDeliveries | Implemented |
| DescribeDeliveryDestinations | Implemented |
| DescribeDeliverySources | Implemented |
| DescribeDestinations | Implemented |
| DescribeExportTasks | Implemented |
| DescribeFieldIndexes | Not implemented |
| DescribeImportTaskBatches | Not implemented |
| DescribeImportTasks | Not implemented |
| DescribeIndexPolicies | Not implemented |
| DescribeLogGroups | Implemented |
| DescribeLogStreams | Implemented |
| DescribeLookupTables | Not implemented |
| DescribeMetricFilters | Implemented |
| DescribeQueries | Implemented |
| DescribeQueryDefinitions | Not implemented |
| DescribeResourcePolicies | Implemented |
| DescribeSubscriptionFilters | Implemented |
| DisassociateKmsKey | Implemented |
| DisassociateSourceFromS3TableIntegration | Not implemented |
| FilterLogEvents | Implemented |
| GetDataProtectionPolicy | Implemented |
| GetDelivery | Implemented |
| GetDeliveryDestination | Implemented |
| GetDeliveryDestinationPolicy | Implemented |
| GetDeliverySource | Implemented |
| GetIntegration | Not implemented |
| GetLogAnomalyDetector | Not implemented |
| GetLogEvents | Implemented |
| GetLogFields | Not implemented |
| GetLogGroupFields | Not implemented |
| GetLogObject | Not implemented |
| GetLogRecord | Not implemented |
| GetLookupTable | Not implemented |
| GetQueryResults | Implemented |
| GetScheduledQuery | Not implemented |
| GetScheduledQueryHistory | Not implemented |
| GetStorageTierPolicy | Not implemented |
| GetTransformer | Not implemented |
| ListAggregateLogGroupSummaries | Not implemented |
| ListAnomalies | Not implemented |
| ListIntegrations | Not implemented |
| ListLogAnomalyDetectors | Not implemented |
| ListLogGroups | Implemented |
| ListLogGroupsForQuery | Not implemented |
| ListScheduledQueries | Not implemented |
| ListSourcesForS3TableIntegration | Not implemented |
| ListSyslogConfigurations | Not implemented |
| ListTagsForResource | Implemented |
| ListTagsLogGroup | Implemented |
| PutAccountPolicy | Not implemented |
| PutBearerTokenAuthentication | Not implemented |
| PutDataProtectionPolicy | Implemented |
| PutDeliveryDestination | Implemented |
| PutDeliveryDestinationPolicy | Implemented |
| PutDeliverySource | Implemented |
| PutDestination | Implemented |
| PutDestinationPolicy | Implemented |
| PutIndexPolicy | Not implemented |
| PutIntegration | Not implemented |
| PutLogEvents | Implemented |
| PutLogGroupDeletionProtection | Not implemented |
| PutMetricFilter | Implemented |
| PutQueryDefinition | Not implemented |
| PutResourcePolicy | Implemented |
| PutRetentionPolicy | Implemented |
| PutStorageTierPolicy | Not implemented |
| PutSubscriptionFilter | Implemented |
| PutSyslogConfiguration | Not implemented |
| PutTransformer | Not implemented |
| StartLiveTail | Not implemented |
| StartQuery | Implemented |
| StopQuery | Not implemented |
| TagLogGroup | Implemented |
| TagResource | Implemented |
| TestMetricFilter | Not implemented |
| TestTransformer | Not implemented |
| UntagLogGroup | Implemented |
| UntagResource | Implemented |
| UpdateAnomaly | Not implemented |
| UpdateDeliveryConfiguration | Not implemented |
| UpdateLogAnomalyDetector | Not implemented |
| UpdateLookupTable | Not implemented |
| UpdateScheduledQuery | Not implemented |