This pattern implements the Lambda Architecture using native AWS services. Kinesis Data Streams handles the high-throughput, real-time ingestion and processing for immediate alerts. Persisting the raw stream to Amazon S3 provides a durable, cost-effective data lake for the batch layer. AWS Glue can catalog the data, and Athena enables serverless SQL queries for historical analysis, perfectly separating the speed and batch layers.

S3 Object Lock in compliance mode provides the strongest WORM (Write Once, Read Many) protection, preventing any user (including the root account) from deleting or overwriting an object version before its retention period expires. S3 Intelligent-Tiering automatically moves objects between frequent and infrequent access tiers based on usage patterns, optimizing costs without performance impact or retrieval fees, which is ideal for a mix of active and archived audit data.

In Redshift, proper table design is paramount for performance. Defining `customer_id` as a SORTKEY enables efficient zone maps, drastically reducing the amount of data scanned for range-restrictive queries. Setting a DISTKEY on a related, frequently joined column (like `customer_id` itself if joins are customer-based) co-locates joined data on the same slices, minimizing network traffic during query execution. This addresses the root cause of the full scans.

AWS Glue is a fully managed, serverless ETL service designed specifically for data preparation and loading. Its visual editor and built-in transforms (like `ApplyMapping` for schema correction, `DropNullFields`, and `Filter`) can handle common cleansing tasks such as format standardization and null handling without extensive coding. It natively connects to S3 for source data and Redshift as a target, making it the most integrated and efficient choice.

A sliding window is defined by a time interval relative to the current row’s timestamp. `WINDOW … RANGE INTERVAL ‘1’ HOUR PRECEDING` correctly creates a window that, for each row, includes all rows from the same `user_id` partition whose `event_time` is within the last hour. This is a sliding window. TUMBLE creates tumbling (non-overlapping) windows. ROWS BETWEEN defines a window based on a fixed count of rows, not a time interval.

Amazon MWAA is a managed service for Apache Airflow, the industry-standard workflow orchestration tool. It is ideal for complex, multi-step data pipelines like a medallion architecture. It allows you to create directed acyclic graphs (DAGs) that can reliably orchestrate diverse tasks—such as running an AWS Glue job for deduplication, invoking a Lambda for validation, and executing SQL for SCD logic—in a specific order with dependency management, retries, and monitoring.

Amazon Redshift Federated Query allows you to directly query live data in an RDS PostgreSQL database (and other sources) from within Redshift. You create an external schema that maps to the remote database, and then you can write SQL JOINs that combine local Redshift tables with remote RDS tables. This provides a unified view with minimal setup, no data movement for the remote data, and low latency for on-demand queries.

In Kinesis Data Streams on-demand mode, throughput is automatically scaled. However, if a single partition key is responsible for a very high percentage of the data (a ‘hot key’), all records for that key go to a single shard, which has throughput limits per shard (1 MB/sec write, 2 MB/sec read). The error indicates the writes for that specific shard are being exceeded. The solution is to improve the partition key design (e.g., adding a suffix) to distribute the load more evenly across shards.

For a dimension table that is frequently joined with large fact tables, the ALL distribution style is often optimal. This places a full copy of the table on every compute node in the cluster. During a join, the fact table’s rows can be joined locally on each node without any network data redistribution, which significantly speeds up query performance. This is suitable for tables that are not excessively large, as it uses more storage space.

This pattern implements the Lambda Architecture using native AWS services. Kinesis Data Streams handles the high-throughput, real-time ingestion and processing for immediate alerts. Persisting the raw stream to Amazon S3 provides a durable, cost-effective data lake for the batch layer. AWS Glue can catalog the data, and Athena enables serverless SQL queries for historical analysis, perfectly separating the speed and batch layers.0