AIS · eu-central-1 · Serverless
Fully serverless. AWS-managed delivery replaces the abandoned stream-processing engine below; two Lambda functions carry all transformation logic.
Source
AIS API
wss://aisstream.io
Producer
Python Script
ingest_ais_data.py
Stream
Kinesis Data Streams
ais-stream-v1
Compute
Lambda #1
Consumer
Storage
S3 Raw Archive
JSONL, Hive-partitioned
Queue
Amazon SQS
decouples consumer
Compute
Lambda #2
SQS → Firehose bridge
Delivery
Kinesis Firehose
Direct PUT
Search
OpenSearch
ais-index-vessel-data-v3
View
OS Dashboards
Geo Map · KPIs · charts
First attempt: stream processing inside Apache Flink SQL, run via Zeppelin, sinking directly to OpenSearch. Scrapped after the sink connector proved incompatible.
Source
AIS API
wss://aisstream.io
Producer
Python Script
ais_to_kinesis.py
Stream
Kinesis Data Streams
ais-stream-v1
Processing
Flink SQL
via Zeppelin
Sink
OpenSearch Sink
connector, ES7 JARs
View
OS Dashboards
never received data
Why it failed: Flink's OpenSearch sink connector depends on Elasticsearch 7 JARs, incompatible with the deployed AWS OpenSearch version. Jobs ran without errors but silently wrote nothing — traced to the connector layer, not the SQL logic.
What broke, and the fix that stuck — the operational detail a diagram alone doesn't carry.
OpenSearch defaults to Deny for every principal on domain creation — dashboards are unreachable until the security config is switched to Allow.
Firehose reported success with zero documents indexed — its service role wasn't mapped into OpenSearch's own backend roles, separate from IAM.
SQS has no native Firehose target. Lambda #2 exists purely to bridge the two — and doubles as the transform step.
Geo Map needs a single geo_point field, not separate lat/lon floats. Lambda #2 composes {"lat","lon"} before the Firehose hand-off.
MMSI, not ship name, is the stable join key — vessel names repeat across the fleet, transponder IDs don't.
Any field change needs a new index version plus a Firehose destination update and a new Dashboards index pattern — three moving parts, not one.