markdown-viewer__skills/data-analytics/examples/etl-pipeline.md

ETL Pipeline

Glue-based ETL pipeline with crawlers, data quality checks, and multi-format data integration.

Key Elements

Component	Stencil
Glue	mxgraph.aws4.glue
Glue Crawlers	mxgraph.aws4.glue_crawlers
Glue Data Catalog	mxgraph.aws4.glue_data_catalog
Glue Data Quality	mxgraph.aws4.aws_glue_data_quality
S3	mxgraph.aws4.s3
Lambda	mxgraph.aws4.lambda_function
Redshift	mxgraph.aws4.redshift

Example

Multi-stage ETL with data quality gates and automatic schema discovery:

@startuml
left to right direction

rectangle "Raw Data" {
  mxgraph.aws4.s3 "JSON\nLogs" as json
  mxgraph.aws4.s3 "CSV\nExports" as csv
  mxgraph.aws4.s3 "Parquet\nArchive" as parquet
}

rectangle "Discovery" {
  mxgraph.aws4.glue_crawlers "Schema\nCrawler" as crawler
  mxgraph.aws4.glue_data_catalog "Data\nCatalog" as catalog
}

rectangle "Transform" {
  mxgraph.aws4.glue "ETL Job\n(Spark)" as etl
  mxgraph.aws4.aws_glue_data_quality "Quality\nCheck" as dq
}

rectangle "Targets" {
  mxgraph.aws4.s3 "Curated\nData (S3)" as curated
  mxgraph.aws4.redshift "Redshift" as rs
  mxgraph.aws4.lambda_function "Alert on\nFailure" as alert
}

json --> crawler
csv --> crawler
parquet --> crawler
crawler --> catalog
catalog --> etl
etl --> dq
dq --> curated : "pass"
dq --> alert : "fail"
curated --> rs
@enduml

Pattern Notes

1. Crawlers first: auto-discover schemas from raw S3 data, register in catalog before ETL reads
2. Data Quality gate: Glue Data Quality validates row counts, null checks, and custom rules mid-pipeline
3. Branching on quality: passed data flows to curated bucket → Redshift; failures trigger Lambda alerts
4. Multi-format: crawlers handle JSON, CSV, Parquet, Avro — ETL normalizes to Parquet for curated layer

Related Icons

Icon	Stencil	Use When
Glue DataBrew	mxgraph.aws4.glue_databrew	Visual data preparation
Step Functions	mxgraph.aws4.step_functions	Orchestrating multi-stage ETL
EventBridge	mxgraph.aws4.eventbridge	Schedule or event triggers
CloudWatch	mxgraph.aws4.cloudwatch	Job monitoring & alerts
MWAA	mxgraph.aws4.managed_workflows_for_apache_airflow	Airflow-based orchestration
Athena	mxgraph.aws4.athena	SQL validation on curated data