Optimising AWS Glue with Apache Airflow: A Cost-Effective Data Orchestration Case Study

Introduction

In modern cloud-native architectures, efficiently scheduling and orchestrating backend workflows is as critical as the application logic itself. From recurring stored procedure calls to time zone-sensitive container executions, every organisation faces the challenge of automation at scale.

For one of our clients, we previously designed a scheduling framework combining Apache Airflow, Amazon ECS, Amazon EventBridge, and RDS stored procedures to create a robust, maintainable, and time zone-aware orchestration layer for a transit data platform. This case study highlights a subsequent enhancement to that platform, where we optimised costs by transitioning from fixed-schedule AWS Glue jobs to event-driven Apache Airflow DAGs — significantly reducing operational expenses while maintaining reliability.

The Client & Context

A smart-transit platform processing regional fare and ridership data, relying on IoT devices for payment collection and transit monitoring. These devices generate raw data stored in Amazon S3 buckets, which is then processed by AWS Glue jobs triggered via Amazon SQS for daily ridership and financial reporting. As data volume grew, two problems compounded: Glue spend ballooned and pipelines repeatedly ran on data that hadn't changed.

The Challenge

Initially, the Glue jobs were scheduled to run at fixed intervals (e.g., every 15 or 30 minutes) regardless of whether new data was available in the S3 buckets. This approach led to:

• High operational costs due to unnecessary Glue job executions when no new data was present.
• Inefficient resource utilisation, as compute resources were allocated for idle jobs.
• Complexity in managing scattered cron-based schedules across environments.
• Lack of centralised visibility into job execution status and dependencies.
• Timezone drift in pipelines that needed to align with regional business hours.

Manual processes and rigid scheduling were causing inconsistent processing cycles and increased administrative overhead, prompting the need for a cost-effective, scalable, and data-driven orchestration solution.

The Architecture

To address the cost and efficiency challenges, SharkOps' senior DevOps engineer Priyanshu Singh redesigned the orchestration layer using Apache Airflow as the central workflow management platform. The new architecture leverages event-driven DAGs (Directed Acyclic Graphs) to trigger AWS Glue jobs only when new data is available in S3, integrating seamlessly with the existing AWS ecosystem.

Key components of the architecture:

• Amazon S3 — stores raw data from IoT devices, such as payment transactions and ridership metrics, uploaded in real-time.
• Amazon SQS — queues notifications of new S3 data uploads, acting as a buffer to signal data availability.
• Apache Airflow — orchestrates workflows using DAGs that monitor SQS for new messages and trigger Glue jobs only when data is present. Airflow provides centralised visibility, logging, and retry mechanisms.
• AWS Glue — executes ETL (Extract, Transform, Load) jobs to process raw data into structured formats for reporting.
• Amazon RDS — stores processed data and supports stored procedures for generating daily ridership and financial reports.
• Amazon EventBridge — triggers time zone-specific workflows for containerised services running on Amazon ECS, ensuring compatibility with the platform's scheduling needs.

Reference Flow

1. Source data lands in S3.
2. S3 Event Notification publishes to EventBridge.
3. EventBridge rule pushes the event into SQS.
4. Airflow sensor consumes the queue and triggers the matching DAG.
5. The DAG invokes AWS Glue with the right job parameters and waits for completion.
6. Results are written back to S3; pipeline state is persisted in RDS.

Solution Implementation

The core of the solution was to replace fixed-schedule Glue jobs with Airflow DAGs that respond dynamically to data availability. We configured Airflow's S3Sensor to monitor S3 buckets for new file uploads and the SQSSensor to check for SQS messages indicating data readiness. When new data is detected, the DAG triggers the relevant Glue job, passing necessary parameters such as file paths and job configurations.

Each DAG was designed to handle a specific set of Glue jobs based on the type of data (e.g., payment transactions or ridership metrics) and the required processing logic. Dependencies between jobs were explicitly defined in the DAG, ensuring that downstream tasks (e.g., stored procedures in RDS) only execute after successful data processing. This eliminated redundant executions and optimised resource usage.

Airflow's web interface provided a centralised dashboard for monitoring DAG runs, tracking failures, and reviewing logs — reducing administrative overhead. The retry and alerting mechanisms ensured robust error handling, with notifications sent to the operations team via Amazon SNS for critical failures.

The key idea: compute is triggered by data, not by time. When relevant files land in S3 or upstream signals fire, EventBridge routes them through SQS into Airflow, which decides whether (and how) to invoke Glue.

Results

The transition to an Airflow-based orchestration layer yielded significant benefits:

• Cost Reduction — by triggering Glue jobs only when new data was available, we reduced AWS Glue usage by approximately 60%, as idle job executions were eliminated.
• Efficiency — event-driven DAGs ensured compute resources were allocated only when necessary, optimising processing cycles.
• Visibility — Airflow's centralised interface provided real-time insights into workflow status, simplifying monitoring and debugging.
• Extensibility — the modular DAG structure allowed easy addition of new workflows to accommodate growing data sources and processing needs.
• Reliability — built-in retry and alerting mechanisms minimised downtime and ensured consistent reporting cycles.
• Timezone-aware pipelines matching the client's actual business cadence.

Technology Stack

Conclusion & Key Takeaway

By leveraging Apache Airflow's event-driven orchestration capabilities, we transformed the client's transit data platform into a cost-efficient, scalable, and reliable system. The shift from fixed-schedule AWS Glue jobs to data-driven DAGs addressed the challenges of high costs and inefficient resource utilisation while maintaining the platform's ability to deliver timely ridership and financial reports.

For analytics pipelines where source data arrives unpredictably, fixed schedules are quietly expensive. Switching to an event-driven model — orchestrated by Airflow, fed by EventBridge and SQS — pays for itself quickly. The 60% AWS Glue cost reduction here is not exotic optimisation; it's the natural consequence of letting compute follow data instead of the clock. This case study demonstrates the power of combining cloud-native tools like Airflow with AWS services to build flexible, cost-optimised workflows for modern data platforms.