GCP Data Engineer
Infosys
Noida, Uttar Pradesh, India1d agovia LinkedIn
About the Role
As a GCP Data Engineer at Infosys, you will work on cutting-edge data engineering and analytics initiatives for global clients across industries within the Data & Analytics unit.
Key Responsibilities
- Design and develop scalable GCP Data Engineering solutions and cloud-native data platforms.
- Build batch and real-time pipelines using BigQuery, Cloud Storage, Dataflow, Pub/Sub, Dataproc, Python, PySpark, and Databricks.
- Develop and optimize Databricks, Delta Lake, Spark, and Structured Streaming workloads for large-scale processing.
- Implement high-throughput streaming solutions using Pub/Sub and Dataflow, including windowing, watermarking, late-arriving data, duplicate handling, and dead-letter queues.
- Handle high-volume TB/PB-scale data with focus on performance, scalability, reliability, and cost optimization.
- Develop complex SQL transformations, data models, and analytics-ready BigQuery datasets using partitioning and clustering.
- Orchestrate pipelines using Cloud Composer / Airflow and implement CI/CD and Terraform-based Infrastructure as Code.
- Ensure data quality, governance, security, monitoring, alerting, and production support.
- Collaborate with architects, stakeholders, and cross-functional teams in an Agile delivery model.
- Troubleshoot and optimize end-to-end GCP, Databricks, and streaming data pipelines.
Requirements & Tech Stack
- GCP BigQuery
- Cloud Storage
- Dataflow
- Pub/Sub
- Dataproc
- Batch and real-time streaming
- Databricks
- PySpark
- Python
- Strong SQL
- Databricks with Delta Lake
- Structured Streaming
- Cloud Composer / Airflow
- CI/CD
- Terraform