Data Engineer

Agivant Technologies

Key Responsibilities

• Execute massive-scale asset rationalization, pipeline migrations, and programmatic metadata implementations using enterprise query, visualization, and telemetry tools to streamline enterprise data operations.
• Asset Rationalization: Execute the migration, consolidation, or retirement of hundreds of legacy enterprise dashboards (Tableau, Power BI) and data pipelines based on strict usage telemetry.
• Programmatic Definitions: Write code that pushes metric definitions, dimensions, and owner mapping programmatically into central semantic catalogs via APIs.
• Data Health Enforcement: Monitor real-time observability dashboards and resolve automated escalations regarding pipeline freshness lags or volume drops within strict SLAs.

Requirements & Tech Stack

Core Technologies

• Query & Databases: Hive, Advanced SQL, distributed datasets (Hive / Presto / Trino).
• Programming & Orchestration: Python (for data manipulation), Airflow (or equivalent orchestration tools).
• Visualization & Telemetry: Tableau / Enterprise Visualization tools, telemetry logging.

Technical Depth

• Proven hands-on experience querying massive distributed datasets.
• Strong capability to trace complex data lineage manually through code.

Bonus Skills

• Experience with Databricks workflows and PySpark optimization.
• Practical experience integrating automated testing within dbt (data build tool).
• Familiarity utilizing AI coding assistants for rapid SQL script generation and optimization.