Graph & ML Platform Engineer with Freight MNC

About the Role

Seeking a Data Engineer who can meticulously model data and manage infrastructure, possessing experience with revisions and point-in-time correctness in live pipelines, and having operated a graph database in production.

Key Responsibilities

• Data Landing:
• Implement a connector framework with data contracts, a delivery ledger, validation, and quarantine mechanisms.
• Develop connectors for licensed vendor files and open-source data.
• Ensure every record is stamped with its three clocks: period, knowledge time, and question time.
• Point-in-Time Store on Object Storage (Delta Lake):
• Manage append-only revisions and vintages.
• Implement an "as-of" read path that prevents returning later-published information.
• Incorporate staleness caps and lineage tracking.
• Reference and Identity Layer:
• Model entities such as airports, lanes, corridors, and regions, including index hierarchies.
• Perform identity resolution across sources with validity windows (Splink or rule-based).
• Graph Layer (Owner):
• Generate the property-graph schema from the ontology using schema-as-code (LinkML).
• Create deterministic projections per "as-of" date.
• Load and operate Neo4j (Aura) with a read-only product projection.
• Write graph features back into the feature store as "as-of" columns.
• Enforce that no licensed raw value or forecast leaks into the graph.
• Maintain the ability to rebuild the graph with a single command and keep the GPU/graph-ML path open.
• Platform:
• Utilize Terraform on Google Cloud (or Azure/AWS).
• Manage container jobs.
• Configure PostgreSQL with row-level security.
• Orchestrate with "as-of" partitions (Dagster or comparable).
• Integrate with MLflow.
• Implement CI/CD gates.
• Establish observability, secrets management, backups, and cost control.
• Ensure the repository is agent-ready (dev container, fixtures, contract tests, leak scans).

Requirements & Tech Stack

• Python and SQL proficiency.
• Experience with DuckDB/Polars/Pandas.
• Expertise in managing larger data pipelines with data contracts, schema evolution, and data quality checks.
• Proven ability to handle point-in-time data correctness: "as-of" reads, publication lags, vintages, revisions, and reproducibility.
• Production experience with Neo4j and Cypher: property-graph data modelling, loading at scale, query performance, and operating a managed instance. Understanding of the graph as a projection of tabular truth, not a system of record.
• Familiarity with Docker, GitHub Actions, Terraform, one major cloud (GCP preferred), PostgreSQL, Parquet/Delta Lake, Dagster/Airflow, MLflow, logging, and monitoring.
• Ability to translate requirements like "build a new macro-data connector" into controlled agent workflows: spec → connector → fixtures → contract tests → data quality → CI → documentation → pull request.