Senior Data Engineer

About the Role

The Senior Data Engineer is responsible for designing, building, and maintaining the scalable data pipelines and ingestion frameworks that power the Digital Banking department. This role focuses on translating disparate, high-volume raw data streams—from mobile apps, internet banking portals, and payment gateways—into structured, clean, and highly optimized data stores.

The Senior Data Engineer ensures that data is consistently available, accurate, and structured to support real-time reporting, advanced business intelligence, and production-ready machine learning models.

Principle Responsibilities

• Design, implement, and optimize scalable batch and real-time data ingestion pipelines using distributed computing frameworks like PySpark.
• Build and maintain resilient data lakes and warehousing environments, managing storage formats (e.g., Parquet, Delta) and metadata cataloging systems such as a Hive Metastore backed by PostgreSQL or object storage.
• Structure and partition large datasets to ensure low-latency query performance for downstream consumers (BI Analysts and Data Scientists).
• Implement strict data contract definitions, schema registries, and quality validation checks within pipelines to catch upstream system changes before they break downstream models or reports.
• Ensure data pipelines adhere to strict banking data privacy regulations, masking sensitive customer details, managing access control levels, and archiving historical logs securely.
• Maintain a highly transparent, clear data catalog mapping out data lineage from core systems to final analytics tables.
• Work to deploy data pipelines via containerized environments (Docker/Kubernetes).
• Serve as the primary technical point of contact for the BI Team and Data Science Team, translating their business analytics requirements into optimized backend data assets.
• Enforce clean, modular, and optimized SQL and Python coding standards for data engineering, ensuring thorough version control (Git) and documentation.

Qualifications Required

• Bachelor's degree in Computer Science, Software Engineering, Information Systems, Data Science, Statistics, Mathematics, or any related field.
• Minimum of 3 years of professional experience as a Data Engineer or Core Database Developer, with a proven track record of managing production-grade data pipelines.
• Advanced, hands-on experience using PySpark/Spark to extract, transform, and load massive, complex datasets.
• Deep understanding of managing decoupled data environments, file storage types (Parquet etc), and metadata catalogs (Hive Metastore).
• Expert-level proficiency in writing and optimizing complex queries, indexing, and modeling data structures within relational engines (e.g., PostgreSQL, Oracle).
• Strong familiarity with container tools (Docker) and modern data orchestration workflows (e.g., Apache Airflow or cron-based job scheduling).
• Dedication to automation, building resilient architectures that can recover from network timeouts, api failures, or source data spikes without manual intervention.
• An obsessive eye for identifying performance bottlenecks in queries and pipeline steps to minimize computing costs and execution time.
• Excellent technical communication skills, allowing for seamless collaboration with data consumers to understand exactly how the data needs to be shaped.
• Flexible and adoptive to market dynamics and experimentation.
• Customer-centric mindset.
• Self-driven and problem-solving skills.

CRDB Commitment

CRDB Bank is dedicated to upholding Sustainability and ESG practices and encourage applicants who share this commitment. The Bank also promotes an inclusive workplace, hence applications from women and individual with disabilities are encouraged.

It is important to note that CRDB Bank does not charge any fees for the application or recruitment process, and any requests for payment should be disregarded as they do not represent the bank’s practices.

Only Shortlisted Candidates will be Contacted.