Lead Data Engineer

Job Description:

We are looking for a skilled Sr. Data Engineer with a solid background in data engineering, data management, and database administration to join our team. This role is pivotal in establishing, maintaining, and optimizing our data systems and processes, particularly focusing on finance-related datasets. The ideal candidate will bring a mix of technical acumen and strategic insight, working to design and implement high-performance data solutions.

Responsibilities:

• Database Design & Management: Establish and uphold database standards, guidelines, and quality control for deliverables such as conceptual design, logical database structure, capacity planning, and data security.
• Platform Expertise: Build and optimize data solutions on Databricks, Spark, Snowflake, and SQL-based environments. Use Python to build data pipelines and automate processes.
• Data Architecture & Quality Assurance: Document and communicate database architecture, ensuring robust quality assurance practices across systems and applications.
• Application & UI Prototyping: Assist in designing user interfaces and prototypes for business applications, providing the technical structure to support these solutions.
• Data Storage Optimization: Collaborate to adapt business requirements, ensuring efficient and appropriate storage structures and formats.
• Cross-Functional Support: Work with data modeling teams and systems engineers to establish data specifications, database attributes, and relationships between tables and fields within a relational database model.

Qualifications:

• Experience: 5-10 years in data engineering, preferably with a background in finance.
• Technical Skills:
  • Strong SQL expertise is essential
  • Proven experience with Databricks, Spark, Python, and Snowflake
• Database Proficiency: Skilled in conceptual and logical database design, including capacity planning, security, and maintenance.
• Analytical Mindset: Able to evaluate database systems and storage solutions to meet performance and data integrity requirements.