Senior Data Engineer (Databricks, Spark)

Job Responsibilities

- Own the end-to-end data architecture ensuring the platform it set up for future scale and SLAs by implementing engineering best practice design patterns.

- Build and implement a roadmap of functionality from the business and gratefully integrate into the architecture while efficiently managing and paying down technical debt.

- Develop pipelines and software using CI/CD best practices and train team to follow proper engineering and deployment standards

- Lead and participate in code reviews and team mentorship.

- Design and implement Data Quality identification & resolution systems to quickly identify data quality problems and lead them to resolution without SLA disruption.

- Provide inputs to major architectural designs to ensure fault tolerant ingestion and maintainability.

- Automate ingestion of various data sources into Databricks and MongoDB.

- Implement software enhancements to our Data Transformation and ETL platforms adhering to software requirements and design specifications.

- Build and optimize data pipelines of highly scalable big data stores.

- Integrate with other utilities (authentication, API gateway) to provide a comprehensive service to our users.

- Build infrastructure automation (e.g. Terraform) to support rolling updates.

- Handle support escalations from users requiring troubleshooting with existing applications.

What you bring

- Degree in a Computer Science or related field.

- At least 5-7 yrs. of relevant experience in building and owning data pipelines, with 2+ years working hands-on with multi-terabyte datasets.

- Experience in at least 2 of the following platforms: Spark, Databricks, AWS EMR, Kafka/Kinesis, MongoDB.

- Experience in building end-to-end infrastructure centric applications using two or more of the following languages Java, Scala, Python, SQL, SQL (SQL is a must have).

- Familiarity with AWS solutions.

- General knowledge of cloud-centric architectures and technical standards, agile software development concepts and processes.

- General knowledge of BI tools (e.g., Power BI, Tableau, etc.).

- Excel at working with users to gather, prioritize, plan and execute solutions to business requirements as it relates to data sources, ingestion and search patterns.

- Experience as an engineering team lead or manager for a team of 2+ engineers.

- Strong understanding of database structures, principles, and design patterns.

- Practiced familiarity with Lambda and Kappa architectures.

‍

‍