Data Engineer

As a data engineer, you will be responsible for ideating, designing, developing and maintaining all the engineering building blocks which are required to keep the data consistent, reliable and consumable.

We expect our data engineers to

• Architect the Data lake and Data warehouse infrastructure to enable the users to execute business intelligence, data analytics, data science and ML/AI workloads

• Design, develop and maintain data pipelines for optimal extraction, transformation, and loading of data from a wide variety of data sources.

• Build workflow orchestration tools to enable users to run periodic jobs on our data infrastructure

• Build frameworks for data processing jobs to run decision science workloads such as personalisation, fraud detection, optimisation algorithms and more

• Building observability over the data pipelines to maintain consistent data with an agreed upon latency

• Ideate processes to maintain the health of the infrastructure such as cpu, memory monitoring, query monitoring, workload monitoring and educate the teams to follow the best practices. Set the standards high.

• Work closely with decision scientists, data analysts and product managers to understand their data needs, their technical issues and help them in performance tuning.

• Keep the data secure from the external world and have fine grained access control within the organization to make sure that only relevant data is available to every user.

• Create or integrate with tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.

To cater to the above responsibilities, we need our data engineers to be equipped with

• Proven experience in Data engineering role with at least 1-5 years of experience

• High sense of ownership and strong decision making skills backed by first principles thinking

• Strong analytical skills to break down the complex problems and build the right solutions

• Good knowledge on data engineering concepts such as ETL pipelines, building data lake, building data warehouses/ data marts, Stream and event processing, distributed processing of large scale data.

• Experience building and optimising big data' data pipelines, architectures and data sets.

• In depth knowledge in data warehousing concepts and familiarity with at least one of the warehousing tools such as Amazon Redshift, Snowflake and more.

• Strong SQL knowledge and understanding of performance tuning techniques

• Good knowledge of big data tools such as Hadoop, Apache Spark, Apache Druid, S3, Glue, Athena, Flink, Airflow, etc.

• Good exposure on streaming technologies like Kafka, SQS, Kinesis

• Good understanding of SQL and NoSql databases

• Experience with Amazon web services or Google Cloud Platform.

• Proficient in Java, Scala or Python.

• Experience in machine learning workloads is a plus

• Experience in supporting and working with product managers, decision scientists and data analysts in a dynamic environment