Responsibilities

• Review and understand business requirements ensuring that development tasks are completed within the timeline provided and that issues are fully tested with minimal defects
• Partner with a software development team to implement best practices and optimize performance of Data applications
• Collaborate across the company and interact with our customers to define, design and showcase new concepts and solutions
• Research on new Big Data technologies, assessing maturity and alignment of technology to business and technology strategy
• Collaborate with other developers to ensure that client needs are met at all times
• Work in a rapid and agile development process to enable increased speed to market against a backdrop of appropriate controls
• Implement good development and testing standards to ensure quality of deliverables
• Rapidly understand and translate clients’ business challenges and concerns into a solution-oriented discussion

Requirements

• At least 4+ years of experience in design and development using Data Engineering technology stack and programming languages
• Mandatory experience on following areas:
• Python/Scala/Java
• Hadoop, HDFS, MR
• Spark SQL, Dataframes, RDDs
• SQL
• Hive / Snowflake/SQL Server/Bigquery
• Good knowledge of the current landscape and ability to visualize technology and industry trends
• Working knowledge of Big Data Integration with Third party / in house built Metadata Management, Data Quality, Master Data Management Solutions.
• Active community involvement through articles / blogs / speaking engagements at conferences

Good to Have

• Spark Streaming, Spark ML
• Kafka/Flume
• Apache NiFi
• Apache Airflow/Oozie
• Worked with Cloud Based Data Platforms 
• NoSQL Databases ­ HBase/Cassandra/Neo4j/MongoDB
• Elastic Search