Resources
Post a Job
 Special Offer 

Jumpstart your hiring with a $75 credit to sponsor your first job.*

Sponsored jobs are 4.5X more likely to result in a hire.**
  • Invite top candidates to apply with Instant Match
  • Get more visibility in search results
  • Appear to more candidates longer
*The $75 Sponsored Job credit offer is only available for new accounts in the US that post a job and expires one year after account creation. Upon expiration of the credit, users are charged based on Sponsored Job budget. Terms, conditions, and quality standards apply **Indeed data, worldwide

Data Engineer Job Description: Top Duties and Qualifications

Last updated: September 27, 2023

A Data Engineer, or Data Systems Engineer, is responsible for developing and maintaining data processing software like databases. Their duties include coordinating with company Executives and other professionals to create unique data infrastructure, running tests on their designs to isolate errors and updating systems to accommodate changes in company needs.

Build a Job Description
Are you a Job Seeker? Find Jobs

Data Engineer Duties and Responsibilities

In addition to creating and maintaining an optimal pipeline architecture, typical duties and responsibilities for a Data Engineer position may include: 

  • Assembling large, complex sets of data that meet non-functional and functional business requirements
  • Identifying, designing and implementing internal process improvements including re-designing infrastructure for greater scalability, optimizing data delivery, and automating manual processes
  • Building required infrastructure for optimal extraction, transformation and loading of data from various data sources using AWS and SQL technologies
  • Building analytical tools to utilize the data pipeline, providing actionable insight into key business performance metrics including operational efficiency and customer acquisition
  • Working with stakeholders including data, design, product and executive teams and assisting them with data-related technical issues
  • Working with stakeholders including the Executive, Product, Data and Design teams to support their data infrastructure needs while assisting with data-related technical issues
Build a Job Description
Are you a Job Seeker? Find Jobs

Data Engineer Job Description Examples

Example 1

At Intellibus, we are an innovative technology consulting firm made up of passionate software engineers.

We are currently seeking an ETL Developer with strong experience in SQL, Python, Java, Data Warehouse, and building scalable ETL pipelines in AWS.

You will be working with individuals with Global Industry Experience to build Serverless Data Pipelines in AWS Cloud.

Qualifications
  • 8+ Years Relevant Experience

  • Strong Experience in Design, Development, Implementation, Unit Testing, Troubleshooting and Support of ETL Process Using Talend and Other In-House Tools

  • Experience in AWS Environment

  • Strong Experience in SQL Query Design and Implementation

  • Experience in DWH Concepts and other DWH Related Terms

  • Experience in SQL Performance Tuning and Query Optimization by Indexing, Partitioning and Denormalization

  • Well Versed with Migration

  • Work with Stakeholders and Understand their Requirements

  • Working Knowledge on BI Reporting (preferably Jasper)

    Job Types: Full-time, Contract

    Pay: $50.00 - $75.00 per hour

    Schedule:
  • Monday to Friday

    Experience:
  • SQL: 5 years (Preferred)
  • DWH: 5 years (Preferred)
  • UNIX: 5 years Location: One location

    • Example 2

    Are you a Data Engineer working at a Large Financial Institution and being told by your leadership that you are too hands-on or detail-oriented or think and work like a start-up?

    Imagine working at Intellibus to engineer platforms that impact billions of lives around the world. With your passion and focus we will accomplish great things together!

    We are looking forward to you joining our Platform Engineering Team.

    Our Platform Engineering Team is working to solve the Multiplicity Problem. We are trusted by some of the most reputable and established FinTech Firms. Recently, our team has spearheaded the Conversion & Go Live of apps that support the backbone of the Financial Trading Industry.

    We are looking for Engineers who can

  • Create Data modeling
  • Work on Snowflake modeling roles, databases, schemas, ETL toolswith cloud-driven skills
  • Work on SQL performance measuring, query tuning, and database tuning
  • Handle SQL language and cloud-based technologies
  • Set up the RBAC model at the infra and data level.
  • Work on Data Masking / Encryption / Tokenization, Data Wrangling / ECreLT / Data Pipeline orchestration (tasks).
  • Setup AWS S3/EC2, Configure External stages and SQS/SNS
  • Perform Data Integration e.g. MSK Kafka connect and other partners like Delta lake (data bricks)

    We work closely with

  • Data Wrangling
  • ETL
  • Talend
  • Jasper
  • Java
  • Python
  • Unix
  • AWS
  • Data Warehousing
  • Data Modeling
  • Database Migration
  • ECreLT
  • RBAC model
  • Data migration

    Our Process

  • Schedule a 15 min Video Call with someone from our Team
  • 4 Proctored GQ Tests (
  • 30-45 min Final Video Interview
  • Receive Job Offer

    If you are interested in reaching out to us, please apply and our team will contact you within the hour.

    Job Type: Full-time

    Pay: $60.00 - $80.00 per hour

    Schedule:
  • Monday to Friday



    Experience:
  • Data Wrangling: 7 years (Preferred)
  • SnowFlake: 7 years (Preferred)
  • ETL: 7 years Location: In person

    • Example 3

    Antra started its services in the year 2008, since then we have been diligently serving every need of our clients professional and software solutions. With our unique amalgam of innovation and inspiration, backed by Agile processes, we deliver world-class services with our core values to our clients. We identify unique strategies to transforming organizations through picking their sweet spot for innovative thinking and helping them in implementing in-depth cultural shift; ultimately augmenting the innovation process to the next level fostering the innovation race that never ends.

    This role requires the design, development and implementation of data solutions to business problems. An Azure Data Engineer will be expected to perform duties such as: evaluating the performance of current data solutions, designing and implementing cloud and hybrid data solutions. Ability to adapt and learn new technologies per business requirements is also needed.

    Responsibilities:

  • Design and implement data solutions using industry best practices.

  • Performs ETL, ELT operations and administration of data and systems securely and in accordance with enterprise data governance standards.

  • Monitor and maintain data pipelines proactively to ensure high service availability.

  • Works with Data Scientists and ML Engineers to understand mathematical models and optimize data solutions accordingly.

  • Continuous development through training and mentorship programs.

  • Create scripts and programs to automate data operations.

    You meet our must haves for this role if you have:

  • Minimum Bachelor's degree in Data Science, Business Intelligence, Computer Science or related fields, or the equivalent combination of education, professional training, and work experience.

  • 0-3 years of experience working with one or more languages commonly used for data operations including SQL, Python, Scala and R.

  • Experience working with relational databases such as SQL Server, Oracle and MySQL.

  • Experience working with NoSQL databases such as Redis, Mongo DB, Cosmos DB.

  • Excellent problem-solving skills and ability to learn through scattered resources.

  • Thorough understanding of the responsibilities and duties of a data engineer, as well as established industry standards/best practices and documentation guidelines.

  • Outstanding communication skills, and the ability to stay self-motivated and work with little or no supervision.

  • Willing to relocate to any US location on Antra projects location.

  • Authorization(s) to work lawfully in the United States (OPT/CPT is acceptable).

    Plus, if you meet any the of requirements:

  • Experience with cloud-based data technologies.

  • Experience with distributed systems utilizing tools such as Apache Hadoop, Spark or Kafka.

  • Working experience in Agile Scrum environments.

  • Experience with source control tools such as Git, SVN and TFS.

    The intent of this position description is to provide a representative summary of the major duties and responsibilities performed by incumbent(s) in this position. Incumbent(s) may not be required to perform all duties in this description and incumbent(s) may be required to perform position-related tasks other than those specifically listed in this description.

    Antra Inc. is an equal opportunity employer, providing equal employment opportunities (EEO). All employees and applicants for employment will receive consideration for employment without regard to race, color, religion, sex, national origin, age, disability, genetics, sexual orientation, gender identity or expression, pregnancy, protected veteran status or other status protected by law. This policy applies to all terms and conditions of employment.

    Job Types: Full-time, Contract

    Pay: $60,000.00 - $68,000.00 per year

    Benefits:
  • 401(k)
  • Dental insurance
  • Health insurance
  • Health savings account
  • Life insurance
  • Paid time off
  • Professional development assistance
  • Referral program
  • Relocation assistance
  • Vision insurance
    Schedule:
  • Monday to Friday

    Ability to commute/relocate:
  • Washington, DC: Reliably commute or planning to relocate before starting work (Preferred)

    Education:
  • Bachelor's Location: Multiple Locations

    • Example 4

    Title: Data Engineer
    Location: Remote
    Type: C2C
    Visa Status: Any
    Experience: 8+Yrs

    Requirements :

  • Bachelor's or Master's degree in Computer Science, Information Systems, Engineering or equivalent.

  • 5-6+ years of relevant experience with detailed knowledge of data warehouse technical architectures, infrastructure components, ETL/ ELT and reporting/analytic tools.

  • 3+ years of experience with Cloud based DW such as Redshift, Snowflake etc.

  • 3+ years' experience in Big Data Distributed ecosystems (Hadoop, SPARK, Hive & Delta Lake)

  • 3+ years experience in Workflow orchestration tools such as Airflow etc.

  • 3+ years' experience in Big Data Distributed systems such as Databricks, AWS EMR, AWS Glue etc.

  • Leverage monitoring tools/frameworks, like Splunk, Grafana, CloudWatch etc.

  • Experience with container management frameworks such as Docker, Kubernetes, ECR etc.

  • 3+ year's working with multiple Big Data file formats (Parquet, Avro, Delta Lake)

  • Experience working on CI/CD processes such as Jenkins, Codeway etc. and source control tools such as GitHub, etc.

  • Strong experience in coding languages like Python, Scala & Java

    Knowledge and Skills

  • Fluent in relational based systems and writing complex SQL.

  • Fluent in complex, distributed and massively parallel systems.

  • Strong analytical and problem-solving skills with ability to represent complex algorithms in software.

  • Strong understanding of database technologies and management systems.

  • Strong understanding of data structures and algorithms

  • Database architecture testing methodology, including execution of test plans, debugging, and testing scripts and tools.

  • Strong analytical and problem-solving skills.

    Nice to Have

  • Experience with transformation tools such as dbt.

  • Have experience in building realtime streaming data pipelines

  • Experience in pub/sub streaming technologies like Kafka, Kinesis, Spark Streaming etc.

    Job Type: Full-time

    Salary: $55.00 - $60.00 per hour

    Experience:
  • Spark,Python and SQL: 5 years (Preferred)
  • Scala,ELT,ETL and AWS: 5 years (Preferred)
  • AWS Glue,Docker and Kubernetes: 5 years (Preferred)
  • Data Engineer: 5 years All IT: 8 years (Preferred)

    • Example 5

    Senior Data Engineer

    Location: Remote

    Salary: $105,000 - $130,000

    Targetsmart Senior DATA Engineer

    A TargetSmart Senior Data Engineer's role is to lead and implement data engineering projects, support and maintain data pipelines, and provide expertise and best practices regarding data engineering for staff across the company. Typical data engineering projects focus on improving performance and adding feature to existing data pipelines. As needed, the Senior Data Engineer will design and develop new data engineering pipelines as part of the Data Engineering Team. Further, the Senior Data Engineer will help decide how and implement improvements to pipeline, systems, and infrastructure.

    Senior Data Engineers are expected to have an in-depth understanding of data engineering best practices as well as expertise with the tools needed to debug and diagnose issues. Senior Data Engineers contribute as a team member to testing, QA, and documentation of data pipelines and systems.

    The TargetSmart Data Engineering Team is a small agile team of motivated individuals who welcome challenges, adapt quickly, strive to acquire new knowledge, learn new technologies, accept new responsibilities, and work well individually, as a team, and with other teams within the organization. You will work closely with the Data Engineering Lead, product managers, workflow developers and other software engineers to advance projects and meet objectives.

    We seek an experienced engineer that is interested in pursuing an advanced individual contributor technical career track over a technical management career track.

    Senior Data Engineer Expectations: *

    Maintain in-depth understanding of relevant data engineering best practices

    Display expertise with tools needed to debug and diagnose issues

    Given a medium to large understood problem, can design, implement, and deploy a solution

    Show initiative and help when needed without being asked

    Deliver feedback in a constructive manner

    Provide guidance to other engineers and workflow developers

    Work well with technical leads, incorporating feedback as needed

    Support TargetSmart's culture and mission as a leading Democratic & Progressive political data and data services firm.

    Required Skills:

    AWS: S3 Data Lake, Athena, Redshift, EMR, Glue, ECS

    Proficiency with Python in a data engineering context. Ex: Pandas, PySpark, Petl

    Source control using Git

    Proficiency with SQL

    Proficiency with workflow orchestration concepts

    Adaptable to Windows, Linux, and container-based deployment environments

    Workflow performance, scaling, and optimization. Big Data solutions in some cases.

    Security & privacy principals

    Desired Skills & Qualifications

    Domain knowledge related to United States electoral data and consumer data

    Workflow orchestration platforms: Prefect ([website])

    ETL productivity tools: Alteryx ([website])

    About TargetSmart Communications

    TargetSmart is a leading provider of political data and technology that enables campaigns and organizations to successfully communicate with large audiences, personalize outreach, and create lasting relationships. Our superior politically-focused, consultative approach combines consumer data, databases, data integration and consulting solutions for personalized multichannel marketing strategies. TargetSmart leverages over 25 years of experience in data management to deliver high-performance, reliable data products and solutions.

    Job Type: Full-time

    Pay: $105,000.00 - $130,000.00 per year

    Benefits:
  • 401(k)
  • 401(k) matching
  • Dental insurance
  • Employee assistance program
  • Flexible spending account
  • Health insurance
  • Life insurance
  • Paid time off
  • Parental leave
  • Professional development assistance
  • Vision insurance
    Schedule:
  • Monday to Friday
    Supplemental Pay:
  • Bonus pay

    COVID-19 considerations:
    This is a remote position

    Experience:
  • SQL: 5 years (Required)
  • AWS: 5 years (Required)
  • Python: 5 years Location: Remote

    • Example 6

    Requirements:

    Complexity - Performs varied work that may be somewhat difficult in character, but usually involves limited responsibility. Works on problems and projects of moderate scope and complexity.
    Knowledge- Demonstrates a broad working knowledge of discipline-specific principles. May have detailed knowledge in a specific area.
    Discretion- Interprets policy and exercises discretion and independent judgment on most tasks.
    Supervision - Works under broad supervision of manager.
    Experience- May have Bachelor degree or equivalent with approximately 3 - 5 years professional work experience. May have some work toward postgraduate degree in related field.
    *Educational Requirements:Bachelor's degree in Information Technology or Computer Science
    Certifications required - Microsoft SQL Server Certifications Desired: Microsoft Certified Azure Data Engineer Associate
    Summary of Position: Performs activities related to design, development, as well as day-to-day administration and maintenance of various database systems. Designs and coordinates changes to databases. Codes, tests, and implements physical databases for relational and non-relational database solutions including CosmosDB, MongoDB, SQL Server, etc. Handles installations, upgrades, backups, and configuration. Confers with appropriate staff regarding problems with and capabilities of systems.
    Accountability:
    20% Design logical and physical databases and review changes to design to understand how changes affect physical databases (data storage in terms of physical aspects such as location, amount of space, and access method.)
    20% Design, implement and monitor MongoDB, CosmosDB, and SQL Server workloads across multiple enterprise applications. Implement Power BI solutions along with other Azure Data solutions where necessary
    15% Perform performance tuning by modifying database programs to increase processing efficiency. Calculate optimum values for database parameters
    10% Configure and integrate databases with other information systems. Design and write complex SQL Server stored procedures
    10% Work with vendors and database developers to correct recurring problems with applications and systems
    10% Implement security procedures to ensure that databases remain protected. Specify user access level for each segment of one or more data items such as insert, replace, retrieve, or delete data.
    10% Provide technical support to various departments and users with database problems in order to maximize database utilization.

    5% Review contracts for database support, services, and products. Research, evaluate, and recommend new databases.

    Job Type: Full-time

    Pay: $83,465.00 - $110,000.00 per year

    Benefits:
  • 401(k) matching
  • Dental insurance
  • Employee assistance program
  • Employee discount
  • Flexible schedule
  • Flexible spending account
  • Health insurance
  • Health savings account
  • Life insurance
  • Paid time off
  • Referral program
  • Tuition reimbursement
  • Vision insurance
    Schedule:
  • Monday to Friday

    Experience:
  • Informatica: 1 year (Preferred)

    License/Certification:
  • Microsoft SQL Server Certifications (Required)
  • Microsoft Certified Azure Data Engineer Associate Location: Remote

    • Example 7

    Job description

    Responsibilities:

    Provides plan with data, reporting, and analyses that enable data-driven decision-making.
    Provides summary analyses in written and oral presentation settings.
    Builds database from scratch. And prepares complex presentations.
    Develops system test cases and documents results, researches system issues and documents findings.

    Requirements:
  • Understanding of database objects (entities, attributes, constraints, relationships, joins, etc.)
  • Good logic-building skills.
  • Ability to write DDL/DML/DQL (any RDBMS) queries.
  • Good technical presentation & delivery skills.
  • Exposure to any programming language (C, C++, VB, Python, R, etc.)
  • Ability to translate business requirements into non-technical, lay terms.
  • High-level written and verbal communication skills.
  • Experience with other relational databases, BI reporting and data discovery tools are preferred (PowerBI, Tableau, SQL Server, Snowflake, MS Azure).

    Education:
  • Bachelor's or Master's degree in Science, Technology, Engineering, or Mathematics.
  • Work Location: Multiple locations
  • Work Remotely (Temporarily due to COVID-19)

    Job Types: Full-time, Contract, Internship

    Salary: $60,000.00 - $72,000.00 per year

    Benefits:
  • 401(k)
  • Dental insurance
  • Health insurance
  • Life insurance
  • Referral program
  • Relocation assistance
  • Vision insurance
    Schedule:
  • 8 hour shift
  • Monday to Friday

    COVID-19 considerations:
    Due to COVID-19, the Bootcamp will be remote

    Experience:
  • SQL: 1 year (Preferred)
  • DBMS Concepts: 1 year (Preferred)
  • Data modeling: 1 year Location: Multiple Locations

    • Example 8

    Title: Sr Data Lead Engineer (Apache Flink/AWS/Big Data)

    Work location: Jersey city, New Jersey

    Duration:12 months

    Position type: W2 contract

    Job Description:

  • We are looking for min 12 to 15 + years of experience for the Data Lead engineer position.
  • Expert in Cloud Services (Apache, Flink/AWS/Big Data )and design technique as well as experience working across large environments with multiple operating systems/infrastructure for large-scale programs(e.g., Expert Engineers) starting to be firm-wide resources.
  • working on projects across Business.
  • Is multi-skilled with expertise across software development lifecycle and toolset.
  • May be recognized as a leader in Agile and cultivating teams working in Agile frameworks.
  • Sought out as coach for at least one technical skill.
  • Strong understanding of techniques such as Continuous Integration, Continuous Delivery, Test Driven Development, Cloud Development, resiliency, security.
  • Stays abreast of cutting-edge technologies/trends and uses experience to influence application of those technologies/trends to support the business; may give speeches and outside the firm, writes Articles.

    Job Type: Contract

    Salary: $51.66 - $87.12 per hour

    Experience level:
  • 11+ years
    Schedule:
  • 8 hour shift
  • Monday to Friday



    Ability to commute/relocate:
  • Jersey City, NJ 07302: Reliably commute or planning to relocate before starting work (Required)

    Application Question(s):
  • Comfortable to Work on W2
  • Only local candidate of Jersey City/New York

    Experience:
  • Apache: 8 years (Preferred)
  • Big Data: 8 years (Preferred)
  • AWS: 8 years Location: In person

    • Example 9

    Role:Data Engineer with GCP and Hadoop

    , TX *(100%Onsite only)*

    Role Description

    Duration: 1 Yr



    Strong experience in Spark, need to be able to tune the spark jobs.

    Strong python programmer.

    Very good experience Bigdata( In GCP it is data proc), programming and hadoop admin level.

    Google cloud platform experience like BQ, Dataproc,servveless,cloud funtions, GCR etc

    Good at orchestration tolls like cloud composer

    Good experience Kbernetes (GKE or EKS )

    Experience in Docker images and corresponding repositories.

    CICD experience like Jenkins.

    Need good experience in how machine learning models work , what are different algorithms, how each model performance will be measured etc. Entire life cycle of machine learning model.

    Job Type: Contract

    Salary: $91,482.45 - $186,496.98 per year

    Benefits:
  • 401(k)
  • Dental insurance
  • Health insurance
  • Vision insurance
    Compensation package:
  • Bonus opportunities
    Experience level:
  • 10 years
  • 11+ years
  • 8 years
  • 9 years
    Schedule:
  • 10 hour shift

    Work In person

    • Example 10

    Junior Big Data Engineer



    Company Overview:

    We are a leading technology-driven company based in Atlanta, GA, dedicated to providing innovative solutions that leverage big data and advanced analytics. Our mission is to transform raw data into actionable insights, empowering our clients to make data-driven decisions for their business growth and success. We are currently seeking a highly skilled and motivated Big Data Engineer to join our talented team and play a pivotal role in building and maintaining our big data infrastructure.

    Job Description:

    As a Big Data Engineer, you will be responsible for designing, developing, and maintaining our big data platform and solutions. You will collaborate with cross-functional teams to understand data requirements, implement data pipelines, and optimize data processes to ensure scalability, reliability, and performance. Your expertise will be instrumental in enabling efficient data processing, storage, and analysis to support various business initiatives.

    Key Responsibilities:

  • Design, develop, and maintain our big data infrastructure, including data ingestion, processing, and storage systems
  • Implement and manage data pipelines for collecting, storing, and processing large volumes of structured and unstructured data from various sources
  • Work closely with data scientists and analysts to understand data requirements and ensure the availability of clean, high-quality data for analysis and reporting
  • Optimize and fine-tune big data systems for maximum performance, efficiency, and scalability
  • Troubleshoot and resolve data-related issues and performance bottlenecks in a timely manner
  • Collaborate with the data security team to implement appropriate data privacy and security measures
  • Research and evaluate emerging big data technologies and tools to enhance the company's data infrastructure and capabilities
  • Work with cross-functional teams to gather and define data requirements for new projects and initiatives
  • Develop and maintain documentation related to data engineering processes and systems.

    Qualifications:

  • Bachelor's or Master's degree in Computer Science, Software Engineering, or a related field
  • 4-6 years of experience working in a technology-based role
  • Proficient in languages such as Python, Java, Scala, or similar for developing data pipelines and data transformation
  • Hands-on experience with big data technologies such as Apache Hadoop, Apache Spark, Apache Kafka, and related ecosystems
  • Proficiency in working with NoSQL databases (e.g., MongoDB, Cassandra, HBase) and distributed storage systems (e.g., HDFS)
  • Solid understanding of data modeling, data warehousing, and ETL processes
  • Experience with cloud-based data platforms (e.g., AWS, Azure, GCP) is a plus
  • Familiarity with containerization technologies (e.g., Docker, Kubernetes) for deploying and managing big data applications
  • Strong problem-solving skills and the ability to troubleshoot complex data-related issues
  • Excellent communication and collaboration skills to work effectively within a team and across different departments

    Perks and Benefits:

  • Competitive salary and performance-based incentives
  • Comprehensive health, dental, and vision insurance plans
  • Generous paid time off
  • Opportunities for career growth and professional development
  • Access to cutting-edge technologies and tools in the big data space
  • Collaborative and inclusive work environment fostering innovation and creativity

    If you are a passionate Big Data Engineer with a strong background in designing and managing big data solutions, and you are excited about the potential of big data to drive business success, we invite you to apply and join our dynamic team in Atlanta, GA.

    Job Type: Full-time

    Pay: From $32.00 per hour

    Benefits:
  • 401(k)
  • 401(k) matching
  • Dental insurance
  • Health insurance
  • Life insurance
  • Paid time off
  • Professional development assistance
  • Referral program
  • Vision insurance
    Compensation package:
  • Bonus opportunities
  • Hourly pay
  • Performance bonus
    Experience level:
  • 1 year
  • 2 years
    Schedule:
  • 8 hour shift



    Education:
  • Bachelor's Location: In person

    • Example 11

    JOB Description:

    Industry: Insurance

    Job Category: Information Technology - Computer Network Security

    With a startup spirit and 90,000+ curious and courageous minds, we have the expertise to go deep with the world s biggest brands and we have fun doing it. Now, we re calling you all who see the world differently and are bold enough to reinvent it. Transformation happens here. Come, and be a part of our exciting journey!

    Qualifications:

  • 10+ years of experience in data engineering

  • 5+ years of experience managing a team of data engineers.

  • Strong understanding of SQL Server SSIS

  • Excellent communication and interpersonal skills

  • Ability to work independently and as part of a team.

  • Ability to meet deadlines and work under pressure.

  • Strong desire to be hands-on with the development and implementation of data pipelines and data warehouses.

  • Experience working with stakeholders to develop product backlog grooming, sprint planning data engineering, and QA Testing

  • Migration of SQL Server to Azure is desired.

  • Experience with Azure Data Factory, ADLS, and CI/CD highly desired

    The Hands-On Data Engineering Lead is responsible for leading a team of data engineers in the design, development, and implementation of data pipelines and data warehouses. The ideal candidate will have a strong understanding of data engineering principles and practices, as well as experience with SQL Server SSIS.

    The Hands-On Data Engineering Lead will also be responsible for managing the team's workload, mentoring junior engineers, and communicating with stakeholders. Additionally, the Hands-On Data Engineering Lead will be expected to be hands-on with the development and implementation of data pipelines and data warehouses and to have experience working with stakeholders to develop product backlog grooming, sprint planning data engineering and QA Testing.

    JOB Responsibilities:

  • Lead the design, development, and implementation of data pipelines and data warehouses.

  • Develop and implement data pipelines and data warehouses using SQL Server SSIS

  • Troubleshoot data pipelines and data warehouses.

  • Communicate with stakeholders to understand their needs and ensure that projects meet their requirements.

  • Manage the team's workload and ensure that projects are completed on time and within budget.

  • Mentor junior data engineers and help them develop their skills.

  • Stay up to date on the latest data engineering technologies and best practices.

  • Provide support to production teams and QA teams.

  • Work with stakeholders to develop product backlog grooming, sprint planning data engineering and QA Testing

  • Working with other engineers to design and implement data pipelines and data warehouses using Azure Data Factory, ADLS, and CI/CD

  • Experience in migrating SQL servers to Azure

  • 30% team management and 70% hands-on

    Compensation:

  • Full-time

  • Benefits - Full

  • Relocation Assistance Available - No

  • Commission Compensation - No

  • Bonus Eligible - No

  • Overtime Eligible - No

  • Interview Travel Reimbursed - No

    Candidate Details

  • 10+ to 15 years experience

  • Seniority Level - Mid-Senior

  • Management Experience Required - No

  • Minimum Education - Bachelor s degree

  • Willingness to Travel - Occasionally

    Job Type: Full-time

    Pay: $130,000.00 - $145,000.00 per year

    Experience level:
  • 10 years



    Application Question(s):
  • Are you willing to share with us at least 2 recommendation letters? We can provide you with Location: On the road

    • What Does a Data Engineer Do?

    Data Engineers typically work for corporations in various industries to help them manage and store important data regarding company operations. They work closely with other data professionals and members of the information technology department to design and implement these systems across departments. Their job is to use their knowledge of different programming languages to code and update data systems. They may also be responsible for implementing cybersecurity

    Data Engineer Skills and Qualifications

    A job description for a Data Engineer should contain the following skills and qualifications requirements:

    • Ability to build and optimize data sets, ‘big data’ data pipelines and architectures
    • Ability to perform root cause analysis on external and internal processes and data to identify opportunities for improvement and answer questions
    • Excellent analytic skills associated with working on unstructured datasets
    • Ability to build processes that support data transformation, workload management, data structures, dependency and metadata

    Data Engineer Salary Expectations

    The average annual salary expectations for a Data Engineer is $127,339 per year. This information is based on 4,091 anonymously submitted salaries to Indeed from employees and users along with past and present Data Engineer job postings on Indeed within the last 36 months.

    Data Engineer Education and Training Requirements

    A Data Engineer position usually requires a hybrid of educational requirements, starting with a degree in information technology or computer science with additional vendor-specific certification.

    Google’s Certified Professional-Data-Engineer certification establishes that an individual is familiar with the principles related to data engineering and can function as an associate or professional within the industry.

    Considered a gold standard by many in the industry, the IBM Certified Data Engineer – Big Data certification focuses on applications specific to big data instead of general skills.

    Specific to Cloudera’s solutions, the CCP Data Engineer for Cloudera certification shows the individual has proven experience in ETL analytics and tools.

    Secondary certifications, including the Microsoft Certified Solutions Expert (MCSE), include a wide range of topics with specific sub-certifications including MCSE: Data Management and Analytics.

    Data Engineer Experience Requirements

    Many companies look for candidates with more than five years of industry experience who have a graduate degree in information systems, informatics, statistics, computer science or another quantitative field.

    Applicants should also have a demonstrated understanding and experience using software and tools including big data tools like Kafka, Spark and Hadoop; relational NoSQL and SQL databases including Cassndra and Pastgres; workflow management and pipeline tools such as Airflow, Luigi and Azkaban; AWS close services including Redshift, RDS, EMR and EC2; stream-processing systems like Spark-Streaming and Storm; and object function/object-oriented scripting languages including Scala, C++, Java and Python.

    Data industry events are excellent sources of information and provide an excellent networking opportunities.

    Job Description Samples for Similar Positions

    If this job description isn’t exactly what you’re looking for, consider these samples for similar positions:

    Ready to Hire a Data Engineer?Build a Data Engineer Job Description

    Frequently asked questions about Data Engineers

    What is the difference between a Data Analyst and a Data Engineer?

    The difference between a Data Analyst and a Data Engineer is their areas of job focus. For example, Data Analysts use data systems like databases to pull data regarding customer service, sales quotas, revenue streams or employee satisfaction. In contrast, Data Engineers use their coding skills to develop and update databases and other types of data infrastructure. Data Analysts work closely with data professionals like Data Engineers to streamline the data collection and retrieval process.

    What are the daily duties of a Data Engineer?

    On a typical day, a Data Engineer starts by reviewing upcoming deadlines and checking their email or phone messages to determine changes in assignment needs. Throughout the day, they participate in meetings with company Executives, IT personnel and department heads to establish the need for new databases or other data infrastructure to aid in data storage. Data Engineers use downtime in their office to code frameworks for new data systems or code updates to existing systems. They also determine the success of new systems or updates by visiting with individual departments and getting feedback.

    What qualities make a good Data Engineer?

    A good Data Engineer uses their knowledge of programming languages in combination with business needs to design, monitor and update data systems for corporations. They also have excellent interpersonal communication, allowing them to speak with employees from a range of departments to address technical problems or database needs. Further, a good Data Engineer always seeks to enhance their coding skills by taking certification courses and participating in training opportunities. A good Data Engineer also needs to have an investigative mindset that enables them to troubleshoot issues with data systems and isolate defects in data software.

    Job Description Examples

    Need help writing a job description for a specific role? Use these job description examples to create your next great job posting. Or if you’re ready to hire, post your job on Indeed.

    No search results found

      *Indeed provides this information as a courtesy to users of this site. Please note that we are not your recruiting or legal advisor, we are not responsible for the content of your job descriptions, and none of the information provided herein guarantees performance.