They have their overlaps, but data scientist and data engineering jobs are distinct in the needs they meet. While a data scientist analyses data, a data engineer designs and maintains the technological systems that make this analysis possible.
Both are absolutely essential in the data ecosystem, as this example would illustrate.
“A person employed with Google working on the search algorithm would be a data scientist. But the one who implements the algorithm across servers to ensure queries are answered in a split second is a data engineer,” explains Kunal Jain, founder and CEO, Analytics Vidhya.
The demand for data engineering is particularly on a sharply rising trend. The Dice 2020 Tech Job Report described data engineering as the fastest-growing career option in technology in 2019, citing a 50 per cent year-on-year growth in the number of open positions.
“A mix of skills (data science and data engineering) or a lot more of the latter is now preferred by companies,” says Kunal, an aero-science engineer turned data scientist. He cites the example of companies that were looking for talent in data science in a big way, two years ago. Now, the same companies have realised the necessity of having on board, data professionals who combine data science/machine learning with data engineering skills.
He adds that “for every three data scientists in an organisations, there is a need for 10 data engineers.”
Domain Knowledge
Viju Ravichandran, Education Technical Evangelist at MathWorks India, says data science will continue to move from theory to practice as more engineers and scientists understand the tools and techniques.
Domain expertise is key to understanding how everything can be done.
“Data science skills combined with domain expertise will play a crucial role in helping expand the reach of machine learning to new applications and fields. Additionally, simulation techniques that have historically been used for engineering design are now being used to generate synthetic data for data science workflows in cases where data is scarce or unavailable,” says Viju and adds that a data scientist can build more accuracy and intelligence into the system by incorporating relevant domain knowledge.
With some innovation in how they design their courses, universities can bridge the existing talent gap. A few years ago, All India Council for Technical Education (AICTE) directed all affiliated universities to start offering multi-disciplinary courses with special focus on emerging technologies.
- * The ability to to solve real-world problems across industry domains
- * Being well grounded in developer skills
- * PYTHON, R, SQL databases, cloud computing form the knowledge base
- * Hackathons and other competitions to solve real-world challenges can hone the abilities of budding data engineers
“Data science and machine learning will soon become an essential skill for engineers, and every discipline should look at introducing them in the curriculum,” says Viju.
Solving real-world problems through hackathons is another way of getting people interested in analytics.
Hiring parameters
For an upcoming “Job-a-Thon” organised by Analytics Vidhya from June 26 to July 4, HDFC Bank, one of the participating companies, is looking to hire data engineers with the following educational requirement: “MCA/MTech/MSc Statistics or Economics/BTech/MBA or equivalent in Computer Science/Statistics or related areas with strong inclination towards machine learning algorithms”.
But that is not to say every company defines the role the same way.
Arvind Saraf, head of applications engineering, Drishti Technologies, points out that statistics and mathematics are not essential for someone to be a data engineer, but they can certainly be helpful.
“What we look for is the candidate’s ability to build data science models and solve pressing business problems. Educational background is important but exposure to real-world problems would be a good launch pad for data engineers,” says Arvind.
The company also recruits candidates who have completed recognised data analytics/ data science certifications.
Says Arvind, “We encourage people who have handled various products to take up internal mobility in data science roles.”
