Data Engineer carrier path

[Lena10]

Hello,

I want to start a career as a Data Engineer. I have read extensively about it and possess foundational skills in PostgreSQL and Pandas. Currently, I am learning Scrapy. However, I struggle to envision how these tasks translate into real job scenarios. For instance, if my chief assigns me the task of collecting prices from an online shop, and I use Scrapy to gather the data, should I create tables in Pandas or in PostgreSQL? Moreover, what if a data scientist works with MySQL or MongoDB? What is the most convenient program for the chief or clients to receive the data?

 

[GatorsFan]

A data engineer builds the pipelines to connect data input to analysis, how you build it is up to you. The most convenient is the one your most comfortable with, while some companies may want you to build in certain ways

Data Engineer | Codecademy

A data engineer builds the pipelines to connect data input to analysis. Includes **Python 3**, **SQL**, **pandas**, **PySpark**, **Git**, **MongoDB**, and more.

www.codecademy.com

 

[Lena10]

A data engineer builds the pipelines to connect data input to analysis, how you build it is up to you. The most convenient is the one your most comfortable with, while some companies may want you to build in certain ways

Data Engineer | Codecademy

A data engineer builds the pipelines to connect data input to analysis. Includes **Python 3**, **SQL**, **pandas**, **PySpark**, **Git**, **MongoDB**, and more.

www.codecademy.com

GatorsFan, thank you!

I understood the core concept and program that I need to learn. I was thinking that ChatGPT would replace data engineers. But now I see that I was worried for nothing. I see lots of jobs in this area.