Today I decided to dive into the world of the Data Engineer and learn about what they do. I am calling myself a Data Engineer due to my extensive background of doing lots of ETL (Extract, Transform, and Load) work using SQL throughout my career in Data Management. The title of Data Engineer seems to fit a lot of what I used to do.
However, it is a brave new world, indeed, and the role of the Data Engineer is constantly changing. The term "Data Engineer" wasn't very common a few short years ago—even though it is exactly what I was doing. The best description I’ve heard for the Data Engineer role comes from a class I’m taking on Pluralsight, called "Becoming a Data Engineer". They describe the Data Engineer as a digital librarian, and that actually makes perfect sense.
If you consider what librarians do, they manage the books in a library. They organize, categorize, preserve the books—ordering new books when needed and making sure the right Dewey Decimal number is attached. Their entire career is based on curating and caring for books—ensuring they are ready, available, and easily accessible for anyone who needs them.
That is exactly what a Data Engineer does for data—except their work is all done digitally and can be automated to a significant extent through scripts and even AI. If you think about what Data Engineers do, they source data from multiple places (like flat files or API calls). They clean the data to remove errors, duplicates, and inconsistencies. They organize and categorize the data for other systems and/or people. Finally, they load the data into the appropriate locations.
See, Data Engineers are digital librarians. Ever since I was a kid, I always thought being a librarian would be a cool job. Today, I am drawn to a position that performs many of the same tasks—not for books, but for massive amounts of data. Not only that, but I get to play around with some very cool technology. It seems that if I am going to keep my fingers on the pulse of the tech world, being a Data Engineer might not be a bad place to start. After months of waiting and wondering what's next, this feels like a way to use what I know to learn something new.
God loves good stewardship, and a Data Engineer is all about being a good steward with the data they work with. The result of their effort is cleansed and organized data that can be used to drive real decision-making. Seems like a good use of talent, time, and technology to me. As I have acknowledged before, I have many interests across a broad spectrum. Data engineering seems to check a lot of boxes and is certainly something I plan on looking into further as a potential career path. I am starting with a few more Pluralsight models and maybe a small personal ETL project to test the waters.
What do you think? Do you think the librarian example fits with the Data Engineer? Why or why not? Have you ever found a role that unexpectedly fit your past experience like a glove? I'd love to hear what you think.
—Daniel