I have been trying to use my time without a day job as wisely as I can. There is the job searching and resume posting that I have been doing. Those two tasks are no small feat. I have also been honing my skills as a writer, as evident by the number of articles I have posted here. There is a third thing I have been doing lately as well. I have been doing a deep dive into Data Engineering as a technical position and career path.
After evaluating my 25+ year work experience, I felt it was appropriate to call myself a Data Engineer. I have been working with data in some form for most of that 25 years. I have done database maintenance, re-written more SQL queries and procedures than I can count. I have crafted SQL based steps to bring data into a database and conform it to specific database standards. Finally, I have written countless custom reports for data quality and analytics.
For all the work I have done in the data space, I believe that assuming the title of Data Engineer is appropriate. However, most of my work has been in aging or even obsolete technologies. So, while I might have a good grasp of the fundamental concepts of Data Engineering, I am out of date when it comes to tools and more modern ways of working.
So, I decided to dive in and learn what a Data Engineer is defined as today. It has been an eye-opening exercise so far. I mentioned this before, but the best analogy I found for a Data Engineer is a digital librarian. I wrote about this article. But as I investigate more, it goes deeper.
The Data Engineer is responsible for the sourcing, cleansing, transforming, and loading of data so it can be used. This includes every step in the ETL process—Extract, Transform, and Load. But it is so much more than this. Data Engineers need to be able to evaluate data sources to see if they are a good fit for the project. They need to do all the steps to make that data usable.
With the amount and variety of data available these days, SQL is just not enough anymore. I used to use SQL for the ETL work and VBA for the data visualization, but data and the tools have become so much more sophisticated that those older tools aren't enough. Data Engineers now have to deal with cloud databases and cloud data lakes. They have to be able to understand data modeling and data warehousing, and how to automate the pipelines to get the data from the source to the target. Cleansing and transforming the data along the way.
This demands more significant programming skills in languages like Python, R, Scala, and JavaScript. It also requires knowledge of both structured and unstructured database technologies like MongoDB and Snowflake. You also have to be familiar with big data technologies like Apache Spark and real time processing with applications like Amazon Kinesis. This is just a small sample.
So, I am diving in and trying to learn some of these technologies. Taking online classes is priority number one. However, beyond that, I have to begin working on my own pet projects in order to build an effective portfolio to share my work and show my knowledge. It is a slow process, but it will be well worth the effort.
What do I hope to get out of all of this effort? As a life-long learner, knowledge is always a great payoff. With this, I hope to either land the position of my dreams or be able to jump into the world of freelance consulting and become a solopreneur. Ideally, working for myself would be the ultimate end goal. The reality of that is yet to be seen. I will continue to work towards that goal, but if the right position comes along, I would be short-sighted if I didn't at least consider it.
That's where I am today. Honing my craft as a Data Engineer. The writing will remain a constant. It's too much fun and I find it very therapeutic. Not to mention I really like sharing my thoughts with others. Always hoping that what I write you find informative and entertaining. As for the data stuff, it's hard to just throw away years of experience. I just don't want to trash all that tech experience, so I need to update and bring myself up to today's standards. That's the best I can do, and let God handle the rest.
Do you have any skills or experience that you thought were obsolete? Were you able to take those skills and enhance them to enhance yourself and find new and exciting opportunities? Sharing your story would make my day.
—Daniel