Improving Clay's Data Platform

Clay had several poorly performing Databricks jobs that needed improvement, and wanted to include Elasticsearch in their data platform. Due to many changes within the development team there was no consensus on how to build and maintain the solution.

As a data engineer I was responsible for building a component for Elasticsearch, improving the development practices, improving CI/CD and reworking the poorly performing Databricks jobs.