Databricks Developer – Resume Sample in PDF

Advised on Data Lake maintenance and expansion (banking sector, EU, as Lead Data Engineer):
• Apache Spark / Airflow / AWS development (process / code / architecture) for Data Lake.
• Built analytical platform upon Databricks on AWS (resolving matters like scalability/data security in the cloud/ IaC automatization)
• Reduced prod AWS EMR processing costs by 25% and decreased downtime by 37%.
Built a data processing framework for FHIR format compliant data (medical sector, US).
• Developed FHIR format – Azure – Databricks integration framework (also automated cucumber / pytest-bdd test framework)
• Troubleshot Delta Live Tables jobs
Implemented a PoC for Azure Databricks-based Data Lake (e-commerce, PL).
• Designed ELT processes (pyspark, Databricks Workflows).
• Created CICD processes for schema migrations, workflows, cluster pools, etc. Designed Apache Airflow architecture for an MFT business case (energy sector, PL).

Developed custom Apache Spark listeners (FMCG)
• Led project.
• Gathered logs produced by Spark jobs on Databricks.
• Visualized and pointed out weak spots, cost generators, and suboptimal queries.
Master Data Engineering (FMCG)
• Migrated SAP-based ETL to Microsoft Azure.
• Built from scratch data processing engine (Databricks + Airflow + ADLS + Docker).
• Built REST APIs connecting the engine’s components.

Big Data Engineering (Financial Services)
• Developed a solution responsible for orchestrating workflows from data vendors (public and private sources, both structured and unstructured) to a machine learning engine.
• Reviewed pull requests, distributed tasks to subordinates, and supervised them.
• Planned and executed data migration from HDFS to Azure Blob Storage.
• Optimized Apache Spark jobs and HDFS storage.

Created store chain expansion model (Retail):
• Designed and implemented a machine learning workflow responsible for the prediction of store income based on geographical and internal data.
Cloudera Hadoop cluster administration:
• Configured nodes / roles, installed / updated software.
• Performance monitoring, and troubleshooting.
• Prepared and maintained a working environment for Data Scientists (JupyterHub, Cloudera Data Science Workbench, mlflow, RStudio Server, etc.)
• Completed Cloudera Administrator Training for Apache Hadoop

• Developed KPIs tracking Shiny application
• Developed a process responsible for handling loans assignment to external debt collectors.
• Refactored an LGD calculation model from Excel based to a standalone Shiny dashboard.

• Assisted in the development and maintenance of various financial and operational reports, ensuring accurate data collection and presentation for internal review and decision-making processes.
• Conducted data analysis to support various departments, identifying trends and insights to aid in strategic planning and operational improvements.
• Provided administrative support to the team, including scheduling meetings, preparing documentation, and assisting with project management tasks to ensure smooth and efficient operations.