Location
PolandRate
Years of experience
5+About
I am an experienced Data Engineer with over five years of diverse experience in Information Technology. My expertise includes the development and implementation of various Big Data applications, primarily in cloud environments, and fostering a DevOps culture with CI/CD practices. I excel in both team settings and independent work, thanks to my strong communication skills and problem-solving abilities. I have successfully designed and implemented complex ETL pipelines, automated data processing, and created CI/CD pipelines in Azure DevOps, significantly improving efficiency and reliability. My career includes roles at BI4ALL, Procter & Gamble, Atos, and Nissan Sales CEE, where I developed scalable Big Data applications, optimized databases, and built data pipelines using tools like Databricks, Azure, and AWS. I hold multiple certifications, including Databricks and Microsoft credentials, and a Master’s degree in Computer Science with a specialization in Data Engineering from Warsaw University of Technology. Additionally, I am proficient in several programming languages, databases, and data visualization tools, and have a keen interest in automotive, sports, and emerging technologies like blockchain and cryptocurrencies.Tech Stack
Cloud, AWS, Azure, Big Data, Docker, Python, SQLExperience
- Designed and implemented complex ETL pipelines using Databricks (Python, PySpark, Spark SQL) and Azure Data Factory to efficiently ingest and transform data from multiple sources into a Data Lake and Azure SQL Database.
- Developed and maintained a highly scalable Big Data application using Python, Spark SQL, PySpark, and deployed it on Azure Kubernetes Service with Spark pods implementation, handling 100 million records per processing.
- Built and managed Data Pipelines and Jobs with Databricks (Delta Live Tables, PySpark, Python, SQL) and Azure Data Factory to streamline data processing.
- Created CI/CD pipelines in Azure DevOps to automate development implementation with Databricks and Azure Data Factory, enabling faster and more reliable deployment of data pipelines.
- Led a team of Data Engineers in refactoring the application architecture and implementing the Big Data App solution, utilizing Agile methodology.
- Optimized PostgreSQL and MSSQL databases to improve query performance and efficiency in data processing tasks.
- Automated data processing and improved efficiency using Databricks Jobs and Data Pipelines (DLT) with Spark, enhancing overall data processing accuracy.
Employment history
● Successfully designed and implemented a complex ETL pipeline usingDatabricks(Python,PySpark,SparkSQL)andAzureData Factory to efficiently ingest and transform data from multiple sources into a Data Lake and Azure SQLDatabase.
● Automated data processing using Databricks Jobs and Data Pipelines (DLT), improving overall efficiency and accuracy of the data processing with Spark.
● Created CI/CD pipelines in Azure DevOps to automate developmentimplementationwithDatabricksandAzureData Factory, enabling faster and more reliable deployment of data pipelines.
● Developed and maintained a highly scalable Big Data application using Python, Spark SQL, PySpark and deployed it on Azure Kubernetes Service with Spark pods implementation (100M records per processing)
● Created a Docker-based app development environment (Python, Spark, YAML) and integrated it with Redis Cache for efficient app development.
● Created customlog queries usingAzureLogAnalyticsQuery Language (KQL) to search, filter, and analyze app performance.
● Ingested data with Python, PySpark, Spark SQL and optimised query, processing using Spark.
● Optimized PostgreSQL database to improve query performance.
● Built and managed Data Pipelines and Jobs with Databricks (Delta LiveTables,PySpark,Python,SQL) and Azure DataFactory to streamline data processing.
● BuiltandmaintainedCI/CDpipelinesusingAzureDevOpsand managed infrastructure environments with Terraform.
● Led a team of 2 Data Engineers in the refactoring of the application architecture and implementation of the Big Data App solution, utilizing Agile methodology.
● Designed and implemented an ETL pipeline withDatabricks (Python, Pyspark, Spark SQL) and Azure Data Factory to load data into MySQL database.
● Created reports and dashboards with Tableau and PowerBI for data-driven decision-making.
● Assisted increating MachineLearningmodelsforpredictive analytics.
● UtilizedAzureSynapseAnalyticsandAzureLogAnalyticsto troubleshoot and resolve data processing issues
● Built scripts using Python, PySpark and Spark SQL to schedule Databricks Jobs for financial calculations and task automation.
● Developed and maintained data models in MSSQL database, migrating to Azure SQL Database.
● Built data pipelines with Azure DataFactory for data processing.
● Implemented Azure Functions with Python and Spark for data extractions.
● BuiltAzureDevOpsprojectstructure andCI/CDpipelinesfor consistent deployment of code and infrastructure. Integrated environments
Education history
● Master's thesis: "NoSQL databases in Big Data solutions".
● Engineering's thesis: "Project of software to automatically select electric cars".
● Head of Software department in Scientific Association Proton Dynamic and Aria.