Data Platform Development - International
This project is a SaaS solution operating in the logistics domain, which has been stable since early 2021. It processes more than 100 million records daily.
Implement a Data Platform based on Datalake, Delta Lake, Spark/DataBricks, and Aiven.io.
Apply the GitOps model starting early 2022, utilizing Git, Codefresh, and ArgoCD.
Increase availability and predictability of test environments used by three development teams.
Create over 50 custom Ansible roles and playbooks for software installations and application deployments.
Ensure availability of Production and Development systems, primarily based on Amazon EC2/ECS and RKE2.0.
Build data pipelines on the Spark Framework using Python, Databricks, Kafka streaming, and Aiven.io.
Develop applications using Django and PHP.
Technologies: Datalake, Deltalake, Spark/DataBrick, Aiven.io, GitOps, Git/Codefresh/ArgoCD, Ansible Roles, Playbooks, Amazon EC2, ECS, RKE2.0, Spark Framework/ Python, Databrick, Kafkastreaming, Aiven.io, Django, PHP