Prepare dashboards and reports using Elasticsearch to enhance data visibility and facilitate decision-making. Optimize Elasticsearch indexes to improve query performance and reduce latency. Design and develop conceptual and logical data models, complete with detailed flowcharts to visualize data relationships and processes. Install and configure Apache Airflow on Kubernetes as the orchestration tool for the data platform. Develop APIs to securely share data across stakeholders, ensuring seamless data access and collaboration. Design and implement a robust ETL pipeline to transform over 1TB of unstructured data into structured formats, enabling efficient querying and analysis in data warehouses. Proactively monitor and enhance the performance of the legacy general ledger system by optimizing SQL queries, resulting in substantial reductions in job execution times. Develop and maintain data models in FSCD (SAP) pipelines with DBT and Airflow.

Successfully onboarded three finance metrics datasets into the client's data ecosystem, leveraging Snowflake, AWS Glue, AWS Lambda, and GitHub for seamless integration. Implemented a robust ETL process to ingest data from Excel files received via the client's existing mail server, ensuring efficient and accurate data transformation and loading into Snowflake. Conducted comprehensive testing, including manual, automated, and end-to-end tests, to ensure the reliability and accuracy of data pipelines, achieving seamless integration with existing datasets. Collaborated with cross-functional teams to integrate new data sources with existing systems, enhancing data accessibility and utility for business intelligence and analytics purposes.

Spearheaded the development and ongoing management of cutting-edge data models within FSCD (SAP) pipelines, utilizing the power of DBT and Airflow to ensure seamless data orchestration and transformation. Pioneered the creation and continuous maintenance of data ingestion processes from Jira and Confluence sources to construct a dynamic data mart, enhancing developers' performance visibility and decision-making capabilities. Proactively monitored and enhanced the performance of the legacy general ledger system by meticulously optimizing SQL queries, resulting in a substantial reduction (3-hour reduction) in long-running job execution times, significantly contributing to operational efficiency and data processing speed. Developed and maintained data models in FSCD (SAP) pipelines with DBT and Airflow.

Spearheaded the development and continuous optimization of an ETL pipeline using AWS Glue, resulting in a 50% reduction in both runtime and operational costs. Played a pivotal role in crafting and maintaining intricate Airflow DAGs, ensuring a reliable and robust workflow orchestration system to support complex data processes. Led the development and ongoing maintenance of data marts within Redshift, facilitating the creation of insightful dashboards and business intelligence reports using DBT, enabling data-driven decision-making. Processed real-time streaming customer behavior data from the website backend using DynamoDB and Kafka, enabling rapid insights into user actions and preferences, ultimately enhancing customer experience. Spearheaded the creation of interactive dashboards to analyze customer behavior, utilizing key metrics such as CTR/CVR across all web pages, leveraging tools like Metabase and QuickSight, fostering data-driven strategies to improve user engagement. Implemented a robust notification system, enabling timely alerts and communication on pending bookings, data ingestion failures, and more, through email and Slack channels, contributing to seamless data operations and issue resolution. Drove research and implementation efforts for prompt engineering with ChatGPT, resulting in a notable 15-20% reduction in development time.

Orchestrated the development and continuous maintenance of a high-performance ETL ingestion pipeline, seamlessly connecting a Transactional Database (Azure) with the Data warehouse using AWS Lambda, AWS Step Function, and S3, resulting in efficient data transfer and synchronization across cloud platforms. Streamlined daily business operations by automating tasks using Logic Apps, significantly reducing manual effort by 60% and enhancing productivity across the organization. Contributed to the development and relentless optimization of data models in PowerBI, utilizing PowerQuery and DAX to craft impactful analytics solutions that facilitate data-driven decision making. Spearheaded the development and maintenance of data models within Redshift, deploying DBT to modernize architecture, ensuring scalable and efficient data transformation processes with a team of three Data Engineers and one Data Analyst. Provided support across various business units by addressing ad-hoc data requests, using data insights to solve complex issues and inform strategic decisions. Implemented a sophisticated notification system to promptly alert engineers of data mismatches between source systems and the data warehouse, reducing data quality issues reported by business users by 30%. Designed and executed data models specifically for an Inventory Management application, showcasing proficiency in crafting data structures optimized for inventory tracking and management.

Proficiently designed and managed ETL pipelines using SQL in conjunction with Pentaho, ensuring smooth data extraction, transformation, and loading processes, contributing to data accuracy and accessibility. Spearheaded the development and ongoing maintenance of Spark jobs, optimizing big data transformation using on-premise clusters of Spark and Hadoop, resulting in enhanced data processing capabilities and improved analytics outcomes. Conducted extensive research into the functionality and architecture of Customer Data Platforms (CDP), demonstrating a commitment to staying at the forefront of innovative data management solutions, ultimately contributing to informed decision-making. Prepared data for the creation of dynamic dashboards and Business Intelligence (BI) reports in Tableau, leveraging telecommunications data to provide valuable insights and support data-driven strategies within the organization.

Practice batch processing with Hadoop and HiveQL using advertising data. Program automatic tasks using the Luigi workflow management framework. Create an on-premise environment to handle structured data from various sources. Research and implement cookies web technology. Implement fingerprinting on web/mobile applications.