DataIntel Water Leak Detection System - Australia
2015 - 2021
The DataIntel IoT system processes real-time data from a vast array of sensors embedded within water pipe networks to perform analytics and detect water leaks. This solution has been successfully deployed in regions including Singapore, South Australia, New Zealand, Malaysia, Sharjah, and the UK.
Designed and transitioned the existing system from a monolithic structure to microservices using Vert.x, Spring Boot, MongoDB, ElasticSearch, and Docker, enhancing the scalability and performance to handle an increased number of sensors for each customer.
Suggested and devised a new ETL system using Talend to replace old data importers, enabling connectivity to diverse data sources such as MQTT, SFTP, MongoDB, SQL DB, APIs, and MAXIMO, with minimal development effort and improved data transformation efficiency for high-volume data.
Developed a new ETL jobs management system using Airflow to replace previous Linux crontab scheduling, improving maintenance ease with features like a friendly UI, auto-retry, email notifications, smart logging, and comprehensive reporting.
Led the drive for new feature development and technology selections through all stages of the product lifecycle.
Collaborated closely with product owners to understand requirements and deliver appropriate solutions.
Implemented APIs using MuleSoft to facilitate data access for third parties and customers.
Provided expertise directly to customers, helping them identify features that deliver maximum business value.
Technologies: Vert.x, Spring Boot, MongoDB, ElasticSearch, Docker, Talend, MQTT, SFTP, MongoDB, SQL DB, APIs, MAXIMO, Airflow, MuleSoft