Nowadays, natural disasters pose a challenge due to their increasing frequency and destructive scale, which results in human and economic losses. One of the reasons for this damage is the slow response time due to a lack of information during the disaster. This gap can be filled by spatial imagery, which contains important disaster insights that can be analyzed using the latest advances in deep learning, which is much faster than manual interpretation by humans. These disasters can be classified into three categories based on their type and characteristics. The first is the hydro-meteorological category, which includes floods and cyclones, representing 73.5% of the disasters reported in 2024. We propose a Hydro-Meteorological Damage Assessment Backbone (HMDAB-Net) which combines a GA-optimized scene semantic segmentation core and a disaster-focused Mixture of Experts (MoE). Experimental results highlights an ensemble model which shows promising results towards facilitating rescue plan.