Rapid urbanization leads the building sector to embrace a strategic role in energy conservation and carbon reduction. While the urban 3D compact form influences building carbon emissions by modulating the microclimate, existing studies lack a systematic quantification of the “form–microclimate–carbon emissions” pathway and effective spatial optimization strategies. This study developed a 1-km gridded database of urban 3D compact form indicators (NVCI), Urban Heat Accumulation (UHA), and building energy consumption carbon emissions (BECCE) from 1995 to 2020 in Xiamen. A Bayesian-parameter-convolution-optimized Random Forest model (RFBPC) was proposed to analyze the pathway. The results showed that (1) the RFBPC model consistently demonstrated superior performance over RF and XGBoost across all test cases, with testing R² values reaching as high as 0.993 (vs. 0.879 for RF) and RMSE reductions of up to 75% (from 0.583 °C to 0.144 °C); (2) UHA exhibited a significant mediating effect between NVCI and BECCE, accounting for 13.88% of the variation in BECCE; (3) the optimal compactness transition path was identified using a geographical detector, showing that shifting NVCI from Level 5 (high compactness) to Level 2 (low compactness) could yield BECCE reduction of 401.74 kg per grid unit; and (4) scenario projections indicated that such compactness optimization could advance the building sector’s carbon peak in Xiamen by 2.56 years. This study provides a quantifiable and actionable modeling framework and planning reference for coastal cities to achieve carbon-peak goals.