Collaborative Perception (CP) enables multiple agents to share environmental information to improve autonomous driving. Among various approaches, intermediate collaboration offers a good balance between perception accuracy and communication efficiency, yet still suffers from information loss caused by feature compression. Furthermore, existing methods rely on class-agnostic confidence pruning, which disproportionately removes features from certain object classes, and they experience significant performance degradation when collaborating agents become scarce. Despite their wide field of view and stable observations, infrastructure sensors are typically used only as auxiliary sources, leaving their advantages underutilized. This paper proposes an infrastructure-centric collaborative perception framework that incorporates semantic information to overcome these limitations. The framework consists of two components: Post- Semantic Fusion, which applies learnable class-importance weights to restore weakened features after fusion, and Prior-Semantic Distillation, which transfers semantic priors from vehicles to enhance infrastructure features, particularly in low-agent scenarios. Experiments on the V2X-REAL dataset demonstrate improvements of 5.7% for the vehicle class and 3.9% for the pedestrians class, with up to 8.5% gain in sparse-agent settings. The proposed methods also maintain stable performance under localization noise. In summary, this work presents a semantic-aware, infrastructure-centric CP framework that enhances robustness and scalability for cooperative autonomous driving, while providing a foundation for future research in multi-modal and class-balanced dataset generation learning extensions.