The deployment of Convolutional Neural Networks (CNNs) for plant species classification in agricultural and biodiversity monitoring applications requires robust interpretability to ensure reliable real-world performance. This study presents the first systematic analysis of visual bias in plant classification models using Gradient-weighted Class Activation Mapping (Grad-CAM) across five CNN architectures: Baseline CNN, Improved CNN, VGG16, ResNet50, and DenseNet121. We evaluated these models on plant species datasets to investigate potential biases in feature attribution patterns. Our analysis reveals a consistent and previously unreported bias toward light-colored plant features across all tested architectures, with models systematically focusing on bright leaves, flowers, and background elements while underutilizing darker plant components for classification decisions. This light-color dependency presents significant implications for deployment in diverse environmental conditions where lighting variations are common. Statistical analysis confirms the bias is architecture-independent, suggesting a fundamental limitation in current CNN training approaches for botanical applications. We provide methodological guidelines for bias detection in specialized computer vision domains and discuss implications for responsible AI deployment in agricultural systems. These findings highlight critical considerations for explainable AI in plant classification and establish a framework for identifying similar biases in domain-specific applications.