Improved fusion method based on ambient illumination condition for multispectral pedestrian detection
着卷积神经网络的发展, 基于多光谱图像的行人检测研究取得了巨大进步, 且应用广泛.最近研究表明, 融合由多光谱相机(可见光和热成像相机)捕获的图像信息可以使行人在强光和弱光条件下的检测都变得鲁棒.然而, 如何根据光照条件有效地融合图像◥信息仍缺乏进一步的研究.本文提出了一种多层次特征提取方法, 旨在从不同特征层提取有用信息.同时, 还提出一种置信度融合机制, 测量多光谱图像的光照情况.采用一个融合函数对双流网络输出的分类结果和RPN输出的分类结果进行融合, 提高行人检测的性能.通过实验将所提出的多¤光谱光照感知检测R-CNN(MIAD-RCNN)与现有的多光谱行人检测器进行比较, 该方法在全天候均具有较低的漏检率和较快的速度.Abstract:
With the development of convolutional neural network, the research of pedestrian detection based on multi-spectral images has made great progress and has been widely used. Recent studies have shown that the fusion of image information captured by multispectral sensors (visible and thermal imaging cameras) makes pedestrian detection robust in both good and poor lighting conditions. However, there is still a lack of further research on how to effectively fuse image information according to lighting conditions. This paper presents a novel multi-feature extraction method aiming at extracting useful information from different feature levels. Further, this paper also proposes a novel classification score fusion mechanism. By measuring the illumination in-formation of the input multispectral images and using a fusion function, the classification results output from the two-stream classification network and RPN are combined to improve the performance of pedestrian de-tection. In conclusion, compared with the latest multispectral pedestrian detectors through experiments, the proposed multispectral illumination-aware detection R-CNN (MIAD-RCNN) achieves a lower miss rate and a faster speed during both day time and night time.
表 1 不同层级的特征图的光照感知分类的准确性比较
白天 夜晚 多层次特征图 96.96% 94.48% 第五层特征图 98.90% 97.99%
表 2 和原始RPN+下游CNN分类器的漏检率对比
Reasonable- all Reasonable- day Reasonable- night 可见光图ω 像流分类 39.89% 32.75% 54.53% 红外图像流分类 32.50% 34.10% 30.24% 置信度融合 25.67% 26.75% 24.84% 置信度融合+多层次特征提取 24.92% 25.21% 24.17%
表 3 和原始RPN+下游CNN分类器的漏检率对比
Reasonable- all Reasonable- day Reasonable- night ACF+T+THOG 54.80% 51.97% 61.19% Halfway Fusion 37.19% 37.12% 35.33% Fusion RPN+BDT 29.68% 30.51% 27.62% FRPN-Sum+TSS 26.67% 26.75% 25.24% IATDNN+IAMSS 26.37% 27.29% 24.41% MIAD-RCNN 24.92% 25.21% 24.17%
表 4 不▓同检测方法的速度对比
方法 Halfway Fusion Fusion RPN+BDT FRPN- Sum+TSS IATDNN+ IAMSS MIAD- RCNN 时间 0.398 s 0.780 s 0.230 s 0.226s 0.218 s
 ZHANG L L, LIN L, LIANG X D, et al. Is faster R-CNN doing well for pedestrian detection?[C]//14th European Conference on Computer Vision. Amsterdam: Springer, 2016: 443-457. DOI: 10.1007/978-3-319-46475-6_28.  REN S Q, HE K M, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems. Cambridge: ACM, 2015: 91-99.  BRAZIL G, YIN X, LIU X M. Illuminating pedestrians via simultaneous detection and segmentation[C]//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 4950-4959. DOI: 10.1109/ICCV.2017.530.  HWANG S, PARK J, KIM N, et al. Multispectral pedestrian detection: benchmark dataset and baseline[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 1037-1045. DOI: 10.1109/CVPR.2015.7298706.  WAGNER J, FISCHER V, HERMAN M, et al. Multispectral pedestrian detection using deep fusion convolutional neural networks[C]//24th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. Bruges: ESANN, 2016.  LIU J J, ZHANG S T, WANG S, et al. Multispectral deep neural networks for pedestrian detection[M]//WILSON R C, HANCOCK E R, SMITH W A P. Proceedings of the British Machine Vision Conference. British: BMVA Press, 2016. DOI: 10.5244/C.30.73.  GUAN D Y, CAO Y P, YANG J X, et al. Fusion of multispectral data through illumination-aware deep neural networks for pedestrian detection[J]. Information Fusion, 2019, 50: 148-157. DOI: 10.1016/j.inffus.2018.11.017.  KONIG D, ADAM M, JARVERS C, et al. Fully convolutional region proposal networks for multispectral person detection[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu: IEEE, 2017: 49-56. DOI: 10.1109/CVPRW.2017.36.  GUAN D Y, CAO Y P, YANG J X, et al. Exploiting fusion architectures for multispectral pedestrian detection and segmentation[J]. Applied Optics, 2018, 57(18): D108-D116.