用于视频会议的多模态主动式扬声器检测和虚拟电影技术（multimedia）
- 2020 年 2 月 15 日
- 筆記
通过自动平移、倾斜和缩放视频会议摄像机，主动式扬声器检测(ASD)和虚拟电影技术(VC)可以显著改善视频会议的远程用户体验:用户对专家视频摄影师的视频的主观评价显著高于未编辑的视频。我们描述了一种新的自动化的主动式扬声器检测和虚拟电影技术，它在0.3 MOS的范围内执行一个专业的电影摄影师基于主观评分1-5级。该系统采用4K宽幅摄像机、深度摄像机和麦克风阵列;它从每种模式中提取特征，并使用AdaBoost机器学习系统来训练主动式扬声器检测，该系统非常高效并实时运行。虚拟电影技术也同样使用机器学习来优化整体体验的主观质量。为了避免分散房间参与者的注意力和减少切换延迟，系统没有移动部件——虚拟电影技术通过剪切和缩放4K宽视频流来工作。该系统使用广泛的众包技术进行了调优和评估，并在N=100个会议的数据集上进行了评估，每个会议时长为2-5分钟。
原文题目：MULTIMODAL ACTIVE SPEAKER DETECTION AND VIRTUAL CINEMATOGRAPHY FOR VIDEO CONFERENCING
原文：Active speaker detection (ASD) and virtual cinematography (VC) can significantly improve the remote user experience of a video conference by automatically panning, tilting and zooming of a video conferencing camera: users subjectively rate an expert video cinematographer’s video significantly higher than unedited video. We describe a new automated ASD and VC that performs within 0.3 MOS of an expert cinematographer based on subjective ratings with a 1-5 scale. This system uses a 4K wide-FOV camera, a depth camera, and a microphone array; it extracts features from each modality and trains an ASD using an AdaBoost machine learning system that is very efficient and runs in real-time. A VC is similarly trained using machine learning to optimize the subjective quality of the overall experience. To avoid distracting the room participants and reduce switching latency the system has no moving parts – the VC works by cropping and zooming the 4K wide-FOV video stream. The system was tuned and evaluated using extensive crowdsourcing techniques and evaluated on a dataset with N=100 meetings, each 2-5 minutes in length.
原文作者：Ross Cutler, Ramin Mehran, Sam Johnson, Cha Zhang, Adam Kirk, Oliver Whyte, Adarsh Kowdle
原文链接：https://arxiv.org/abs/2002.03977
