Researchers believe they have eliminated the problem of camera operators appearing in shot during live sports games.
A team from Kaunas University of Technology in Lithuania has come up with an AI model that will remove cameramen in real time if and when they appear in a shot.
The scientists say that when a camera operator appears on screen, it “detracts from critical game moments” and could lead to “revenue losses for broadcasters because of viewer dissatisfaction.”
To solve this visual distraction during the game, researchers have developed an “end-to-end system” that detects video operators and removes them in a similar way a photo editor might remove an object from a photograph.
But to eliminate an object from a live picture is a huge challenge and to meet it, the researchers turned to a state-of-the-art object detection system called YOLOv8 which is known for its speed and accuracy.
YOLOv8 is short for “You Only Look Once”. It can detect and classify objects in a single pass, making it ideal for real-time events such as live sports broadcasts.
“It works by dividing the image into a grid and predicting bounding boxes, class probabilities, and segmentation polygons for each grid cell. This enables it to identify and segment cameramen,” says member of the research team Serhii Postupaiev.
To help guide the YOLOv8 model to accurately detect and segment cameramen during games, the team created a dataset.
“I created this dataset to include a diverse range of cameramen with different sizes, shapes, and types of equipment, captured under various conditions and at different stages of the game. Now YOLOv8 uses this dataset to identify where cameramen are in the video frames,” says Postupaiev.
The team utilized video inpainting which is a type of deep learning that operates much like the Spot Healing Brush Tool.
Artificial intelligence (AI) and computer vision-based technology analyze the video frames to detect unwanted cameramen and fills the removed areas with relevant background details. The modified frames are then streamed back to viewers.
Thanks to the slight delay of a live broadcast, the algorithm processes the recorded image before it is shown live on air just a few seconds later.
“The broadcast will feel more polished and professional without disruptions caused by cameramen appearing where they shouldn’t. This improvement will reduce the number of cases where important moments of the game are missed due to distracting shots,” highlights Postupaiev, who received a Master’s degree from Kaunsas University of Technology with this project.
While the project mainly looked at soccer, the technology applies to all dynamic games including basketball, football, and ice hockey.
Postupaiev says that it could also be added retrospectively into old recordings of classic games.
It’s not the first time broadcasters have looked at hiding cameramen during sports broadcasts, a skating cameraman camouflaged in white has been going out onto the ice during hockey games.
The full paper can be found here.
Image credits: Header photo licensed via Depositphotos.