Researchers develop AI tech that can monitor pitchers using low-resolution video captured by smartphones

University of Waterloo researchers have developed new artificial intelligence (AI) technology that can accurately analyze pitcher performance and mechanics using low-resolution video of baseball games.

The system, developed for the Baltimore Orioles by the Waterloo team, plugs holes in much more elaborate and expensive technology already installed in most stadiums that host Major League Baseball (MLB), whose teams have increasingly tapped into data analytics in recent years.

Those systems, produced by a company called Hawk-Eye Innovations, use multiple special cameras in each park to catch players in action, but the data they yield is typically available to the home team that owns the stadium those games are played in.

To add away games to their analytics operation, as well as use smartphone video taken by scouts in minor league and college games, the Orioles asked video and AI experts at Waterloo for help about three years ago.

The result is a comparatively simple system called PitcherNet, which overcomes challenges such as motion blurring to track the movements of pitchers on the mound, then yields data on metrics including pitch velocity and release point from standard broadcast and smartphone video.

A paper on the project, PitcherNet: Powering the Moneyball Evolution in Baseball Video Analytics, was presented at the 2024 IEEF/CVF Conference on Computer Vision and Pattern Recognition and is available on the arXiv preprint server.

“The Orioles approached us with a problem because they weren’t able to analyze pose positions and, subsequently, the biomechanics of their pitchers at games that may not have access to high-resolution cameras,” said Dr. John Zelek, a professor of systems design engineering and co-director of the Vision and Image Processing (VIP) Lab at Waterloo.

“The goal of our project was to try to duplicate Hawk-Eye technology and go beyond it by producing similar output from broadcast video or a smartphone camera used by a scout sitting somewhere in the stands.”

To help train AI algorithms at the heart of the technology, researchers created three-dimensional avatars of pitchers so their movements could be viewed from numerous vantage points.

Information from video processed by the system is provided to biomechanics analysts for the Orioles, who have committed to jointly funding the project for another year.

That data can be used to adjust how pitchers throw the ball to improve performance or avoid injuries, and assess the future success and durability of pitching prospects.

“Existing technology has already improved baseball analytics,” said Jerrin Bright, a Ph.D. student who had a leading role in the project. “Since it’s limited to home games, however, there is a real need for solutions that work in any setting, especially for scouting. That’s where our system comes in.”

Researchers are now exploring the application of the underlying idea—AI analysis of player poses using standard broadcast and smartphone video—to other professional sports, including hockey and basketball, in addition to other aspects of baseball, such as batting.