The CHI 2021 extended abstract 'Development of Blind Football Play-by-play System for Visually Impaired Spectators: Tangible Sports' (Ohshima, Kobayashi, and Shimada) introduced an automated play-by-play pipeline for blind football: a single elevated panoramic 4K camera and a deep-learning detection stage feed a 32 × 48-pin refreshable tactile display, with pin-vibration frequencies labeling teams and ball. The source research program's own statements — the 2018 ICCHP paper (Kobayashi, Fukunaga, and Shimada), the CHI 2021 abstract, and the authors' 2023 follow-up — define the open research problem: acquiring full-field positions in a form deployable for local matches, given the stated drawback that the position of a camera should be high enough to capture the entire field at once and that setting a tall pole to carry the camera might be dangerous and difficult to establish in a local game; combining the tactile channel with synthesized voice, the explicit future work of the CHI 2021 paper; and establishing kinesthetic (motion) presentation methods that support both team identification and continuous tracking, given the observed conflict between statically discriminable and easily tracked vibration frequencies — the transmission of kinesthesia that the authors' 2023 follow-up calls a major task of human interface research. Later literature — the authors' 2023 interface-improvement poster, audio-based tennis and audiotactile golf broadcasting systems, a human-operated vibrotactile football maquette, and 2024–2026 systematic reviews — has advanced adjacent pieces but not resolved this core. The problem is to build and validate a deployable, automated, real-time, multimodal (audio–tactile) play-by-play system that lets visually impaired spectators follow blind football, including the underlying presentation-method science and local-match-deployable full-field position acquisition.

Background

Blind football (also called 5-a-side football or blind soccer) is the Paralympic adaptation of football for players with visual impairment, governed internationally by the International Blind Sports Federation (IBSA); it is played five-a-side with an audible ball. Although the game is designed for blind players, spectators with visual impairment cannot follow it visually and depend on live play-by-play commentary — a running verbal description of players, ball, and game events. Asakawa and Hurst (ASSETS 2021 — the ACM SIGACCESS conference on computers and accessibility) interviewed 43 people with vision impairment about watching sports and documented both the information needs (who is where, what just happened) and the demand for non-visual channels beyond audio. Human commentary, however, does not scale to local matches, and verbal description conveys spatial layouts only serially. A research program at the Tokyo Metropolitan Industrial Technology Research Institute and Tsukuba University of Technology therefore pursued an automated play-by-play system for visually impaired spectators. Its first study (Kobayashi, Fukunaga, and Shimada, ICCHP 2018 — the International Conference on Computers Helping People with Special Needs) acquired player positions with the Quasi-Zenith Satellites System (QZSS — a Japanese satellite positioning augmentation system giving higher accuracy than standalone GPS around Japan) and showed them on a refreshable tactile display; that paper also stated the drawback of camera-based acquisition that "the position of the camera should be high enough to capture the entire field at once and setting a tall pole to carry the camera might be dangerous and difficult to establish in local game," and planned as future work "to add image processing to the input method and synthesized voice to the output method." The CHI 2021 (ACM Conference on Human Factors in Computing Systems) work by Ohshima, Kobayashi, and Shimada replaced dedicated instrumentation with broadcast-style sensing: a single professional panoramic camera records 4K video at 30 frames per second of the entire field; an image-processing and deep-learning pipeline detects the 10 players, 2 referees, and ball, converts detections to two-dimensional field coordinates, and streams them to a refreshable tactile display. A refreshable tactile display (RTD) is a device whose grid of raised pins is raised and lowered dynamically so that a blind reader perceives an updatable tactile 'picture' with the fingertips; pin vibration can add further labels. The 2021 hardware tiled four KGS SC-10 piezoelectric bimorph modules (display modules whose pins are driven by thin bending piezoelectric ceramic elements) into a 32 × 48 pin surface (2.4 mm pin spacing, roughly 50 ms full-surface refresh), rendering each player as a single pin and the ball as a 3 × 3 pin glyph. Two perceptual experiments on this hardware set the initial design knowledge: (i) pairwise subjective discrimination of statically vibrating 8 × 8-pin squares identified the subset {0, 1, 2, 8, 32} Hz (from the available set {0, 1, 2, 3, 4, 8, 16, 32} Hz) as discriminable, while adjacent higher pairs such as 8 vs 16 Hz and 16 vs 32 Hz were poorly distinguished (mean about 1.9 on a five-point scale); (ii) ranking of the ease of continuously tracking a single moving pin gave the order 16 Hz > 32 Hz > 8 Hz > 0 Hz > 4 Hz > 3 Hz > 2 Hz > 1 Hz, with lower frequencies making pins seem to disappear. These results expose the central perceptual tension: frequencies that are easy to discriminate statically are not the ones that are easy to track in motion. In an end-to-end trial the system rendered a single 7-second goal scene from a real match with the mapping 'scoring team 32 Hz, other team 8 Hz, ball 0 Hz'; six of seven blind or visually impaired participants reportedly identified team members and one reconstructed the formation. Kinesthesia — here, the conveyance of motion and continuously changing spatial layout through dynamic tactile stimulation — is the mechanism such a system depends on, and the authors call its transmission a major task of human-interface research. What the source program leaves open constitutes the present problem: the 2018 study stated the tall-pole drawback of camera-based capture quoted above, and the 2021 system relies on such a single high-mounted panoramic camera; the output is tactile-only, with an audio–tactile combination explicitly deferred to future work in the 2021 abstract; and the transmission of kinesthesia remains, in the authors' 2023 follow-up, a stated major task of human-interface research. The 2021 vision pipeline was characterized on footage of one event with ad hoc annotation thresholds, and its evaluation used a handful of participants and one brief scene.

Problem Statement

Complete the automated blind-football play-by-play system for visually impaired spectators so that it is deployable and validated, addressing the open directions stated by the source research program itself (Kobayashi, Fukunaga, and Shimada, ICCHP 2018; Ohshima, Kobayashi, and Shimada, CHI EA 2021; Ohshima, Kobayashi, and Shimada, HCI International 2023). The research objective is to build and validate one integrated, real-time pipeline that realizes all three components (1)–(3) on real blind-football match content. Within that pipeline, component (1) is a capture configuration that attains the one-pin-cell accuracy on real match content while remaining deployable for local matches, satisfying an operational deployment constraint (for example maximum mounting height, camera count, setup personnel and time, and on-site calibration) that excludes the tall-pole setup the source drawback identifies as dangerous and difficult to establish in a local game; component (2) is an audio–tactile rendering that specifies the content and timing of each modality and, as measured by the spectator evaluation described below, conveys the players, teams, and ball better than each single-modality baseline by a stated margin, without cross-modal conflict and without an increase in reported cognitive load; and component (3) is a vibration-encoding and update scheme for which the visually impaired-spectator evaluation described below supplies evidence that a single scheme supports static team identification and continuous tracking of moving players simultaneously. The common objective is one system: an automated, real-time channel that acquires the positions of all players and the ball across the entire field and presents them so that visually impaired spectators can follow game developments without a human commentator. Within that objective, three source-stated open components must be addressed. (1) Full-field capture: how can the positions of all players and the ball across the entire field be acquired in real time, with an accuracy set by the display's fixed spatial resolution (positions accurate to within one pin cell of the true field position on the 32 × 48 grid, i.e. about one pin-spacing at field scale), without requiring instrumentation carried by the players, in a form deployable for local matches — in light of the drawback stated in the 2018 ICCHP paper that the position of a camera should be high enough to capture the entire field at once and that setting a tall pole to carry the camera might be dangerous and difficult to establish in a local game? (2) Multimodal output: how should tactile rendering and synthesized voice be combined in content and timing so that game developments are conveyed better than either modality alone, overcoming the tactile channel's cognitive limits without overloading attention — the audio–tactile graphic system that the CHI EA 2021 paper explicitly defers to future work? (3) Kinesthetic presentation: which presentation methods for motion — vibration-encoding mappings, motion rendering, and update schemes on refreshable pin displays — simultaneously support static identification (such as team membership) and continuous tracking of moving players, given the observed conflict between the statically discriminable subset ({0, 1, 2, 8, 32} Hz) and the frequencies easiest to track (16 and 32 Hz) — the transmission of kinesthesia that the authors' 2023 follow-up frames as a major task of human-interface research? Validation follows the source program's own evaluation practice. The system is to be validated both technically and with visually impaired spectators. The technical validation runs the completed system end to end on real competitive blind-football match content and measures component (1)'s capture accuracy against the documented positional ground truth, with each component's criterion stated above met from the system's own measured output. The validation with visually impaired spectators has visually impaired spectators follow real match content rendered through the completed system, assesses their comprehension of game developments against at least the single-modality baselines (tactile-only and audio-only), and supplies the spectator-level evidence for component (3) that one encoding and update scheme supports static team identification and continuous tracking of moving players simultaneously, with the solver assembling that participant cohort. No match content or positional ground truth is supplied with the task: the solver must obtain real competitive blind-football match content and produce the positional ground truth by a documented annotation procedure that is independent of the completed system — annotated by people blind to its output or taken from an externally sourced or held-out reference, and may not adopt the pipeline's own detections as the evaluation reference — reporting the content's source and the annotation protocol. The capture-accuracy validation of component (1) must be run on match content captured with the solver's own deployable configuration, so that that configuration and its deployability are actually exercised; publicly archived full-length recordings of competitions such as the IBSA Blind Football World Championships and Paralympic blind-football matches are suitable only for the spectator validation, not for measuring capture accuracy.

References

  1. Hiroyuki Ohshima, Makoto Kobayashi, Shigenobu Shimada (2021). Development of Blind Football Play-by-play System for Visually Impaired Spectators: Tangible Sports. Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–6. DOI: 10.1145/3411763.3451737, https://doi.org/10.1145/3411763.3451737
  2. Makoto Kobayashi, Yoshiki Fukunaga, Shigenobu Shimada (2018). Basic Study of Blind Football Play-by-Play System for Visually Impaired Spectators Using Quasi-Zenith Satellites System. Computers Helping People with Special Needs (ICCHP 2018), Lecture Notes in Computer Science, pp. 23–27. DOI: 10.1007/978-3-319-94274-2_4, https://doi.org/10.1007/978-3-319-94274-2_4
  3. Hiroyuki Ohshima, Makoto Kobayashi, Shigenobu Shimada (2023). Improvement of User Interface of Blind Football Play-by-Play System for Visually Impaired Spectators-Tangible Sports. HCI International 2023 Posters, Communications in Computer and Information Science 1833, pp. 345–351. DOI: 10.1007/978-3-031-35992-7_48, https://doi.org/10.1007/978-3-031-35992-7_48
  4. Gaurav Jain, Basel Hindi, Connor Courtien, Xin Yi Therese Xu, Conrad Wyrick, Michael Malcolm, Brian A. Smith (2023). Front Row: Automatically Generating Immersive Audio Representations of Tennis Broadcasts for Blind Viewers. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST 2023), pp. 1–17. DOI: 10.1145/3586183.3606830, https://doi.org/10.1145/3586183.3606830
  5. Gaurav Jain, Basel Hindi, Connor Courtien, Conrad Wyrick, Xin Yi Therese Xu, Michael C. Malcolm, Brian A. Smith (2023). Towards Accessible Sports Broadcasts for Blind and Low-Vision Viewers. Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–7. DOI: 10.1145/3544549.3585610, https://doi.org/10.1145/3544549.3585610
  6. Haerim Kim, Minho Chung, Eunchae Kim, Yongjae Yoo (2024). Automatic Video-to-Audiotactile Conversion of Golf Broadcasting on A Refreshable Pin Array. Proceedings of the 2024 ACM Symposium on Spatial User Interaction, pp. 1–8. DOI: 10.1145/3677386.3682092, https://doi.org/10.1145/3677386.3682092
  7. Ana Grasielle Dionísio Corrêa, Gabriel Belapetravicius Dias, Raphael Gueleri Kalaes, Bruno da Silva Rodrigues (2025). Desenvolvimento de uma Maquete Vibrotátil para Acompanhamento Imersivo de Futebol por Deficientes Visuais: Um Relato de Experiência em IHC. Anais do XXIV Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais (IHC 2025). DOI: 10.5753/ihc.2025.10903, https://doi.org/10.5753/ihc.2025.10903
  8. Chutian Jiang, Emily Kuang, Mingming Fan (2024). How Can Haptic Feedback Assist People with Blind and Low Vision (BLV): A Systematic Literature Review. arXiv:2412.19105, https://arxiv.org/abs/2412.19105 (journal version: ACM Transactions on Accessible Computing 18(1), 2025, pp. 1–57. DOI: 10.1145/3711931, https://doi.org/10.1145/3711931)
  9. João Franco, Francisco Marques, Roberto Ribeiro, Anabela Marto, Alexandrino Gonçalves, Pedro Gago, Telma Garção, Joana Sousa, Nuno Rodrigues (2026). A systematic review of assistive technologies for blind and low-vision viewers of broadcasting sports. Computers in Human Behavior Reports 22, 101128. DOI: 10.1016/j.chbr.2026.101128, https://doi.org/10.1016/j.chbr.2026.101128
  10. Saki Asakawa, Amy Hurst (2021). “What just happened?”: Understanding Non-visual Watching Sports Experiences. Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2021), pp. 1–3. DOI: 10.1145/3441852.3476525, https://doi.org/10.1145/3441852.3476525
  11. Leona Holloway, Swamy Ananthanarayan, Matthew Butler, Madhuka Thisuri De Silva, Kirsten Ellis, Cagatay Goncu, Kate Stephens, Kim Marriott (2022). Animations at Your Fingertips: Using a Refreshable Tactile Display to Convey Motion Graphics for People who are Blind or have Low Vision. Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2022), pp. 1–16. DOI: 10.1145/3517428.3544797, https://doi.org/10.1145/3517428.3544797
  12. Samuel Reinders, Matthew Butler, Ingrid Zukerman, Bongshin Lee, Lizhen Qu, Kim Marriott (2025). When Refreshable Tactile Displays Meet Conversational Agents: Investigating Accessible Data Presentation and Analysis with Touch and Speech. IEEE Transactions on Visualization and Computer Graphics 31(1), pp. 864–874 (presented at IEEE VIS 2024). DOI: 10.1109/TVCG.2024.3456358, https://doi.org/10.1109/TVCG.2024.3456358
  13. Boryi Alexander Becerra Patiño, José Pino-Ortega, Jorge Olivares-Arancibia (2025). Exploratory review of the scientific production of 5-blind soccer with the Bibliometrix tool [Revisión exploratoria sobre la producción científica del fútbol 5 ciegos con la herramienta Bibliometrix]. Retos 68, pp. 1025–1047. DOI: 10.47197/retos.v68.115790, https://doi.org/10.47197/retos.v68.115790
  14. Yuta Ochiai, Ayumu Tsuji, Shimpei Aihara, Hiroyasu Iwata (2025). Virtual Reality Assessment of Auditory Spatial Perception and Navigation in Blind Football Players. 2025 IEEE International Workshop on Sport, Technology and Research (STAR), pp. 215–220. DOI: 10.1109/STAR66750.2025.11264764, https://doi.org/10.1109/STAR66750.2025.11264764