Extending text-driven video montage beyond themed text: arbitrary text scope, professional fine-grained control, and scalable cinematography-aware assembly (the Write-a-Video future-work agenda)

Write-a-Video (Wang, Yang, Hu, Yau, and Shamir, ACM Transactions on Graphics 2019) generates a cinematographically coherent video montage from a themed narrative text and a video repository by cascaded keyword filtering plus visual-semantic embedding retrieval, followed by hybrid dynamic-programming assembly under cinematographic idiom controls. Its Discussion states two future-research directions — supporting a wider scope of input texts (highly specific texts about rare people, objects, or actions; abstract or multi-themed texts) through more advanced visual-semantic matching, and finer-grained artistic control including intentional violations of cinematographic guidelines, with deterministic export as an Edit Decision List (EDL) for professional post-editing — while its optimization section reports that the assembly solver is only a tractable heuristic (worst-case complexity $O(L\bar{T}^2|\bar{S}|!)$, mitigated by tenth-frame cut sampling and a four-shot-per-segment cap). Later research has partially absorbed each thread — keyword-free weakly-supervised retrieval and real-time sequencing (Transcript to Video), arbitrary-script montage (ICMR 2023), learned cut plausibility (Learning to Cut), language-driven editing agents (LAVE, ExpressEdit, VideoDiff, Vidmento), and the generative text-to-video paradigm (Make-A-Video, Imagen Video) — but no work delivers the integrated, repository-grounded system the agenda describes.

Background

Video montage — a sequence of short shots assembled to illustrate a narrative or theme, familiar from music videos and travel films — is traditionally produced with frame-level editing software that requires both manual labor and cinematographic expertise. A research line on computational video editing automates parts of this process: QuickCut (Truong et al., UIST 2016) supports interactive editing of narrated video through a fixed script, and Leake et al. (ACM TOG 2017) introduced idiom-based editing for dialogue-driven footage, where a cinematographic idiom is a named editing convention (for example a preference for camera-movement continuity or a target shot length) that the editor sets and the system enforces.

Write-a-Video (Wang et al., ACM TOG 2019, SIGGRAPH Asia) is a tool in this line that takes a themed narrative text and a related video repository (from online websites or personal albums) and produces a montage mostly through text editing. It segments the text, retrieves candidate video shots — a shot being one continuous camera take between two cuts — by first filtering with user-supplied keywords and then ranking with a visual-semantic embedding, i.e. a learned joint text–image feature space in which semantically matching content lies close together (the paper uses VSE++, an embedding trained with a triplet loss that explicitly mines the hardest negative examples). Assembly is posed as a hybrid optimization over a graph of shots with shot-level terms (semantic match, camera stability, tone), cut-level terms (avoiding jump cuts — accidental near-identical consecutive framings — opposite camera motion, and tonal discontinuity), and segment-level idiom terms (movement and shot-duration preferences), solved by dynamic programming. To keep optimization interactive, candidate cuts are only considered at every tenth frame and at most four shots per segment are used; the paper reports the unrestricted worst-case complexity as $O(L\bar{T}^2|\bar{S}|!)$, where $L$ is the number of text segments, $\bar{T}$ the average temporal length of segments, and $|\bar{S}|$ the average number of candidate shots per segment, so the practical solver is a heuristic without approximation guarantees.

The authors' Discussion section states the limitations that define this problem. The tool targets themed texts that are neither too specific nor too abstract: if the input text is too specific — involving particular people, subjects, or actions — good video matches may not be found in the repository, and if the text is too abstract or combines several themes, the text-to-video embedding space is hard to learn; the authors propose that more advanced visual-semantic matching algorithms could support a wider scope of texts. Professional cinematography sometimes intentionally breaks standard guidelines to express artistic style (for example, intentional jump cuts with small shot lengths, favored in selfie videos), which the fixed idiom set and weights cannot express; the authors call for more fine control and note the outputs could be exported as an Edit Decision List (EDL) — the standard machine-readable list of clip decisions, in/out points, and transitions exchanged between editing systems — for further professional frame-based adjustment. Together with the reported combinatorial worst case of the assembly optimizer, these limitations define an agenda for repository-grounded, text-driven montage generation.

The surrounding field has since transformed: weakly-supervised retrieval, learned cut plausibility, LLM-based editing agents, and above all generative text-to-video models (Make-A-Video, Imagen Video, and successors) that synthesize footage for arbitrary prompts. Understanding this problem requires distinguishing progress on retrieval-based, repository-grounded montage — where footage must come from an existing collection — from the adjacent capability of generating new footage.

Problem Statement

Stated source-faithfully from Write-a-Video's stated limitations (Wang et al., ACM TOG 2019, Discussion section, together with the complexity facts of its optimization section), the problem is to remove the three limitations of repository-grounded, text-driven montage generation as one coherent objective:

1. Wider scope of supported text. Extend the visual-semantic matching between narrative text and a video repository so that montage generation works for input texts beyond moderately themed ones — specifically (a) highly specific texts involving particular people, subjects, or actions, for which matching shots may be unavailable in the repository, and (b) abstract texts or texts combining several themes, for which the text-to-video embedding space is hard to learn — while handling the case where suitable footage does not exist.

2. Fine-grained artistic control with professional interoperability. Provide richer and finer control than a fixed set of cinematographic idioms and weights — in particular, controls that support professional cinematography's intentional violations of standard guidelines (such as deliberate jump cuts with very short shot lengths or unconventional transitions) — together with deterministic export of the montage as an Edit Decision List (EDL) for further professional frame-based adjustment in commercial software.

3. Generalization and scalability of assembly. Make the cinematography-aware assembly optimization scale beyond the original heuristic regime — whose worst-case complexity is $O(L\bar{T}^2|\bar{S}|!)$ (where $L$ is the number of text segments, $\bar{T}$ the average temporal length of segments, and $|\bar{S}|$ the average number of candidate shots per segment) and whose interactive behavior relies on sampling candidate cuts at every tenth frame and capping shots at four per segment — to very long narratives and extremely large candidate sets.

These three capabilities are to be achieved jointly for montage generation from a given video repository; the single overarching objective is to make text-driven, cinematography-aware montage generation applicable to arbitrary narrative text, to professional workflows, and to large-scale repositories. Generative text-to-video synthesis, which creates new footage rather than assembling an existing repository, does not by itself answer this question.

References

1. Miao Wang, Guo-Wei Yang, Shi-Min Hu, Shing-Tung Yau, Ariel Shamir. Write-a-video: computational video montage from themed text. ACM Transactions on Graphics 38(6), Article 177, 2019. DOI: 10.1145/3355089.3356520. https://doi.org/10.1145/3355089.3356520
2. Anh Truong, Floraine Berthouzoz, Wilmot Li, Maneesh Agrawala. QuickCut: An Interactive Tool for Editing Narrated Video. Proceedings of the 29th ACM Symposium on User Interface Software and Technology (UIST 2016), 497–507. DOI: 10.1145/2984511.2984569. https://doi.org/10.1145/2984511.2984569
3. Mackenzie Leake, Abe Davis, Anh Truong, Maneesh Agrawala. Computational Video Editing for Dialogue-driven Scenes. ACM Transactions on Graphics 36(4), 130:1–130:14, 2017. DOI: 10.1145/3072959.3073653. https://doi.org/10.1145/3072959.3073653
4. Fartash Faghri, David J. Fleet, Jamie Ryan Kiros, Sanja Fidler. VSE++: Improving Visual-Semantic Embeddings with Hard Negatives. BMVC 2018. arXiv:1707.05612. https://arxiv.org/abs/1707.05612
5. Yu Xiong, Fabian Caba Heilbron, Dahua Lin. Transcript to Video: Efficient Clip Sequencing from Texts. Proceedings of the 30th ACM International Conference on Multimedia (MM '22), 2022, 5407–5416. DOI: 10.1145/3503161.3548268, arXiv:2107.11851. https://doi.org/10.1145/3503161.3548268
6. Guoxing Yang, Haoyu Lu, Zelong Sun, Zhiwu Lu. Shot Retrieval and Assembly with Text Script for Video Montage Generation. Proceedings of the 2023 International Conference on Multimedia Retrieval (ICMR '23), 2023, 298–306. DOI: 10.1145/3591106.3592247. https://doi.org/10.1145/3591106.3592247
7. Alejandro Pardo, Fabian Caba Heilbron, Juan León Alcazar, Ali Thabet, Bernard Ghanem. Learning to Cut by Watching Movies. IEEE/CVF International Conference on Computer Vision (ICCV 2021), 6838–6848. DOI: 10.1109/ICCV48922.2021.00678, arXiv:2108.04294. https://doi.org/10.1109/ICCV48922.2021.00678
8. Bryan Wang, Yuliang Li, Zhaoyang Lv, Haijun Xia, Yan Xu, Raj Sodhi. LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing. Proceedings of the 29th International Conference on Intelligent User Interfaces (IUI '24), 2024, 699–714. DOI: 10.1145/3640543.3645143, arXiv:2402.10294. https://doi.org/10.1145/3640543.3645143
9. Bekzat Tilekbay, Saelyne Yang, Michal Lewkowicz, Alex Suryapranata, Juho Kim. ExpressEdit: Video Editing with Natural Language and Sketching. Proceedings of the 29th International Conference on Intelligent User Interfaces (IUI '24), 2024, 515–536. DOI: 10.1145/3640543.3645164, arXiv:2403.17693. https://doi.org/10.1145/3640543.3645164
10. Mina Huh, Dingzeyu Li, Kim Pimmel, Hijung Valentina Shin, Amy Pavel, Mira Dontcheva. VideoDiff: Human-AI Video Co-Creation with Alternatives. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. DOI: 10.1145/3706598.3713417, arXiv:2502.10190. https://doi.org/10.1145/3706598.3713417
11. Catherine Yeh, Anh Truong, Mira Dontcheva, Bryan Wang. Vidmento: Creating Video Stories Through Context-Aware Expansion With Generative Video. Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. DOI: 10.1145/3772318.3791337, arXiv:2601.22013. https://doi.org/10.1145/3772318.3791337
12. Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, Yaniv Taigman. Make-A-Video: Text-to-Video Generation without Text-Video Data. arXiv:2209.14792, 2022. https://arxiv.org/abs/2209.14792
13. Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, Tim Salimans. Imagen Video: High Definition Video Generation with Diffusion Models. arXiv:2210.02303, 2022. https://arxiv.org/abs/2210.02303
14. Jia-Qi Zhang, Xiang Xu, Zhi-Meng Shen, Ze-Huan Huang, Yang Zhao, Yan-Pei Cao, Pengfei Wan, Miao Wang. Write-An-Animation: High-level Text-based Animation Editing with Character-Scene Interaction. Computer Graphics Forum 40(7), 217–228, 2021. DOI: 10.1111/cgf.14415. https://doi.org/10.1111/cgf.14415
15. Dennis Quandt, Philipp Altmeyer, Wolfgang Ruppel, Matthias Narroschke. Automatic Text-based Clip Composition for Video News. Proceedings of the 2024 9th International Conference on Multimedia and Image Processing (ICMIP 2024), 106–112. DOI: 10.1145/3665026.3665042. https://doi.org/10.1145/3665026.3665042