Scribe: Deep integration of human and machine intelligence to caption speech in real time

Communications of the ACM

Volumn 60, Issue 9, 2017, Pages 93-100

  Lasecki, Walter S ^a   Miller, Christopher D ^b   Naim, Iftekhar ^b   Kushalnagar, Raja ^c   Sadilek, Adam ^b   Gildea, Daniel ^b   Bigham, Jeffrey P ^d  

^a UNIVERSITY OF MICHIGAN   (United States)

^b University of Rochester   (United States)

^c GALLAUDET UNIVERSITY   (United States)

^d CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; AUDITION;

AUTOMATED ASSISTANCE; DEEP INTEGRATIONS; HARD OF HEARINGS; HIGH-ACCURACY; INTERACTIVE SERVICES; MACHINE INTELLIGENCE; MULTIPLE SEQUENCE ALIGNMENTS; SEGMENT LENGTHS;

USER INTERFACES;

EID: 85028647072     PISSN: 00010782     EISSN: 15577317     Source Type: Journal    
DOI: 10.1145/3068663     Document Type: Article

References (32)

1. Bernstein, M.S., Brandt, J.R., Miller, R.C., Karger, D.R. Crowds in two seconds: Enabling realtime crowd-powered interfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST '11 (New York, NY, USA, 2011). ACM, 33-42.
2. Bigham, J.P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R.C., Miller, R., Tatarowicz, A., White, B., White, S., Yeh, T. Vizwiz: Nearly real-time answers to visual questions. In Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, UIST '10, (New York, NY, USA, 2010). ACM, 333-342.
3. Cooke, M., Green, P., Josifovski, L., Vizinho, A. Robust automatic speech recognition with missing and unreliable acoustic data. Speech commun. 34, 3 (2001), 267-285.
4. Driedger, J. Time-scale modification algorithms for music audio signals. Master's thesis, Saarland University, 2011.
5. Edgar, R. Muscle: multiple sequence alignment with high accuracy and high throughput. Nucleic acids research 32, 5 (2004), 1792-1797.
6. Elliot, L.B., Stinson, M.S., Easton, D., Bourgeois, J. College students learning with C-print's education software and automatic speech recognition. In American Educational Research Association Annual Meeting (New York, NY, 2008), AERA.
7. Flowerdew, J.L. Salience in the performance of one speech act:the case of definitions. Discource Processes 15, 2 (Apr-June 1992), 165-181.
8. Metze, F., Gaur, Y., Bigham, J. P. Manipulating word lattices to incorporate human corrections. In Proceedings of INTERSPEECH, (2016).
9. Gaur, Y., Lasecki, W.S., Metze, F., Bigham, J.P. The effects of automatic speech recognition quality on human transcription latency. In Proceedings of the 13th Web for All Conference (2016) ACM.
10. Glass, J.R., Hazen, T.J., Cyphers, D.S., Maliouto-, I., Huynh, D., Barzilay, R. Recent progress in the MIT spoken lecture processing project. In Interspeech (2007), 2553-2556.
11. Gordon, M., Bigham, J.P., Lasecki, W.S. Legiontools: A toolkit+ UI for recruiting and routing crowds to synchronous real-time tasks. In Adjunct Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (2015) ACM, 81-82.
12. Gordon-Salant, S. Aging, hearing loss, and speech recognition: stop shouting, i can't understand you. In Perspectives on Auditory Research, volume 50 of Springer Handbook of Auditory Research. A.N. Popper and R.R. Fay, eds. Springer New York, 2014, 211-228.
13. Jensema, C., McCann, R., Ramsey, S. Closed-captioned television presentation speed and vocabulary. In Am Ann Deaf 140, 4 (October 1996), 284-292.
14. John, B.E. Newell, A. Cumulating the science of HCI: from svR compatibility to transcription typing. ACM SIGCHI Bulletin 20, SI (Mar. 1989), 109-114.
15. Kadri, H., Davy, M., Rabaoui, A., Lachiri, Z., Ellouze, N., et al. Robust audio speaker segmentation using one class SVMs. In IEEE European Signal Processing Conference (Lausanne, Switzerland, 2008) ISSN: 2219-5491.
16. Kushalnagar, R.S., Lasecki, W.S., Bigham, J.P. Captions versus transcripts for online video content. In Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility, W4A '13, (New York, NY, 2013), ACM, 32:1-32:4.
17. Kushalnagar, R.S., Lasecki, W.S., Bigham, J.P. Accessibility evaluation of classroom captions. ACM Trans Access Comput. 5, 3 (Jan. 2014), 1-24.
18. Lasecki, W. Bigham, J. Online quality control for real-time crowd captioning. In International ACM SIGACCESS Conference on Computers & Accessibility, ASSETS 2012, 2012.
19. Lasecki, W., Miller, C., Sadilek, A., Abumoussa, A., Borrello, D., Kushalnagar, R., Bigham, J. Realtime captioning by groups of nonexperts. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology, UIST '12, (2012), 23-34.
20. Lasecki, W.S., Gordon, M., Koutra, D., Jung, M.F., Dow, S.P., Bigham, J.P. Glance: rapidly coding behavioral video with the crowd. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST '14, (New York, NY, 2014). ACM, 1.
21. Lasecki, W.S., Homan, C., Bigham, J.P. Architecting real-time crowd-powered systems. Human Computation 1, 1 (2014).
22. Lasecki, W.S., Kushalnagar, R., Bigham, J.P. Helping students keep up with real-time captions by pausing and highlighting. In Proceedings of the 11th Web for All Conference, W4A '14 (New York, NY, 2014). ACM, 39:1-39:8.
23. Lasecki, W.S., Miller, C.D., Bigham, J.P. Warping time for more effective real-time crowdsourcing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '13 (New York, NY, 2013). ACM, 2033-2036.
24. Lasecki, W.S., Murray, K., White, S., Miller, R.C., Bigham, J.P. Real-time crowd control of existing interfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST '11, (New York, NY, 2011). ACM, 23-32.
25. Lasecki, W.S., Song, Y.C., Kautz, H., Bigham, J.P. Real-time crowd labeling for deployable activity recognition. In Proceedings of the 2013 conference on Computer supported cooperative work (2013) ACM, 1203-1212.
26. Marschark, M., Sapere, P., Convertino, C., Seewagen, R. Access to postsecondary education through sign language interpreting. J Deaf Stud Deaf Educ. 10, 1 (Jan. 2005), 38-50.
27. Naim, I., Gildea, D., Lasecki, W.S., Bigham, J.P. Text alignment for real-time crowd captioning. In Proceedings North American Chapter of the Association for Computational Linguistics (NAACL) (2013), 201-210.
28. Salisbury, E., Stein, S., Ramchurn, S. Real-time opinion aggregation methods for crowd robotics. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems. International Foundation for Autonomous Agents and Multiagent Systems (2015), 841-849.
29. Turner, O.G. The comparative legibility and speed of manuscript and cursive handwriting. The Elementary School Journal (1930), 780-786.
30. Wald, M. Creating accessible educational multimedia through editing automatic speech recognition captioning in real time. Interactive Technology and Smart Education 3, 2 (2006), 131-141.
31. Wang, L., Jiang, T. On the complexity of multiple sequence alignment. J Comput Biol. 1, 4 (1994), 337-348.
32. World Health Organization. Deafness and hearing loss, fact sheet N300. http://www.who.int/mediacentre/factsheets/fs300/en/, February 2014.