Civil War Photo Sleuth
The American Civil War (1861–1865) was the first major conflict to have been extensively photographed, with the images being widely displayed and sold in large quantities. Around 4,000,000 soldiers fought the war, and most of them were photographed at least once. After 150 years, thousands of these photographs have survived, but most of the identities of these soldiers are lost.
We introduce a web-based platform called Civil War Photo Sleuth (www.civilwarphotosleuth.com) for helping users identify unknown soldiers in portraits from the American Civil War era. This system employs a novel person identification pipeline by leveraging the complementary strengths of crowdsourced human vision and face recognition algorithms.
- V. Mohanty, D. Thames, S. Mehta, and K. Luther. Supporting Historical Photo Identification with Face Recognition and Crowdsourced Human Expertise (Extended Abstract). Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence Sister Conferences Best Papers. Pages 4755-4759. (Invited Submission)
- V. Mohanty, D. Thames, S. Mehta, and K. Luther. Photo Sleuth: Identifying Historical Portraits with Face Recognition and Crowdsourcing.
ACM Transactions on Interactive Intelligent Systems (TiiS), 10(4), 1-36 (Invited Submission)
- V. Mohanty, D. Thames, S. Mehta, and K. Luther. Photo Sleuth: Combining Human Expertise and Face Recognition to Identify Historical Portraits.
ACM Conference on Intelligent User Interfaces (IUI 2019), Los Angeles, CA, USA, 2019. (25% acceptance rate) (BEST PAPER AWARD)
- V. Mohanty, D. Thames, and K. Luther. Photo Sleuth: Combining Collective Intelligence and Computer Vision to Identify Historical Portraits. ACM Conference on Collective Intelligence (CI 2018), Zurich, Switzerland, 2018. (32% acceptance rate for oral presentations)
- BEST PAPER AWARD at the ACM Conference on Intelligent User Interfaces (IUI) 2019
- GRAND PRIZE WINNER (25000 USD) of the Microsoft Cloud AI Challenge 2018
- Placed 3rd in the "Best Tool or Suite of Tools" category at the Digital Humanities Awards 2018
- Crossed 10000 Registered Users (March 2019)
- News Coverage: Smithsonian [Read here ]
- News Coverage: Roanoke Times [Read here ]
- News Coverage: Slate [Read here ]
- Public Launch of the website www.civilwarphotosleuth.com at the National Archives Building in Washington, DC [Related Article ]
- Initial Launch of the website www.civilwarphotosleuth.com at the 45th Civil War Artifact and Collectibles Show in Gettysburg, PA
As AI-based face recognition technologies are increasingly adopted for high-stakes applications like locating suspected criminals, public concerns about the accuracy of these technologies have grown as well. These technologies often present a human expert with a shortlist of high-confidence candidate faces from which the expert must select correct match(es) while avoiding false positives, which we term the "last-mile problem."
We propose Second Opinion, a web-based software tool that employs a novel crowdsourcing workflow inspired by cognitive psychology, seed-gather-analyze, to assist experts in solving the last-mile problem. We evaluated Second Opinion with a mixed-methods lab study involving 10 experts and 300 crowd workers who collaborate to identify people in historical photos. We found that crowds can eliminate 75% of false positives from the highest-confidence candidates suggested by face recognition, and that experts were enthusiastic about using Second Opinion in their work. We also discuss broader implications for crowd–AI interaction and crowdsourced person identification.
The presentation for this project can be viewed here.
- V. Mohanty, K. Abdol-Hamid, C. Ebersohl and K. Luther. Second Opinion: Supporting Last-Mile Person Identification with Crowdsourcing and Face Recognition
Vol 7 No 1 (2019): Proceedings of the Seventh AAAI Conference on Human Computation and Crowdsourcing (25% acceptance rate)
- V. Mohanty, D. Thames, and K. Luther. Are 1,000 Features Worth A Picture? Combining Crowdsourcing and Face Recognition to Identify Civil War Soldiers. AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2018), Zurich, Switzerland, 2018. [POSTER LINK] (BEST POSTER/DEMO AWARD)