Sort Story: Sorting Jumbled Images and Captions into Stories. (arXiv:1606.07493v5 [cs.CL] UPDATED)

Temporal common sense has applications in AI tasks such as QA, multi-document summarization, and human-AI communication. We propose the task of sequencing -- given a jumbled set of aligned image-caption pairs that belong to a story, the task is to sort them such that the output sequence forms a coherent story. We present multiple approaches, via unary (position) and pairwise (order) predictions, and their ensemble-based combinations, achieving strong results on this task. We use both text-based and image-based features, which depict complementary improvements. Using qualitative examples, we demonstrate that our models have learnt interesting aspects of temporal common sense.

from cs.AI updates on arXiv.org http://ift.tt/29dQVhk
via IFTTT