COCO’s Memory Palace: A Strange Fantasia

Project 1: Color Poems

The first project I’ll show was inspired by a wonderful color summarizer API by Martin Krzywinski. About 6 months ago I was impressed enough by the “image in words” section — see right side — that I asked him if I could make a bot that used the API for poetic purposes. He told me to do it and really double down on the poetry angle.
My Color Poems result (nouns from Vision API are probably “moisture” and “kelp”
An early result before all the CSS fixes, dropshadows, etc. Google loves “products.”

Different Photos, Different Text

I started thinking about other images: holiday photos, selfies, and family pics, like this one of my sister and niece from years ago.

Family photo. Rockin’ that do.
From Talking About Pictures: A Case for Photo Elicitation by Ben Harper
Fludd’s Memory Palace for Music (link)

Project 2: Image Recognition Poems

There are 2 ways to go about image recognition by tool right now: general API labeling of the photo’s broad “content” like Google Cloud Vision and Amazon Rekognition, and object identification that localizes things IN the photos (the part below on COCO). My results from the general labeling services were in the end most successful as texts, probably because they were better at zeroing in on the photos’ contents.

Using Amazon Rekognition (in red) and Google Vision API (black text) from a flickr image, with a tracery.js grammar.
Image local to KIKK Festival talk :)
ML #fail rather than poetry.


I originally wanted to make the poems interactive: as you click on objects in them, you get a new poetic sequence, possibly even a story.

from the MS COCO paper
sample image
“sandwich” hits in Allison’s Gutenberg poetry corpus
COCO doesn’t see the road and trees. She sees clocks in the branches.
Words most similar to “castle_NOUN” in English gigaword over my local flask API
The last line is from the click on the outline on the right of the standing boy. He doesn’t have a recliner, that’s for sure.

Poems About Paintings

There are some very famous poems about paintings. I wondered how my toys would work on them. My favorite is “Musée de Beaux Arts” by W.H. Auden.

The Old Guitarist (Picasso) [His guitar is not blue.]

Keeping Things Blurry for the Poets

We are told the results of machine learning in domains like image and speech recognition are outpacing human performance.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store