Moving to San Francisco; joining Occipital

Two days ago, I finished the second-to-last draft of my thesis and, a couple hours later, my first full watch-through of Star Trek: The Next Generation. And now it’s time for another transition!

YYZ to SFO

I’ve lived in Toronto for 6 years while doing grad school. I got to participate a ton in the improv, theatre, and nerd communities and they are all fantastic. Alas, I’m moving on to somewhere else. Pending US VISA stuff, I’ll be getting on a truck with my stuff and navigator/housemate Sean in a week and a half and heading to San Francisco to work for the amazing company Occipital.

Occipital makes the Structure Sensor, a small-format depth sensor that currently fits on the back of an iPad for scanning people and interior architecture. They ran a wildly successful Kickstarter almost a year ago, and have one of the creators of OpenCV on their board of directors. During my interview process we talked about some of their future plans in the pipe, and I’m very excited. I’m their 19th employee, and my formal title will be Spatial Interaction Engineer. I’m looking forward to developing really cool interactions that take advantage of having spatial knowledge about our bodies and the world around us.

Coming out of grad school and figuring out what to do next is pretty bewildering. I’ll write more on that when I have real hindsight. While I like developing cool, new stuff, exploiting unusual sensors and extending art practices, I’ve always had trouble with academic writing. It just isn’t as enjoyable or enriching to me as crafting something new and interesting, which is why I got into academia in the first place. I’m into creative writing (like this blog), but academic writing has always felt like hard work, like I’m wading through a complex network of academic frameworks, justifications, and citations. This would be fine if writing or documentation was part of the job, but if you’re an academic professionally, it’s the primary way your worth is measured (I might be putting my foot in my mouth if I somehow change my mind and end up back in academia a years from now). During grad school, I’d find myself procrastinating writing often, and getting depressed over it. I have much to thank of the post-docs at my lab for forcing teaching me to manage my time better. I thought of trying to be an independent developer or artist for a bit, but I’m coming out of grad school with debt, and I’d like to get my hands dirty on creating for new technology as quickly as possible. I interviewed at quite a few places, and I loved the vibe of the people at Occipital, and their excitement about really affecting change in how the space around us is understood and manipulated.

Writing a PhD is an endurance slog. You can’t hero through it in a day. Here’s my commits to it in the last year. I’ll talk more about this later, when it’s done and publicly available.
phd_thesis github stats

If you’re based in Toronto, and want to hang out before I head off, let me know! I’ll probably be announcing a goodbye thing in a couple days.

Onwards and Upwards,
Dustin

Posted in me-news, travel | 2 Comments

Books that Stuck with Me

I meant to assemble ten (10) works of fiction or non-fiction that had been influential to me at some point. This proved hard, and for a while I tried to rank them so I could eliminate any 11+. I settled on splitting my life thus far into 3 periods: Childhood, Adolescence and Adulthood, and the books that were influential to me therein. The edges of these periods are pretty soft-edged, and I think, for me, I consider my adolescence going until 19 or 20 or something.

CHILDHOOD

Collaboration – Mark C. Jarvis
This is a pretty obscure short story. I listened to its audio version on a booktape my Dad gave me. A lot. So much so, I can’t be sure if it is objectively good, but I feel like it’s pretty great. Several researchers discover that cetaceans can not only perceive the 3D world from audio returned from an echolocation, but can also project the 3D audio that represents such a return, and that is their method of communication, i.e. happiness = the image of a fish entering and being digested in a stomach. The researchers happen on a dolphin that is a wandering bard. Cultural exchange occurs. There’s a lot about this that was formative to me.

My Old Man and the Sea: A Father and Son Sail Around Cape Horn – David & Daniel Hays
Another book on tape my Dad made for me; a story of a father and son that built a sailboat and sailed around Cape Horn. From this, a got a love of travel, which I’ve done lots of, solitary thought, which I’ve also done lots of, and sailboats, which I haven’t done as much.

Extraterrestrials: A Field Guide for Earthlings – Terence Dickinson
This book is absolutely fantastic. I don’t think one should worry too hard whether it is grounded in science, but the book follows the pattern of presenting several possible worlds that are slightly different from earth, positing the kind of life forms that would live there, and then hiring an artist to draw them in a very serious, polished way that you could imagine illustrations in any natural field guide. Really formative for me.

The Search for Snout – Bruce Coville
Bruce Coville wrote several books that I read while on the emotional & adventurous roller coaster of being a young guy, and his characters often went through personally intense moments, which were obviously fantastical, but seemed so damn relatable in that clever way of knowing that young people will more easily think about their own lot if told indirectly about it, say, involving several humans and aliens working together to find an old friend.

Time and Again – Clifford Simak
On the side of the road in Surrey, in the suburbs of Vancouver, I came across a large cardboard box of sci-fi paperbacks. I don’t know where these came from, whether someone decided they outgrew them, or someone died, but this massively influenced my early development. This particular book was a formative book that I definitely read before the right age when one is supposed to. I’ve probably read it about 6 times in my life, and might again. 1000s of years into the future or so, an author makes a stand for the rights of artificial humans, which isn’t that interesting of itself, except that he finds out that he will write a book about it, before he even knew he had an opinion on it, thanks to some wibbly wobbly time travel stuff. Manifest Destiny as a philosophical idea appears. Good brain food. I should read it again.

Callahan’s Crosstime Saloon – Spider Robinson
Another book from my found cardboard box stash. It introduced me to alternative forms of non-hard-sci-fi, made me realize that good fiction should enable interesting ways of examining concrete life experiences. Spider Robinson isn’t my favourite author, but I’ve read many, many of his books since. My love of puns and wordplay is directly traceable to this book.

ADOLESCENCE

Mars Trilogy (Red Mars, Green Mars, Blue Mars) – Kim Stanley Robinson
Another book series I read really early, and then had to re-read later to properly get. I should probably read it again. These books started my decade-long aspiration to become an astronaut. I’ve since decided I’m not going to pursue the astronaut thing unless it’s colonization, not exploration, on the agenda. This book felt like hard sci-fi in a good way; focusing on the people. There’s a passage in there where a ship psychologist draws out a map where he sees the personality of everyone on the ship fitting in.

Calculating God – Robert J. Sawyer
Like The Search for Snout, this book seemed to hit me in just the right way, emotionally. I’ve since read lots more of Robert J. Sawyer’s work. I appreciate his work both because the character’s lives seem to be honest and imperfect, despite working, often, in a fantastical situation. And it’s almost always Can-Con.

Dirk Gently’s Holistic Detective Agency – Douglas Adams
Hitchhiker’s Guide to the Galaxy is great, but it feels like several little scenes that are sketches on their own. I think the Dirk Gently books are far superior, as holistically, everything combines together to the end (into a literal symphony in this book). Damn, what a great book, which I need to read again. The situation with the couch stuck in the hallway is something I’ve always wanted to do a research project on.

Stranger in a Strange Land – Robert A. Heinlein
A great perspective-distorting book that was pretty influential to me. It’s odd that, in the modern era, sections could definitely be considered backwards, definitely misogynist, but it influenced me in thinking of other ways to be and live.

The Name of the Rose – Umberto Eco
I can trace my love of libraries and their architecture to this book. I was introduced to the movie first, which we watched in my undergrad Semiotics class. A good mystery book set in a monastery, before it was generally accepted that people can learn about the world by reasoning about it. A lost book of Aristotle’s Poetics, on Comedy, is a major plot point and got me interested in the philosophy of comedy. I’ve read a few of Umberto Eco’s books since, notably totally failing to finish Foucault’s Pendulum. Baudolino is pretty good.

Catch-22 – Joseph Heller
Definitely my favourite book. I think I’ve read it 3 or 4 times. It feels like a British Comedy, written by an American, with it’s deconstruction of bureaucrat thinking. It’s very, very funny – I have memories of dry heaving laughter during the first parts of the book. It’s non-linear narrative made James Joyce’s books more palatable when I got to them later. While the Rome sequence in Catch-22 is very depressing, the book ends on such a hopeful note I get a sense that the entrappings of bureaucracy are always, eventually, defeatable.

Impro for Storytellers – Keith Johnstone
I think Jim Davies is responsible for putting this book in my hands. I had performed improv for 3 or 4 years at this point, and for at least a year of that I was in a weekly show. This book really opened up analyzing the structure of improv scenes as a philosophy for me, and it all started coming together in my mind. An astonishing amount of theatre practice, especially unscripted theatre practice, is only passed down in an oral tradition, since there is never a need to write it down. I think that Charna Halpern and Del Close’s Truth in Comedy is a much better book, but it didn’t influence me as much by the time I got to it.

ADULTHOOD

The Satyricon – Petronius
The timelessness of some of the classic comedy works is amazing. I’m also a big fan of Aristophanes, but I still think the Satyricon is a more impressive work. It doesn’t have to invoke animal metaphors (no problem with that, of course) to do its comedy. It’s just 2 dudes one a goofy adventure. It’s American Pie. It’s the under-rated Dude Where’s My Car? in ancient Rome. It’s great.

The Cyberiad – Stanisław Lem
Lem writes science fiction that feels like a fairy tale, justifying plot events by using scientific terminology, but in a hand-wavey way that lets you know it’s all for fun, but in a way that doesn’t feel offensive or stupid. I’m sure Stanisław Lem could write a script for The Core that would actually be good. His short stories are fall and full of ideas, and are great meditative brain food.

The City & The City – China Miéville
One could criticize this book for being high-concept: a murder plot MacGuffin set in a city that is actually two legally-separated cities, in legally different countries. The cities are separated culturally, not spatially, and occupants of either are able to size each other up and tell, by clothes or posture, what city they are in, and whether they are allowed to interact. This gave me some interesting perspective of the urban and social spaces I find myself in, such as standing on top of a glass skyscraper in India in a white-collared shirt and staring down at a slum. These sort of separated situations are not wrong inherently, certainly, just interesting to be aware of.

Schild’s Ladder – Greg Egan
Catch-22 is my favourite book, but Greg Egan is my favourite author. Schild’s Ladder is the second book of his I read, and the one that caught me. Permutation City was the first, but it left me feeling oddly depressed. I’d like to think of Greg Egan’s books, usually dealing with semi-artificial minds, as their own genre of fiction (mindpunk?). The moment in Schild’s Ladder that really caught me is when two agender lovers, finally undressing for the first time, discover the genitals of their artificial bodies are undergoing transformations to figure out a way to fit together, as unique snowflakes that represent each other, but also the relationship they share. That and the theoretical math foundations in the books is great. If you want a more palatable start to Greg Egan, Quarantine contains all the usual bells and whistles without being too confusing to the uninitiated.

Understanding Comics – Scott McCloud
This book gave me an appreciation for three things: (1) Visual communication, (2) Different ways to structure narrative, and (3) Concise explanations using the thing you’re explaining. #3 is the most-lauded part of this book. McCloud describes comics while drawing himself running around and point at different parts of the page. It’s as clever and illustrative as the definition of “Circular Reference” in a Computer Science dictionary pointing to “see Reference, Circular”. For any talks I do where I’m presented an technology I developed, I try to make as much of the talk a live demo as possible (I did this for my talk on LACES at CHI 2014)

Cryptonomicon & The Baroque Cycle – Neal Stephenson
I devour Stephenson’s extremely high-detailed descriptions of people, processes, events and ideas with voracity and endurance that no other author can match. Snow Crash is a great intro book to him. I’m currently in the midst of Reamde. One thing authors, or artists in general can do, is point to something we consider mundane, accepted, that we are used to passing over, and say “let me talk to you for a bit about why this is interesting”. This is Neal Stephenson. I gobble that shit up.

Y: The Last Man – Brian K. Vaughan & Pia Guerra
The first graphic novel series I really, really liked. I’m reading Vaughan’s Saga now and it is also excellent.

The Road to Mars – Eric Idle
Man, there’s lots of books on this list that have a “theory of comedy” as their theme. This is another one, combined with science fiction. I’d like to think that if I turned into a novelist somehow, this book most resembles the one I’d write. Two comedians with a spaceship of their own are travelling across the solar system doing comedy shows while on the run from the law and a terrorist organization. Interspersed are chapters from the perspective of their C3PO-like robot butler companion, who is working on a thesis on comedy, and questioning whether a robot could maybe, one day, be funny. If you liked Firefly or Cowboy Bebop, a bunch of different characters stuck in a tiny ship, you will absolutely love this book. This is all abstracted to another level as this book is intended as the apocrypha documents of a writer a generation after the main events in the novel, who is trying to rewrite what actually happened. And there’s a zero-G sex scene. This book has everything.

JPod – Douglas Coupland
I watched the TV show before the book. And liked it so much I read the book, and found it was entirely different.The characters and their starting positions are the same, but how the plot follows diverges, and both directions are reasonable. I’ve never found such a satisfying split in other works. It’s like Coupland put the effort into the character creation (the characters are great), and then set them loose, improvisationally writing plot. The characters, employees of a large video game studio that is blatantly EA in Burnaby, feel so obviously Canadian, but it’s hard to put your finger on why they do.

Posted in commentary, improv | Leave a comment

Appears to Take on Meaning…

I’m in the midst of writing up my thesis, an unnaturally large document that I both want to just finish, but also have the urge to encapsulate every thought I’ve had in my life to this date. Some of my spare thoughts that don’t appear in that formal academic document will appear here.

Humans are storytellers. A human being telling a narrative story to you is active, and if you’re listening, you’re passive. But, humans can’t help but do storytelling of their own when confronted with random input. This is as unavoidable as turning off your sense of hearing.

I’ve observed that humans create meaning for seemingly unconnected things things. This can be a random collection of items at a checkout, which contain no explicit story, but a real human can’t help ask questions. A toothpaste and a toothbrush make sense and don’t contain an interesting story. Toothpaste and a drill imply some sort of amateur dentistry. Discarded pants on the ground aren’t interesting of themselves, but found early Saturday or Sunday morning, they appear to tell a different story.

What I find so interesting about this process is that you can’t help it. My thesis system, Improv Remix, is about video remix of theatre performance. While mashing up music can be primarily rhythmic, mashing up video of people speaking or moving feels primarily semantic. Although it’s an over-simplification, I like to think of these as little semantic tokens that can be re-arranged with different orders and relationships. This arrangement can be random, or it can be curated, or semi-curated. But the interpreter (observer if you want to think of meaning as resolved from lack of meaning in a quantum sense, or audience if you want to think of meaning as performed) is going to create an internal explanatory story either way.

I played with structured arrangements of semantic tokens a long time ago, with Rock Paper Scissors Infinity.

Semantic tokens is the term I’m using here, but I’m sure other people have used different terms elsewhere (if you know of one, comment please!).

Lee Kuleshov observed the Kuleshov Effect in film editing, where the meaning of a scene as interpreted by the audience as different based on the other scenes before or after it. There’s a couple ways to take this, and one really negative one is that “Oh no! There is no meaning and everything is constructed and artificial!” Whether this is true or not, I don’t think there’s cause for alarm. Meaning can certainly be manipulated, although I don’t like to use that word because it implies evil puppeteering, but there’s also joy in feeling your brain construct it. I’m a grad student, and some times I wake up without having bought breakfast yet. I don’t know why I felt that being a grad student adds to this story, but let’s leave it there. I walk to the corner store and very clearly buy something breakfast-y. Sometimes cereal and milk, sometimes eggs and bacon. That’s the entire contents of what I put on the counter, and I always feel, as if I need to justify myself to the cashier. “I swear I have my life together; I just forgot to have breakfast purchased.” But then, sometimes, on other days, I actually decide to go shopping in the morning, and then realize I only really need breakfast stuff, despite having already eaten breakfast. I feel the same urge to explain all this to the cashier, but then realize how insane that would make me sound, and my face takes on that pained grimace of someone trying to prevent himself from giggling during the entire transaction.

Here’s an example from one of Improv Remix’s performers, Oliver Georgiou, re-mixing semantic content.

The ability to record a scene while playing a previous one allows single performers to construct complex scenes, but exploiting timing and alignment. We will describe one example where a performer recorded a series of scenes to that were interesting to build, yet surprising in combination.

Scene 1: Acting like a duck, the performer waddles from stage right to stage left, occasionally looking behind itself. Finally, it turns around and waddles back slightly faster, to stand up and kiss an empty spot in the air.

The audience watches with curiosity. What is the duck doing?

Scene 2: [Recorded with Scene 1 playing] The performer stands on far stage right, repeating, as endearingly as possible, “Come here duck!” and beckoning as the duck, from Scene 1, walks away. Finally, the performer loses his patience and loudly yells “Hey duck!”, at which point the duck in the video turns around and starts coming back. The performer non-verbally encourages it, and picks it up and kisses it, saying “You’re so cute!”.

The audience laughs as the glances of the duck in Scene 1 are now explained. The performer leaves and returns to the stage wearing a heavy coat and carrying a newspaper.

Scene 3: [Recorded with Scene 2 playing] A dishevelled man stands on stage left reading a newspaper. The exuberant man from Scene 2 repeats “Come here duck!”, distracting the dishevelled man from reading the newspaper. Initially, the dishevelled man looks around, discerns that the exuberant must be not be speaking to him, and then looks back to his paper. As the exuberant man continues, the dishevelled man looks at him more angrily. Finally, the exuberant man from Scene 2 yells “Hey duck!” and the dishevelled man drops to the ground, ducking from possible danger. Seeing there is none, he charges the exuberant man, saying “Hey, buddy, what’s the big idea!?”. The exuberant man kisses him and the dishevelled man slaps him in response.

Later, the dishevelled man from our Constructed Scene was reloaded alongside our beat-boxer, where his angry looks now appear directed at the beat-boxer. I call examples of these surprising interactions Failed Dissonances: here, dissonance between two semantic tokens should occur, but it does not, it fails because our brains find meaning in them.

Posted in art, commentary, research, theatre | Leave a comment

Improv Remix Dailies

Inspired by The Wooster Group, we’re going to start making dailies for each day* leading up to the showcase. Below are the first two! Check Twitter or my YouTube channel for more as they come out.

* Not absolutely every day.

Posted in me-news, theatre | Leave a comment

Improv Remix Call For Performers: July 28th – August 3rd, 2014

Dustin Freeman and Montgomery Martin are looking for 3-5 performers of any age and demographic with a background in improv or comedy to collaboratively playtest a live video remix system. Performers will interact with an array of motion sensors and cameras to construct a series of original scenes from live video recordings to be presented August 1st to 3rd.

We are also seeking a volunteer stage manager and front of house for the public showcase, as well as volunteer technicians to assist with the assembly and operation of the technology and lighting during the week.

For a demonstration of the technology at work see this video:

And this website:
http://dustinfreeman.org/jb/thesis/

The commitment includes approximately 16-20 hours of rehearsal and performance (flexible) time, located at the Storefront Theatre in downtown Toronto.

For those interested, please send a brief description of any relevant experience.

Dustin Freeman
dustin@cs.toronto.edu
Montgomery Martin
montgomery.c.martin@me.com

Posted in art, creations, me-news, research | Leave a comment

Metadata Secret Friends

In the future, there will be friendships consisting entirely in verbal form, with no digital record. These will be special, intimate, beyond anything analyzable by the machines we use on a regular basis today.

Kids will start doing this. It will feel fun and covert at first, but will gain a special meaning. Parents will be concerned about these special, secret friendships. If you have nothing to, why hide your knowledge of this person from the grand scheme of things?

We’ll have to meet these people in person, outside the view of digital security cameras. The most accidental of meet-cutes, without any digital pre-planning. We can trade addresses or names, but can’t trade email, or Facebooks, or Twitter handles, or phone numbers. The machines don’t own dates and times, so we can use those to coordinate, as long as we leave our phones at home.

“Future”, etc., what have you.

Posted in commentary | Leave a comment

Big Blimpin’: Getting the Right Feel of Flying an Airship for an Installation

The Hive

This is an install in a 3 x 8 metre room with stereoscopic projection (a Hive) in the Communitech Hub in Kitchener, sponsored by Christie Digital and CAFKA. The project was started by Ruth Gibson and Bruno Martelli (aka Igloo). There is more about the conception and themes of the project on their page. This doc is a description of the interface and behaviour used to fly the airship around the abstract landscape. I (Dustin) worked on the programming and collaborated on some of the physical interface design.

General Info and Visiting Instructions

We have a high-resolution scan of Svalbard, an island far north of Norway. The island is very secretive – it is suspected there are a few military bases nearby – and if you look at the satellite view in Google Maps, the resolution of the data is very low. While you can fly around most other places in the world at high-resolution using Google Maps, Svalbard is separate, still mysterious and far away. Also, if you want to leave a city in Svalbard to wander around the island, you are legally required to carry a gun to defend against polar bears. In a world that’s generally well-mapped and understood, Svalbard stands apart.

Svalbard via Google Satellite

We represent the island as a wireframe. This visualization originated to give us a sense of the resolution of the data, but when we noticed the size of the polygons changed drastically depending on detail of the landscape, the aesthetics of it were quite nice. While it looks somewhat retro, Tron-ish, that is not exactly the feeling we are aiming for. It is hard to aesthetically explain the feel of this without experiencing the landscape in the Hive. A hill made of raw polygons somehow draws one in much more than a hill that is filled in and shaded. It is very unusual for polygons to be regularly arranged, and you are drawn towards small-polygon, high-detail areas. Whatever algorithm was used to simplify them from the source data creates a Jackson Pollock-like arrangement of triangles that is visually enticing.

Abandoned Polygon Closeup

Personally, I’ve always been drawn to representations of cold, north, mysterious places. Out of the whole Group of Seven, I have always been drawn most to Lawren Harris, who entices me with his simplified, iconic representations of north. Somehow, an iconic form of the landscape feels more true and effective than one that is literally photographic.

Lawren Harris' Lake and Mountains (1928)

The Status Quo of Navigation

In the commercial space where our Hive is set up, it is mostly used as a tech demo to tour groups, and rented by architects to visualize building plans for clients. The space is large enough to walk around a digital car. The current tool to navigate across 3D space is a smooth-future-plastic-molded object with a trigger and a joystick, called a view tracker. The curved plastic sections hold retroreflective markers visible to the 3D motion tracker cameras set above the Hive.

The Hive's AR Tracker Navigation Tool

A forward push on the joystick of the view tracker lurches us forwards in the direction it is pointing. While the velocity is proportional to the push on the joystick, the acceleration is instant, which makes navigation a disorienting stop-and-start affair, similar to Flight Simulator’s slew mode.
The nature of this navigation doesn’t lend itself to the slow, meandering navigation we wanted to achieve with our piece.

The Feel of North and Exploration

I don’t want to put words in Bruno and Ruth’s mouths, but I’ll summarize what I got out of them from conversation, and their previous work in virtual spaces. Much of their work looks like video games, but they are very careful to avoid making things gamey. Explicit visual goals should be avoided, as should a sense of progression. Anything of the sort puts exhibition attendees, whether or not they have video game experience, in the mindset of trying to figure out what the world is wanting them to do. They don’t like their work being compared to Dear Esther (which explores an island in the Hebrides), or to Tale of Tales’ body of work (http://www.tale-of-tales.com/videogames.php). Both of these tend to fudge the definition of “game” – which, if you believe Salen and Zimmerman, require a “quantifiable outcome”. Dear Esther isn’t “hard” in the game sense – the player (experiencer?) just meanders about an island discovering sound bites. However, these weave together into a primarily-linear story, and yield a sense of finish in the end. Bruno and Ruth want to avoid the sense of progression altogether in their virtual environments; more wandering, perhaps, than exploration. I remember one moment, in the middle of playing Dear Esther, where I was unsure where to go next because I had not discovered any more sound bites, and ended up switching from a meandering state to more frantic running around the edge of the thus-far explored map.

(see Glenn Gould’s “The Idea of North” chapter in his Solitude Trilogy))

What does this mean for the sense we want to make of exploring Svalbard? The north, or any sort of lonely exploration, is cold, hard and effortful. The immediate slew mode given by the view tracker felt just wrong. The amount of detail we have in our scenery is enough that you can leave the Hive in any one position on the map, and spend minutes looking around. It is unclear if this is because of the (literally physical) immersion of the Hive, the island model, or the natural curiosity of the occupants (us). The high speed given by the view tracker is almost boring. I recall games that use fast travel as a mechanic you must depend on, and I always try to avoid it when playing them initially, but end up begrudgingly giving in as points of interest are so far apart. I also recall how Jon Blow describes trying to make the level design in The Witness feel exploration-y in the sense that the most interesting stuff in any location is almost immediately in front of you, and that you don’t get a sense that you are ever in a between, filler place.

Our fundamental idea for navigating the space emerged as a floating airship. Many of the initial explorations over the northern polar regions were by airship, and they often departed from Svalbard, so this worked thematically. Additionally, Bruno and Ruth found a really nice wooden wheel in a nearby shop that we wanted to put to use. Thus began the interesting design problem of taking the dinky, finger-movement navigation that currently is used in the Hive and turning it into something that felt more like it should feel.

The Wheel!

A for Effort

There’s an optimization in Human-Computer Interaction to accomplish more with less effort. Get through more list (accurately, of course) with less physical and mental exertion. However, as I’ve said above a couple times, that just didn’t feel right in our case, and we sought to make an interface that did. We needed to control:
- Turning left and right (the wheel, obviously)
- Going up and down (floaty, like a blimp, as opposed to like a plane)
- Speed (Not too important, but people should be allowed to stop at points of interest)

With video games, you want to have a high degree of control. You go fast, turn fast, generally do things that would make you throw up in real life. Our goal is to produce slow, contemplative, meandering motion. The controls could not be too responsive as, in the Hive, that can lead to motion sickness, and fast controls imply that the controller has somewhere to go. Managing the controls themselves and observing the landscape as it is passing by should be its own delight.

It was hard to come up with a setup for the steering wheel that required the right amount of effort to turn, yet still turned smoothly. In the meantime, I implemented WASD-style controls for the airship, AD for left and right, WS for elevation, Shift to go faster and Control to brake to zero speed. I set it up so that if no speed controls were pressed, then the speed of the airship would slowly tend towards an “idle” speed, like a car with automatic transmission. This turned out to be really nice and it made it so that when exhibitors would first come across the exhibit, the ship would flying slowly, and the interest of the scene was high enough that it could be watched for several minutes. Having a non-zero starting speed also satisfied some of Bruno and Ruth’s goals in the sense of wandering. If the airship starts with zero speed, it requires user interaction to start things going, and one cannot interact without an intention. This is the odd problem of deciding where to go once you get in a car or on a bike. Frustratingly, unless you live in the middle of a flat desert, you need to think about where you’re going to go before you start going there. This thinking is usually tedious and not very fun. Having a passively-moving ship you can gradually seize control of is much more interesting.

With our basic WASD controls set up, I prototyped some of the feel of the airship’s flight while others hunted down hardware for the physical controls. The first device we got was a Logitech racing wheel, including two levers right behind the wheel, and a gas/brake pedal. The wheel had a nice torque feedback where it pulled back to centre, but its look and feel was far too much like a racing car. We also found a joystick, which we felt would work well for controlling climb and descent. The racing wheel came with gas pedals, but we felt it would be uncomfortable to use these while standing. Putting a chair in there would encourage visitors to stay longer, which you don’t want in a gallery.

Gas and Brake by Exertion

We had an interaction by turning something (the wheel, changing heading), and an interaction by a control column (the joystick, changing elevation) and I really wanted to find something of a different character for speeding up and slowing down. Ideally, speeding up and slowing down would feel different from each other, but I would be happy with anything that was different from the other existing controls. We were excited about using an engine telegraph to control throttle, but it would cost $800+ to buy a real one that was large enough, and making one and running the cables to a separate object would be too much work.

I became really infatuated with a linear pulling motion, the kind that feels like you’re pulling on an emergency drag chute, or, opening a valve that lets a torrent of rocket fuel into your engine. I thought of finding a screen door piston (a novelty to the British), and we looked at a few, but found they required too much force. During a trip to Toronto’s Active Surplus, I came across a whole bin of paint-spraying tools, which provided both damping and a pulling-back spring. These would be perfect.

While it would be nice to have a linear measure of how far these tools were pulled, we opted for a set-up where a rubber band would pull on a binary switch mounted on the top. Using a rubber band, instead of a less stretchy rope, ensures we can travel over the entire length of the tool’s extension without breaking the switch.

The effortful piston assembly

These tools were mounted from the ceiling, and we hung down a length of nautical-looking wire. To visually and physically differentiate gas and brake, we used different knots, and stained wooden knobs stolen from different tools. I like how the brake pull cord looks and feels a little bit more like an “emergency” option because it is easier to grab onto and yank on.

gas and brake setup

The tension required to pull these down felt quite perfect – you can hold onto them and rest the whole weight of your arm without actuating them – it requires sustained effort to gas or brake.

The effortful piston assembly

The Wheel and The Stick

For the wheel, we used a large AC motor to provide resistance and a solid shaft to spin around. If you give it a good spin, it rotates a few times before slowing to a stop, like on Wheel of Fortune. We got a local carpenter to build a solid wooden box that the motor sits in, which sits on top of a very sturdy metal stand that we cannibalized from a table.

For the stick, we disassembled a joystick and inset it in the wooden box so the stick came through a slot, restricting its movement to only forwards and backwards. We removed the molded curvy plastic shaft and drilled into a wooden dowel to create a longer shaft, and shoved it on the end of the stick. This creates a nice, heavy feel to moving the stick. Unfortunately, the stick doesn’t have much damping when it returns to zero, so it bounces slightly.

Climb/Dive Stick

To get rotation signals from the wheel, we had Michael Yan set up an Arduino Leonardo with a rotation encoder. To keep it simple, when the Arduino was plugged in a USB port, it acts like a mouse (after a hiccup where Windows XP tries to identify it every time), and we send mousewheel events when the wheel turns. The stick was already a USB joystick, so we just had to plug it in. To get the signals from the gas and brake switches, we ran wires across the ceiling girders, down the wall and across the floor, soldering them to buttons on the joystick circuit board. This hack works nicely as we don’t need to build any more circuitry (on an Arduino) and we can test the gas and brake behaviour by pressing the built-in buttons on the circuit board.

Overall, the final set-up is quite nice. I wish we had come up with something a little cooler for the joystick, like something at hip level on either side, but we ran out of time and didn’t want an extra thing on the floor that people would trip over and would be have to be moved each day by the gallery manager.

Feedback of Gradual Changes

I’m going to talk about interface and motion design of the blimp since I’ve described the entire physical interface. Realistically, the motion design and physical interface were being designed side-by-side throughout the process.

Turning, climbing and descending, and acceleration and braking are all very slow. Being in a gallery space, and without giving any clear goals, our interface needs to be walk-up-and-use, and this runs counter to an interface that responds slowly. I conceptualized the motion design as having a gradual outcome, but requiring immediate feedback. The immediate feedback needs to be designed so that what the gradual outcome will be is obviously hinted.

For climbing and descending, we pitched the ship up or down.
For gas or brake, we thought of doing camera shake or sound effects, but could not come up with something in time.
For turning, we rolled the ship – except this was harder to figure this out, and I’ll explain it in the next section.

Climb/Dive Feedback

These are fairly obvious and align with people’s intuition towards airplane movement. For the stick and the pull knobs, these control a rate of change of a variable. The immediate feedback is of what that rate is.

However, as I was working on the wheel, I ran into a design problem: it has no true zero. We wanted a ship’s wheel that would spin several rotations. If we had had a gearbox, we maybe could have put physical stoppers so that its entire range of motion was, say, 5 rotations. This is how a rudder operates, and to get back to zero turning of the ship, you have to turn the wheel back to zero. Now I could solve this by creating stoppers programmatically, where the wheel will stop responding after you spin it past a certain threshold, but this is confusing for the user, and I was running across a second problem. Unlike the stick and the pull knobs, you could turn the ship with the wheel without exertion, by leaving it off-zero, and the ship would have to keep turning in that direction. Maybe we could have had some force-feedback wheel that pulls itself back to zero, but we did not. However, an exhibit where a user could spin the wheel to one side and leave it at max turn was not acceptable.

We needed to throw away realism and figure out a better way to operate the wheel.

An Unrealistic Approach to Turning

I think that an interface that is dramatic is much more fun that an interface that’s realistic. I was inspired by the scene in Star Trek IV where the whalers are frantically spinning the wheel of the ship to turn it away from the Klingon warbird. If our gallery visitors wanted the airship to turn very quickly, I wanted them to work for it.

Whalers spinning a wheel in Star Trek IV
(Incidentally, they are speaking a dialect of Northern Norwegian in this scene)

The obvious approach is to map wheel turn rate to ship turn rate. However, it’s easy to make the ship turn TOO fast if you really spin the wheel. You could limit the rate read by the ship, but really spinning the wheel should still be significant somehow. The approach I went with was that turning the wheel incremented a “turn queue” variable, an amount of turning the ship was required to do, at its own pace. Turning the wheel an amount to starboard could put 20 degrees on the turn queue, which the ship would “eat up” as it turned to starboard. While in the middle of this starboard turn, the wheel operator, could turn the wheel to port, eating up some of the queue, and returning the turn queue value to 0. Then, the ship will quickly stop its turn, and continue on its current heading. This approach of using a turn queue meant that the ship wasn’t very responsive when a turn was starting, which is what we wanted, but was responsive when the turn was stopping. Anything else would likely be frustrating for our visitors.

The consequence of this turn queue approach is that to keep the ship continually rotating, you need to be continually spinning the wheel, exactly like what the whalers are doing in the Star Trek Video.

In Control/Not in Control

The blimp is very hard to fly with one person, both in terms of reach, coordination and exertion. We’ve found in our playtests so far that when more than one person are in the Hive, they help each other with the controls. There’s enough cognitive and physical effort in a single control for any one person. If you have one person trying to operate, i.e. do something significant, with all controls simultaneously, then it is quite nice and dramatic.

I really fantasize about people doing it as a group. It reminds me of when I’ve spent time on a tallship, and commands are being yelled up and down the deck. Even the simple act of turning about is a complex chore, but one that somehow feels satisfying once you’ve finished it.

General Info and Visiting Instructions

Credits

2014 In search of Abandoned CAFKA14 Christie, Communitech HIVE Kitchener, Canada

In search of Abandoned
Concept & design Gibson/Martelli
Sound – David Jensenius
Programming: Dustin Freeman
Additional programming: Jeremy Sioui, Brandon Ryan
Interface: Michael Yan, Rex Lingwood, Dustin Freeman

Special thanks to: Marian Wihak, Jesse Scott, Kristina Foster, Sarah Kernohan, Gordon Hatt, Dylan Reibling, & all of the CAFKA & Christie & Communitech crews

In search of Abandoned is supported by a CAFKA / Christie Residency Program

Posted in art, creations, me-news, technical | Leave a comment

Real Escape Game: Review

This Saturday, myself and 9 friends participated in the Real Escape Game TO. Its structure would be familiar to any avid video gamer or puzzle solver: you are in a locked room, and need to follow a series of (not obvious) clues to get out. As soon as I heard about it, I recruited friends over Facebook, and with extended friends that came to a total of 10 people. The slot we booked was for 11 people, and in my mind this was perfect, as we had one guy who knew noone – a wildcard – at least, as I fantasized about it. Probably a plant. Probably we’d have to kill him, or his surprise ability to factor large numbers would come in handy.

This review contains no spoilers for the puzzle itself. The most spoiler-y thing I talk about is some high-level strategy.

My expectations varied wildly going in. I thought they’d be so high that I would inevitably be disappointed, but I enjoyed it very much and would definitely do it again. This particular room was the first “volume” of a series of rooms the same company will be doing in the future.

My puzzle-solving lineage goes back to playing Myst and Riven in the olden days. I played a bit of Sierra games, mostly Space Quest, but that felt like less puzzle solving some times and more hunting around in pixels, or arbitrarily combining objects together. For the puzzles in The Room, it involved pen, paper and coordination.

I was curious how you could keep 11 people busy and not bored for the 1 hour we had to complete the room. We did not succeed, but we were very close. At the time we played, only 1/24 groups in Toronto had beat the room, and apparently the winning group only did with 1 minute to spare. In San Francisco, there is a duplicate of this room, and the win rate there is 18/1000. We managed to finish the big first stage about 30 minutes in, which is supposed to be quite good.

When you’re initially let into the room, it looks like a normal living room, slightly larger than an average, with an assortment of furniture, paintings, tables, books everywhere. There was nothing immediately obvious that screamed of a clue, like a disembodied head or something, except there was a door at the far side we were instructed to open. Of course, we were all hyper-actively thinking of what to do. Once we were allowed to start going, I, almost as a joke, started disassembling the piece of furniture that was immediately in front of me, and that turned out to be spot-on. The game is very well designed, staging clues from nearly immediately obvious, to much more rewarding if you try harder and look deeper. All of us were busy most of the game, either taking things apart or searching to find clues, or assembling these clues into puzzles and solving them, yielding results that would help us unlock (often literally) more puzzles.

Unlike a video game environment, there was no interactive logic puzzles, like magical levers for you to pull that created some behaviour you had to solve. The only real mechanical stoppages were things that were hard to find (i.e., hidden in furniture) or locks, the combinations for which we had to solve as part of puzzles.

Interestingly, the orientation of the items in the room did not matter, and if you rearranged the room the same puzzle could be run, as long as it was just as difficult to physically find certain clues as it was before. With 11 people moving around the room, I find it difficult to imagine how you would make alignment of objects in the room relevant. As we “processed” objects, there was the brief suggestion that we should declare a corner of a room for objects that were “finished” and had had all their clues extracted. This somewhat worked in practice, but often details were missed and having a pile like that turned out to be too strict of a rule.

I brought my GoPro and planned to record our playthrough, and edit the best bits together, in the style of Chinchilla Dave’s Skyrim playthroughs. Totally unsurprisingly, they did not let us bring in cameras or other recording devices, which is fair.

I highly recommend it. It is very hard, but there were many satisfying partial victories along the way that made it gratifying.

Posted in commentary, games | Leave a comment

Coolest of CHI 2014

Every year around this time my research community has the CHI conference, both standing for Computer-Human Interaction and a pun on the greek χ, representing empirical data. Here are the projects I saw at the conference that were the coolest. I’ll update this as I read papers I didn’t get to see in person.

Haptic turk: a motion platform based on people

This was my favourite overall – to avoid the difficulty of body-size hardware for body-size haptics, this project used friends holding you while you played a game, the most prominent example using an Oculus Rift in a hang-gliding simulation. The people holding you up could shake you or lift you up or down as instructed by DDR-style visualizations. The visualizations were clever in that they gave the people holding the player (the Turkers) enough warning to react on time to effects in the environment.

Walk this Way: Musically Guided Walking Experiences

This project studied a few ways of constructing music that changed when a user navigated a space. Given a certain location, as detected by GPS, a variation of the same musical theme would play. I initially felt the topology the authors explored was overly simple — concentric circles around points of interest, and there were, for example, no long channels of the same audio character. When I asked about this, they said the project was in preparation for a sculpture garden that were commissioned to work on, so moving towards and away from singular objects is the experience they were interested in. They also pointed out, and this is pretty interesting, that it would be really easy to walk over and through an audio channel without being able to orient yourself. I’m not sure – would love to see more work on it.

WaaZam! Supporting Creative Play at a Distance in Customized Video Environments

A system for creating a shared video environment across a distance. The authors used the Kinect to make it a shared 3D environment, and played with changing scale of the people in the environment as well. It looks like they are working with Open Frameworks to clean up their networking system a bit (which is very good and useful).

Exertion in the Small: Improving Differentiation and Expressiveness in Sports Games with Physical Controls

The premise for this work is that, in sports video games, players with the same stats run equally quickly across the field. There is no opportunity for a player to exert themselves wildly and dramatically in a key moment. While some games model fatigue and stamina, it is quite simplified and doesn’t yield the tense and exciting moments we get in the real world. The game presented (Jelly Polo) uses a novel control of thumb-flicking: the faster you thumb-flick, the faster you are in the game!

And of course, MY work:
LACES: Live Authoring through Compositing and Editing of Streaming Video

Posted in research | Leave a comment

“That’s very compelling, but are you the only one who can use it?”

I was asked this question when demoing a research interface at a presentation recently. It’s a good question, and I think one we in the research HCI field tip-toe around a lot.

The answer I gave was that concept interfaces are meant to explore a new idea. A successful concept presents an idea what is compelling enough to warrant future work. Any interface, ever, is going to require a ton of polish. I specifically cited the 80/20 ratio, where 20% of the project appears to take 80% of the work. That final 20% of the project, of the design of your new interface, is going to take so much work that you’re out of the realm of research and into regular UI engineering.

It’s really hard to figure out where to draw the line between research and polish. If you’re trying to describe a new way of adaptively moving buttons around, it needs to be polished enough to be obvious, even if the buttons aren’t their final sizes. However, it hurts a little when people note that the button sizes aren’t quite right when, you know, I’m trying to figure out how to make them move.

Posted in commentary, research | Leave a comment