I have spent a lot of time in academia, and the last four years at commercial startups, including running a few myself. This gives me a special vantage point on upcoming computing areas – I want to jump over what’s interesting now to what will be interesting in 5 years. My current job hunt has been a really great excuse to do an industry overview and talk to everyone; I honestly wish I could do a two-month sabbatical each year and go spend a day talking to every single interesting company or research organization.
In the next decade, the most interesting computing trends to me are:
– Synchronicity and Liveness
– Cyborgized Cognition
– Telepresence Embodiment
I want to work on these problems whether in research, or commercially. If you work in this area and want to just chat, or hire me, give me an email, or tweet.
I’m going to talk about the next decade shortly, but first here are a few trends I participated in that changed significantly over the last decade:
Tabletop Computing used to be a big deal, with the Microsoft Surface, DiamondTouch and interactive wall displays. My Master’s Degree was all about tabletop computing. However we don’t really have anything exciting happening in this area now – nobody does day-to-day computing on or around horizontal surfaces. The only interesting game in town is maybe TiltFive.
Spatial Computing as a term has managed to get out of tech and academic circles and into general discussion. This is great – it’s a useful summary of all the interesting work happening in Computer Vision, AR, VR and Volumetric Capture. Previous attempts to summarize the field fell into useless 360 vs Stereo or AR vs VR vs XR vs “Cinematic Reality” hair-splitting.
Gestural Interaction was how I thought (and many people still think) we’d interact with computers in the future. The idea of implicit input, or that maybe people will learn American Sign Language to talk to computers, is such an enticing idea that it’s drawn too many people in. But gestural interaction has failed to get real traction. There’s just too much noise in real-life gestural environments. Most gestures actively used in hand-tracking UIs are indexical instead of symbolic, and keyboards and voice interaction still being most important.
Deep Learning unfortunately means too many things right now. Anything that involves some statistical analysis is now called Deep Learning. Marketers that used to use SPSS to analyze Likert scores instead use differently-branded software packages to do deep learning, with an unclear change in actionability. Doing an analysis of an image based on edge detection could be construed as using a small CNN, so it is now called Deep Learning, too. I’m not trying to gatekeep, but I do wish that in general people were more specific, and didn’t use an obfuscated deep learning technique when a much clearer tool like linear regression works totally fine and is easy to understand.
Let me tell you what I’m excited about for the next decade of computing
Synchronicity and Liveness
We are so used to the convenience of asynchronous communication that it’s now common to text or schedule in advance of calling, and that calling someone without warning is considered either rude and requires an apology, or a potential emergency. In timezone-distributed tech circles, almost every meeting is a Google Calendar invite. Messages are sent slightly before or after the scheduled time, with either an apology that someone is only slightly late, or seeking confirmation that the meeting is still on. There’s just too much overhead.
To have quality synchronous time, people still fly in to meetings. This has been a bit bizarre during my current job hunt as companies have spent probably mid-five-figures now flying and hotelling me various places, only to have many of my “onsite” meetings happen in a single-person conference room with a remote employee over Zoom.
Slack usage is pervasive at tech companies of all sizes, since instant messaging is better than emails for quick, time sensitive communication. However, if an employee gets overwhelmed with notifications, they tend to turn them off. Incentives for communicating honestly about your availability are misaligned, and in-person cues about availability aren’t properly translated.
In real-person meetings at someone’s desk, you can get a sense of how busy they are, or if they’re talking to someone else that you can interrupt if you need to. Currently our interaction states are binary: I’m in a call, or I’m not. I’m “online” in Slack, or I’m “offline”, and I may have set the “offline” status artificially because I need to concentrate on something and can’t be bugged at the moment.
I hope new research or businesses explore making synchronicity at a distance more friendly. Calendly has been incredibly useful for me to schedule time to speak with recruiters, as it reduces all the back-and-forth scheduling iterations to a single click. I wish when I called someone, and they’re busy, that we could set up a system to automatically initiate a call again when their current call is done, or we’re both free again. Dialup – sort of a synchronous, audio-based social network, is doing very interesting things with calls that aren’t invoked by either callee. There’s interesting research or a product to be done in communicating and requesting availability – this is handled well in-person with body language, open/closed doors, quick, ephemeral interruptions without disrupting ongoing meetings. Computing doesn’t handle this well at all. I almost want an always-on video conference setup that people can request to join, and then a queue if the join isn’t immediate.
When I was running ticketing for The Aluminum Cat, one of the surprises was there was no online ticketing service that handled time zones well. Turns out, most tickets purchased are for in-person events, so this use case isn’t handled well at all. I’d love a company to build an equivalent of Doodle + eBay + calendly for bidding on group activities, where a booking transaction occurs automatically once quorum (financial, headcount) is met.
Cyborgized Cognition
The average person still refers to artificial intelligences, algorithms and machine learning as “the other”. These often feel a lot like “the other” because when we interact with them, they’re run by large, impersonal, companies whose interests aren’t the same as ours. An AI may be branded as a decision engine, but if that’s done by a company that also serves ads, it doesn’t generate great consumer trust that my behaviour isn’t being influenced outside my best interests.
Once we fix this trust problem (which might require fixing capitalism, tbh), we can start by forming better more useful, actionable relationships with AI – a cyborgization.
The AIs I actively engage with most on a regular basis are my Gmail spam/important filters, and my Facebook and Twitter feeds. Training my Gmail spam/important filters feels like gardening, in that I know it takes extra work to mark messages as important or not instead of deleting or archiving them, but I think it will make my inbox more manageable in the future. When I tell Facebook, Twitter or Instagram that an ad or a post isn’t relevant for me, I don’t have the same relationship; because I know these social networks prioritize engagement (ad views) over what I may want. “We’ll show you less of these in the future” is the best possible phrase to describe the change, but it still feels passive aggressive, since we know the feed algorithms aren’t ultimately trying to serve you.
I have enough presence of mind to spend time gardening algorithms, but from when I talk to other less tech-savvy people, they don’t understand (or believe) that algorithms can change. There is definitely interesting ethnographic research to be done on how the average user can be educated about interacting with algorithms – currently they’re treated like omniscient oracles, or malicious agents able to read your mind. I’d also love to have a browser plugin that lets me know when a news feed is changing on me, sort of like sousveillance for A/B testing.
For cyborgized cognition, I think eventually our day-to-day lives will more resemble a continually negotiated tradeoff between tasks your AI handles, and you handle. Cyborg chess is a good model for this. There are startups that use the term “personal assistant” or “concierge” which I feel a bit grossed out by, since I grew up middle-class and those are things that only useless rich people use. I think there will eventually be lots of interesting research and design work to be done in the area of users and an AI negotiating implicit permission and agency, but none of that can be effectively be used in the wild yet until the trust relationship is better handled.
Maybe Apple will figure it out first. Historically, engaging with an algorithm you don’t own with large compute resources has meant that you give up a significant chunk of your privacy, however there’s a recent Apple Machine Learning Journal publication that has demonstrated remote model training that’s privacy-conscious.
Telepresence Embodiment
From spatial computing, and remote presence via drones such as Double, or wall displays such as Tonari, or live volumetric capture such as Mimesys (acquired by Magic Leap), people are going to spend more time interacting with other people and things at a distance with varying degrees of embodiment.
There’s interesting problems to be solved around scheduling, which I already covered in my section on Synchronicity, but embodiment and expression is an interesting area as well.
Embodiment could be for the purposes of self-expression, such as has been common in online games for a long-time. Wearing a different avatar is like simply wearing a mask. To appear older or of a different gender in voice chat, people have often used audio filters. In social spatial computing environments with live motion capture such as VRChat or Rec Room, it could be foreseeable that users may want motion filters – to appear more masculine, or less clumsy, or mobile at all when they may be quadraplegic.
For human-negotiated attention in remote presence, eye contact is important. Pluto is currently exploring using depth sensor data to correct remote eye contact during video calls so that face-to-face eye contact is preserved.
A short demonstration of eye contact with Pluto iOS and a comparison to the standard selfie camera. #arkit #TrueDepth #sharedpresence pic.twitter.com/N9m5ZqKHz7
— PlutoVR (@PlutoVR) June 14, 2019
Update: Attention Correction is included in the upcoming iOS 13:
How iOS 13 FaceTime Attention Correction works: it simply uses ARKit to grab a depth map/position of your face, and adjusts the eyes accordingly.
Notice the warping of the line across both the eyes and nose. pic.twitter.com/U7PMa4oNGN
— Dave Schukin 🤘 (@schukin) July 3, 2019
This works pretty immediately in a one-on-one video conference setup, if you want to preserve literal eye contact. But what if you’re in a multi-way video chat, like you’ve just formed Voltron:
In this case, it becomes clear that just faithfully reproducing eye contact is overly literal. If I’m person A, I may be able to see when person B is looking at me, but when person B looks at person C, I may not be able to tell. It may be better to adjust person B’s video, as it appears to person A, so that person B’s eyes are oriented towards person C.
The interesting conclusion here is that with remote presence, there’s many reasons to transmit more higher-fidelity expressions of embodiment, but not to reproduce them literally at the remote end.
Taekwondo does not support Double-Jumping
I started Taekwondo in November 2018. I’ve wanted to try a martial art for years, but it’s been one of those hobbies in the backlog that required a chance encounter to nucleate.
I used to do more cardio and cognitive intensive physical activities: gymnastics, figure skating and parkour (amateurishly), but the last few years have only been biking, weightlifting and rock climbing.
Being a programmer, my body goes through long daily phases of neglect where it’s kept catatonic while my mind is in Infinite Fun Space. This takes a toll over time, and weightlifting has been great to work my body through a range of motion. Whenever I let my weightlifting regime slack, I can tell that it has been keeping my posture in check.
Unfortunately, I get bored easily with exercise. With both rock climbing and weightlifting I…somehow drift away since my mind feels unengaged, and I stop pushing myself as hard as I could. I listen to podcasts, but they put me in a contemplative state, not an engaged state.
I love biking at high speeds through urban environments. If I could somehow program while also doing this, I think this would be peak activation of all my pleasure centres at once (Sidenote: I should prototype this, and maybe I can make a more visceral version of Mecha Trigger).
Sadly, biking is not really possible through the winter, so I’ve been antsy for proper exercise for months.
But back to our main topic, which is Taekwondo. My first major observation is that punching and kicking consistently is surprisingly difficult. It reminds me of the time I was learning archery. It was only two or three months into Taekwondo that I got to properly spar with someone, where we’re both actively trying to kick and punch each other, in a “fire at will”.
To my surprise, jumping is way less effective than video games and dreams have been teaching me my entire life.
You see, I’m energetic, and flighty, so jumping excitedly out of the way when I’m in danger is a natural response. A lifetime of playing videogames where this is rewarded and encouraged has not helped, but rather has reinforced this instinct. In most of my dreams, I fly. My dream flight takes two different forms: a) the muddy, drifting hovering a metre above the ground b) soaring on air currents high in the sky. Over my life, I’ve become highly familiar with how to fly in these two configurations. I’ve even had Inception-style recursive dreams, where I’m flying, then wake up and discover I can still fly, and celebrate that all this practice has paid off and I can fly in real life. And then I wake up one more time, back in the real world, and find myself staring at the ceiling, asking why I would do this to myself.
To my surprise, when you jump out of the way of a punch or kick in real life, you, midair, are bound by Newton’s First Law, and have negligible effect over your momentum. There’s no double-jumping, mid-air steering, or glide mechanics at all.
I’ve discovered this thanks to every time I jump in the air, I get kicked or punched in the fucking ribs, and land, on my side, on the ground.
I have had heated conversations with my instructor where he’s informed me that there is no way to “train enough to charge up my chi so I can fly Dragonball Z style”. Harrumph.
So my current dodge strategy has had to change to sliding abruptly across the ground. This is way less cool, but has meant I don’t get knocked out of the air as much, which is nice. When you’re in the middle of a floaty jump, you’ve effectively removed yourself from combat, so now that I’m spending more time in combat, I can opportunistically make crazier moves. My current favourite is disruptive axe kicks that force my opponent to take a step back and blink a couple times.
My feet have had trouble dealing with all the sliding, and so I’ve had to get special foot moisturizer for them. My strength and reflexes are apparently fine, so currently the three major things holding me back in Taekwondo are:
– flexibility
– feet aren’t moist enough
– my instinct to point my fingers dramatically