The 3D mouse in your pocket

Think back to when you last decorated an Easter egg. Egg in left hand, brush in right. Left hand rotates, right hand adds paint. Repeat. The do-rotate-do method has been the way we’ve been making tools the last 2,600,000 years. And then, around 40 years ago we stopped doing it that way, just like that. We started designing our tools on 2D screens and paper. Here’s how I tricked my brain into thinking I was rotating an object in my hand, like decorating an Easter egg.

Let’s back up for a second. I made an app — Rotato. It lets designers make screenshots and animations of their apps. It started out as an augmented reality app with an infinitely scalable iPhone that you could dump on any surface, like a desk. No particular reason, it just looked cool.

The resulting images and videos were as shaky and as messy as the real world, so I decided it was not going to get me the quality I was going for. It did strike me how natural it was to position the virtual camera and get the right shot. A bit sad, I started disentangling the AR code. I promised myself I’d get back to it for positioning the camera down the road. Then one day, Evgeniy, a Rotato user, said he had a crazy idea. “What if you could rotate the phone on the screen by rotating a phone in your hand?”

I opened it, and immediately felt weird. A part of my brain understood I was holding a brown phone in my hand, but it felt like I was holding the virtual silver phone in my hand.

It was like re-finding a great book and remembering you hadn’t read the last chapter, so I pulled in the old code and started connecting things. I opened it, and immediately felt weird. A part of my brain understood I was holding a phone in my hand, but it felt like I was holding the virtual phone on the screen in my hand.

A mouse with a touch screen

As you can see on the video, I removed the video feed from the traditional ARKit experience and added my own design. There’s probably a ton of more things you could do with a mouse that’s also a pocket super computer, but this is what I did for now:

A mouse with a touch screen

As you can see on the video, I removed the video feed from the traditional ARKit experience and added my own design. There’s probably a ton of more things you could do with a mouse that’s also a pocket super computer, but this is what I did for now:

Painting my virtual Easter egg phone

With this new super power, I was able to quickly rotate the virtual phone to the back, change its color, and rotate it back to the front. Like inspecting a newly decorated Easter egg.

A faster way to animate

It also got a lot easier to find good angles for animation keyframes.

Avoiding motion sickness

Roughly, AR works similar to humans. It gets its sense of orientation from a set of motion sensors (the balance system in our ears) and a camera (our eyes). When the two systems don’t match up, we get nauseated from the sensory conflict (to see this in action, go see people trying the Oculus Rift at the Microsoft Store.)

Since the 3D mouse app uses the back-facing camera, and the back facing camera most often looks at the ever changing laptop screen, the phone itself experiences a digital version of nausea, and that affects the tracking quality.

Facing the camera

But a phone also has a camera on the front — and even better, next to it, a 3D camera. This means we can measure the distance between the front of the phone and the face of the user. Move your face closer to the phone to zoom, just like you’d zoom on that physical Easter egg.

That helped, but it obviously only works when the front-facing camera can see my eyes. The gray ball on my face is my phone telling me where it thinks my face is. I don’t look like that in real life.

Visiting other p̶l̶a̶n̶e̶t̶s̶ apps

Rotato is obviously not the only 3D app on 2D screens. I hooked up the iPhone app to Cinema 4D just to try it out. There’s a lot of hacks (ie. brutally inelegant math) involved, so I’m quite surprised by how well it came out. I mean, check out how the now decommissioned Earth Observing-1 comes to life in my hand!

Sculpting the Easter egg

And now we’re back to decorating those eggs. In this case, I’m a (terrible) digital artist, sculpting in 3D. Do, rotate, do.

10 times better

If the rumors are true about Apple adding long-distance depth cameras, this technique simply gets ten times better over night, and we can avoid the digital nausea altogether. So maybe then is a good time to pick up the proverbial 3D mouse in my pocket again.

If you want to play around with this, let me know on Twitter which part is interesting to you.

You can follow Morten and Rotato on Twitter.