All the material has been adapted to use audio in place of haptics to get the idea across. Make sure to set the volume to a comfortable level before unmuting the videos. Enjoy :)

The virtual space of GUIs

If you were to try and grab the objects you see on a screen, you would soon realise that they're in fact intangible. This obvious observation highlights the fundamental difference between the physical objects we touch and those that we interact with through our devices. The project takes this reflection as its starting point and develops into an exploration of ways to make it feel like your hand was touching the virtual space. It does so through an augmented mouse and a series of digital sketches that each look into distinct features of the digital space.

Objects with substance

Think of the times you have resized an object on screen, be it a window or a layer in a photo editing software. When you resize graphical elements, you’re essentially changing that object’s available space to exist in front of you. This happens without the object putting up any resistance against your manipulation. But what if an object did resist manipulation? In this sketch, a vibration pattern of increasing intensity delivered through the mouse conveys the feeling of an object that wants to expand against manipulation.

Putting up against obstacles

The cursor lets you effortlessly move around the digital space as you transition between windows, applications and even monitors. Nothing you see on the screen is unreachable with your mouse. But what if the cursor encountered resistance, as if bumping against physical barriers? This sketch explores this concept with a combination of sharp tactile stimuli and a temporary movement locking of the cursor that create the sensation of the mouse pushing against a boundary.

Feeling the third dimension

Designers put a lot of careful work into making graphical interfaces look like they have depth, as it helps with clarifying what objects are active and in the foreground. Thus, what appears as depth in the digital space is simply a visual representation. In this sketch, tactile feedback is used to reveal information that is not perceivable with sight, such as the thickness of objects that would otherwise look indistinguishable from one another. Sharp tactile stimuli of varying intensity suggest the idea of the cursor "stepping" onto shapes of different heights.

A simple prototype

The sketches are made with vanilla HTML, CSS and JavaScript. Vibrations are triggered by an Espruino Pico which is controlled through web serial. The vibration motor is attached to an off-the-shelf mouse using some sticky putty. The actuator is an Eccentric Rotating Mass vibration motor, which creates strong vibrations that can easily propagate throughout the mouse. The sketches can be tried out on any computer, although the vibrations have been replaced with sounds that give a general impression of how it should work.

Play with the sketches

What do people think?

The sketches were tested with five people, who were given specific tasks in order to experience the vibrations. The tests bring up a few interesting learnings.

1. Ambiguous visual treatment can open up interpretation for what the vibrations mean. In the sketch where the cursor can be moved across a boundary, some people felt like they were moving the mouse on top of a small ridge, while others thought it felt like a gap between two adjacent surfaces.
2. Users needed to be in the mindset of "listening" to vibrations in order to understand them as part of the experience. Some users were not sure the vibrations of the mouse were intentional until they were given a full explanation of what was being tested.