Sketching With Sound
This project explores how interactive digital sketching can be used to support music ideation. Two fully functional prototypes were built and tested with users. The test results are used to formulate design recommendations for music sketching interfaces.
Year
2025
Duration
12 weeks
Type
Individual
Tools
p5.js, Tone.js, JavaScript, HTML, CSS
Keywords
haptics, GUIs, interactivity
The prototypes
Scribbler
Scribbler allows musicians to sketch melodies by drawing lines where the horizontal axis represents time and the vertical axis represents pitch. It includes features like pitch quantization and the ability to move and erase strokes
Looper
Looper enables the creation of musical loops by connecting nodes on a grid. The distance between nodes defines the temporal interval, while a per-node slider defines the pitch. Multiple loops can be created and played simultaneously.
Learnings and recommendations
Based on user testing and insights from the design process, three key recommendations for designing digital music sketching interfaces emerged.
Tight coupling of visual and auditory feedback
In early prototypes, the visual feedback was directly tied to the user's input while the auditory feedback reflected all the changes applied by the pitch quantization. This disconnect led to confusion about the state of the sketch. The recommendation is to ensure any computation can modify the user input instead of applying changes after the fact.
Diagram showing how non-destructive computation works in the context of music sketching interfaces. It illustrates the flow from user input to visual updates and selective application of changes.
Spotlighting as support for focused ideation
Users expressed the need to focus their ideation on specific parts of the sketch. This can be achieved bu letting people select on what part of the sketch they want to work on, either by looping a section or by muting parts of the composition. With these additions, digital sketching becomes less immediate but more powerful than sketching on paper.
Two examples of how spotlighting can be used to focus on one part of the sketch at a time. On the left, a section of the sketch is looped. On the right, a part of the sketch is muted.
Performance as sketching practice
Music develops on a temporal axis, while the sketching process develops on a spatial one. An interface that appropriately supports music ideation should support ways of inputting data with live performance and preview to capture ideas that are not yet fully formed and hard to translate into a visual representation.
Two examples of how performance and preview can be used to support the composition process. On the left, a keyboard is used to input notes into the sketching interface. On the right, the user can perform their idea directly in the interface using touch.