Future Apple Vision Pro may get Bob Ross-style virtual painting tools

Posted:
in Future Apple Hardware

If you've ever wanted to unleash your inner Bob Ross, you may be soon able to make your own happy accidents in Apple Vision Pro.

Hand drawing a pine tree on a canvas with a pencil, illustration simplified with numbered labels.
Become a virtual Bob Ross or Monet with Apple's new technology.



We recently reported on rumors that it is testing the Apple Pencil for AR and VR, but it looks like the company hopes to eventually ditch the stylus entirely.

Schematic drawing of an elongated stylus with numbered labels, triangular shaded areas at both ends, and dotted outlines suggesting additional positions or movements.
Light emitters embedded in a stylus would allow an AR system to track with precision.



Apple's proposed technology is based on two patents allowing users to draw or paint in augmented reality. Users would be able to paint on a virtual canvas. However, the technology also allows the creation of art in the real world, where artwork could be superimposed over actual elements.

A sketch of a tablet displaying a beach scene with overlay arrows pointing to real-world objects, captioned 'Augmented Virtuality'.
Want to add a hat and birds to the real world? Strap on your headset and start painting.



The first patent application, "Stylus-based input system for a head-mounted device," describes exactly what the rumors suggested: Apple is working on using a stylus to interact with a virtual world.

While the Apple Pencil seems the natural choice for an input device, the patent describes a new device with a light emitter (likely infrared) to allow the headset to track the device's position and orientation.

A second patent "Techniques for enabling drawing in a computer-generated reality environment" takes this even further.

This illustration shows Apple's plan to track a user's hand. Pinch your fingers as if holding a paintbrush, and one will appear superimposed over your hand. Switch tools and the Apple Vision Pro would change the display of what you're holding.

You won't even need a paintbrush to create art with Apple's tech.
You won't even need a paintbrush to create art with Apple's tech.



While it wasn't mentioned in the application, adding gestures for different media would be easy. You might be able to virtually spray paint using the gestures of holding a can and pushing a button or work in finger paint by extending your fingers.

Apple wants to create a virtual simulation of an artist's workflow with unique gestures for selecting colors. With this technology, you could pick and mix colors by flipping over one hand and "dipping" a virtual brush into virtual paint. The experience would mimic artists holding a paint pallet in one hand and the brush in another.

Drawing in a 3D space isn't unique. Anyone with a headset like the Quest can use Tilt Brush or other art programs to create virtual artwork.

However, those programs use controllers and are, at best, awkward. Pushing a button isn't a natural way to draw or paint, and the controllers get in the way of the experience.

A hand in front of a canvas displaying a simple pine tree sketch, surrounded by circular diagrams with intersecting lines, enclosed in a rectangular frame.
Create art in the palm of your hand, literally, and virtually.



In addition to virtual painting, Apple's patent suggests that audio feedback could let a user hear the brush strokes or pencil lines on paper. Immersive experiences like this could transform artwork creation in AR/VR.

Audio feedback could also be used to take virtual art classes. Put on a headset, listen to soothing music, and follow the prompts of a painting instructor. The result could be an art class in a headset.

We can imagine this being combined with other Apple patents for AR/VR, which would allow multiple users to view and interact with the same virtual objects, even if they are not in the same space. For example, you could take a class with a real instructor and paint alongside them on the same virtual canvas.

As with all patent applications, there's no guarantee that these technologies will come soon, or even ever. But combined with the rumors that Apple has been testing the Apple Pencil in Apple Vision Pro, it's safe to say that some of this technology will be available very soon, perhaps as soon as WWDC 2024.

Apple's patent for using a stylus for input is credited to Killian Moore, the lead applicant on numerous AR/VR patents filed by Apple. The patent for drawing in AR/VR was created by Edwin Iskander, Senior Manager of Apple's visionOS.



Read on AppleInsider

Comments

  • Reply 1 of 8
    tmaytmay Posts: 6,329member
    Why stop there.

    I'd assume a virtual Bob Ross could paint anything that I could imagine, based on inputs I give to an AI app.

    Then, I just sit back and watch "Bob" paint.

    Or, how about Jackson Pollock if you have a big space to view into?
    watto_cobra
  • Reply 2 of 8
    chasmchasm Posts: 3,294member
    I must start perma-frizzing my hair IMMEDIATELY!
    watto_cobra
  • Reply 3 of 8
    dewmedewme Posts: 5,362member
    Will happy trees be coming to the Vision Pro soon?
    watto_cobra
  • Reply 4 of 8
    tmay said:
    Why stop there.

    I'd assume a virtual Bob Ross could paint anything that I could imagine, based on inputs I give to an AI app.

    Then, I just sit back and watch "Bob" paint.

    Or, how about Jackson Pollock if you have a big space to view into?
    Well, that's not you painting anything.
    watto_cobra
  • Reply 5 of 8
    Let's hope the idea is that you're painting against a REAL surface.  

    Because "arms get tired".
    watto_cobra
  • Reply 6 of 8
    jdwjdw Posts: 1,335member
    Well, Vision Pro certainly would become more comfortable if it shipped standard with Bob Ross hair as a band.
    watto_cobra
  • Reply 7 of 8
    danoxdanox Posts: 2,849member
    Apple Iteration in time.....
    watto_cobra
  • Reply 8 of 8
    tmaytmay Posts: 6,329member
    tmay said:
    Why stop there.

    I'd assume a virtual Bob Ross could paint anything that I could imagine, based on inputs I give to an AI app.

    Then, I just sit back and watch "Bob" paint.

    Or, how about Jackson Pollock if you have a big space to view into?
    Well, that's not you painting anything.
    No, it isn't, and I'll bet that a lot of the people that watched Bob Ross on PBS never more than finger-painted in their life. For them, it was entertainment. 
Sign In or Register to comment.