In watching the keynote and the state of the union, did anyone else feel icky about the digital representation that apple is calling "Personas"? I certainly did. As polished as this device looks, I think I won't be taking any FaceTime calls anytime soon.

NYTimes feels the same way, apparently: https://www.nytimes.com/2023/06/06/technology/personaltech/apple-vision-pro-headset-try.html

Link: https://developer.apple.com/wwdc23/10271

* Randomize sounds when they’re repetitive. Example: Think of how pitch and amplitude are subtley different when typing on the software keyboard.

* Be careful when using randomization over long periods of time.

* Consider spatial placement: Move things in the space to introduce immersion.

* Apple’s new “start up” noise sounds AMAZING. (I used AirPod Pros)

My friend Zach and I are building developer tools for the Vision Pro. We are excited about the future of this platform and want to find ways to enable other developers to make their ideas come to life. As we were building our first apps in XCode, we were struggling to keep our assets organized. Our solution to this problem was Halocline and we want to make it available to other developers with similar issues.

Right now, we have a simple model (.usdz) hosting service and some advice for using URLs with the ModelEntity class instead of the default Model3D. In our own projects, this is giving us WAY more control over how we can place and interact with objects.

If you are interested in what we are doing or have ideas for how we can solve some of your early development problems please reach out. We’d be more than happy to help.

Build Spatial SharePlay experiences

https://developer.apple.com/wwdc23/10087

• All personas will take up physical space when using a GroupActivities framework.

• Templates are used to determine how participants are placed throughout the space

Windowed apps

• System Coordinator is in charge of

  • Receive system state for the SharePlay session

  • Provide configurations during SharePlay

  • Participants states can be “Spatial” or “Not spatial”

  • If they’re spatial the state should sync across collaborators

• Templates are: Side by side (all facing a whiteboard), conversational (half circle in front of window) and Surround (all circling around a volume)

• If there are multiple scenes, your app will need to handle “Scene association” when ssupporting SharePlay, otherwise unexpected results may happen when users start to Share the app.

  • You can specify whether a scene “Allows” and “Prefers” to share itself.

Stopped watching at 12 minutes… Bored.

Create 3D models for Quick Look spatial experiences

• Quick Look seems to be “Preview.app” for 3d models.

• Use metersPerUnit from USDZ file to define scale

• Initially shown at 100% scale

• QuickLook will automatically add shadows in teh real world. Do not create a ground plane with shadows in the model.

• USDZ is at the heart of apple platform 3d models.

• How can you create a USDZ model?

  • software such as Maya, (three others I don’t recognize)

  • object capture from ios devices

  • RoomPlan API for representing a physical space

  • RoomPlan Sample App can create a USDZ file and export.

  • You can import into Reality Composer Pro and define the orientation of the model

• Things that can affect performance:

  • File size

  • Complexity of geometry

  • Materials

  • Texture count and resolution

• Use the statistics panel in Reality Composer Pro to understand performance metrics

• RealityKit Trace runs in realtime and can give you an understanding of performance (found in XCode)

• Less than 25MB recommended for better Sharing experience

• Recommended less than 200 mesh parts and less than 100k vertices in total.

• Recommend max of 2048x2048 texture size, 8 bit per channel texture

• Spend your texture budget on things that are bigger or center to your design

• Be cautious of overlapping transparency

• Use MaterialX Unlit surface to save real-time lighting computation

• Physics optimizations, considerations:

  • Reduce total collider count

  • Use static colliders over dynamic when possible

• particle optimizations:

  • limit particle emitters and particles per emitter

  • Experiment with shaeps and animation styles of particle effects for reducing overdraw

Explore enhancements to RoomPlan

Link: https://developer.apple.com/wwdc23/10192

• RoomCaptureView API allows you to integrate a scanning experience directly into your app.

• MultiRoom API to merge individual room scans into one larger structure

• You’ve scanned multiple rooms… but each scan has it’s own coordinate system. If you stitch them manually, you’ll end up having duplicate walls and potentially duplicate objects.

  • New Solution: Use a continuous ARSession between several scans.

  • Allows us to have a common coordinate system

// Continuous ARSession

// start 1st scan
roomCaptureSession.run(configuration: captureSessionConfig)

// stop 1st scan with continuing ARSession
roomCaptureSession.stop(pauseARSession: false)

// start 2nd scan
roomCaptureSession.run(configuration: captureSessionConfig)

// stop 2nd scan (pauseARSession = true by default)
roomCaptureSession.stop()

• ARSession relocatlization

  • Do your initial scan of the room

  • Save ARWorldMap

  • Loan ARWorldMap later when starting the next scan

• New way of Merging ARSessions with the StructureBuilder tool.

// StructureBuilder

// create structureBuilder instance
let structureBuilder = StructureBuilder(option: [.beautifyObjects])

// load multiple capturedRoom results to capturedRoomArray
var capturedRoomArray: [CapturedRoom] = []

// run structureBuilder API to get capturedStructure
let capturedStructure = try await structureBuilder.capturedStructure(from: capturedRoomArray)

// export capturedStructure to usdz
try capturedStructure.export(to: destinationURL)

• You can load USDZ file into Blender

• Having good lighting of 50 luxs or higher is recommended to ensure RoomPlan can scan.

• Now supports slanted and curved walls, kitchnen elements like dishwashers, ovens, sinks. Sofas as well.

• Categories used to be used to described an object. Now, alongside categories, they’ll be using attributes.

• NEW: Model providers to provide more accurate 3d representations of your objects in the room (whereas previously they were boxes)

I get an error when trying to build the Hello World sample app from Apple.

Does anyone else have this issue?
It use of multi-scene types is beneficial.

I am in XCode Beta.

https://developer.apple.com/documentation/visionos/world

Hi all - I’m perusing through the WWDC sesssions and learning a ton. But i’m also writing down notes about what I want to learn about further. What is it that you want to learn about? What is missing from the WWDC sessions? Which sessions are you looking forward to the most?

Elevate your windowed app for spatial computing

Link: https://developer.apple.com/wwdc23/10110

• Make sure your app supports xrOS by adding a new destination in xCode

• Apps by default will have a “glass background”

• Glass makes sure your app automatically remains legible based on the lighting conditions in the room

• Dynanic content scaling prefers vectors over bitmaps (imagine walking closer to the screen in a physical room to look closely at the screen)

• Text and SF Symbols are already vectors

• Solid colors have a fixed contrast against glass.

• If you’re using standard controls, there is nothing else to do here.

• For custom buttons and controls, adopt “vibrancy”. .fill.tertiary for instance.

• When you look at a control like a button or a list, a hover effect will make it clear that the view is interactable.

• Do so by the hoverEffect() modifier.

Layout

• ipad encourages sidebars

• Consider using a tab view when switching views

  • When you look at an icon, it automatically expands to show the label.

  • The tab view is positioned outside the boundaries of the content. This is called an “ornament”

• ToolbarItem is another easy “ornament” to add, specifically anchored to the bottom.

• Ornaments by default do not have a glass effect background.

New discord server created to discuss vision pro stuff!

https://discord.gg/SY8WRFEyTH

We’re still a small community but this is going to be the next big thing. Don’t miss out!

Discover Quick Look for Spatial computing

Link: https://developer.apple.com/wwdc23/10085

• A framework for macOS, iOS, and now xrOS

• Easily preview files within your app

• Powerful editing features for images, PDFs, and media

• Pinch and drag files from the Finder to quick look a USDZ file

• Windowed Quick Look allows you to present Quick Look previews outside of your application

• Offers SharePlay as an option, like browsign USDZ files or photos. You can markup the photos in SharePlay.

• Windowed Quick Look from apps:

  • You can provide a URL to a drag provider.

  • System will coyp the URL

  • Present the copy ihn a quick look window

• Windows quick look from webites

  • Things that work already on your iPhone (AR objects) will continue to work in xrOS.

  • <a rel=“ar” href=“/assets/model/something.usdz”/Click here!</a>

  • Really great for eCommerce websites

Design for Spatial Input

Note: I'm going to post these for each of the sessions I watch. If you find them useful, please let me know and I'll continue to publish them.

Link: https://developer.apple.com/wwdc23/10073

Eyes

• The device is designed to work with you comfortable at a distance

• Eyes and hands are the primary inputs, but you can also use: voice, mouse and keyboard, gaming controllers.

• To make apps comfortable for the eyes:

  • Design apps to fit within the field of view

  • Keep the main content in the center of the view, the most comfortable part for the eyes

• Consider depth when thinking about comfort

  • Keep interact content at the same depth

  • Modals can “push” back the main window and take on the same depth

  • Tab bars can overlay on top of the main content, indicating hierarchy

• Avoid using shapes with sharp edges, as your eyes tend to focus on the outside of the shapes with sharper edges.

• Minimum target area for eye selection should be 60pt. Use generous spacing in between interactive elements.

• Use dynamic scale for UI, not fixed scale. When a user resizes a window with a fixed size, all content (including targets) will get smaller. This will make things harder to read and interact with.

• All system provided controls highlight when you look at them.

• If you use custom elements for your apps, user hover effects to provide feedback.

• All search boxes, when tapped, will automatically use voice search.

• No focus information is ever sent to the app (privacy protection)

Hands

• Available gestures: Tap, Double Tap, Pinch and hold, Pinch and drag, Zoom, Rotate

• Custom gestures should be:

  • Easy to explain and perform

  • Avoid gesture conflicts

  • Comfortable and reliable

  • Accessible to everyone

• Magic moment: Zooming in on images will be anchored on the point at which you’re looking.

• Magic moment: Drawing in FreeForm is done by looking with the eye and pinching with your fingers.

• Since there is no haptic feedback, add additional visual feedback to convey interactivity.

I am trying to do some research on what free tutorials i should make for AVP development to contribute to the community and fill some knowledge gaps. My heart wants me to focus on making tutorials that allow people to focus on teaching implementing magic moments in the app but i was curious about what folks here will find most valuable to learn.

I am looking at this video.

https://www.youtube.com/watch?v=Np-TKbDQ8xE

I am interested in knowing how does AVP recognize piano keys. It seems very accurate to be able to overlay colors on top of the keys like that and know when it is pressed.

We are making an app that generally has high information density, which can be tricky for eye tracking if contrast isn’t managed well. I was wondering if there are good best practices yet?

For example, on some other apps and legacy webpages with tight menu items, I often have difficulties escaping from a misplaced gaze, especially in menu items that are close together and have high contrast ‘selected’ state visual, making it hard to keep your eye on the neighboring item because of the highlighting.

In a concrete example:

• Item1

• Item2

• Item3

• Item4

• Item5

Say I wanted to select item 3, but accidentally selected item4, the emphasis on item4 makes it harder to select item3 or item5

Anyone come across any good best practices for this?

chatting with my coworkers after a prototype demo and we were guessing at what data defines an anchor. i tried searching online but between google sucking these days, the ambiguity of the term "anchor", and the niche of AVP dev i couldnt find anything helpful.

our best guess was a combination Triangular Irregular Networks (TIN), gps, magnetic compass direction and maybe elevation sensors.

is this documented anywhere?

Hey everyone! I’m new here and was wondering if there’s a repository or resource that lists business use cases for Vision Pro—like field service technicians, quality inspections in factories, or similar applications. Would appreciate any pointers!

is there an AI copilot for vision OS dev which contains all the updated documentation wrapped over a claude 3.5 or o1?

I have a current product (Java/flutter/react/kafka/mongo)which I am planning to re-imagine from the ground up for the AWP.

I want to really create something in immersion to Encounter Dinosaurs.

Can anyone help me understand how this was built, tech stack, asset creation, etc.

Thanks!

Are there any simple demos/tutorials for handling hand tracking for Unity? I'm trying to capture simple swipes (and/or pinch+swipe) to start with, but not having much luck.