Reality Kit Tutorial

A QUICK GUIDE FOR GETTING STARTED WITH REALITY KIT

Apple announced a new framework at WWDC2019 called Reality Kit which makes it much easier for building Augmented Reality apps in iOS13.

Reality Kit relies heavily on using Apple’s new Reality Composer app to create the 3D/AR content. I recently published article on importing custom usdz files into Reality Composer

Building A Simple Reality Kit App

What you need to get started:

    1. macOS Catalina Latest Beta
    2. Xcode 11 Beta 4
    3. An iPad or iPhone installed with iOS13Beta or iPadOS to test our Augmented Reality iOS app

Apple makes it super easy to start building a Reality Kit app by providing a Augmented Reality Kit template in Xcode 11.

Augmented Reality Kit Template

We want to achieve 4 main things in this tutorial

    1. Anchor a custom usdz animation file into our App on a horizontal surface
    2. Setup behaviours action sequence for playing usdz animation audio files
    3. Add tap gesture in our App, so we can control simple behaviours
    4. Launch our App and test it on iOS13 or iPADOS device.

In Xcode we will only need to work with three areas I’ve circled in Red.

    1. Edit ViewController.swift
    2. Edit Experience.rcproject
    3. Open Reality Composer

      Build Reality Kit App in Xcode 11

      Let’s first look at ViewController.swift which adds our box to the AR View on our device.

      
import UIKit
import RealityKit

class ViewController: UIViewController {

@IBOutlet var arView: ARView!

override func viewDidLoad() {
super.viewDidLoad()

// Load the "Box" scene from the "Experience" Reality File
let boxAnchor = try! Experience.loadBox()

// Add the box anchor to the scene
arView.scene.anchors.append(boxAnchor)
}
}

      Without any changes if you attach your iOS13 device to your Mac, and build the project to the target device. The app will insert the basic primitive cube into the AR View inside your room. However, it is just a boring cube! Therefore, to make it more interesting… I want to show you how to import a custom usdz file with animation into Reality Composer and setup an audio sound effect, some physics forces on a ball and control everything with a tap gesture.

      For this example we’ll use the cannon firing animation & “BANG” signage for our custom usdz. Click on each image to go to download page.

      Download button is located on the right sidebar (circled in red below).

      Click on Download Button Circled in Red

      After downloading the two usdz files. Go back into Xcode and click on the Experience.rcproject file in our Xcode project (left sidebar) and Open
      Reality Composer inside Xcode (top right header bar).

      Once inside Reality Composer – delete the cube by clicking on it with right click on the mouse. Then, click on the Objects + button at the top and import the two custom usdz files.

      Now by exiting Reality Composer and going into Xcode will update the AR scene inside Xcode. But we won’t do that yet. Let’s quickly set up the rest of the scene to fire the cannon with sound effects and cannon ball.

      You scale and position the scene to look like this.

      Position the cannon onto the origin. Whilst you can place the “Bang” signage just over the front of the cannon where the ball will fire out. Add a primitive sphere the size of a cannon ball and place it just in front of the cannon where it comes out.

      Next, click on Behaviours tab on the top left corner of Reality Composer. Then click the “+” button and add a “Start Hidden” behaviour followed by a “Custom” behaviour. Next we will add action sequences to each. First sequence is easy. Just select the cannon ball & bang sign as those objects are hidden at the start of the scene and won’t appear until we tap the cannon.

      Setup “Start Hidden” behaviour for cannon ball and bang sign

      The second sequence is more complicated. It is a custom sequence with several actions.

        1. Add + select action tap action and select cannon. It should turn green when selected.
        2. Add + select action usdz animation and select cannon again.
        3. Add + select action -show and select cannon ball & bang sign.
        4. Add + select action Force and select cannon ball. Set to 18 KM/hr you may have to adjust that as required. We also have to set the direction of our force by rotating the force arrow in the right direction. Otherwise the cannon ball will fly off in the wrong direction!

        make sure the force on cannon ball is pointing in the right direction!

        5. Add + select hide and select the bang sign. Set 1 second for duration.
        6. Add + select play sound and select cannon ball or cannon. explosion.mp3 To download the audio file – use the right-click button and select download file. 7. Add + select move by and select cannon. add 5 cm to x-coordinate to push it back as it fires.

      All these actions should be merged by dragging each action onto the first action in the sequence. By merging them together, they will occur at the same time when the animation plays.

      Add tap, show, usdz animation, force, hide, audio sequence

      Add Play sound + move by action to cannon

      Note: One gotcha is objects are only clickable in Reality Click if it has collision body. In Reality Composer set the cannon to physics participate – as fixed.

      Make sure the cannon is setup with physics – fixed. Otherwise it won’t be tappable in Reality Kit

      Now we can test our scene inside Reality Composer by selecting the “Play” button on the top right header. Play the scene and make sure it seems like its working as expected.

      We can now close Reality Composer go back into Xcode and the new scene with all the behaviours setup should be ready in the Xcode scene view.

      Now we have to setup a tap gesture in ViewController.swift so that when our AR scene loads in the room it plays the animation as expected. Replace the ViewController.swift code with the following code snippet. Then attached your iOS13 or iPadOS device and build and run app. Once the cannon anchors on horizontal surface…tap on the cannon for the animation sequence to play.

      Note: There is a bug with some mp3 audio files at the moment which can crash the app. So for testing purpose use this sound effect which works as expected explosion.mp3 – Hover over link & use the right-click button and select download file. 

      
import UIKit
import RealityKit

class ViewController: UIViewController {
    
    @IBOutlet var arView: ARView!
    
    override func viewDidLoad() {
        super.viewDidLoad()
        
        // Load the "Box" scene from the "Experience" Reality File
        let cannonAnim = try! Experience.loadBox()
        
        // Setup horizontal Anchor and add to arView
        
        let anchor = AnchorEntity(plane: .horizontal, minimumBounds: [0.2, 0.2])
        
        arView.scene.addAnchor(anchor)
        
        // setup tap gesture recognizer & add to arView
        
        let tapGesture = UITapGestureRecognizer(target: self, action:#selector(onTap))
        
        arView.addGestureRecognizer(tapGesture)
               
        // Add the the cannon animation to arView
        
        arView.scene.anchors.append(cannonAnim)
        
    }
    
    
    @IBAction func onTap(_ sender: UITapGestureRecognizer) {
            
            let tapLocation = sender.location(in: arView)
            
            // Get the entity at the location we've tapped, if one exists
            
        if let cannonFiring = arView.entity(at: tapLocation) {
               
              print(cannonFiring.name)
              print("firing Cannon")
        
              }
        }
    }

      Next Tutorial => Occlusion in Reality Kit