Build an Augmented Reality Experience app with FioriAR

Before you start following the first step, notice that if you want to go the UIKit based approach, go and follow the Set Up the SAP BTP SDK for iOS tutorials in order to generate an app using the SAP BTP SDK for iOS Assistant. The first step is solely for the SwiftUI based approach!

In this step you will use the Xcode Project creation flow to create a SwiftUI based app project.

1. Open up your instance of Xcode, here using Xcode Version 13.2.1 (13C100).

   !

   

2. Create a new App project.

   !

   

3. Give the app project a name and make sure you’ve selected SwiftUI as the Interface. You can choose the name freely.

   !

   

4. Create the project in a directory of your choice.

   !

   

5. After you’ve created the project you should see the project opening up.

   !

   

The FioriAR GitHub Repository package is open-source and available to you in the SAP-Samples on GitHub. You can not only consume the package through the Swift Package Manager but also contribute to the project yourself by forking the repository. For more information on contribution check the Contributing guide.

You will use the Swift Package Manager within Xcode to pull and embed the FioriAR package.

1. In your Xcode project, open the Add Swift package… menu item under File in the Menu Bar.

   !

   

2. Use the Search Field to paste in the GitHub repository address (1):

   Code Copy

   https://github.com/SAP/cloud-sdk-ios-fiori-ar.git

3. The Swift package should show up (2), make sure to change the Dependency Rule to Up to Next Major Version (3) in order to use the latest release.

   !

   

4. Click on Add Package. This will start the fetching of the Swift package.

   !

   

5. After the fetching progress has finished, select the FioriAR package product and click on Add Package.

   !

   

You can choose between two options when embedding the FioriAR Swift package to your project. The one you have select for this tutorial will also fetch and embed all of FioriAR's package dependencies for you including the SAP BTP SDK for iOS packages. If you choose the FioriAR-requiresToEmbedXCFrameworks, you need to embed the SAP BTP SDK for iOS packages yourself using the SAP BTP SDK for iOS Assistant.

If you click on Package Dependencies in the project file, you can see that the FioriAR Swift package got added to your project. If you look on the left-hand side you can see the dependencies of the FioriAR package.

!

The way the FioriAR package and it’s APIs work is through providing a generic ARCard model conforming to the CardItemModel protocol. The CardItemModel protocol is there to be implemented by a model to represent an AR annotation card.

So what is an AR annotation card?

AR annotation cards are UI elements being rendered in an AR Scene on your iOS mobile devices. These UI elements are provided by the FioriAR package to make your life easier and take the burden away to create these yourself with Reality Composer and other 2D/3D modeling tools.

!

1. Create a new Group in the Project Navigator. Name it Models.

2. Add 3 new Swift files and call them:

   • ExampleCardItem
   • StringIdentifyingCardItem
   • ARScene

   !

   

The tutorial will go into detail on what these files will individually take responsibility of.

Both, the ExampleCardItem as well as the StringIdentifyingCardItem models must conform to CardItemModel. The difference between them is that the ExampleCardItem has an Integer based ID property and the StringIdentifyingCardItem uses a String based ID. For your implementation you are free to use either depending on your implementation. For this tutorial you will rely on the Integer based model because the SAP Mobile Services service you will be calling supports Integer based models.

1. Open the ExampleCardItem.

2. Make it conform to the CardItemModel protocol:

   Swift Copy

   public struct ExampleCardItem: CardItemModel { }

3. Add import statements for SwiftUI and FioriAR:

   Swift Copy

   import SwiftUI
   import FioriAR

The import Foundation is not needed in this model implementation.

Usually, model implementations in Swift and SwiftUI are done using structs. The reason is that structs are Value Types which means they are created as true copies of the original when being initialized, copied or mutated. This comes in handy when working with data models. If you want to learn more about the differences between structs and classes visit the official documentation Classes and Structs - The Swift Programming Language.

1. Implement the model’s properties:

   Swift Copy

       public var id: Int
       public var title_: String
       public var subtitle_: String?
       public var detailImage_: Data?
       public var image_: CardImage?
       public var actionText_: String?
       public var actionContentURL_: URL?
       public var icon_: String?
       public var position_: SIMD3<Float>?

   As you can see the ID is of type Int for the ExampleCardItem.

2. Repeat the process 3-6 for the StringIdentifyingCardItem:

   Swift Copy

   public struct StringIdentifyingCardItem: CardItemModel {
       public var id: String
       public var title_: String
       public var subtitle_: String?
       public var detailImage_: Data?
       public var image_: CardImage?
       public var actionText_: String?
       public var actionContentURL_: URL?
       public var icon_: String?
       public var position_: SIMD3<Float>?
   }

3. Safe both Swift files.

Each time, the user creates a new AR Scene it gets an ID assign automatically after it has been published to SAP Mobile Services. In a productive environment you want to have the AR Scene as an Entity in your Database holding the Scene information. You should make sure to have an relationship between the Scene entity and some other entity it refers to. Imagine an ARScene might be displayed in case a user wants to see how a product from a product catalogue might look like. The user wants to place an augmented version of the product on a table. In that case your AR Scene entity as an 1-1 relationship to the Product entity. And there might be many cases where this is true for other entities.

In this tutorial you will use the ARScene model to store, the ID which is getting returned by the SAP Mobile Services API, in the UserDefaults. Of course this is not suitable for a production use case as the UserDefaults only exist in memory as long as the app is alive. But it is an easy persistence for this tutorial.

1. Open the ARScene Swift file.

2. Make the ARScene conform to Identifiable and Codable:

   Swift Copy

   struct ARScene: Identifiable, Codable { }

   For SwiftUI to do proper data binding you need to conform to the protocol Identifiable. The codable typealias, is needed for you to encode and decode an object which we need for persisting the ARScene. In the case of Codable, the typealias combines both, the Decodable and Encodable protocols.

   Please read the official documentation for more information.

   • Identifiable - Apple Developers
   • Codable - Apple Developers

3. Implement a property needed for storing the ARScene ID from SAP Mobile Services:

   Swift Copy

   var id: Int

This tutorial uses a number of utils used to do different type of tasks for your app. The util Swift files are provided in this tutorial:

1. Download the Swift Util files:

   • AuthenticationParams
   • FileManager+Extensions
   • SAPURLSession+Extension
   • UserDefaultsHandler

2. Extract the files into your project folder and drag them into the File Navigator of Xcode.

   Here you get a quick overview of what the different utils are there for:

• FileManager+Extension.swift

  A simple extension on FileManager - Apple Developers taking care of retrieving the documents directory, creating a directory in the documents directory and saving data to a directory. These are necessary to load, save and retrieve reality files stored within your app project.

• SAPURLSession+Extension

  An extension written to create an OAuth session with the SAPURLSession. Also the extension contains a method for attaching an OAuth Observer to the SAPURLSession in case you want to do deeper debugging on the network traffic.

• AuthenticationParams

  There are many ways to work with static information in your iOS apps, one way would be to use a property list (plist), another is having an enum. For this tutorial, because it is easier, you’re using an enum to store static information for establishing an OAuth connection challenge.

• UserDefaultsHandler

  This tutorial doesn’t use a true persistence layer for persisting information, here you will use the UserDefaults - Apple Developers. The UserDefaults give you an interface to interact with the defaults system of the OS. Now you can store, at runtime, information within the UserDefaults. Usually used for storing user preferences but in this tutorial a welcome alternative for using a true database. A static array would’ve done the job too but using the UserDefaults is just a bit smoother.

NOTE: Make sure that the above mentioned files are added correctly to your project and are included in the build target.

In the last step you learned about the different types of utils being used in this tutorial. One of the util files is the AuthenticationParams holding static information about the authentication parameters used to establish an OAuth challenge against SAP Mobile Services. These parameters are provided through SAP Mobile Services and can be simply copied and pasted into the AuthenticationParams enum.

1. Open SAP Mobile Services.

   !

   

2. Navigate to Mobile Applications and Native/Hybrid.

3. Click on New to create a new mobile app definition.

   !

   

4. Enter an ID and Name. Click on Next.

   !

   

5. Assign the Mobile Augmented Reality feature. Click on Finish.

   !

   

The creation process might take a minute.

If the creation was successful you can access the mobile app definition to retrieve the OAuth information you need for the app implementation.

1. Open the created mobile app definition.

   !

   

2. Navigate to Security.

   !

   

   In the Security tab you have all information needed to copy them in the AuthenticationParams.

   !

   

3. Copy the following fields to the matching enum cases within AuthenticationParams:

   • Client ID
   • Redirect URL
   • OAuth Authorization
   • OAuth Token
   Swift Copy

     static let clientID = "<Your-Client-ID>"
     static let redirectURL = "<Your-Redirect-URL"
     static let authURL = "Your-Auth-URL"
     static let tokenURL = "Your-Token-URL"

4. Save the AuthenticationParams.

The Scene Authoring View is a SwiftUI view provided by the FioriAR package and it is there for the user to create AR Annotation cards and add them to a AR Scene. This is usually done through an app called Reality Composer provided by Apple. If you want to enable your users to create such scenes directly from within your app or make it possible for you to create scenes without using Reality Composer the FioriAR package got you covered.

If you look at the initializer of the SceneAuthoringView(_ title:, serviceURL:, sapURLSession:), it expects a String title for the NavigationBar, a service URL which is the SAP Mobile Services API endpoint and the same as the redirect URL, and a SAPURLSession in order to establish the connection.

  import FioriAR
  import SAPFoundation
  import SwiftUI
  import UIKit

  struct SceneAuthoringWithUIKitView: View {
      var body: some View {
          SceneAuthoringControllerContainer()
              .navigationBarTitle("Using UIKit")
      }
  }

  // Implemented just to present SceneAuthoringController in Test App which is SwiftUI based
  struct SceneAuthoringControllerContainer: UIViewControllerRepresentable {
      func makeUIViewController(context: Context) -> ARCardsAuthoringControllerVC {
          ARCardsAuthoringControllerVC()
      }

      func updateUIViewController(_ uiViewController: ARCardsAuthoringControllerVC, context: Context) {}
  }

  class ARCardsAuthoringControllerVC: UIViewController {
      var presentSceneAuthoring: UIButton!

      private var sapURLSession = SAPURLSession.createOAuthURLSession(
          clientID: IntegrationTest.System.clientID,
          authURL: IntegrationTest.System.authURL,
          redirectURL: IntegrationTest.System.redirectURL,
          tokenURL: IntegrationTest.System.tokenURL
      )

      override func viewDidLoad() {
          super.viewDidLoad()

          self.presentSceneAuthoring = UIButton()
          self.presentSceneAuthoring.setTitle("Present Scene Authoring", for: .normal)
          self.presentSceneAuthoring.setTitleColor(.white, for: .normal)
          self.presentSceneAuthoring.layer.cornerRadius = 10
          self.presentSceneAuthoring.backgroundColor = .systemBlue

          view.addSubview(self.presentSceneAuthoring)
          self.presentSceneAuthoring.translatesAutoresizingMaskIntoConstraints = false
          self.presentSceneAuthoring.centerXAnchor.constraint(equalTo: view.centerXAnchor).isActive = true
          self.presentSceneAuthoring.centerYAnchor.constraint(equalTo: view.centerYAnchor, constant: -100).isActive = true
          self.presentSceneAuthoring.widthAnchor.constraint(equalToConstant: 250).isActive = true
          self.presentSceneAuthoring.heightAnchor.constraint(equalToConstant: 60).isActive = true
          self.presentSceneAuthoring.addTarget(self, action: #selector(self.presentSceneAuthoringAction), for: .touchUpInside)
      }

      // Use navigationController?.pushViewController()
      // Modal presenting not supported
      @objc func presentSceneAuthoringAction(sender: UIButton) {
          let sceneAuthoringController = SceneAuthoringController(title: "Annotations",
                                                                  serviceURL: URL(string: IntegrationTest.System.redirectURL)!,
                                                                  sapURLSession: self.sapURLSession,
                                                                  sceneIdentifier: SceneIdentifyingAttribute.id(IntegrationTest.TestData.sceneId),
                                                                  onSceneEdit: self.onSceneEdit)
          self.navigationController?.pushViewController(sceneAuthoringController, animated: true)
      }

      func onSceneEdit(sceneEdit: SceneEditing) {
          switch sceneEdit {
          case .created(card: let card):
              print("Created: \(card.title_)")
          case .updated(card: let card):
              print("Updated: \(card.title_)")
          case .deleted(card: let card):
              print("Deleted: \(card.title_)")
          case .published(sceneID: let sceneID):
              print("From SceneEdit:", sceneID)
          }
      }
  }

There is no instance of the SAPURLSession being created within the app, for the connection to SAP Mobile Services this needs to be created. Because it is nice to have one centralized session available to the whole app, a good approach can be to initialize the session in the AppDelegate. In SwiftUI there is no AppDelegate available from the get go but you can add one, SwiftUI knows how to handle it. Where you would usually implement the AppDelegate would be the App.swift file.

1. Open the FioriARSceneExampleApp.swift file.

   If you have named your app differently, the name of the app file is named after the project. The name displayed in this tutorial might not match with what you have.

2. In the app file, add the AppDelegate class definition below the app struct:

   Swift Copy

     class AppDelegate: NSObject, UIApplicationDelegate {
         static var sapURLSession = SAPURLSession.createOAuthURLSession(clientID: AuthenticationParams.clientID,
                                                                 authURL: AuthenticationParams.authURL,
                                                                 redirectURL: AuthenticationParams.redirectURL,
                                                                 tokenURL: AuthenticationParams.tokenURL)
   
         func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]? = nil) -> Bool {
             // Register FioriNext Fonts
   
             Font.registerFioriFonts()
   
             // activate logging either for FioriAR specific logger or for all loggers (incl. SAPFoundation)
             // Logger.shared(named: "FioriAR").logLevel = .debug
   
             Logger.root.logLevel = .debug // all loggers, incl. SAPFoundation
   
             return true
         }

If you look closer, you can see that in the application(_ didFinishWithOptions:) method the Fiori fonts are set and the root level of the SAP logger is being set. That makes the logger more detailed and let’s you see a lot which is happening within your app. You can set this to error or warning at a later point if you want to.

The SAPURLSession is being initialized using the extension util you have added to the project. This creates an OAuth session for you which can be used within your app because the session being static.

At the moment the ARSceneAuthoringContentView is not being called onto the navigation stack. You need to make the view reachable within your app. A good place to do this is from the ContentView as it is already hooked into the navigation flow of the app, to be precise, it is the initial view being displayed by any SwiftUI app.

1. Open the ContentView.swift file.

2. Add the FioriAR import to it:

   Swift Copy

   import FioriAR

3. Within the body, add the following code in order to add a NavigationView and the navigation link to the ARSceneAuthoringContentView:

   Swift Copy

   NavigationView {
               List {
                   NavigationLink(destination: ARSceneAuthoringContentView()) {
                       Text("Create a new AR Scene")
                   }
               }.navigationBarTitle("AR Cards Example")
           }.navigationViewStyle(StackNavigationViewStyle())

   The reason why this is embedded in a List view is because you will add another navigation later on. Your initial view is a stack navigation view pointing to two views later on.

4. Compile and run the app on your physical iOS device and try out the navigation flow.

As previously mentioned the AR scenes the user creates are being published to SAP Mobile Services. On SAP Mobile Services these scenes can be edited or you can create additional localizations if needed.

Important to know is that if you’re using 1SAP BTP Trial version1, your SAP Mobile Services instance will get stopped over night and you need to restart it before using your app.

1. Open the app on your iOS device and create a AR Scene.

   1.1 Create a new AR Card and fill out the form fields.

   !

   

   !

   

   !

   

   1.2 Add an image anchor. Download the QR Code code included but you can use anything you want as an image anchor e.g. a gaming controller.

   !

   

   !

   

   1.3 Tap on Go to AR Scene.

   !

   

   1.4 Scan the image anchor, and add your AR Card anchor. With pinching you can in-/decrease the size of the anchor.

   1.5 Save the changes.

   1.6 Return to the main view.

2. Publish it to SAP Mobile Services using the Publish button on the SceneAuthoringView.

   !

   

3. Open SAP Mobile Services.

4. Navigate to your mobile app configuration.

5. Open the Mobile Augmented Reality feature.

   !

   

6. You should see your published AR scene there. Click on Details.

   !

   

You can see all information added to the scene through the SceneAuthoringView.

To further enhance the app, you will implement additional views to display the created scenes per session and display a selected scene as well.

NOTE: There is no API provided to fetch all created AR scenes from SAP Mobile Services because it is recommended to store the scene information within a database on your backend.

1. Open up your Xcode project.

2. Create 2 new SwiftUI views. Call them:

   • ARSceneListContentView
   • ARSceneContentView

3. Open the ARSceneListContentView.

   The ARSceneListContentView is a simple list view displaying the AR scenes the user has created during a session. Remember, the scenes are stored in the UserDefaults and might not be available anymore after you have taken the app out of memory.

4. Add a new view called SceneRow above the ARSceneListContentView struct declaration and below the import statement:

   Swift Copy

   struct SceneRow: View {
       var scene: ARScene
   
       var body: some View {
           NavigationLink(destination: ARSceneContentView(sceneID: scene.id)) {
               Text(String(scene.id))   
           }
       }
   }

   This is the row view you will display in a list view.

5. In the body of the ARSceneListContentView, add a navigation view and a list view:

   Swift Copy

   NavigationView {
           if let sceneIDs = UserDefaultsHandler.decodeSceneIDs() {
               List(sceneIDs) { sceneID in
                   SceneRow(scene: sceneID)
               }
               .navigationBarTitle("Available AR Scenes")
           } else {
               Text("No scenes available")
           }
       }.navigationViewStyle(StackNavigationViewStyle())

   It is important to check if a scene is stored in the UserDefaults in order to inform the user if there is not. In that case you display a simple Text view.

6. Save the file.

The ARSceneContentView is the actual view displaying the AR scene. To do so, the FioriAR package provides you a view taking care of this. To load the AR scene, using the static SAPURLSession is being used to connect against SAP Mobile Services.

1. Open the ARSceneContentView.

2. Add the following import statements:

   Swift Copy

   import SwiftUI
   import FioriAR
   import SAPFoundation

3. Add a new view struct definition:

   Swift Copy

   struct ARSceneContentView: View { }

4. In the view add 2 StateObject - Apple Developer and a sceneID property:

   Swift Copy

   @StateObject var arModel = ARAnnotationViewModel<CodableCardItem>()
   @StateObject private var asyncStrategy: ServiceStrategy<CodableCardItem>
   var sceneID: Int

5. Create an initializer with the sceneID as a parameter:

   Swift Copy

   init(sceneID: Int) {
       self.sceneID = sceneID
       self._asyncStrategy = StateObject(wrappedValue: ServiceStrategy<CodableCardItem>(
         serviceURL: URL(string: AuthenticationParams.redirectURL)!,
         sapURLSession: AppDelegate.sapURLSession,
         sceneIdentifier: SceneIdentifyingAttribute.id(sceneID)))
   }

6. In the view’s body, initialize the ARAnnotationsView, and on appearance of the view, load the initial data:

   Swift Copy

   ARAnnotationsView(arModel: arModel,
                   cardAction: { id in
                   // set the card action for id corresponding to the CardItemModel
   
                   print(id)
   })
   .onAppear(perform: loadInitialData)

7. Implement the loadInitialData() method:

   Swift Copy

   func loadInitialData() {
       do {
           try self.arModel.loadAsync(loadingStrategy: asyncStrategy)
       } catch {
           print(error)
       }
   }

1. Open the ContentView.

2. Add another navigation link to the list view:

   Swift Copy

   NavigationLink(destination: ARSceneListContentView()) {
       Text("Show available AR Scene(s)")
   }

3. Save the project.

In case you’ve killed your app in-between steps your UserDefaults are probably gone which means you need to create another scene.

You can check if there are available scenes by navigating to the ARSceneContentView. If there are none, create one.

1. Tap on the available scene to display the AR scene.

The scene gets loaded from SAP Mobile Service into your app.

Congratulations, you’ve successfully implemented an AR ready app.

What you have learned in this tutorial is to use the FioriAR Swift package to create and display AR scenes and communicate with the AR service of SAP Mobile Services.

What you might be interested in, is the implementation for locally stored AR scenes in form of a Reality File. The project you have as a result of this tutorial is suitable to be extended with that functionality. If you’re interested in going a step further and implement that feature, go to the SAP TechEd YouTube channel and watch the Developer Keynote section where this is explained or read the available blog post. Of course all of this is documented in the GitHub repository above mentioned and there is even a sample app available for you to test.

Links:

• SAP TechEd Developer Keynote
• Building modern business ARKit ready iOS apps with FioriAR