Skip to Content

Implement Image classification Using a Trained Core ML Model and Vision Framework.

test
0 %
Implement Image classification Using a Trained Core ML Model and Vision Framework.
Details
// Explore More Tutorials

Implement Image classification Using a Trained Core ML Model and Vision Framework.

10/08/2019

Import a trained Core ML model and use the Vision framework to prepare a certain image for the classification; you will then use the classification result to fetch products of a certain product category from the sample data service.

You will learn

  • How to import a trained Core ML model into Xcode
  • How to use the Vision framework to process an Image and feed it to the classification model
  • How to take the classification result and fetch products for the classified product category


Step 1: Understand the Core ML model

For this tutorial you will import the TechEd2019.mlmodel Core ML model into your Xcode project. This model has been trained on product data like notebooks and office chairs. The data service your app is talking to has similar products in the catalogue. The goal for this tutorial is to later feed an image of a MacBook as well as an image of an office chair to the model. The Core ML model should classify those two images the correct way and your app will load similar products from the data service and display them in a Table View.

Log on to answer question
Step 2: Import Core ML model

In order to use the TechEd2019.mlmodel Core ML model you have to add it to your Xcode project. The model is located on the desktop of the TechEd machine in the Tutorial Content folder.

Go into that folder and drag & drop that model into the Project Navigator of Xcode. Xcode will bring up a dialogue, make sure Copy Items if needed and Create folder references is selected and click on Finish.

Import Model

The model will now be referenced in your Xcode app project and can be initialized within the app code.

Log on to answer question
Step 3: Set up Product Classification Table View Controller for data loading

The Product Classification Table View Controller is supposed to display products of the classified product category. You have implemented similar code before so this should look familiar to you.

Add the needed import statements below the import UIKit statement above the class declaration:


import SAPFiori import SAPOData import SAPCommon

Add the following properties right above the viewDidLoad(_:) method:


private var dataService: ESPMContainer<OnlineODataProvider>? private let appDelegate = UIApplication.shared.delegate as! AppDelegate private let logger = Logger.shared(named: "ProductClassificationTableViewController") private var products = [Product]()

Because the classification as well as the data loading will take some time, you should display a loading indicator to let the user know that your app is currently busy working on those tasks.

Every Xcode project generated by the iOS Assistant has a convenience protocol, making it easy for you to display and hide a SAPFioriLoadingIndicator.

Let the ProductClassificationTableViewController.swift class conform to the SAPFioriLoadingIndicator protocol:


class ProductClassificationTableViewController: UITableViewController, SAPFioriLoadingIndicator

The protocol wants you to add the FUILoadingIndicatorView as a property to your class.

Add the following line of code right above the dataService property:


var loadingIndicator: FUILoadingIndicatorView?

You will use an FUIObjectTableViewCell to display the products in the Table View, also you will need access to the data service.

Add the following lines of code to the viewDidLoad(_:) method:


tableView.estimatedRowHeight = 80 tableView.rowHeight = UITableView.automaticDimension tableView.register(FUIObjectTableViewCell.self, forCellReuseIdentifier: FUIObjectTableViewCell.reuseIdentifier) guard let dataService = appDelegate.sessionManager.onboardingSession?.odataController.espmContainer else { AlertHelper.displayAlert(with: "OData service is not reachable, please onboard again.", error: nil, viewController: self) logger.error("OData service is nil. Please check onboarding.") return } self.dataService = dataService
Log on to answer question
Step 4: Classify the input image

To classify an image, you can use the Vision framework to prepare the image for classification.

Import the Vision framework by adding the import statement right below the UIKit import above the class declaration:


import Vision

You will use a so called VNCoreMLRequest which will use an instance of the Core ML model for image classification.

Implement the following code right below the viewDidLoad(:), read the inline comments carefully:


lazy var classificationRequest: VNCoreMLRequest = { do { // Instantiate the Core ML model let model = try VNCoreMLModel(for: TechEd2019().model) // Create a VNCoreMLRequest passing in the model and starting the classification process in the completionHandler. let request = VNCoreMLRequest(model: model, completionHandler: { [weak self] request, error in self?.processClassifications(for: request, error: error) }) // Crop and scale the image request.imageCropAndScaleOption = .centerCrop return request } catch { fatalError("Failed to load Vision ML model: \(error)") } }()

Next you want to implement a method performing the requests.

Implement a method called updateClassifications(for:):


/// - Tag: PerformRequests func updateClassifications(for image: UIImage) { // show the loading indicator self.showFioriLoadingIndicator("Finding similiar products...") // make sure the orientation of the image is passed in the CGImagePropertyOrientation to set the orientation of the image let orientation = CGImagePropertyOrientation(image.imageOrientation) // Create a CIImage as needed by the model for classification. If that fails throw a fatalError. guard let ciImage = CIImage(image: image) else { fatalError("Unable to create \(CIImage.self) from \(image).") } // Dispatch to the Global queue to asynchronously perform the classification request. DispatchQueue.global(qos: .userInitiated).async { let handler = VNImageRequestHandler(ciImage: ciImage, orientation: orientation) do { try handler.perform([self.classificationRequest]) } catch { /* This handler catches general image processing errors. The `classificationRequest`'s completion handler `processClassifications(_:error:)` catches errors specific to processing that request. */ print("Failed to perform classification.\n\(error.localizedDescription)") } } }

Right now the code above will cause compile time errors. You need an extension on CGImagePropertyOrientation to manually match the orientations of UIImage to the CGImagePropertyOrientation.

Create a new Swift class with the name CGImagePropertyOrientation+UIImageOrientation in the Project Navigator.

Create Swift ClassCreate Swift Class

In that extension class, replace the code with the following lines:


import UIKit import ImageIO extension CGImagePropertyOrientation { init(_ orientation: UIImage.Orientation) { switch orientation { case .up: self = .up case .upMirrored: self = .upMirrored case .down: self = .down case .downMirrored: self = .downMirrored case .left: self = .left case .leftMirrored: self = .leftMirrored case .right: self = .right case .rightMirrored: self = .rightMirrored } } }

Go back to the ProductClassificationTableViewController.swift class and implement a method processing the image classification. This method will also fetch the products according to the classification result.

Implement the following method directly below updateClassifications(for:) and read the inline comments carefully:


/// - Tag: ProcessClassifications func processClassifications(for request: VNRequest, error: Error?) { // Use the main dispatch queue DispatchQueue.main.async { // Check if the results are nil and display the error in an Alert Dialogue guard let results = request.results else { self.logger.error("Unable to classify image.", error: error) AlertHelper.displayAlert(with: "Unable to classify image.", error: error, viewController: self) return } // The `results` will always be `VNClassificationObservation`s, as specified by the Core ML model in this project. let classifications = results as! [VNClassificationObservation] if classifications.isEmpty { AlertHelper.displayAlert(with: "Couldn't recognize the image", error: nil, viewController: self) } else { // Retrieve top classifications ranked by confidence. let topClassifications = classifications.prefix(2) let categoryNames = topClassifications.map { classification in return String(classification.identifier) } // Safe unwrap the first classification, because that will be the category with the highest confidence. guard let category = categoryNames.first else { AlertHelper.displayAlert(with: "Unable to identify product category", error: nil, viewController: self) self.logger.error("Something went wrong. Please check the classification code.") return } // Set the Navigation Bar's title to the classified category self.navigationItem.title = category // Define a DataQuery to only fetch the products matching the classified product category let query = DataQuery().filter(Product.category == category) // Fetch the products matching the defined query self.dataService?.fetchProducts(matching: query) { [weak self] result, error in if let error = error { AlertHelper.displayAlert(with: "Failed to load list of products!", error: error, viewController: self!) self?.logger.error("Failed to load list of products!", error: error) return } // Hide the loading indicator self?.hideFioriLoadingIndicator() self?.products = result! // You will display the product images as well, for that reason create a new array containing the picture urls. self?.productImageURLs = result!.map { $0.pictureUrl ?? "" } self?.tableView.reloadData() } } } }

Add the productImageURLs property right below the product array property above in the class:


private var productImageURLs = [String]()

The last step would be to call the updateClassifications(for:) method inside the viewDidLoad(_:) method as the last line of code:


updateClassifications(for: image)

Your viewDidLoad(_:) should now look like this:


override func viewDidLoad() { super.viewDidLoad() tableView.estimatedRowHeight = 80 tableView.rowHeight = UITableView.automaticDimension tableView.register(FUIObjectTableViewCell.self, forCellReuseIdentifier: FUIObjectTableViewCell.reuseIdentifier) guard let dataService = appDelegate.sessionManager.onboardingSession?.odataController.espmContainer else { AlertHelper.displayAlert(with: "OData service is not reachable, please onboard again.", error: nil, viewController: self) logger.error("OData service is nil. Please check onboarding.") return } self.dataService = dataService updateClassifications(for: image) }

That’s all you need to do to classify an image with Vision and a pre-trained Core ML model.

Continue with the tutorial to implement the displaying of products in the Table View.

Log on to answer question
Step 5: Implement the Table Views Data Source methods

To display the products, you will implement the data source methods directly in the class like you have done before.

Replace the existing numberOfSections(in:) and the tableView(_:numberOfRowsInSection) methods with the following code:


// Return one section. override func numberOfSections(in tableView: UITableView) -> Int { return 1 } // The number of rows is dependant on the available products. override func tableView(_ tableView: UITableView, numberOfRowsInSection section: Int) -> Int { return products.count }

Next implement the tableView(_:cellForRowAt:) method and read the inline comments carefully:


override func tableView(_ tableView: UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell { // Get the correct product to display let product = products[indexPath.row] // Dequeue the FUIObjectTableViewCell let cell = tableView.dequeueReusableCell(withIdentifier: FUIObjectTableViewCell.reuseIdentifier) as! FUIObjectTableViewCell // Set the properties of the Object Cell to the name and category name cell.headlineText = product.name ?? "" cell.subheadlineText = product.categoryName ?? "" // If there is a price available, format it using the NumberFormatter and set it to the footnoteText property of the Object Cell. if let price = product.price { let formatter = NumberFormatter() formatter.numberStyle = .currency let formattedPrice = formatter.string(for: price.intValue()) cell.footnoteText = formattedPrice ?? "" } // Because you're using a lazy loading mechanism for displaying the product images to avoid lagging of the Table View, you have to set a placeholder image on the detailImageView property of the Object Cell. cell.detailImageView.image = FUIIconLibrary.system.imageLibrary // The data service will return the image url in the following format: /imgs/HT-2000.jpg // In order to build together the URL you have to define the base URL. // The Base URL is found in the Mobile Services API tab. let baseURL = "https://hcpms-d061070trial.hanatrial.ondemand.com/com.sap.edm.sampleservice.v2" let url = URL(string: baseURL.appending(productImageURLs[indexPath.row])) // Safe unwrap the URL, the code above could fail when the URL is not in the correct format, so you have to make sure it is safely unwrapped so you can react accordingly. You won't show the product image if the URL is nil. guard let unwrapped = url else { logger.info("URL for product image is nil. Returning cell without image.") return cell } // You will use an image cache to cache all already loaded images. If the image is already in the cache display it right out of the cache. if let img = imageCache[unwrapped.absoluteString] { cell.detailImageView.image = img } // If the image is not loaded yet, use the loadProductImageFrom(_:) method to load the image from the data service. else { // The image is not cached yet, so download it. loadProductImageFrom(unwrapped) { image in cell.detailImageView.image = image } } return cell }

Before you will implement the loadProductImageFrom(_:) method, you have to define an image cache. As a cache you will simply use a dictionary.

Add the following property definition to below the private var productImageURLs = [String]() property:


private var imageCache = [String:UIImage]()

Next implement the loadProductImageFrom(_:) method to actually fetch the product images.

Implement the method right below the viewDidLoad(_:) method and read the inline comments carefully:


private func loadProductImageFrom(_ url: URL, completionHandler: @escaping (_ image: UIImage) -> Void) { // Retrieve an instance of the SAPURLSession if let sapURLSession = appDelegate.sessionManager.onboardingSession?.sapURLSession { // Use a data task on the SAPURLSession to load the product images sapURLSession.dataTask(with: url, completionHandler: { (data, response, error) in if let error = error { self.logger.error("Failed to load image!", error: error) return } if let image = UIImage(data: data!) { // safe image in image cache self.imageCache[url.absoluteString] = image DispatchQueue.main.async { completionHandler(image) } } }).resume() } }
Log on to answer question
Step 6: Run app and classify some products

Let’s classify some images!

Run your app on the iOS Simulator, because the iOS Simulator doesn’t have a camera you have to import the product images into the iOS Simulator’s Photo Library. The needed images are located on the Desktop inside the Tutorial Content folder.

Select the MacBook.png and the Office Chair.png image and drag them onto the iOS Simulator. The iOS Simulator will open up the Photo Library and add the images.

Create Swift Class

Open your app by tapping on the SalesAssistant navigation on the top-left corner.

Create Swift Class

In your app enter the App passcode if necessary.

Create Swift Class

In the Overview View Controller of your app, tap on the implemented Bar Button Item in the Navigation Bar and select Find Product Based on Photo.

Create Swift ClassCreate Swift Class

Next the Photo Library of the iOS Simulator will open up, please tap on Recently Added.

Create Swift Class

Select the MacBook in there.

Create Swift Class

The Photo Library will close and the Product Classification Table View Controller will open up and start the classification process as well as the fetching of the products.

The Table View should display all available notebooks available through the data service. The product images should lazy load and appear as well.

Create Swift Class

Tap on Done to go back to the Overview View Controller.

Repeat that process using the Office Chair image to see if the classification also works with other products.

Create Swift Class

The Product Classification Table View Controller will show all available Chairs in the data service.

Create Swift Class

Congratulations! You’ve successfully used a pre-trained Core ML model and Vision to classify product images.

Which Apple Framework is used in this tutorial to process the image for the Core ML classification? Choose the correct answer.
×

Next Steps

Prerequisites

  • Development environment: Apple Mac running macOS Mojave or higher with Xcode 11 or higher
  • SAP Cloud Platform SDK for iOS: Version 4.0 SP00
Back to top