Use the Image Classifier API on SAP BTP

For this tutorial we will create a Xcode project from scratch via Xcode and connect to the API manually. Please go ahead and create a new Xcode Project with the following configuration:

For the other properties choose something which match your preferences.

Now we want to find the right API for us to classify images with. Fortunately the SAP API Business Hub offers such an API. Please go to SAP API Business Hub and search for SAP Leonardo ML - Functional Services. Click on the SAP Leonardo Machine Learning Foundation - Functional Services to go to the available APIs.

We want to use the Product Image Classification API, for that please search for Image. Click on the Product Image Classification API to see the API details.

The SAP API Business Hub offers code snippets for the API implementation in several languages like JavaScript, Java, Swift or SAPUI5. Please click on Code Snippet.

In the code snippet pop-up, select the language Swift and click on Copy and Close. We have the needed code in the clipboard.

A typical 200 response from the API would look like this:

{
  "_id": "string",
  "error": "string",
  "request": "string",
  "predictions": [
    {
      "name": "string",
      "results": [
        {
          "label": "string",
          "score": 0
        }
      ]
    }
  ],
  "status": "QUEUED",
  "tenantName": "string",
  "error_description": "string"
}

First open the ImageClassifierTableViewController class and add the following import statements:

  import UIKit
  import Photos
  import Foundation
  import SAPFoundation
  import SAPCommon
  import SAPFiori

Next change the class declaration to inherit from:

• UITableViewController
• UINavigationControllerDelegate

Also add the following properties to your class, this code will give us a logger, an AppDelegate instance, an UIImagePickerController instance for the Photo Library and a dictionary to safe the response of the classifier:

  var classifications = [Dictionary<String, Any>]( )
  let picker = UIImagePickerController()
  private let appDelegate = UIApplication.shared.delegate as! AppDelegate
  private let logger: Logger = Logger.shared(named: "ImageClassifierTVC")
  var loadingIndicator: FUIModalLoadingIndicatorView?

At last, add the following lines of code to the viewDidLoad(:):

  logger.logLevel = .info
  picker.delegate = self

Before we go and implement the logic for calling the classifier’s API, we will implement the Table View logic. First we want to register a FUIObjectTableViewCell at our Table View. Go to the viewDidLoad(:) and add the following lines of code:

  tableView.estimatedRowHeight = 80
  tableView.rowHeight = UITableView.automaticDimension
  tableView.register(FUIObjectTableViewCell.self, forCellReuseIdentifier: FUIObjectTableViewCell.reuseIdentifier)

With that we make sure the cell is registered on the table view and the rows will be displayed the correct way. Implementing the UITableViewDataSource will make sure that the products are going to be displayed the correct way. Implement the following lines of code:

  // MARK: - Table view data source

  override func tableView(_ tableView: UITableView, numberOfRowsInSection section: Int) -> Int {
        let numRows = classifications.count

        if numRows == 0 {
            let noDataLabel: UILabel  = UILabel(frame: CGRect(x: 0, y: 0, width: tableView.bounds.size.width - 50, height: tableView.bounds.size.height))
            noDataLabel.text          = "Select an image or take a picture"
            noDataLabel.textColor     = UIColor.lightGray

            noDataLabel.textAlignment = .center
            tableView.backgroundView  = noDataLabel
        }
        else {
            tableView.backgroundView = nil
        }
        return numRows
    }

    override func tableView(_ tableView: UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell {
        let item = classifications[indexPath.row]

        let cell = tableView.dequeueReusableCell(withIdentifier: FUIObjectTableViewCell.reuseIdentifier) as! FUIObjectTableViewCell

        cell.headlineText   = item["label"] as? String
        cell.footnoteText   = "Confidence: \(String(describing: Int(round(Double(((item["score"]) as! NSNumber) as! Double) * 100))))) %"

        return cell
    }

You remember we created those IBActions all the way in the beginning, now you’re going to implement those to actual show the camera and the Photo Library.

In your ImageClassifierTableViewController locate the two action methods. Replace the code of the didTapLibrary(_ : UIBarButtonItem):

  picker.allowsEditing = false

  // select the source
  picker.sourceType = .photoLibrary

  // define the needed media types
  picker.mediaTypes = UIImagePickerController.availableMediaTypes(for: .photoLibrary)!

  // present the UIImagePickerController
  present(picker, animated: true, completion: nil)

Also replace the code of the didTapCamera(_ : UIBarButtonItem):

  // if there is a camera, present it
  if UIImagePickerController.isSourceTypeAvailable(.camera) {
      picker.allowsEditing = false
      picker.sourceType = UIImagePickerController.SourceType.camera
      picker.cameraCaptureMode = .photo
      picker.modalPresentationStyle = .fullScreen
      present(picker,animated: true,completion: nil)
  } else {

      // if there is no camera available (Simulator), show an UIAlertController
      let alertVC = UIAlertController(
          title: "No Camera",
          message: "Sorry, this device has no camera",
          preferredStyle: .alert)

      let okAction = UIAlertAction(
          title: "OK",
          style:.default,
          handler: nil)

      alertVC.addAction(okAction)

      present(
          alertVC,
          animated: true,
          completion: nil)
  }

We want to make some resizing to the images, so we implement some logic for that:

extension UIImage {
  func resized(toWidth width: CGFloat) -> UIImage? {
      let canvasSize = CGSize(width: width, height: CGFloat(ceil(width/size.width * size.height)))
      UIGraphicsBeginImageContextWithOptions(canvasSize, false, scale)
      defer { UIGraphicsEndImageContext() }
      draw(in: CGRect(origin: .zero, size: canvasSize))
      return UIGraphicsGetImageFromCurrentImageContext()
  }
}

This resizes any image from the camera or the Photo Library to width of 600 pixels, and scales the height proportionally. Since the API Hub does not allow files exceeding 1 megabyte, in this particular case resizing the image to a smaller size is preferred over increasing the image compression.

If you now build run the app on the simulator, your app should look like this:

You won’t be able to use the camera on simulator, run it on a physical iOS device to use the camera.

Now everything is ready to take pictures or pick one from the Photo Library. Let’s implement the logic for connecting to the API and get the photo classified.

Please take the sample code from the SAP API Business Hub.

Let’s implement a method which will connect to the API and send an image with it.
Please add the following method to your class:

private func sendImage(image: UIImage, filename: String) {
  //adding request headers
   let boundary = "Boundary-\(UUID().uuidString)"
   let headers = [
       "Accept": "application/json",
       "APIKey": "0dj0kYVf4a7CK6AXKk5JLcBtdoAm7NWB",
       "Content-Type": "multipart/form-data; boundary=\(boundary)"
   ]

   var request = URLRequest(url: URL(string: "https://sandbox.api.sap.com/ml/prodimgclassifier/inference_sync")!,
                            cachePolicy: .useProtocolCachePolicy,
                            timeoutInterval: 10.0)
   //setting request method
   request.httpMethod = "POST"
   request.allHTTPHeaderFields = headers
   request.httpBody = self.createBody(
       boundary,
       data: image.jpegData(compressionQuality: 0.8),
       mimeType: "image/jpg",
       filename: filename)

   let session = SAPURLSession()

   //sending request
   let dataTask = session.dataTask(with: request) { data, response, error in
       guard let data = data, error == nil else {
           // check for fundamental networking error
           return
       }

       do {
           let json = try JSONSerialization.jsonObject(with: data, options: .mutableContainers) as AnyObject


           self.logger.info("response :\(json)")
           let rootKey = json.allKeys[0]
           let dictArray = json[rootKey] as! [NSDictionary]

           // retrieve 'results' node from JSON and store results in classifications field
           self.classifications = dictArray[0].value(forKey: "results") as! [Dictionary<String, Any>]


           DispatchQueue.main.async {
               self.tableView.reloadData()
               self.loadingIndicator?.dismiss()
           }
       }
       catch let error as NSError {
           self.logger.error("error : \(error)")
       }
   }

   dataTask.resume()
 }

Okay this code does a lot, let’s take a look. In this method, first the required HTTP headers are created. The image is sent as multipart/form-data, and the response is retrieved in JSON format. The APIKey header expects your personal API Hub key, which can be retrieved from the API Hub by clicking the key icon in the top-right of the REST API page.

Because the request is sent as multipart/form-data, a boundary string needs to be constructed which will be used in the request body. This HTTP body is populated in method createBody(:).

The REST API URL is then set in the request and an URLSession is created. After a successful response, the returned data is serialized to a JSON object. The classifications array will then be populated with the JSON object’s results node, and the table view is reloaded.

Now let’s implement the createBody(_ : String, data: Data?, mimeType: String, filename: String) -> Data? method:

private func createBody(_ boundary: String, data: Data?, mimeType: String, filename: String) -> Data? {
  var body = Data()

  let boundaryPrefix = "--\(boundary)\r\n"

  body.append(Data(boundaryPrefix.utf8))
  body.append(Data("Content-Disposition: form-data; name=\"files\"; filename=\"\(filename)\"\r\n".utf8))
  body.append(Data("Content-Type: \(mimeType)\r\n\r\n".utf8))
  guard let data = data else {
    self.logger.error("Data is nil! Can't create body")
    return nil
  }

  body.append(data)
  body.append(Data("\r\n".utf8))
  body.append(Data("--".appending(boundary.appending("--")).utf8))

  return body
}

Everything is ready to use now, there is one piece missing which is the implementation of the UIImagePickerControllerDelegate. Please add another extension to the ImageClassifierTableViewController which is implementing the UIImagePickerControllerDelegate protocol.

extension ImageClassifierTableViewController: UIImagePickerControllerDelegate {
  func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [UIImagePickerController.InfoKey : Any]) {
      let  pickedImage = info[UIImagePickerController.InfoKey.originalImage] as? UIImage

      // API Hub doesn't allow images submitted over 1MB in size.
      // Resizing the image to a width of 600px should suffice.
      let resizedImage = pickedImage?.resized(toWidth: 600.0)

      loadingIndicator = FUIModalLoadingIndicator.show(inView: self.view, animated: true)
      self.sendImage(image: resizedImage!, filename: "image.jpg")

      dismiss(animated:true, completion: nil)
  }

  func imagePickerControllerDidCancel(_ picker: UIImagePickerController) {
      dismiss(animated: true, completion: nil)
  }
}