Skip to Content

Set Up the Metaflow Library for SAP AI Core

Requires Customer/Partner License
Explore different ways of logging metrics during training. Compare generated models.
You will learn
  • How to create a sandbox Python or Docker environment.
  • How to set up the Metaflow Python package for SAP AI Core
  • How to run a local test of your Metaflow pipeline.
karimmohrazKarim MohrazSeptember 14, 2022
Created by
helenaaaaaaaaaa
September 7, 2022
Contributors
karimmohraz
helenaaaaaaaaaa

Prerequisites

  • You have Docker Desktop installed.
  • You have created your first pipeline with SAP AI Core, using this tutorial.

Discover how Metaflow assists you, with diagrams and visualisations from production to deployment. For more information, see the the Metaflow documentation.

  • Step 1
  • Step 2

    Set your environment variables.

    SHELL
    Copy
    # set username if not set
    export USERNAME=tutorialuser
    
    # set location to find settings (configuration) for metaflow, customizable
    export HOME=/root
    
  • Step 3

    Install the Metaflow package.

    SHELL
    Copy
    pip install sap-ai-core-metaflow
    
  • Step 4

    Create a configuration file for Metaflow. Metaflow uses settings from configurations to store snippets of pipeline in your AWS S3 storage.

    SHELL
    Copy
    mkdir -p $HOME/.metaflowconfig
    touch $HOME/.metaflowconfig/config.json
    

    Edit the following snippet and the paste in the config.json file, created above. Replace <YOUR_S3_BUCKET_NAME> with AWS S3 bucket ID that you want Metaflow to use to store files.

    JSON
    Copy
    {
        "METAFLOW_DATASTORE_SYSROOT_S3": "s3://<YOUR_S3_BUCKET_NAME>/metaflow",
        "METAFLOW_DATATOOLS_SYSROOT_S3": "s3://<YOUR_S3_BUCKET_NAME>/metaflow/data",
        "METAFLOW_DEFAULT_DATASTORE": "s3"
    }
    
  • Step 5

    Create a credentials file for your AWS S3 Object Store. The file is used by the Metaflow package access your AWS S3 store.

    SHELL
    Copy
    mkdir -p $HOME/.aws
    touch $HOME/.aws/credentials
    

    Edit the following snippet and paste it in your credentials file. Replace <YOUR_S3_ACCESS...> with your AWS S3 credentials, do not enclose your credentials within quotes("").

    TEXT
    Copy
    [default]
    aws_access_key_id = <YOUR_S3_ACCESS_ID>
    aws_secret_access_key = <YOUR_S3_ACCESS_KEY>
    

    INFORMATION the Metaflow library for SAP AI Core uses AWS, however you may skip the installation of AWS CLI.

    Why use Metaflow along with your AI Code?

  • Step 6

    Create a file hellometaflow.py with following contents.

    PYTHON
    Copy
    # You should divide the contents of the steps of the pipeline file into different python modules.
    # However the contents within each step may have snippets imported from separate python packages/modules.
    from metaflow import FlowSpec, step
    
    
    class HelloFlow(FlowSpec):
        """
        A flow where Metaflow prints 'Hi'.
    
        Run this flow to validate that Metaflow is installed correctly.
    
        """
    
        @step
        def start(self):
            """
            This is the 'start' step. All flows must have a step named 'start' that
            is the first step in the flow.
    
            """
            print("HelloFlow is starting.")
            self.next(self.hello)
    
        @step
        def hello(self):
            """
            A step for metaflow to introduce itself.
    
            """
            print("Metaflow says: Hi!")
            self.next(self.end)
    
        @step
        def end(self):
            """
            This is the 'end' step. All flows must have an 'end' step, which is the
            last step in the flow.
    
            """
            print("HelloFlow is all done.")
    
    
    if __name__ == "__main__":
        HelloFlow()
    

    INFORMATION You may discover more snippets in the official Metaflow tutorials.

    Why use Metaflow along with your AI Code?

  • Step 7

    Inspect the steps of the Metaflow pipeline hellometaflow.py using following snippet.

    SHELL
    Copy
    python hellometaflow.py show
    

    Run this snippet locally.

    SHELL
    Copy
    python hellometaflow.py run
    
    metaflow

    Why use Metaflow along with your AI Code?

  • Step 8
Back to top