Set Up the Metaflow Library for SAP AI Core

Set your environment variables.

# set username if not set
export USERNAME=tutorialuser

# set location to find settings (configuration) for metaflow, customizable
export HOME=/root

Install the Metaflow package.

pip install sap-ai-core-metaflow

Create a configuration file for Metaflow. Metaflow uses settings from configurations to store snippets of pipeline in your AWS S3 storage.

mkdir -p $HOME/.metaflowconfig
touch $HOME/.metaflowconfig/config.json

Edit the following snippet and the paste in the config.json file, created above. Replace <YOUR_S3_BUCKET_NAME> with AWS S3 bucket ID that you want Metaflow to use to store files.

{
    "METAFLOW_DATASTORE_SYSROOT_S3": "s3://<YOUR_S3_BUCKET_NAME>/metaflow",
    "METAFLOW_DATATOOLS_SYSROOT_S3": "s3://<YOUR_S3_BUCKET_NAME>/metaflow/data",
    "METAFLOW_DEFAULT_DATASTORE": "s3"
}

Create a credentials file for your AWS S3 Object Store. The file is used by the Metaflow package access your AWS S3 store.

mkdir -p $HOME/.aws
touch $HOME/.aws/credentials

Edit the following snippet and paste it in your credentials file. Replace <YOUR_S3_ACCESS...> with your AWS S3 credentials, do not enclose your credentials within quotes("").

[default]
aws_access_key_id = <YOUR_S3_ACCESS_ID>
aws_secret_access_key = <YOUR_S3_ACCESS_KEY>

INFORMATION the Metaflow library for SAP AI Core uses AWS, however you may skip the installation of AWS CLI.

Create a file hellometaflow.py with following contents.

# You should divide the contents of the steps of the pipeline file into different python modules.
# However the contents within each step may have snippets imported from separate python packages/modules.
from metaflow import FlowSpec, step


class HelloFlow(FlowSpec):
    """
    A flow where Metaflow prints 'Hi'.

    Run this flow to validate that Metaflow is installed correctly.

    """

    @step
    def start(self):
        """
        This is the 'start' step. All flows must have a step named 'start' that
        is the first step in the flow.

        """
        print("HelloFlow is starting.")
        self.next(self.hello)

    @step
    def hello(self):
        """
        A step for metaflow to introduce itself.

        """
        print("Metaflow says: Hi!")
        self.next(self.end)

    @step
    def end(self):
        """
        This is the 'end' step. All flows must have an 'end' step, which is the
        last step in the flow.

        """
        print("HelloFlow is all done.")


if __name__ == "__main__":
    HelloFlow()

INFORMATION You may discover more snippets in the official Metaflow tutorials.

Inspect the steps of the Metaflow pipeline hellometaflow.py using following snippet.

python hellometaflow.py show

Run this snippet locally.

python hellometaflow.py run