Bundle data (via JavaScript) SAP Data Hub, developer edition 2.4

Right click on the JS String Operator and click on Open Script to display the JavaScript snippet which is executed by the JS String Operator. The JavaScript snippet opens in a new tab.

Currently the JavaScript snippet creates an incremental counter every time it receives data via the input port and sends the counter to the output port.

Replace the code with the following snippet to ensure that “bundles” of 30 sensor records are sent to the output port.

var counter = 0;
var bundle = "";

$.setPortCallback("input",onInput);

function onInput(ctx,s) {
    counter++;
    bundle = bundle + s;

    if(counter==30) {
      $.output(bundle);
      counter = 0;
      bundle = "";
    }
}

Close the window with the JavaScript snippet. Afterwards press the Save button to save the graph.

Before you execute the pipeline, delete the contents from the /tmp/hdfsManager directory. To find out the container for HDFS, run the the following command:

docker ps

This will provide you a list of all the running containers. From the list, copy the Container ID for the container with the name hdfs. For this example, we are assuming that our Container ID is 8560ee632f69

Now log into the running container.

docker exec -it 8560ee632f69 /bin/bash

Replace 8560ee632f69 with your Container ID while executing

Then enter the following command.

hdfs dfs -rm /tmp/hdfsManager/*

You can check the deletion via http://localhost:50070.

Press the Run button to execute the pipeline.

When the Status tab indicates that the pipeline is running, use the context menu Open UI of the Terminal operator to see the generated sensor data.

Open http://localhost:50070 and display the /tmp/hdfsManager directory. You see that the system does not create a file for each single sensor record, but only for each 30 sensor records.

Stop the pipeline by pressing the Stop button.

To check the content of the created files, log into the running container and enter the following (to see, for example, the content of file test_1.txt ).

hdfs dfs -cat /tmp/hdfsManager/test_1.txt

Attention: You might notice that the first line of file test_1.txt does not necessarily have the counter 0 (that is the first column of the file and in the above screenshot it has the counter 123) as one might expect when looking at the JavaScript snippet which is executed by the Data Generator. The reason for this can be that the Kafka Producer had sent a message to Kafka, but then you most likely stopped the pipeline before the Kafka Consumer consumed the message. When you afterwards restarted the pipeline, the Kafka Consumer first of all processed this “previous” message. For the sake of this tutorial you do not have to bother about this behavior.

Now log into the running container.

docker exec -it 8560ee632f69 /bin/bash

Replace 8560ee632f69 with your Container ID while executing

Then enter the following commands.

hdfs dfs -ls /tmp/hdfsManager/*

You will see a lot of files as seen below.

Now run the below command to open one of files that you see above.

hdfs dfs -cat /tmp/hdfsManager/test_01.txt

Replace 01 with your file number while executing

Copy any row of the file starting from second line and paste it in the frame below and click on Validate.