Generator

Overview

This component is responsible for generating requests and storing them in a format that could be submitted to the server. The user can declare the requests by leveraging the functions inside the library in tests/infra/piccolo/generator.py. The user can generate requests from the library by either calling the command line tool in tests/infra/piccolo/generate_packages.py or by creating a script calling the functions of the library, such as the tests/infra/piccolo/logging_generator.py which contains a sample generation of requests for the logging CCF application.

Run generator

The generator component offers a command-line tool for basic scenarios as well as a sample for an application for more complex scenarios.

Command-Line Tool

For the command line option you need to run the following:

$ python3 generate_packages.py

After executing this command there will be a .parquet file produced in the same directory containing the data with the requests. You can configure the generated requests using the following arguments:

-h, --help show this help message and exit
-hs HOST, --host HOST The host to submit the request. (default: 127.0.0.1:8000)
-p PATH, --path PATH  The relative path to submit the request. (default: /app/log/private)
-vr VERB, --verb VERB The request action. (default: POST)
-r ROWS, --rows ROWS  The number of requests to send. (default: 16)
-rt REQUEST_TYPE, --request_type REQUEST_TYPE The transfer protocol for the request. (default: HTTP/1.1)
-pf PATH_TO_PARQUET, --path_to_parquet PATH_TO_PARQUET Path to the parquet file to store the generated requests (default: ./requests.parquet)
-ct CONTENT_TYPE, --content_type CONTENT_TYPE The Content-Type representation header is used to indicate the original media type of the resource. (default: application-json)
-d DATA, --data DATA  A string with the data to be sent with a request (default: {"id": 1, "msg": "Send message with id 1"})

Scripting Requests

To script the requests, a Messages object must be first created and the two main functions append() and parquet_to_file() get called. The append() function will create a dataframe containing an id for each request and the request in a raw format. The generated-dataframe will first be concatenated to the object-dataframe and then the generated-dataframe will be returned to the user. The user could then, based on the previously returned dataframes, interleave requests or implement as complex scenarios as he wants. The append() requires the server, the path and the request method to create the requests, if a method that requires data is specified then the user can declare the data as well. The content-tye and the request version can also be specified if the defaults are not preferred. Lastly, the user can also declare the number of times the same request is generated by providing the number on the iterations argument.

After finishing the generation of the requests, the to_parquet_file() function must be called to write the concatenated object-dataframe on the file specified in the arguments. Then you can run your script as you would run any python file:

$ python3 logging_generator.py

Parquet files are an easy and well-compressed way of storing requests generated from this component to run the same generated requests multiple times on the same submitter under different circumstances.