Student Implementations

For tracking and managing student implementations, stored in PyBryt as a list of 2-tuples of the objects observed while tracing students’ code and the timestamps of those objects, PyBryt provides the StudentImplementation class. The constructor for this class takes in either a path to a Jupyter Notebook file or a notebook object read in with nbformat.

stu = pybryt.StudentImplementation("subm.ipynb")

# or...
nb = nbformat.read("subm.ipynb")
stu = pybryt.StudentImplementation(nb)

Notebook Execution

The constructor reads the notebook file and stores the student’s code. It then proceeds to execute the student’s notebook using nbformat’s ExecutePreprocessor. The memory footprint of the student’s code is constructed by executing the notebook with a trace function that tracks every value created and accessed by the student’s code and the timestamps at which those values were observed. PyBryt also tracks all of the function calls that occur during execution.

To trace into code written in specific files, use the addl_filenames argument of the constructor to pass a list of absolute paths of files to trace inside of. This can be useful for cases in which a testing harness is being used to test student’s code and the student’s actual submission is written in a Python script (which PyBryt would by default not trace).

stu = pybryt.StudentImplementation("harness.ipynb", addl_filenames=["subm.py"])

To prevent notebooks from getting stuck in a loop or from taking up too many resources, PyBryt automatically sets a timeout of 1200 seconds for each notebook to execute. This cap can be changed using the timeout argument to the constructor, and can be removed by setting that value to None:

stu = pybryt.StudentImplementation("subm.ipynb", timeout=2000)

# no timeout
stu = pybryt.StudentImplementation("subm.ipynb", timeout=None)

PyBryt also employs various custom notebook preprocessors for handling special cases that occur in the code to allow different types of values to be checked. To see the exact version of the code that PyBryt executes, set output to a path to a notebook that PyBryt will write with the executed notebook. You can also access this notebook as an nbformat.NotebookNode object using StudentImplementation.executed_nb This can be useful e.g. for debugging reference implementations by inserting print statements that show the values at various stages of execution.

stu = pybryt.StudentImplementation("subm.ipynb", output="executed-subm.ipynb")

If there is code in a student notebook that should not be traced by PyBryt, wrap it PyBryt’s no_tracing context manager. Any code inside this context will not be traced (if PyBryt is tracing the call stack). If no tracing is occurring, no action is taken.

with pybryt.no_tracing():
    foo(1)

Checking Implementations

To reconcile a student implementation with a set of reference implementations, use the StudentImplementation.check method, which takes in a single ReferenceImplementation object, or a list of them, and returns a ReferenceResult object (or a list of them). This method simply abstracts away managing the memory footprint tracked by the StudentImplementation object and calls the ReferenceImplementation.run method for each provided reference implementation.

ref = pybryt.ReferenceImplementation.load("reference.pkl")
stu = pybryt.StudentImplementation("subm.ipynb")
stu.check(ref)

To run the references for a single group of annotations, pass the group argument, which should be a string that corresponds to the name of a group of annotations. For example, to run the checks for a single question in a reference that contains multiple questions, the pattern might be

stu.check(ref, group="q1")

Checking from the Notebook

For running checks against a reference implementation from inside the notebook, PyBryt also provides the context manager check. This context manager initializes PyBryt’s tracing function for any code executed inside of the context and generates a memory footprint of that code, which can be reconciled against a reference implementation. The context manager prints a report when it exits to inform students of any messages and the passing or failing of each reference.

A general pattern for using this context manager would be to have students encapsulate some logic in a function and then write test cases that are checked by the reference implementation inside the context manager. For exmaple, consider the median example below:

def median(S):
    sorted_S = sorted(S)
    size_of_set = len(S)
    middle = size_of_set // 2
    is_set_size_even = (size_of_set % 2) == 0
    if is_set_size_even:
        return (sorted_S[middle-1] + sorted_S[middle]) / 2
    else:
        return sorted_S[middle]

with pybryt.check("median.pkl"):
    import numpy as np
    np.random.seed(42)
    for _ in range(10):
        vals = [np.random.randint(-1000, 1000) for _ in range(np.random.randint(1, 1000))]
        val = median(vals)

The check context manager takes as its arguments a path to a reference implementation, a reference implementation object, or lists thereof. By default, the report printed out at the end includes the results of all reference implementations being checked; this can be changed using the show_only argument, which takes on 3 values: {"satisfied", "unsatisfied", None}. If it is set to "satisfied", only the results of satisfied reference will be included (unless there are none and fill_empty is True), and similarly for "unsatisfied".

Working with Memory Footprints

Once the notebook has been executed, which happens when the constructor is called, the submission’s memory footprint can be found in the footprint field of the StudentImplementation object. This field contains a MemoryFootprint object, which has fields and method for accessing the values in the footprint, the function calls observed by the trace function, the set of imported modules, and the processed notebook that was executed by PyBryt. See the API reference for MemoryFootprint objects for more information.

Storing Implementations

Because generating the memory footprints of students’ code can be time consuming and computationally expensive, StudentImplementation objects can also be serialized to make multiple runs across sessions easier. The StudentImplementation class provides the dump and load methods, which function the same as with reference implementations. StudentImplementation objects can also be serialized to base-64-encoded strings using the dumps and loads methods.