Important

Jupyter widgets cannot be run within the documentation. To interact with the widget, you must run a mybinder instance. To run a mybinder instance of this notebook, please use this link https://mybinder.org/v2/gh/osscar-org/scicode-widgets/HEAD?labpath=docs%2Fsrc%2Fcheck.ipynb. Note that also the LaTeX rendering is resolved when running the notebook.

Writing checks

The purpose of a check is to give students a way to validate their code solutions. The student’s code can be validated by providing a list of inputs and reference outputs. Once the student presses on the Check Code button, the reference outputs are compared to the outputs of the student’s code. Furthermore, in cases when the reference outputs need to be obfuscated so the student does not see the solution, the outputs can be passed through a fingerprint function before validation. Another supported form of validation is to test functional behavior of the student’s code, for example identity checks. This notebook goes through each of these features and presents an example.

from scwidgets import CodeInput, CodeExercise, Check, CheckRegistry, ExerciseRegistry

import numpy as np

Similar to the ExerciseRegistry, we need to define a CheckRegistry that registers the checks for each exercise.

check_registry = CheckRegistry()

Checks using inputs and output references

def sin(arr):
    import numpy as np
    return np.cos(arr) # oops! wrong solution


check_code_ex = CodeExercise(
    key="sine_with_references_1",
    code=sin,
    check_registry=check_registry,
)

# An assert function returns a string that specifies
# the error message to the student, or is empty if the
# check passes
def my_assert_allclose(outputs, references) -> str:
    if not np.allclose(outputs, references):
        return "Your output is not close to the references."
    else:
        return "" # We use empty strings means it passes

check_registry.add_check(
    check_code_ex,
    asserts=[
        my_assert_allclose
    ],
    inputs_parameters=[{"arr": np.asarray([0., np.pi, 2*np.pi])}],
    outputs_references=[(np.asarray([0., 0., 0.]),)]
)

check_code_ex.run_check()
check_code_ex

Since some asserts are frequently needed across various exercises, we provide a common set of asserts.

from scwidgets import (
    assert_numpy_allclose,
    assert_shape,
    assert_type,
)

def sine(arr):
    import numpy as np
    return np.cos(arr) # oops! wrong solution

check_code_ex = CodeExercise(
    key="sine_with_references_2",
    title="sine",
    code=sine,
    check_registry=check_registry,
)

check_registry.add_check(
    check_code_ex,
    asserts=[
        assert_type, # checks if same type as reference values
        assert_shape, # checks if same shape as reference values
        assert_numpy_allclose, # checks if allclose to reference values
    ],
    inputs_parameters=[{"arr": np.asarray([0., 0.78539816, 1.57079633, 2.35619449, 3.14159265])}],
    outputs_references=[(np.asarray([0., 7.07106781e-01, 1.00000000e+00, 7.07106781e-01, 0.]),)]
)

#check_code_ex.run_check()
check_code_ex

One can adapt the default arguments of the asserts by using partial functions

assert_numpy_allclose?

from functools import partial

custom_assert_numpy_allclose = partial(assert_numpy_allclose, rtol=1e-7)

Testing functional behavior

def sine(arr):
    import numpy as np
    return np.cos(arr) # oops! wrong solution

code_ex_functional_behavior = CodeExercise(
    key="sine_functional_behavior",
    code=sine,
    check_registry=check_registry,
)

def assert_2pi_periodic() -> str:
    out = code_ex_functional_behavior.code([0, 2*np.pi])
    if not np.allclose(out[0], out[1]):
        return "Function is not periodic."
    return "" # empty strings means it passes

check_registry.add_check(
    code_ex_functional_behavior,
    asserts=[
        assert_2pi_periodic,
    ]
)

code_ex_functional_behavior.run_check()
code_ex_functional_behavior

Obfuscating the reference solution with a fingerprint

from scwidgets.check import (
    assert_equal
)

def riddle():
    """
    Please return as string the answer to this riddle:

    What has wings but in the air it not swings.
    I looked to the north, but it was not worth.
    What am I looking for?
    """
    return ""
code_input_sine = CodeInput(riddle)

check_code_ex = CodeExercise(
    key="riddle",
    code=code_input_sine,
    check_registry=check_registry,
)

#def assert_equal(output, reference):
#    return "" if output == reference else "Not correct solution. Hint: it is an animal in the Antarctica."

char_to_num = {char: num for num, char in enumerate("abcdefghijklmnopqrmnstuvwxyz")}
def string_to_int(output):
    return sum([char_to_num[char] for char in output])

check_registry.add_check(
    check_code_ex,
    asserts=[
        assert_equal,
    ],
    fingerprint = string_to_int,
    inputs_parameters=[{}],
    outputs_references=[(93,),],
    suppress_fingerprint_asserts = True # By default we do not show the error message, since it is confusing with the fingerprint
)

check_code_ex.run_check()
check_code_ex

Checking all widgets

The check registry also allows checking of all the widgets simultaneously.

check_registry

For the demo, we simulate a button press using the private function that should not be used

check_registry._check_all_widgets_button.click()