Important

Jupyter widgets cannot be run within in documentation. To be able to interact with the widget you must run a mybinder instance. To run a mybinder instance of this notebook please use this link https://mybinder.org/v2/gh/osscar-org/scicode-widgets/HEAD?labpath=docs%2Fsrc%2Fexercises.ipynb.

Writing checks

Checks intention is to give students a way to validate their code solution. The student’s code can be validated by providing a list of inputs and references comparing the output of the student’s code with the references, or by directy testing certain functional behavior of the code. In cases when reference outputs need to be obfuscated the outputs that are compared can be passed through a fingerprint function. This notebook goes through each of these features and presents an example.

from scwidgets import CodeInput, CodeExercise, Check, CheckRegistry, ExerciseRegistry

import numpy as np

Similar as for the ExerciseRegistry we need to define a CheckRegistry that registers the checks for each exercise.

check_registry = CheckRegistry()

Checks using inputs and output references

def sin(arr):
    import numpy as np
    return np.cos(arr) # oops! wrong solution


check_code_ex = CodeExercise(
    key="sinus_with_references_1",
    code=sin,
    check_registry=check_registry,
)

# An assert function returns a string that specifies
# the error message to the student, or is empty if the
# check passes
def my_assert_allclose(outputs, references) -> str:
    if not np.allclose(outputs, references):
        return "Your output is not close to the references."
    else:
        return "" # We use empty strings means it passes

check_registry.add_check(
    check_code_ex,
    asserts=[
        my_assert_allclose
    ],
    inputs_parameters=[{"arr": np.asarray([0., np.pi, 2*np.pi])}],
    outputs_references=[(np.asarray([0., 0., 0.]),)]
)

check_code_ex.run_check()
check_code_ex

Because there are asserts that are repeatedly needed for almost any kind of exercise, we provide of a set of asserts

from scwidgets import (
    assert_numpy_allclose,
    assert_shape,
    assert_type,
)

def sinus(arr):
    import numpy as np
    return np.cos(arr) # oops! wrong solution

check_code_ex = CodeExercise(
    key="sinus_with_references_2",
    title="sinus",
    code=sinus,
    check_registry=check_registry,
)

check_registry.add_check(
    check_code_ex,
    asserts=[
        assert_type, # checks if same type as reference values
        assert_shape, # checks if same shape as reference values
        assert_numpy_allclose, # checks if allclose to reference values
    ],
    inputs_parameters=[{"arr": np.asarray([0., 0.78539816, 1.57079633, 2.35619449, 3.14159265])}],
    outputs_references=[(np.asarray([0., 7.07106781e-01, 1.00000000e+00, 7.07106781e-01, 0.]),)]
)

#check_code_ex.run_check()
check_code_ex

One can adapt the default arguments of the asserts by using partial functions

assert_numpy_allclose?

from functools import partial

custom_assert_numpy_allclose = partial(assert_numpy_allclose, rtol=1e-7)

Testing functional behavior

def sinus(arr):
    import numpy as np
    return np.cos(arr) # oops! wrong solution

code_ex_functional_behavior = CodeExercise(
    key="sinus_functional_behavior",
    code=sinus,
    check_registry=check_registry,
)

def assert_2pi_periodic() -> str:
    out = code_ex_functional_behavior.code([0, 2*np.pi])
    if not np.allclose(out[0], out[1]):
        return "Function is not periodic."
    return "" # empty strings means it passes

check_registry.add_check(
    code_ex_functional_behavior,
    asserts=[
        assert_2pi_periodic,
    ]
)

code_ex_functional_behavior.run_check()
code_ex_functional_behavior

Obfuscating the reference solution with fingerprint

from scwidgets.check import (
    assert_equal
)

def riddle():
    """
    Please return as string the answer to this riddle:

    What has wings but in the air it not swings.
    I looked to the north, but it was not worth.
    What I am looking for?
    """
    return ""
code_input_sinus = CodeInput(riddle)

check_code_ex = CodeExercise(
    key="riddle",
    code=code_input_sinus,
    check_registry=check_registry,
)

#def assert_equal(output, reference):
#    return "" if output == reference else "Not correct solution. Hint: it is an animal in the antarctica."

char_to_num = {char: num for num, char in enumerate("abcdefghijklmnopqrmnstuvwxyz")}
def string_to_int(output):
    return sum([char_to_num[char] for char in output])

check_registry.add_check(
    check_code_ex,
    asserts=[
        assert_equal,
    ],
    fingerprint = string_to_int,
    inputs_parameters=[{}],
    outputs_references=[(93,),],
    suppress_fingerprint_asserts = True # By default we do not show the error message, since it is confusing with the fingerprint
)

check_code_ex.run_check()
check_code_ex

Checking all widgets

The check registry also provides the possibility to check all the widgets.

check_registry

For the demo to automatically we simulate a button press using the private function that should not be used

check_registry._check_all_widgets_button.click()