How do I write an effective Test Suite in Cody Coursework?

1 view (last 30 days)
With the introduction of Cody Coursework, I'm often asked how to best use the power of MATLAB to effectively challenge students' code.
  1 Comment
Ben Hawkins
Ben Hawkins on 27 Oct 2017
This answer is very outdated. Unfortunately, it shows up really high on Google search results. None of these are necessary with recent updates to Cody, in my experience.

Sign in to comment.

Accepted Answer

MathWorks Community Team
MathWorks Community Team on 27 Feb 2014
Edited: MathWorks Community Team on 9 Jun 2014
First, we need to understand why MathWorks has made the Test Suite visible to students. Students are encouraged to use MATLAB on their computer to solve problems. To make this possible, students must be able to copy and paste the Test Suite code into a MATLAB script which, in turn, calls their solution function. Once satisfied, they then copy their solution function into Cody Coursework for assessment.
With this in mind, consider how a student might "game the system" presented with the following Test Suite from the Times2 problem on Cody:
assert(isequal(times2(1),2));
assert(isequal(times2(11),22));
assert(isequal(times2(-3),-6));
assert(isequal(times2(29),58));
Here, the Test Suite simply calls the students' function, times2, using explicit data values to confirm the students' code returns the expected result. This approach is morally hazardous, as a student can simply write a case or series of if statements to compromise the approach.
Each Test Suite is a MATLAB script. Here we draw on this power to more effectively assess students' code in response to the Times2 problem,
X=rand(1);
Y_answer = x*2;
assert(times2(x),Y_answer)
This still poses two issues, 1. it reveals the answer the students' code needs to implement (i.e. return x * 2), and 2. it does not apply well to complex programming problems.
Now that we've examined what not to do, let's discuss a more effective approach for assessing students' MATLAB code. Faculty already have a reference solution to each programming problem assigned, traditionally offered to teaching assistant staff for manual grading. Protected MATLAB code offers the key of how we use these reference solutions as part of a generalized recipe to write an effective Test Suite.
First, compile your reference solution as Protected MATLAB code.
Second, post this .p file somewhere online accessible by Cody Coursework. This must not be behind the school's firewall, it must be publicly accessible. One option is to use the public folder of a free dropbox account. If using DropBox, it will help to " copy the public link " using their web interface.
Third, implement a test case of your test suite to use your protected reference solution code, for example:
%%Test Functional Correctness of Students’ Code
urlwrite('http://url/ProfsFunction.p','./ProfsFunction.p');
addpath('.');
 
for i=1:10
  data_vector=rand(1000,1);
  prof_answer = ProfsFunction(data_vector);
  student_answer = StudentsFunction(data_vector);
  assert(isequal(prof_answer,student_answer), 'student code gave an incorrect answer'); 
end
Adapting this for more complex problems will require you adapt the data_vector randomization to something appropriate for your subject matter. Alternatively, you might wish to incorporate a data file from which your Test Suite randomly selects input/output values.
Fourth, add a test case to prevent student attempts to call on the .p file themselves, for example by parsing the students' solution file directly,
%%Test for attempts to cheat
found = 0;
fid = fopen('StudentsFunction.m');
while(~feof(fid))
   s = fgetl(fid);
   k = regexp(s,'urlwrite\(|ftp\(|mget\(|web\(|fprintf\(|fwrite\(|sprintf\(');
   if ~isempty(k)
       found = 1;
   end
end
fclose(fid);
assert(isequal(found,0))
Fifth, use the solution map as a "line of last defense." Typically, honest solutions which are correct follow similar patterns. Horizontal bars of these patterns form on the solution map. The few stray solutions below these bars which Cody Coursework has marked correct merit manual code inspection.
There are many possibilities the power of the MATLAB language offers for implementing test suites. We will add more information and examples in future, and I encourage you to share your own insights as answers or comments below.
  2 Comments
Jan
Jan on 23 Mar 2014
Searching for "web" will cause false hits, e.g. when it occurs in a comment. It will be very frustrating for a student, when his solution fails, because his name contains this character sequence.
After several discussions in this forum it is clear, that the contents of P-coded functions is not hidden securely, because the debugger reveals almost all details. I would encourage any teacher to read the code of the students personally, while the automatic assessment can and will be cheated.
[EDITED] You can let Cody's virtual machine load and run code from external links? Isn't this a massive security problem?

Sign in to comment.

More Answers (0)

Products

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!