Mutat(e?) specs.

Mutat = Tatum backwards = COMP2304 automarking backend. 

Marking requirements 

 

• Test cases and meta data are explicitely registered. A hash? of the file is taken. If the file changes between now and when other tests are run then notify the user so that they can re-register it. Once registered, all test case info is retrieved from database rather than files, so that accidently modified files do not harm future marking.
• If test cases are re-registered then any marking that has already been done must be redone. Penalties must be preserved. Only the most recent submission within each resubmission category is marked. Multiple resubmissions are marked and the highest result is associated with the student. Files must be archived in order to ensure that the correct copy of the students code is marked. 
  (This has to be super easy. Changing a test case and ensuring good update results is critical).
• Marking can be haph-hazard 

1. Mark bulk of submissions when they arrive. 
2. Mark some other students code when he hands it in late without penalty. Mark the late submissions with penalties. 
3. Mark resubmissions (again in several batches).
4.  Remark a file for ONE STUDENT who has ALREADY SUBMITTED (but perhaps submitted the wrong copy) and CONFIRM before replacing their result. 
5. Do not let a student submit multiple times without confirmation from a tutor. 

• A system for  applying penalties for late submission and resubmission so that those penalties can be lifted by tutors. The penalty state must somehow? be mirrored on the live server without being killed by the development server.
• Mark assignments without any danger of hurting previous ones.
• Automatic database backup so I can stop using a development and live server. If assignments are re-marked several times due to faulty test cases it should be OK.

Usability requirements

• Too many command line arguments – run through a simple GUI to ensure all arguments are correct.
• Progress bar
• Compile and run each assignment as it goes – no need to compile all first and then find a bug in the running code.
• Ability to resume – if I have marked 50% of assignments do I need to remark them all if one piece of code critical crashes (assuming I have not modified the automarker source or test cases in between – if modified force the ENTIRE GROUP to be marked again – including any previous submissions).
• Reduce the dependence on additional script files (db2csv and dbsync in particular). 

Mark security requirements

• The state of the OFFICIAL marks .csv files should ALWAYS EXACTLY mirror the results stored in the databases. 
• Should not be able to directly edit the .csv file (because then database will not reflect changes)

Time requirements

• < 30 minutes 

Test case requirements

• Minimal new work to write a test case i.e. copy meta data and edit code.
• Must be header guard safe (i.e. not cause any NEW linking errors that students would not have found when carefully including header files in exactly the right places).
• Written in native C++ so that all functionality can be tested.
• Can only award full (true) or none (false) marks to make writing tests easier.
• Should all be contained within one file.
• If possible perhaps students should be able to view the code which is being used to test their code? This is good for openness (especially if automarker is open source).
• Most lenience possible i.e. do not fail if student forgot to hand in one file or failed to implemented one function – run all tests that do not use that function.

Security of marker

• Prevent infinite loops
• Handle crashes nicely
• Prevent? disk writes?
• Do not let students know critical directories to write to if they see the automarker source i.e. do not hard code the results database name and login!

Student testbed

• Allow students to test all their files for the assignment against a core set of example test cases. For each test case inform them of what was being tested, if they passed or failed, and why. 
• Ability to run a subset of tests?
• Shoud? run WITHOUT the benfit of python
• IF THE AUTOMARKER IS OPEN SOURCE AND SOME OF IT IS IN PYTHON AND PYTHON IS ON THE UNI COMPUTERS THEN CAN WE USE PYTHON? IF OPEN SOURCE WE DO NOT CARE IF STUDENTS SEE THE CODE! THIS COULD SAVE A LOT OF EFFORT.
• Should compile files, inject code, and run the minimum set of tests.
• Ideally the smallest possible set of files 

1. A minimal python script – no database – possibly with a GUI
2. An interface between python and dlls (converter.exe?)
3. The list of test cases? If open source, again, does not matter if these are visible. I love openness and honesty.
4. I would like people to write their own test functions in their main etc – will this discourage them? or will they perhaps simply learn to write them as part of this new file. 
5. At least ensures that they meet the minimum standards defined by the examples in the assignments (no silly errors like extra whitespace)

• This means that when writing the python script I should modularize it in to core functions (given to students) and logging functions etc (the database and statistics code) -> database information displayed to stdout. I would rather not have a common class structure one which does database and one which does stdout because I tried this before and it was super annoying to maintain in python (no enforced class structure)

Web Interface (Tatum)

• Alternative to test exe given to students.
• Essentially the test exe accessable through a web interface
• Decorate the exe with enough CGI methods to make it accessable through a web interface. 
• Ability to write test cases – some sort of intermediate language would be nice but makes testing of the full C++ range of ideas (templtaes exceptions etc) would be nice but I think that normal C++ is probably the best bet.
• I really want people writing their own test functions.

In order to address all these issues I am going to need to write the automarker from scratch, identifying the minimal requirements models and double tripple checking that my database schema supports all the operations I would like.