Compile times for COMP2304 Assignment 3

Graph of compilation times for 25 test cases per student, 200 students. First bar shows compilation against Boost Library which was an interesting experiment, and will be used in part in Automarker v3, but which I would never use again as the core testing method because exporting certain things (i.e. pass by pointer) is impossible.

 

Second option is compilation of the same code without boost. This still takes 1.2 hours which just isn’t practicle if you need to be able to remark an entire class often.

 

Third option is time taken if each file needs only be compiled once (3 minutes). This is a huge speed increase (25 x vs 5x for not using boost) so I’ll be concentrating my efforts on working on an efficient and automated way of determining code dependencies and compiling a file a minimum number of times.

One idea is to automatically decorate all the functions and classes in a file with dllspec instructions and then dynamically call them. If an error occurs then the student did not provide the method. This will obviously only work on windows. Unfortunately this won’t work with templates.

 

In fact, the only thing that will work with templates is injecting code directly into the students code before compiling.

 

Another idea is to start by compiling against all test cases and then if the students code does not compile, try compiling all 25 test cases seperately. To fit in a coherent system this would end with a single compilation against the N test cases which succesfully compiled before.  On average I would exepect 30% of students to submit incomplete solutions which would give a runtime of 25 minutes. Closer to acceptable but still not fantastic.

 

An ideal solution would identify exactly which test cases each student could compile against and only compile these ones. This would require:

1.  A way to identify what a given test method depends upon – this can get tricky for things like templates
2. A way to identify what the student has implemented. Could use regex but this almost requires implementing an entire C++ parser because of the huge number of ways there are to implement a function etc. For example, given a constructor, they may have declared it inline or outline. If the latter, you have to find both a prototype and implementation. I’m sure there are worse scenarios.

The final option is to enfoce a complete solution from each student so that it will at least compile. We can tell students that they must implement all prototype, but at least some will spell them incorrectly. Alternately we can provide a header file (this was done when I completed the course) but I believe that this means that the student won’t learn to write a header file themselves (it was certainly the case with some of the people who did the course with me). 

 

 

I’m not really sure where this leaves me. Let me know if you have any ideas! 

---

Possibly related posts: (automatically generated)

• Private college presidents’ pay rose 6.5% in 2008
• Is Cheating the New Studying?
• Jay Mathews: Retest D.C. Classes That Had Dubious Exam Results in ’08