Evaluations

Evaluations are a critical component of the GSoC program. Both at mid-term and at the end, evaluations serve to expose process flaws, assess performance, and precipitate pass/fail decisions. Taking time to evaluate the progress and workflow of the project provides an opportunity to correct course and address underlying issues. Most importantly, it provides a structure in which to give valuable criticism and praise to the students. After all, they are supposed to be actively learning (not just working) and effective learning requires evaluation.

Unlike code critique, which happens openly on mailing lists or other public forums, performance evaluations are personal in nature. Delivering the mentor-student evaluations should be done privately. And more generally, remember the maxim: criticize in private, praise in public.

There are four types of evaluations:

Student evaluations: Each student fills out a mid-term and final evaluation pertaining to their experience and their interaction with mentors.

Mentor evaluations: Each mentor fills out a mid-term and final evaluation covering their participation and their student's performance.

Org Admin evaluations: Each org admin fills out a mid-term and final evaluation about their role in the program.

Program evaluations: Every participant in GSoC answers questions about how the program is operating over all.

These are provided online by Google at specified times with deadlines. The deadlines are important for organizing the payments to students based on the pass/fail decisions by their mentors, so you should ensure that you complete your evaluations on time. Remember, if a student fails at the mid-term, they will no longer be part of the program and will not receive any more payments from Google.

They Won't Know Unless You Tell Them

It is important to note that other than the pass/fail status, students do not have direct access to the evaluations of their performance. Mentors are welcome to make a copy of their evaluation and share it with students directly. Students are welcome to do the same, but it is not required for either the mentor or student to share their evaluations with the other. Make sure to thoroughly check the topics covered by the evaluations regardless of your choice to share them with one another.

Mentors may not have access to student evaluations. Org admins have access to all evaluations submitted by their org's mentors and students.  Mentors have access only to their own evaluations, not to the evaluations submitted by their student.  Thus, if you want to give feedback to your student (and you should!) you need to send this feedback directly to them; you cannot rely on Google to do it for you.  While this may change in the future, it is important for mentors and org admins to coordinate in assessing the student and mentor evaluations and taking appropriate follow-up action.

Evaluations should result in a direct review of the student's progress and should be conducted using a real-time communication medium. This can be by phone, your favorite voice-over-IP service, or in person. The discussion should be frank and in the context of periodic review so that the student is prepared for criticism and to work with you to revise workflows, timelines and habits. Take time to explain the value of learning how to take criticism and praise in a professional setting.

When delivering reviews of student performance, be specific about both positive and negative aspects of a student's performance. Make the suggestions for change or improvement relevant to what your student is currently working on, and provide specific examples. As you prepare for your student's review, you might write it out as though you were sending an email; this will help you to frame your thoughts and to ensure that you are providing a balanced perspective. Make the student aware of the review schedule well in advance. If you provide a written copy of the review, schedule time for discussion immediately after the student reads it.

When in Doubt, Fail the Student Early

This is harsh-sounding advice. However, Leslie Hawthorn reports based on a back-of-the-envelope calculation that more than 85% of the students who are reported as marginal at or before the midterm eventually fail GSoC.  Whatever problems your student is currently having, they are likely to be worse than you currently appreciate, and to get worse rather than better over time.  You are not doing your student, much less yourself, a service by prolonging the agony.  Most GSoC students and their mentors have a great time and get a lot done.  If you are having the other kind of experience, cut your losses and try again next year.

Note that GSoC gives mentoring orgs quite a bit of flexibility and cuts them a lot of slack. In particular, if a student fails during the community bonding period, there is likely an opportunity for the organization to substitute another student, or to give the slot to another organization to do so.  Students need to understand that they are being evaluated from before they are accepted to the end of the program, and that you take the GSoC experience seriously and expect them to do likewise. 

It's Not Impossible to Turn Things Around

A true story: Once upon a time there was a student with limited English. The mentor considered the communication difficulty as a potential issue, but the student was enthusiastic, and hoped to use GSoC as an opportunity to improve his English.

The community bonding went well, but during the first half of coding, progress drifted off track. The student's mentor was much busier with work than was anticipated, and the student became stuck several times with various issues. The student failed to proactively ask for help, and the mentor didn't catch this, so the student became disheartened.

At the midterm, progress was disappointing, and the project came close to failing the student. But as the mentors looked at how they might try to rescue the situation, and after discussion with the student, the project came up with a concrete plan.  The student had a job in the lab for about 8 hours a week, which he agreed to put on hold until the end of GSoC. Each day the student updated a Google document with what he was working on. This included reviewing any outstanding issues which might block progress.  The student would camp on IRC during the hours he worked on his project.

Additionally, a second mentor was brought in, so that a mentor would always be available when the student might be working. Progress was discussed daily. The organization also made it clear that if the plan didn't succeed, that the failure would reflect badly on both the organization and the student. All those involved were happy to work to address any additional issues that might come up.

By the final evaluation, the student was almost back on his original project plan, and had completed all the required goals.

It is possible to rescue a failing student, but you need to really consider the issues, and come up with a plan to address them which everyone involved buys into. You also need to be prepared for the project to fail, and be okay with the extra energy you have invested not resulting in the outcome you wanted.

Mentor, Heal Thyself

This is also an important time for self-evaluation. Are you managing your time adequately? Do you know where the project is at and where it is going? Are you enforcing the deadlines you set? Are you integrating your student into your community?

Take the evaluation period as an opportunity to get feedback from students. Is there any way that you could have helped the student more? How does the student think you be more effective as a mentor? Ask your student directly for feedback.