Numbers

Due: February 27, 2009 23:59:59 EST
You will submit your suggestions in the usual way. The only acceptable file formats are text and PDF (we prefer PDF, but will accept text). Furthermore, please attach your responses as PAPER_CODE.{pdf,txt}. For example, if one of the projects I was reviewing were called "0023", I would send an email to the TA with the attachment "0023.pdf". If this is at all unclear, ask for a clarification.

This assignment is worth 5% of your final grade, i.e. 33% of your "reviewer activities" grade. If you fail to complete this activity, you will have failed to fulfill your duties as a reviewer, which will reflect poorly on you at the end of the semester.

Why?

As good researchers, it is our responsibility to do our due diligence when tackling research problems; that is, we should see what other related work is already out there. This not only helps us to avoid reinventing the wheel, but is also prevents us from looking like complete buffoons when we present an idea as a "new" one when it was actually invented in the 60's. Surveying related literature might also lead to other good research questions, perhaps changing the way we had thought about the original problem.

Either way, this is an important step that often goes unmentioned. The purpose of this activity is for you to do a short survey of related work to help the project groups in their own background research. You will also suggest a set of experiments, possibly (likely?) inspired by the related work you find. The latter is in many ways more important than the former, but you cannot do a reasonable job of the latter without doing a reasonable job of the former. Take this seriously.

What to turn in

• You will receive emails that contain paper names and "paper codes". The paper code will be a 4-digit identifier. This corresponds to the formal proposal(s) for which you will provide feedback. The proposals can be found at: http://www.cc.gatech.edu/~isbell/classes/2008/cs7641_spring/proposals/PAPER_CODE.pdf , where PAPER_CODE is the 4-digit id.

• If you have a conflict of interest (regarding one of the groups/papers you are assigned), you must bring it to our attention ASAP.

• The reviews will be single-blind, which means you will know the groups you are evaluating, but they will not know who you are. Consequently, do not put your name on the reviews.

• Ok, enough administrivia. For each formal proposal you are assigned, you must submit a response which will be handed to the groups to aid them in their own search for related work. The response cannot be longer than 3 pages single-spaced, including references, at, say, 10pt font at a minimum. This is a strict limit (it sounds easy, but is actually kind of hard if you plan on writing a good response). Each group has 3-4 reviewers, so I want the responses short enough to be useful to the groups. Ideally, you will have 3 dense pages of very useful and insightful information. I would hope that your first pass at this will generate more than 3 pages, after which you will edit out all but the essential goodies.

  Each response must contain the following sections:

  1. Overview: This includes a brief summary of the problem being tackled and a brief summary of the approach being taken.

  2. Related Work: You must provide 3 to 4 related works. These cannot include the references from the corresponding formal proposal; however, you should read those references, as they might be good starting points to find other related work. Once you decide on a set of related work to suggest, you should briefly describe each one, and how it relates. Your writing needs to be concise and coherent, otherwise it will be useless to the groups. Good ways to find related work: find one paper that is related and look at the papers it references, ad infinitum; talk with experts in the field; look at old proceedings from conferences such as NIPS and ICML, ....

  3. Suggestions for Experiments: You should suggest 3-4 experiments the group should try, presumably inspired by your scan of related work (though you may end up having a brilliant suggestion totally independent of the related work you find). Each suggestion consists of a precise description of the suggested experiment and a short explanation of what key question(s) the experiment will help resolve, and how.

  4. References: You must provide complete references to the related works you cite. By "complete" all I mean is that you must provide a formal citation, or at least formal-enough so that the groups can track down the papers you suggest.