The motivation of this blog post is from recent discussion on Twitter, where researchers share their experience and taste of research.
But the most original stimulus is Jacob and Lipton’s paper:
and a not so well-known piece of research process abstraction by Jacob:
I want to highlight their main points and to absorb them into my own research taste, as I am special in Natural Language Processing (NLP) or Computational Linguistics (CL), which may be (quite) different from ML research.
This is blog post should not be treated as a methodology discussion on how to do tasteful or influential research, but a retrospect of my own research attitude, which might not be personalized to others who are doomed to have different research experiene.
[1] is written by John Schulman who is a research scientist at OpenAI and his work on policy optimization or policy gradient, e.g. Trust Region Policy Optimization is the one of the most well-known research direction in deep reinforcement learning. His blog post mainly talks about how to conduct research in ML. The article’s main structure is:
How to choose problem?
Understand, compare and criticise what makes a work influential or obscure (obsolete)?
xxx = derive, understand, implement)John advocates goal-driven research instead of idea-driven research, since goal-driven research is the one that you can continually dedicate yourself into.
In my opinion, idea-driven research comes like some hunches which strike on you randomly but causally due to the recent papers you are reading. It is like a brick in a research program or project.
John’s definition of idea-driven research
“Follow some sector of the literature. As you read a paper showing how to do X, you have an idea of how to do X even. better. Then you embark on a project to test your idea.”
But goal-driven research is more like a programme which you conduct and insist on in a long run and may finally make certain unignorable contribution to the community that identifies yourself with what you have done.
John’s definition of goal-driven research
“Develop a vision of some new capabilities you’d like to achieve, and solve problems that bring you closer to that goal. In your experimentation, you test a variety of existing methods from the literature, and then you develop your own methods that improve on them.”
How to make continuous progress on the problem?
Keep a notebook where to record your ideas, observations about the field its super and sub fields. The following excerpts are really motivating.
“I have done this through 5 years of grad school and 2 years at OpenAI”
“I create an entry for each day. In this entry, I write down what I’m doing, ideas I have, and experimental results”
“Every 1 or 2 weeks, I do a review, where I read all of my daily entries and I condense the information into a summary”
“Usually, my review contains sections for experimental findings, insights, code progress, and next steps/future work.”
“I often look at the previous week to see if I followed up on everything I thought of that week.”
Switch problem less often Stick to write a paper, and list the unsolved research questions of the paper, which always motivate better formulation of research questions for you to work upon.
“spend one day per week on something totally different from your main project. This would constitue a kind of epsilon-greedy exploration, and it would also help to broaden your knowledge.”
and How to improve yourself generally during the above procedures?
Above is a summary of [1], which is very handy.
[2] is my favorite paper till now during my PhD persuit, since it gives me standard ordos and donts when look into others or my own research.