Google's LearnLM can be *automatically* evaluated by another LLM-based critic/judge who can be trained to evaluate according to their seven pedagogical benchmarks
I think you may be right. I was very excited to read about these developments. Did you read the report? It gets a little testy at times. At one point, the education field is described as recently entering the 20th century with its emphasis on evidence. Let's just say... not the best moment in the report.
Thanks Nick! Yes, I read the report carefully, as my post above is a review/summary of its contents up to Sections 8 & 9. Indeed, you are correct it does quote the Slavin paper ("At the dawn of the 21st century, educational research is finally entering the 20th century").
I read that as a claim that few pedagogical theories are supported by rigorous, scientific approaches (e.g., randomized controlled trials) of the sort that enable generalizability. It reminds me of them later writing that "human tutoring data is scarce with only four datasets openly available to our knowledge, all of which suffer from limitations", which is why they leaned heavily on synthetic data.
I'm not qualified to evaluate these claims. But their methodology did make me feel like my own approach lacked scientific rigor! I think the achievements of the paper are (i) the evaluation benchmark methodology and (ii) the claim that the fined-tuned "LLM critic" was a successful evaluator of the tutor models (M0 to M4); i.e., LLMs to improve LLM tutors. If you believe in those two things, I think it's easy to envision improvements to their model. Thanks!
I think you may be right. I was very excited to read about these developments. Did you read the report? It gets a little testy at times. At one point, the education field is described as recently entering the 20th century with its emphasis on evidence. Let's just say... not the best moment in the report.
Thanks Nick! Yes, I read the report carefully, as my post above is a review/summary of its contents up to Sections 8 & 9. Indeed, you are correct it does quote the Slavin paper ("At the dawn of the 21st century, educational research is finally entering the 20th century").
I read that as a claim that few pedagogical theories are supported by rigorous, scientific approaches (e.g., randomized controlled trials) of the sort that enable generalizability. It reminds me of them later writing that "human tutoring data is scarce with only four datasets openly available to our knowledge, all of which suffer from limitations", which is why they leaned heavily on synthetic data.
I'm not qualified to evaluate these claims. But their methodology did make me feel like my own approach lacked scientific rigor! I think the achievements of the paper are (i) the evaluation benchmark methodology and (ii) the claim that the fined-tuned "LLM critic" was a successful evaluator of the tutor models (M0 to M4); i.e., LLMs to improve LLM tutors. If you believe in those two things, I think it's easy to envision improvements to their model. Thanks!