Ask HN: Model sycophancy benchmarks?

Posted by gradus_ad 1 day ago

Counter5Comment2OpenOriginal

Model sycophancy is going to become a major issue for individuals and society... Do any benchmarks exist for this? Really needs to become a standard aspect of model quality measures

Comments

Comment by tomveber 1 day ago

Closest I know of are SycEval and the sycophancy evals in Anthropic's 2023 paper, both built on a user pushing back at a correct answer.

Comment by muddi900 1 day ago

I think we should design a "Bullshit Benchmark" that tests for sycophancy and fluff like Opus 5 "two X, but only Y matters" color commentary