Jump in the discussion.

No email address required.

https://old.reddit.com/r/MachineLearning/comments/11rizyb/d_anyone_else_witnessing_a_panic_inside_nlp_orgs/jca3wtv/

It's a darn shame you can't build a Ph.D. career on reproducing (or failing to reproduce) weak-butt SOTA papers that get their performance gains with "stochastic grad student descent" -- that is, have the grad researcher try lots of runs until one randomly hits SOTA, save the seed, and publish.

jfc this is way too real

Jump in the discussion.

No email address required.

If only stemcels could build a career out of reproducing studies to improve replication. Robert Maxwell is in heck rn :marseylaptopsad:

Jump in the discussion.

No email address required.

I know it's an issue related to replication crisis either way, but I feel that STEMcels (as in hard sciences) should at least always guarantee that their results can be replicated. I'm not sure how invested you are in this topic, but this paper made by some google engineers or something tries to compare various transformer architectures across tasks and they come to the conclusion that if you accurately adjust parameters, seed and whatever else you can think of, almost every architecture can get a 'lucky' run resulting in SOTA performance.

tl;dr researchers lie

Jump in the discussion.

No email address required.

I'm in pharma and it was only 10 years ago where bayer did an in house replication study of their own r&d and only around a 1/3 could be replicated.

"Pharmaceuticals company Bayer recently revealed that it fails to replicate about two-thirds of published studies identifying possible drug targets (Nature Reviews Drug Discovery, vol 10, p 712). This rate of failure is higher than rival Amgen which reported that over the past decade its oncology and haematology researchers could not replicate 47 of 53 highly promising results they examined (Nature, vol 483, p 531)."

https://www.newscientist.com/article/mg21528826-000-is-medical-science-built-on-shaky-foundations/

Everything we take for granted could all be just a lie so some academic wagies can keep their grants and tenure. :marseydoomer:

Jump in the discussion.

No email address required.

The need for research has long since been overtaken by the actual amount of research being done. Thus, in order to distinguish yourself amongst the plethora of released findings you basically have to exaggerate or straight up lie to get any sort of attention - regardless in which field you are in. I suppose this is way worse when it comes to fields that rely on empirical methods and observations, but it is still disheartening to read statements such as yours that even 'hard' sciences are plagued by it.

Jump in the discussion.

No email address required.

It's dangerous since these industries and universities are using their research as justification for policymakers and societal/cultural movements. What even is truthful in today's society with this as the backbone?

Climate science,psychology, medicine etc are especially bad for replication. Water management had a meta study done where it found only .6% of studies could be replicated.

"A 2019 study in Scientific Data estimated with 95% confidence that of 1,989 articles on water resources and management published in 2017, study results might be reproduced for only 0.6% to 6.8%, even if each of these articles were to provide sufficient information that allowed for replication."

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6390703/

:marseyhelp:

Jump in the discussion.

No email address required.

but don't forget to Trust the Science™ when it comes to jabs and climate

Jump in the discussion.

No email address required.

Link copied to clipboard
Action successful!
Error, please refresh the page and try again.