Boosted by @trending@homestead.social
If you really think that AI running out of control is a dangerous possibility, then make tech company CEOs legally liable for illegal actions taken by their company's agents.
Remote
Dan Goodman
@neuralreckoning@neuromatch.social
mastodon 4.8.0-alpha.3+glitchI'm a computational neuroscientist and science reformer. I'm based at Imperial College London. I like to build things and organisations, including the Brian spiking neural network simulator, Neuromatch and the SNUFA spiking neural network community.
3049 Followers
328 Following
50 Posts
Joined November 29, 2022
Website:
Youtube:
Brian:
Neuromatch:
SNUFA:
Photography:
You might have read that arxiv is banning people for a year if they post LLM-generated papers and cheered it on. But most of the discussion about this doesn't correctly explain the policy and it is not a good thing.
First up, the policy is that "incontrovertible evidence" of using LLMs and not checking the output is what's at stake. An example given is a hallucinated reference.
Second, the ban will apply to all coauthors of the paper, not just the person submitting.
Third, it's not just a 1 year ban, it's followed by a permanent ban on submitting papers that have not been peer reviewed in a "reputable" journal or conference. Given that arxiv is a preprint server and not a repository for published papers, that makes the ban effectively permanent.
(EDIT: They've subsequently clarified that this isn't a permanent ban but will be lifted after 1 or 3 (the clarification isn't clear) peer reviewed papers. This is still problematic, but much better. The rest of this post left as I originally wrote it.)
So imagine: you are a masters student working on a project with a few other people, and your role is relatively minor. The project leads to a paper and you get your name on it, hurray. The lead author handles the submission and doesn't ask for your permission to send the final version because you're only a masters student. Your supervisor explains that this is how things are done and nothing to worry about. What you didn't know is that someone else on this paper at the last minute made some edits to the grammar of the paper using an LLM because none of you are native English speakers, and the LLM inserted a hallucinated reference. Arxiv picks up on this and you are now permanently banned from using arxiv as a preprint server. Further, every time you try to collaborate with someone else to write a paper and they want to put it in arxiv you have to explain that you can't, and that this means that by collaborating with you, they also can't put it on arxiv. Since arxiv is one of the main channels for distributing papers in your field, soon enough people stop asking you to work with them and your career is effectively over. Because someone else didn't notice that an overenthusiastic grammar checker inserted a fake reference and you weren't in a position of enough power at the time to insist on checking the final version.
Ok that's a long story, but I don't think this is a fanciful situation. Stuff like this happens all the time. It's easy to say - and I've seen a lot of people saying times like this - that everyone should take responsibility for reading the paper, or that it's the responsibility of supervisors to make sure this doesn't happen. But in the world we actually inhabit, power imbalances exist: the masters student can't make the supervisor wait until they read the paper because they're worried about their project grade. Bad supervisors are out there, and it's not fair to punish their students.
This policy will lead to terrible consequences for a lot of innocent people who should not reasonably be held responsible because they weren't in a position of power. I suspect it won't lead to very bad consequences for big name researchers who will just get on the phone to someone at arxiv or one of arxiv's funders and get the decision reversed in their case.
I understand the anger towards LLMs and tech companies, and I share it. I understand the anger towards the people cynically generating whole papers using them, polluting the scientific literature and making all our lives more difficult, and I share it too. But that doesn't mean we should jump to implement extreme and poorly thought out policies that will hurt a lot of people who haven't done anything wrong.
Finally, as an advocate of open science and publishing reform, this is really disappointing from arxiv. By saying that peer reviewed papers in "reputable" journals are ok, they've defined themselves (arxiv) as second class citizens in the world of publishing. This shows such limited ambition, and actively hurts the cause of making the world better by getting rid of the parasitic and harmful publishing industry.
Everyone is praising this article, but I find it a bit disingenuous. He criticises Taylorism ("scientific management") but that's exactly what researchers are subject to, and the glam publication economy is at the heart of it.
Science magazine's business model, and the source of its editor's $700k salary, rely on this exploitation of academic labour. There wouldn't be an AI slop paper problem if it weren't for journals and ranking researchers by publication metrics, because there'd be nothing to gain from generating slop.
https://www.science.org/doi/10.1126/science.aek5570
Replying to
@jonny@neuromatch.social I love how they wrote it because you can just add one line to your adblock filter and it bypasses it.
Replying to
@albertcardona@mathstodon.xyz @grammargirl@zirk.us just the editor in chief, nothing to see here... 😂😭
@curvenote@hachyderm.io and @openrxiv@biologists.social @biorxivpreprint@biologists.social teaming up to make an incredibly cool new tool to read biorxiv preprints as HTML with nice features (particularly on desktop when you hover over links). Try it on any biorxiv preprint, for example my most recent one (with just a couple of bugs that I expect will get ironed out soon):
To me little quality of life features like this can potentially push the visibility and viability of preprints and non-journal infrastructure, so I'm quite excited about this.
Replying to
Link to my more measured post about this article:
https://neuromatch.social/@neuralreckoning/116934923046572869
I think there's some genuinely interesting conversations to be had about machine learning and mathematics. My problem is that every time I see an article on this topic now, it always has such strong branding - for example Tim Gowers' recent article has the product name and version in the title itself - and I can't help but feel like I'm the victim of an intricate (likely not direct) marketing scheme. That makes it hard for me to want to engage in that discussion because I have no interest in helping these companies sell their products.
Replying to
@grammargirl@zirk.us @albertcardona@mathstodon.xyz on the fence about this usage, but I think it's very disingenuous. Longer comment in this post: https://neuromatch.social/@neuralreckoning/116940257495123437
Right, time for a social media break. Not sure how long. See you on the other side (or maybe not, who knows). Might drop back to post new papers.
RE: @thetransmitter@mastodon.social
Great article and I completely agree. If we want to study the interesting thing about the brain ("intelligence") we need to study it doing hard tasks, not easy ones that can be solved with a single neuron or even none. Yes it makes the analysis harder but that's the challenge.
Idea for grant proposal formative assessment. Very short proposal (1 page say) and reviewers are each allowed 3 questions which you get a very limited character count to respond to. Overall effort hugely reduced and lets the proposal evolve within a round rather than needing resubmission. Thoughts?
RE: @gconstantinides@mathstodon.xyz
We're hiring for a project on #SpikingNeuralNetworks and #neuromorphic computing, to start in October this year, for 36 months. Can hire at pre- or post-PhD level. Feel free to email me informally, or apply at the link below. Please do share with your networks if you know someone who would be interested.
#ComputationalNeuroscience
RE: @neuralreckoning@neuromatch.social
In general, academics talking about specifically named LLM products (rather than talking about LLMs in general) is now such a recognisable genre that it's hard to imagine that it's not somehow coordinated and planned as a marketing campaign. Would be fascinating to try to trace how this has happened.
RE: @thetransmitter@mastodon.social
The issues raised in this article bring to mind de Tocqueville's 1835 "Democracy in America" where he argues the reason for the success of American "democratic" science over European "aristocratic" science is that it makes things simple enough for anyone to reuse. We lose that using AI for science.
There may even be a similar social dynamic. Those with the wealth and power to make things more difficult for ordinary people have a strong incentive to do so, because it excludes those ordinary people from taking part and secures their dominant position.
Replying to
@brembs @albertcardona @steveroyle my answer: no! That's the path to having your employer take advantage, as indeed most universities do.
@gepasi@fediscience.org sure, it's better that he's saying this than being gung ho on AI, but many people are saying this sort of thing, including many who are not personally profiting from exploitation of academic labour.
Replying to
@steveroyle @nadege @brembs @albertcardona I'm happy if people want to work as long hours as they're able to, if they want to. The problem only arises when this becomes an expectation and makes it harder for those who can't match these hours to get jobs, grants, promotions, etc. More challenging is to work out a way to allocate jobs/grants/promotions that doesn't create an incentive to overwork.
Replying to
@ahltorp@mastodon.nu I assume that at least there'll be an appeals process for when someone added you without consent. As to the other question: the issue is not technical - because of imbalances of power you may not feel you have the ability to say anything other than giving approval. We can wish it weren't so, but it is.
Replying to
@brembs @albertcardona @steveroyle the 48 hours is because of the European working time directive that makes it illegal for an employee to work more than that, unless they sign a statement waiving that right, and that there is no negative consequence to not signing it. I've never heard of a university that asks anyone to sign this, despite many of them knowing people work more than that (eg one internal survey at a university I know of - but won't name in case it gets me in trouble - found their academic staff working an average of 55 hours per week).
Also that non teaching days are vacation thing sounds probably illegal too. 😂
If you're interested in doing a 2 year Schmidt "AI in science" postdoc fellowship in neuroscience/AI stuff with me starting in July or Oct 2027 take a look at this and get in touch soon. We've had a lot of luck recently getting these fellowships.
https://www.imperial.ac.uk/electrical-engineering/research/fellowships/
Replying to
@steveroyle @StephenCurry I really like the idea of the "silver medal". It would reduce a bit the randomness of hiring and promotion decisions, which would be a good thing. Could even allocate a small pot of money for silver medal winners. Not enough to do the proposed research but maybe enough to fund a further pilot or something.
RE: @thetransmitter@mastodon.social
This is a great article. I still feel less excited about "neural foundation models" than Juan, but I have to say he makes his case very convincingly, both for the advantages and disadvantages. Well worth a read!
Replying to
@julian_kuelshammer@mathstodon.xyz ah I see he tweeted that you could ask to have the ban lifted if you "establish a record" of peer reviewed papers. In some disciplines and depending what they mean by that quantitatively, that could take many years for a junior researcher, and severely impact their chances of getting a job. It's better, but not much.
Also, announcing policy by ambiguously worded tweets is not great. Like, what counts as a "reputable" journal? Will they give a list? And are they ready for the inevitable lawsuits if they do? And if they don't, how will the process be transparent and unbiased? And so on.
Replying to
@drgroftehauge@sigmoid.social I didn't assert that.
Replying to
@steveroyle@biologists.social yeah it's a good question, although there are at least some articles saying they've confirmed it with arxiv, but unclear what that means either.
Replying to
@sanzky@masto.es I won't be able to unsee that now! 😂
Replying to
@danielmk not as far as I know.
Replying to
@steveroyle to me it feels like saying that since racism and sexism is common outside academia that we don't need to address it in academia. I think this logic is just wrong.
Replying to
@steveroyle I guess I'm just proving your point that it's best not to discuss this. 😂
Replying to
@brembs @albertcardona @steveroyle here it differs a lot by institution, but I would say most academics in most countries work longer hours than they're contracted for, and that it would be difficult for them to complete all the tasks they need to do within their contracted hours.
Replying to
@da5nsy@social.coop @elizabethtasker@mastodon.online not really, this is a hard problem. We used Crowdcast but it's not without issues either.
Replying to
@tomstafford as academics we are always talking about the value of formative over summative assessment but we don't apply this to ourselves.
Replying to
@albertcardona @steveroyle agreed, although we ended up running a seminar series in this slot because it was the only time we could regularly book the room. Sigh.
Replying to
@steveroyle yes, same! Also, I'd feel much happier spending my time in a way that helps make the project better rather than just giving a grade.
Replying to
@mpe extraordinary conclusion that since 90% of the cost is borne by applicants, the onus is on applicants to reduce the costs.
Replying to
@jonny@neuromatch.social I don't think an appeals process resolves the issue of the power imbalance meaning that not all coauthors would feel safe to delay the posting of a paper (eg a student will not usually contradict their supervisor). The policy as stated is that they are equally responsible and an appeal can't change this it can only appeal the incorrect application of the rule.
Replying to
@bugaevc@floss.social lots of people use an LLM for this and there's very little control over what they do.
Replying to
@julian_kuelshammer@mathstodon.xyz that's great news, do you have a link to this clarification? I wish they hadn't announced this on X originally.
Replying to
@julian_kuelshammer@mathstodon.xyz much better, although still unclear if it's 1 or 3 since the post says both. Still problematic for junior researchers either way, still has problematic interactions with existing power imbalances, but much better.