Replying to
@futurebird@sauropods.win not to take away from anything you've said, it's definitely true that this sort of approach leads to dismissing novel ideas. but i think there's also another level at which this kind of approach is worrying: by saying they need to sometimes dismiss the machine's output, one admits the machine is not to be trusted, yet they trust it when they can't catch it in what's (seemingly) an obvious lie.
in other words, it's trusting the liar's word unless you already know it to be false. it's trusting the allegorical wolf to take care of the sheep because if you ever see it attacking one you'd stop it.
if you know you can't trust the machine sometimes, then you can't trust the machine. and if your system is then "well i'll just catch the mistakes", then either you only intend to use it in cases where you already know the right answer (in which case, why use it), or you believe you'll somehow figure out something's wrong when you can't tell what "correct" looks like.
it's saying "if it looks correct then it's correct". which is wrong both on the false negatives end and the false positive ends at the same time.
