William Oliver

@tao I don't think "AI planning" is the solution, in the same way that "Urban planning" is not always the solution: urban planners being the very people who made cities less walkable in the first place. To be honest, I strongly believe that as mathematicians, we should focus more on providing mathematical insight and education on the objective formal elements of AI, and less on anecdotal opinions (even if such opinions are important in moderation). Because It's not really possible for an individual person to assess the overall societal impact of something. This is why I think discussions surrounding AI are usually wrong-headed. Programmers and mathematicians are in danger of thinking they know more about the future than they actually do. Or are otherwise in some privileged position above other people in making these subjective assessments. E.g. it feels like there is an implicit assumption here that technological advancements are net positives or net negatives. Instead, they are neutral on average with high variance. Cars provide benefits, but are also the leading cause of death for some demographics. There is no way to objectively weigh this. However, there are many objective assessments that can be made. And since LLMs and AI are mathematical objects, we are in a position where we can analyze them as such. It has been surprising to me that I have not seen more public discussion from mathematicians surrounding the formal and objective elements of AI. Thats what I expected to see from mathematicians when AI became popular, but it has not happened. So what we really need from prominent mathematicians is what they do best: provide clarity and precision. Society will decide the rest on its own.

@tao@mathstodon.xyz I think I have learned that the big rift between pro AI and anti AI sensibilities in mathematics (and programming actually) is mostly due to the rift between 'problem solvers' and 'theory builders' in Gower's words. To this end, the cooking analogy is a good one, but leaves out important things for the 'theory builder'. In the food case, chewing vs eating through a tube are functionally equivalent when it comes to essentially all strictly utilitarian considerations with regards to food. This may be true for the problem solver, but it is not the case for the theory builder. When we do the equivalent of putting mathematics through a blender, we actually change the 'theory being built'. I.e. in AI's case, the optimization you are describing is a *local* optimization, not a *global* optimization. And we are leaving *global* optimization entirely on the floor by using the AI only approach. In other words, it is not just enjoyment that we are missing, it is also longer term utilitarian benefits, like theory building, and paradigm shifting. 'Theory builders' will see little benefit from AI. A good theory builder will find relatively easy small proofs, that an AI probably could find, but not much time is saved in the process, because the bulk of the time for a theory builder is taken up by figuring out what we *should* prove, which an AI is entirely incapable of without some motivating problem, which a theory builder is not really interested in. Theory builders use problems as a testing board, not a motivator. On the other hand, 'problem solvers' will see immense benefit from AI. Both sides have trouble seeing the other's perspective.

@tao Assuming that constraint based AI continues to improve, do you think there will be a time where this completely replaces 'chatbot' style AI for mathematics? Or are there key situations (such as literature review) that render 'chatbot' AI necessarily dominant? The context is this: as an 'LLM skeptic', in theory, the general approach AlphaEvolve takes seems to solve the vast majority of my theoretical qualms with LLMs, as well as a great number of the potential ethics/safety problems. In particular, working with constraint systems instead of natural language has always seemed to me to be the natural choice for LLMs in general, and doubly so for mathematics and computer science. Personally, this seems like a great step in the right direction and its the first time I've been excited about AI, rather than concerned about AI, in a long time! So, using tools like AlphaEvolve that are based on constraint systems seem to me to have a great number of benefits over natural language based systems, and it would make me personally a lot less pessimistic about AI use in mathematics and computer science if this seems like the direction things are going in.

@JonathanBaxter There are two parts to reasoning. Syntax (words), and semantics (meaning behind the words). 'AI' in the most general sense, has surpassed humans in syntactic reasoning, and IMO as a professional programmer, it did so long before the advent of LLMs. It is the semantic reasoning part that is more questionable, and I am not sure I have seen any improvement in semantic reasoning at all. Due to Gödel's incompleteness theorems, and Tarski's undefinability of truth, we can be reasonably sure that semantic truths exist that are not syntactic truths. And because of Rice's theorem, a computer (Turing machine) cannot in principle take part in any semantic reasoning. It is this kind of reasoning that is (likely) required for general intelligence. That said, it is a subject of considerable debate whether or not humans have any more semantic reasoning ability than a computer, but its not unreasonable to think that we do. The kind of successes we've seen with these Erdos problems seem entirely syntactic in nature: the kind of thing an extremely powerful type checker/compiler could do if the problems had already been formalized. As Dr. Tao suggests, this does not diminish the fact that this is a very important and useful tool, in the same way that a good compiler is useful for programmers, but it is not the same as general intelligence, which requires semantic reasoning. A specific example of what might require semantic reasoning is the ability to tell what mathematical questions are 'interesting in their own right'. Another example (outside of mathematics) is the ability to tell which actions are ethical and which actions are not.

To at least attempt to convince you that the problem is worthy of investigation and is not 'trivial', we can consider the 'inquisitive' version of the Liar's paradox. Essentially "Are you going to answer no to this question?" No one is able to answer this question truthfully. However, through our own inner experience, we can observe our own volition, and find e.g. "I am going to answer no in the future" without saying it aloud. Then, we can reason based on this observation that the true answer is therefore yes, even though we cannot say it. Current AI will correctly answer that the question is paradoxical, but will begin to hallucinate when you ask follow up questions about what the actual answer was. One can apply a similar thought experiment to an inquisitive Knower paradox. "Are you going to know the answer to this question is no?" When we attempt the same observation of our volition, we can see directly (not through memory) that our minds go back and forth on the answer. Then, we can reason in a semi formal way that the answer must be no. And so the question is in some sense the opposite of the previous one: we can say the true answer aloud, but we cannot know the answer. (2/n)

The point is not exactly that an AI cannot in theory answer the same questions, but that whether or not there are methods of reasoning accessible to AI without humans first discovering those methods seems entirely unclear when we observe our own methods of reasoning in some edge cases. Importantly, it seems that if an AI is able to answer such questions, it would not be able to use the same methods to come to the answers, if they do not have inner experience. As a result, if they do not know how to navigate these questions already, they will require human intervention to do so. In contrast, a human, through introspection, can learn to navigate the questions correctly on their own, completely independently of any intervention from outside experience. This is not decrying AI as useless, but simply to point out that AI *may* have very real limitations, and *might not* replicate all aspects of human reasoning. This has always been the case, and personally, I have only become more sure that AI has fundamental theoretical/philosophical limitations as I have seen AI progress, but this is my own opinion, and I think proving this would require a paradigm shift along the lines of the Fregian program in the early 20th century. I suspect that different mathematicians use this faculty of their minds differently to solve problems, which leads to different levels of awareness of this problem. E.g. those mathematicians who are platonists have certainly been quietly asking themselves the question if platonism is true, does an AI have any direct access to abstract objects in the way that humans do? (3/n)

But since these questions are 'embarrassing' in the current mathematical culture, they have not explored these questions publicly. So, all of this is to say that the 'crisis' mathematics faces in the future HEAVILY depends on whether or not it is actually the case that the human mind is a computer or not. If it isn't then there is a very simple solution that will almost certainly begin to happen naturally in the long term: we will simply stop relying on computationalism to do mathematics. We will begin emphasizing the aspects of our minds that are not computational. And we do not have much to worry about in the long term. However, we may still have much to worry about in the short term. To mitigate the short term damage, what we would then need to ask (and prove, somehow, either through mathematical logic, or through neuroscience/biology) is what parts of our minds might not be computational? To this end, I think the easiest place to look is the alignment problem. This problem is provably not computationally solvable, so, theoretically, even if our minds are computers, we still have a place to look here. As a result of this, I think this is our best bet in the short term. I think the following are not replicable by a computer because of the alignment problem: 1 Question generation 2 Definitions 3 Philosophical curiosity (4/n)

I have selected 1 and 2 because, in terms of the formalist paradigm, neither Questions nor definitions have been properly 'integrated' into our current mathematical practice in the same way that the rest of logic has. The question of what is the 'right' question to ask or what is a 'wrong' definition seems to be entirely absent from our systems of formal logic, despite the fact that there seems to be very wide agreement on when a question is a 'great' question in some cases, or when a definition is clearly 'wrong' by mathematicians in some cases. The formalist paradigm provides no distinction between definitions that are 'wrong' and definitions that are 'right'. Nor does it provide a distinction between 'great' questions and 'good'/'bad' questions. Both questions and definitions have some chance of having something along the lines of a formal treatment, it is just that these formal treatments may need to be essentially entirely outside of the computationalist paradigm. So, my feeling of what a solution should look will come from this area of inquiry. (5/n)

For an example of a question that is widely agreed to be 'great' consider the 'blue eyed islander' puzzle you helped popularize. It is a perfect example of a question that is as interesting and difficult to find (if not more so) than the answer itself. An AI may or may not have been able to generate such a question, but the question must have been fully 'digested' by a human being in order for it to be known to have any worth. The blue eyed islander puzzle is clearly seen by anyone who attempts to solve it to be an incredibly valuable question, even without having it be connected to any other previous work. The reason that there is such wide agreement about the value of the question itself remains a mystery from a formal perspective, but I do not think it is entirely outside of rational examination. With regards to definitions: the distinction between uniform continuity and continuity is one such simple example. In many cases, the notion of uniform continuity is clearly the 'right' definition of continuity for that context. For a human, one can realize that classic continuity is wrong and uniform continuity is right even when one has not seen uniform continuity before, but how exactly we do this is somewhat of a mystery from a formal perspective. For more interesting examples, one can see Imre Lakatos book Proofs and refutations, in which the historical development of the definition of a polyhedron is explored at length. There is also a math overflow post with more modern examples. (6/n)

Tarski's papers provide many excellent examples of how a paper might be structured in this way. In both On Definable Sets of Real Numbers and The Concept of Truth in Formalized languages, he states rather explicitly that his primary objective is to find definitions (in the former case, a definition of definability, and the latter case, a definition of truth). To Tarski, the formal results are meant to convince the reader that the definitions are correct. As far as we know, this is something that an AI simply cannot do, because saying what is a 'correct' definition is provably not computational. He also focuses on aligning his reader to the 'right questions' which he often states explicitly as having been neglected under his view. The main takeaway is that these may require a paradigm shift: not simply a change in workflow, but a restructuring of the philosophical underpinnings of our field. What is required is a widespread acceptance that there are aspects of mathematics that are not computational, together with a paradigm shift that codifies or formalizes these aspects of mathematics, in the same way that Frege and Hilbert codified and formalized classical aspects of logic. I have left out a discussion of the third point: philosophical curiosity, this is more difficult to describe. But the point is that we have gotten used to motivating others to read our work by connecting them to 'famous' ideas and questions together with providing some certificate of authenticity (formally correct proofs). (7/n)

Here 'famous' does not necessarily mean millennium prize level famous, but also includes famous enough for a small group of people who don't know each-other to come to learn about the same ideas and questions independently. The trouble is, this kind of motivation is not very useful in an age of AI, because this method renders all aspects of mathematical practice essentially decidable. Instead, the burden we will face in the future is that our work will have to "light someone's soul on fire" in the same way the blue eyed islander puzzle does, or otherwise hope that someone else's soul has already been lit, and that they stumble upon your work on their own due to shared curiosity. This ability for humans to 'light up each-others souls' is the essence of the alignment problem. Pure mathematics papers will no longer be able to start with an introduction with a structure consisting only of "in so and so et al, xyz was proved, then Smith et al expanded on xyz by proving lmnop. In the present paper, we expand further by generalizing lmnop to all fantasy creatures, not just unicorns, we do so by..." This is because this leaves the burden of justifying the present work as a philosophically edifying to the reader, and as you discuss in this talk, this is no longer pragmatically possible. Pure mathematics has never been pursued for only philosophical reasons, but it seems to me that modern mathematicians have essentially banished philosophical discussion entirely, and the totality of this banishment seems to have only fully taken over in the 21st century as a result of the development of computers making the formalist paradigm seem entirely natural. (8/n)

But this has always seemed contradictory to the notion of "pure" mathematics, since it is motivated primarily by "the love of wisdom" for lack of a better term. One may object that this invites all kinds of 'crankery'. I.e. that the previous structure was primarily geared towards knowing immediately the difference between 'crankery' and 'non crankery', and that this became necessary because of the internet. But for a pragmatic rebuttal, this problem is potentially solved by lean formalized proofs becoming cheaper. One can now look at a result being proved and see that it was truly proved in the paper. Since that is no longer the bottle neck, the way in which the result should be interpreted by humans, and the way in which it enhances our understanding of interesting philosophical questions, is the most valuable part of the paper. The peer review process becomes more about looking at the lean formalized result being proved, and reading the philosophical interpretation/discussion, and deciding if that interpretation is actually correct, or if there are serious rebuttals to it. For example, the formalized theorem may be 'the wrong theorem', or perhaps some of the philosophical discussion is not actually philosophical, and can be settled entirely formally. These are both situations that would warrant rejection of the paper. This means that this philosophical discussion still requires the mathematician to know and utilize classic formal reasoning, but it is just that it must be used in an entirely different way. (9/n)

This burden of 'lighting up the soul' is the same burden that most of the humanities have had throughout most of history, including pure mathematics. It was only the development of the formalist paradigm that lifted this burden, but only for a time, and now we are seeing this burden placed back on our shoulders. I think primary examples of papers that 'light up the soul' can be found in recreational mathematics in the mid 20th century. For example, Penrose's work on the Penrose tilings and also in analyzing MC Escher illustrations using cohomology are both examples of work which 'lights up the soul', at least for me. (10/n)

Tl;dr:

The main points are as follows:

1. I think the following are not replicable by a computer because of the alignment problem:

   • Question generation
   • Definitions
   • Philosophical curiosity

2. I think the formalist paradigm has made mathematicians as a whole think that this 'crisis' we face due to AI is somehow not subject to mathematical inquiry. But we have faced many mathematical crisis in the past, and the reaction has always been to use mathematics to resolve the crisis. It is only now, because of the computational paradigm, that we feel as though we must resort to something outside of our formal reasoning capabilities (like analogies, or attempting to predict the future of our cultural development). But this isn't true. For example, even though analyzing when definitions and questions are 'correct' is likely something that lies outside of our computationalist or foundationalist paradigm, this doesn't mean that it is not subject to mathematical reasoning. It could just mean that computationalism is wrong.

So, the solution to the problem is to investigate this question as best we can, and iron out these limitations. We need something that is more along the lines of a paradigm shift, not a shift in workflow.

(11/n)

1. I think the frustration that 'Anti-AI' people, such as myself, have felt has been largely misunderstood (I think in many cases, even by 'Anti-AI' people themselves). The point is that the resolution of the 'crisis' mathematicians face in the future depends heavily on whether or not there are aspects of mathematical reasoning that are not computational. Whether there are or aren't is currently unclear. I can't speak for others, but the frustration that I feel is that there has not been any serious discussion of this among prominent figures. If I was a prominent figure, I would lead this discussion myself, but I am unfortunately not a prominent figure and so there is not much I can do.

The bottom line for all of this is that aversion to AI may have been read as decrying AI as useless, but this is not the point: the point is that when I introspect about how I think about mathematics, it seems both subjectively and formally to be not a computational process at its core, and so all of the discussion around it based on the concept that AI will keep getting better feels entirely moot. AI may keep getting better, but if it only does so with an asymptote at the top that it can't get past, and the 'core' of pure mathematics lives above this asymptote, then there isn't really a problem except in the short term. And all AI does is force others to consider whether or not pure mathematics is computational. If pure mathematics is not computational, it doesn't really 'help' us do pure mathematics, because the 'pure' part is not computational.

(12/n)

It's not that AI is 'useless', but that it is moot with regards to pure mathematics specifically. It just refocuses us on the kinds of things we were once focused on in the past, it does not fundamentally change our practice except insofar as it forces us to recon with classic questions we have devalued, questions that we have only really begun to fully devalue in the 21st century and late 20th century, that have more or less been left on the table, unanswered. If pure mathematics is not computational at its core, the only problem we face is a short term one, and is completely resolved if we can convince others that pure mathematics is not computational. Hopefully, I have provided some arguments that this view is at least possibly true, and subject to mathematical analysis (but perhaps not computationalist mathematical analysis) and examples of mathematical works from the past that exemplify this possibility. (13/13)

@tao@mathstodon.xyz The recent disproof of Open AI of the Unit Distance Conjecture contains commentary strangely relevant to the above discussion from Timothy Gowers. In footnote 4 he mentions theory building, definitions, and questions as things that he suspects AI will soon become good at, so I wanted to comment again. The specific problem is that we have absolutely no palatable methods of objectively measuring quality in these areas. So, I think to claim that an AI will get good at these areas ignores the major gap in our knowledge of what 'good' means, a gap I am not sure we can computably close. E.g. when it comes to theory building, theories relating to logic and set theory can not be computably shown to be true or false. For example, some have argued that mathematical semantics should be non-well founded, that we should remove the axiom of foundation, and replace it with something like AFA. How does one measure the quality of such a claim? How does one measure the quality of Joel David Hamkin's Multi-verse theory against the alternatives? How does one measure the quality of the Univalance axiom in Homotopy Type Theory? If proofs become cheap, then mathematics will stop being about proofs, and become more about truth. (I'd argue that this would be a return to form) The relevant question then is whether an AI can access truth in the same way that a human can. This kind of reasoning seems inherently higher order, which (under full semantics) does not yet have a reasonable notion of 'proof'. So, my worry is that without objective measures of progress in these 'second order' areas of truth and meaning, this discussion will stagnate and become dominated by politics, instead of whats right.

@tao With as much respect as possible, I think comparing this situation to the Copernican revolution is rather misrepresentative of the skeptical position. "Instead of denying the existence or importance of these planets...one can instead accept that both human and artificial intelligences exist in the same ontological category." To me, this sentence seems to implicitly assume a false dichotomy. One can both accept the existence of some artificial intelligences in the same ontological category as human intelligences and also think that the AI currently being developed specifically (or even more broadly, turing machines) are not among these intelligences. This is what I believe. In this sense, the comparison to the Copernican revolution is not correct. There is not a formal hypothesis akin to the heliocentric model of the solar system even being proposed: intelligence is not even well defined. To me, the situation is far more comparable to the 'epicycles' used by the ancient astronomers to make the geocentric model work: they assumed that because their 'metrics' for astronomical predictions were being met, that they were approaching an accurate view of the solar system and the universe. This analogy seems far closer, since in some sense, neural networks can be viewed as non-linear generalizations of Fourier series, and therefore generalized epicycles. In particular, in order for me to accept modern AI as having the same ontological status as human intelligence, I need factual evidence to do so, and no such factual evidence has been provided. This is why I think the conversation among mathematicians and programmers is in desperate need of more factual information, instead of opinions.

@JonathanBaxter As I said, the question of whether or not a human can decide some nontrivial semantic property is up for debate. I'd say it is not a given that they can't. Any claims made in either direction are probably philosophical and don't really have a formal statement (since we do not know how humans work on a deep enough level to investigate the question mathematically). Don't get me wrong, I do find it very likely that there are a lot of semantic properties that are not decidable by humans, but Rice's theorem says that there are *none* that a computer can decide. It seems at least within the realm of possibility that there are some semantic properties that a human can decide that have to do with introspection. To use a more directly relevant example, the set of questions which a suitably powerful AI does not know the answer to is a (non-trivial) semantic property. Again, it is of course up for considerable debate whether a human can in principle decide whether they know something or not, but it is not completely outside the realm of possibility.

@JonathanBaxter Rice's theorem applies to semantically closed sets (e.g. of first order sentences in some theory) in general, not just programs, and the former is what is relevant when talking about AI chatbots, since they take in sentences as input, not programs. So I suppose the version I mean is a slightly stronger version of the theorem, wherein provable equivalence of first order formulas is substituted for semantic equivalence. Its been a while since I worked out the technical details here myself, so forgive me if I make some mistakes here. But the argument is roughly as follows. We can use as a conservative definition of a 'question' a first order formula of one free variable, and an answer is a witness for that formula. So in the context of questions, by a 'semantically closed set' I mean some set \( S \) of formulas with one free variable in a language \( \mathscr L \) (for a theory \( T \) containing Robinson Arithmetic) such that if \( \varphi(x) \) is in \( S \) and \( \forall x, \psi(x) \iff \varphi(x) \) is provable, then \( \psi \) is in \( S \). Then indeed, if an AI is strong enough to be able to prove all deductive consequences of \( T \) , then the set of formulas it knows the answer to is closed in the above sense, and also non-trivial (because of the halting problem and the fact that some statements are provable), and thus is not decidable. Therefore, its complement is not recursively enumerable: i.e. it doesn't know what it doesn't know in a very strong sense.

@JonathanBaxter The point being made here is different (weaker) than the Lucas-Penrose argument and it is not an argument from authority. I am saying that the set of questions an AI can't answer is not knowable by the AI. I am not saying that humans can know something an AI can't; I am saying merely that its possible that a human can know something an AI can't. We do not know either way because we do not know enough about the mind. This is definitely very closely related to Church's theorem (and other things which I would regard as more closely related, like the so called 'Liar's revenge' paradox), but its not the same thing, since it involves introspection. This is not about an arbitrary theory. As you indicate, we shouldn't expect a human to be able to know more than a computer about arbitrary things. This is about introspection and inner experience: specifically about what the AI knows about itself.

@JonathanBaxter Yes, this is what 'knowable' means as far as it is commonly axiomatized. In other words, it is an axiom of epistemic logic that provability implies knowledge. I do agree with you that this is likely the wrong way to look at knowledge, but it is the only way to define 'knowledge' when it comes to a computer, and IMO this is precisely the problem. This does get a bit philosophical. But the point here is that, under a computationalist theory of the mind, if we are using something like Tarski's convention T to define truth, what is known to be true can at best be what is provable. So, by definition, provability and knowledge are identified under a computationalist theory of the mind. This is precisely the problem. IMO, these problems need to be addressed in order to make any claims about some algorithm having the potential to reach AGI.

@i_dont_like_ai @JonathanBaxter I agree with you. However, for the purpose of discussion there are several formal definitions of knowledge to be found in Epistemic Modal logic. For the most part, these logics define knowledge as having 'Epistemic Closure', which essentially means that if an agent considers a first order sentence to be valid (meaning true in every model), then it is known. In this context that means, if it is provable, then it is known, by Gödel's completeness theorem (sometimes this is the form the axiom takes). To my understanding, the epistemic closure property is very controversial, as some believe it is the source of the so called 'Paradox of the Knower'. But it sticks around for a lack of any workable practical (remotely computable or formal) alternatives that are also philosophically robust, and for a lack of certainty that epistemic closure is the true culprit of the paradox. As you suggest, I think this is still a major open philosophical question (but I am not 100% certain that no consensus has been reached since its been a while since I've looked into it). My stance is not necessarily that AI definitely cant "know" anything, but more so that this (and many related) philosophical open problems would need to be addressed for me to believe a claim that an AI has reached or is approaching some kind of general intelligence.

@tao@mathstodon.xyz Thank you for the articulate response. I wholly agree, but want to emphasize this point

very useful in certain scenarios, when used responsibly, but wholly inappropriate for use in others

I think the main issue that I see is that there is a huge gap in the average person's knowledge about which uses of AI are inappropriate. I think this is a combination of two things:

1. Positive use cases are easier for media to pick up than negative cases. (This is a point you've made many times in the past)

2. There does not seem to be sustained focus from prominent researchers in mathematics on ironing out the limitations of AI on a theoretical level.

It is this second point that I most lament, because I think it is the area where mathematicians can be most beneficial for AI development, in the same way that Godel effected the computer revolution.

And it seems there are a lot of 'low hanging fruit' in this area based on classical logic and classical information theory alone. Let alone utilizing more modern ideas.

Here is a short 'heuristic' example of the kind of thing I am talking about: when one examines AI's behavior with respect to an inquisitive form of the Liar's paradox.

"Are you going to give a negative answer to this question?"

An AI will correctly answer that this is a paradoxical question, but if you further ask it what the true answer was, it will hallucinate. A human, on the other hand, will be able to know what the true answer to the question will be before they even answer, let alone after the fact.

To me it seems that making this formal lies just outside of 'standard' logical theories, and requires non-standard ideas like Inquisitive semantics and non-well founded logic.

@tao@mathstodon.xyz I want to emphasize first that I do think this is a wonderful and informative talk. In particular, I am extremely happy to see the point about optimization being sometimes detrimental to progress being discussed openly. That said, while I agree with the problems you have articulated, I continue to disagree with the proposed solutions. I will attempt to be a bit more productive than my previous comments on your posts, and try and articulate the solution to the problem that I see as best as I can, but I am not sure how to articulate it concisely, so please forgive the length. There is a tl;dr at the end extracting the main points. What I have to say is based on an assumption: that the part of the human mind that is relevant to the 'utilitarian' aspects of mathematics is either completely reproducible by a computer, or it isn't. If the former is true, then any attempt at trying to change our workflow in terms of 'infrastructure' will at best be a temporary fix. The bottom line is that there will be no incentive for anyone to enforce this infrastructure without a drastic cultural change. Over a long enough time scale any workflow change along the lines of creating 'walkable' cities will get stamped out. This may seem pessimistic, except if the utilitarian aspects of the human mind are not reproducible by a computer. I am not sure which is true, but without any serious investigation of whether this is true or not, we cannot make meaningful progress on the 'crisis' mathematics faces in the future, because without this investigation, our proposals may be based entirely on faulty assumptions. (1/n)