The rapid development of computational and artificial intelligence (AI) systems, particularly in the late 20th century, behooves us to reassess the role of humans in creating knowledge, a role that remains exclusive to humans but may change with further development in AI systems. AI systems are computationally faster and more accurate than humans and artificial gaming intelligences have proven to be undefeatable by human players at some types of games. There is no doubt that computational systems have greatly aided humans in the process of knowledge creation, especially in the fields of mathematical and scientific knowledge. Yet AI systems have failed to fulfill their early promise of independent cognition.
As early as 1980, John Searle described two schools of thought concerning AI systems: weak AI, where AI systems are merely simulations of the human mind, and strong AI, where an AI system that can perfectly simulate human thought would be defined as equivalent to a human mind. By Searle's definition, a strong AI would also need to possess understanding and other cognitive states including consciousness and intentionality.1 We have not succeeded in creating a strong AI system. In "Minds, Brains and Programs"2, Searle tackled the issue of the "philosophical and psychological [significance] of recent efforts at computer simulations of human cognitive functions"; this study will focus on the epistemological significance of such simulations and how the cognitive shortfalls of AI systems determine the role of human intelligence in knowledge construction.
Some background information is necessary before we proceed. Alan Turing3 described a test, known as the Turing test, to answer the question, "Can machines think?" In this test, a human judge converses with a human and a machine without knowing which is which, and must identify the machine from the human. If the judge cannot conclusively identify the machine from the conversation, the machine is said to pass the test. The validity of this test in determining machine intelligence rests on the computationalist belief that the human mind and an information processing system, such as a computer, are analogous.
Searle also advanced the Chinese Room argument against the adequacy of the Turing test to define a strong AI, and that strong AI would have to fulfill much more stringent criteria. Searle suggests the following Gedankenexperiment: suppose he were locked in a room with a rulebook containing a set of rules for correlating one set of formal Chinese symbols with another set, and would return an output of Chinese symbols based on the input given to him and the correlations listed in the rulebook. The rulebook would be designed such that the person inside the Chinese Room would appear to be capable of conversing in Chinese. There is a second part to this Gedankenexperiment, which we will investigate later.
Searle's contention, via this illustration, with the Turing test is that an AI capable of simulating human conversation may not in fact be simulating human intelligence, and thus is not necessarily truly "intelligent".4 Our epistemic perspective must take us one step further: must there not be an intelligent, conscious entity who put the rulebook in the room? The person who wrote the rulebook must, in fact, have a genuine understanding of the rules and the reason for their design; in other words, that person who created the model must be a strong intelligence, artificial or otherwise. What is the role of this intelligence in relation to that of the AI (which, by Searle's definition, is a weak AI)?
It is also important to have a basic understanding of how AI systems are developed. AI systems are essentially models of natural cognitive functions. To appreciate how this affects knowledge creation we must consider how virtually all forms of knowledge employ models to aid knowledge creation: by utilising a correspondent model highlighting particular characteristics of the phenomenon to be studied, we can develop concepts relevant to the natural phenomenon with the benefit of a more manageable scale and with the benefit of predictive utility.5
However, where AI systems are concerned, the models must meet a more rigorous criterion: they must also be formalised6: that is, be put into a rigid mathematical form. This is because of the computational nature of AI systems: they are capable only of quantitative functions. Whether human qualitative functions are merely an extended series of quantitative functions remains to be seen, but let us work from the computationalist assumption that all functions can indeed be quantified. Thus, any AI attempting to simulate the human mind must be capable of formally modeling key human cognitive functions. We shall look at an application of AI that can be considered fairly successful, and investigate how its cognitive shortfalls affect its role in knowledge construction: Chess AI.
As established previously, a function needs to be formalised before an AI system simulating it can be developed. It is not surprising, then, that game-based AI has been a prominent focus of AI development: game theory can function essentially as the study of quantitative formalisation of rational human decision making.
Chess AI systems were initially developed and studied in the hope that they might provide an insight into human cognitive processes. Shannon identified two possible strategies a Chess AI might use to identify the move to be made, Type A and Type B.7 The Type A strategy is one "in which all variations are considered out to a definite number of moves and the move then determined from a formula"8, colloquially known as a "brute force search". The Type B strategy instead "[evaluates] only reasonable positions... [selecting] the variations to be explored by some process so that the machine does not waste its time in totally pointless variations."9
The Type B strategy essentially imitates the thought process of a human chess player, but it is of interest to us for another reason: A working type B strategy would appear to avoid the pitfall of the Chinese Room, because there appears to be no rulebook. The definition and nature of "reasonable positions" would change with each move made, so there is no uniform formalisation being applied by the AI to every legal position at each turn.
Here we diverge a little and explore another possible AI system, based on Bayesian probability10, that also appears to circumvent the Chinese Room problem by formalising the validation of an uncertain proposition (in this case, the rulebook). Such a system would be "taught" by a human or a strong AI whether input evidence corroborated a pre-determined belief11, and the system would produce a numerical value for credence12 upon which the validity of future evidence input into the system would be determined a posteriori. The necessity of human intelligence in "training" Bayesian AI does not seem to invalidate the strength of such a system, considering humans also require a period of training to develop cognition.13
It therefore appears that Bayesian intelligence, like the Type B strategy, avoids the problem of the Chinese Room due to its capability to create a model for the validity of its beliefs (the beliefs here being analogous to the rulebook), but upon closer investigation one realises that such an AI system would merely be a Chinese Room with two rulebooks, such that a fixed set of rules defines and periodically redefines a variable set of rules.
Developmental psychology provides some insight here as to why Bayesian intelligence might not be successful in "modelling a model": there is a formal distinction between learning and development. Learning is "a quantitative process that adapts the current representational structure to the input it receives", as Bayesian intelligence does, but development is a process "by which an inadequate architecture is qualitatively transformed to promote further learning".14 Put simply, the cognitive frameworks for processing knowledge (learning) and interpreting knowledge (development) are different: the former is quantitative, the latter is qualitative. This brings us back to the original epistemic problem of the Chinese Room: which entity engages in this qualitative process that defines the quantitative process expressed in the rulebooks? We must conclude that (1) the computationalist assumption of all functions being essentially quantitative does not hold in the context of developmental psychology, and (2) the role of the human intelligence in knowledge construction is qualitative and cannot be quantified.
Another way of viewing this essential difference is to look at the problem of formalising subjective Bayesian probability as compared to objective epistemic Bayesian probability. The highly subjective nature of Bayesian probability makes it impossible to formalize as it allows for the possibility that different individuals will arrive at a different degree of belief in an uncertain proposition given the same information, a criticism that objective epistemic probability tries to deal with. Objective epistemic Bayesians have been trying to formalise Bayesian probability such that two individuals, given the same information, would calculate the same credence of an uncertain proposition. However, we will see later, when we look at the Harsanyi-Aumann doctrine, that objective epistemic probability does not reflect rationalisable human behaviour. There certainly is presently no adequate formalisation of objective epistemic Bayesian probability that accurately models real-life human behaviour and, as we shall see later, such a formalisation may well be impossible. This inability to formalise a key human cognitive function due to an evaluative element again casts doubt on the quantifiability of all functions.
Attempts at Type B formalisations of chess play were the norm until 1973, when a team from Northwestern University produced a Chess AI based on a Type A strategy, called Chess 3.0.15 They found that in the time needed to evaluate which moves were good enough to be considered, they could simply evaluate the outcomes of all the moves available to the Chess AI. Since then, Chess AI researchers have largely abandoned formalisations based on imitating human thought processes and have instead formalised based on all combinatorial possibilities from the game position.
This would suggest to us that while AI systems have an immense advantage in speed and accuracy over human intelligence in computational and combinatorial functions, AI systems still lack evaluative capabilities that would greatly speed up their processing of information. This is symptomatic of a larger problem: AI systems cannot perform qualitative functions. The role of an AI system in knowledge construction is always strictly quantitative, and therefore the critical role of human intelligence in knowledge construction is qualitative.
Precisely what is it about Type B strategy that makes it so difficult to formalise? To answer this question we must take a closer look at how humans arrive at decisions in Chess. The first thing we notice is that given the same game position, different players are likely to make different moves. The extensive amount of chess literature available on the opening move alone is testament to that fact.
Here we must diverge once more to look at a central doctrine of game theory: the Harsanyi-Aumann doctrine, which holds that:
The prior beliefs of every rational player (who knows the rules of the game) are the same, because all differences in the players' belief arise uniquely from differences in information since each player has the same information given by common knowledge of the rules of the game and each player's rationality. Thus, rational players with common knowledge of rationality will not be able to agree to disagree on the likelihood of any action in the game.16
This doctrine essentially assumes that rational people, given the same information, will draw the same inferences and reach the same conclusions. This assumption, of course, is similar to the outcome that objective epistemic Bayesians have been trying to formalise, without success. One only needs to consider the fact that, in practice, humans will choose one of a variety of rationalisable17 opening moves even in a perfect and complete information situation, to see why such an assumption cannot be a general one even within the context of game theory.18
The reason game theory, as a formalisation, fails to model human behaviour in some situations is because there are cases in which there exists more than one Nash equilibrium19, as well as cases in which rationalisable strategies exist that do not lead to the Nash equilibrium.
Generally speaking, given the same formal information for consideration in decision-making, human players will reach different conclusions. This is clearly impossible to formalise. Indeed, Heap and Varoufakis disagree with the possibility of formalising such a process of inference, on the grounds that "no set of rules can be exhaustive: no set of rules can contain rules for their own application."20 The multiple-rulebook Chinese Room requires a fixed set of rules by which all subordinate rules can be modified, but the fixed rules that permit other rules to be modified cannot apply to themselves.
Therefore, even within a formal study of human rationality such as game theory, an element of creative interpretation is necessary, when "problems lie outside the domain of their determinate operation" - when there are more factors at play that do not lend themselves to a simple causal process of decision-making. It is by such a creative process of judgement that humans choose which opening move to make in Chess and Go given a range of rationalisable strategies.
To determine what attribute human intelligence possesses that AIs do not that allows humans to have this range of choice, we must revisit the Chinese Room. Searle asks us to imagine that, in the Chinese Room, he is fed both Chinese and English input. He responds to Chinese input according to the rulebook, and responds to English input based on his knowledge of the English language. While he would appear to be capable of conversing in (and therefore, by Turing's definition, capable of understanding and thinking in) both English and Chinese, he would only in fact understand English and not Chinese. Thus he rejects Turing's definition of machine intelligence and invites us to question: what is the difference between his processing of English input and that of Chinese input, that he can be said to understand one but not the other?
(Note here that the processing of Chinese input and English input are analogous to the evaluation of moves by a Chess AI and a human player respectively. According to the rulebook, there will only be one possible response to each string of Chinese characters at any given point in time, but this is not so for English input: a range of rationalisable responses is possible.)
The answer, according to Searle, is intentionality: which can be simplistically abbreviated as "aboutness".21 When he converses in Chinese in the Chinese Room, he does not converse "about" the content of his output text, whereas when he converses in English, he is actually conversing "about" his output. Suppose he were to output from his room, "I ate bread". If this output were in Chinese, he would merely be responding based on the rulebook, but if it were in English, he would actually be talking "about" having eaten bread. Human intelligence possesses intentionality, but AI does not. This is the key difference between human and artificial intelligence that allows humans to attach a meaning to their output, and which forms the basis of the process of knowledge creation.
Could intentionality be "created", or come about, via another means? Emergentism suggests that intentionality could be an emergent property; that is, a property that "emerges" as a result of the correct constitution of fundamental cognitive functions.22 If these cognitive functions could be replicated, the same emergent properties would result. However, we must be careful of attributing emergent properties such as creativity, aggressiveness, and the capability to learn to AI systems. Humans have a tendency to attribute too much independent intelligence to AI systems23, when the emergent properties can in fact be traced to well-conceived quantitative functions.24
Precisely because of this, emergentism still fails to explain the very variety of choice that intentionality permits a human mind. Given the same input, the human mind can choose to respond in a variety of ways; this choice on an AI system, no matter now emergent it may appear, can always be traced to quantitative functions. To better illustrate this point, let me provide an example:
Suppose it were found that, upon being asked for his name, a man would reply "Matthew" the first time, "Mark" the second, "Luke" the third and "John" the fourth time, and then "Matthew" again for the fifth, "Mark" for the sixth, and so on. An AI developer has programmed his AI system to receive the man's input and, based on a series of quantitative functions, approximate the man's response. The AI is successful in doing so, and thus appears to have the emergent property of being capable of learning, as well as of having a wide variety of responses.
However, what is being neglected here is the fact that each time the man has four choices, and each time the man answers, he is making an intentional choice to reveal or conceal his name, whatever his real name may be. In contrast, each time the AI answers this question, the AI does not really have the same range of choices; there is only one "right" answer depending on which iteration of the program this is, unlike four rationalisable answers each time for the man. We must not conflate our perception of the man possessing intentionality with that of the AI's ability to mimic the man: the AI does not choose. It simply adopts the rationalisable strategy; should there be more than one, the AI can only choose between them arbitrarily, without intentionality, because it is limited to quantitative functions. Most importantly, even in emergentist circles, it is the AI developer who must ascribe such emergent properties to the machine; the machine cannot produce any proof of knowledge of such emergent properties. Again, human intelligence is required to write and interpret the set of fixed rules from which such properties emerge.
So, to answer the original question: What are the roles of human intelligence and AI in knowledge construction? It is clear that AI can only handle quantitative functions. It can produce simulations of formalised models with greater accuracy and speed than a human, but formalisation of models can only be undertaken by human intelligence. In addition, we have seen that qualitative functions cannot be reduced to quantitative functions in practice because of intentionality, and therefore all qualitative cognitive processes involved in knowledge construction remain strictly in the domain of human intelligence. The inability of AI to understand what it is talking "about" prevents it from conceptualisation and therefore knowledge construction. AI systems are useful tools by which we can better understand the human process of knowledge construction, but like all models, can only serve as a representation of, not replacement for, the process it attempts to model.