AI’s Awakening: 70 Years of Machine Learning’s Turbulent Rise

Machine learning is a subset of artificial intelligence (AI). AI refers to the technologies and methods that enable computer systems to perform tasks typically requiring human intelligence. AI encompasses various technologies and subfields, including machine learning, deep learning, natural language processing, computer vision, and expert systems.

The concept of AI was formally proposed at the Dartmouth Summer Research Project on Artificial Intelligence in 1956. Although small, Dartmouth College is one of the Ivy League universities. The workshop was initiated by John McCarthy, with participation from Claude Shannon and others. John McCarthy was a computer scientist and cognitive scientist, and one of the founders of the AI discipline. He was also the inventor of the famous programming language Lisp, which was used to write many early AI systems and algorithms.

John McCarthy, the initiator of the conference, initially believed that by gathering the best scientists in the U.S., the AI problem could be solved in about eight weeks. However, the grand dream launched in 1956 has, after nearly 70 years of arduous struggle, only reached halfway to its goal.

Along this journey, much like the historical debate between the particle and wave theories of light, there have been two main schools of thought regarding the direction of AI: should models be built based on rules or data? The conflict between these two schools has been tumultuous, but ultimately, with the support of three Chinese scientists, the data-driven camp gained the upper hand, forging the magnificent development of contemporary AI.

AI was first inspired by bionics. In the 1890s, Santiago Ramón Cajal proposed the neuron doctrine, which was later confirmed by Camillo Golgi using Golgi staining. In 1943, Warren McCulloch and Walter Pitts simplified complex neuronal electrochemical processes into relatively simple signal exchanges[^activation], laying a solid foundation for AI bionics.

In 1958, Frank Rosenblatt proposed the Perceptron based on bionic principles. This was one of the earliest neural network models, capable of solving binary classification problems by learning linear classifiers. Rosenblatt’s Perceptron was a brilliant invention because, at the time, computing technology was not yet digital. Training the Perceptron involved manually switching switches. Although primitive, the Perceptron eventually achieved reliable shape recognition capabilities through training.

The Perceptron was once hailed as a major technological breakthrough. However, the invention was too advanced for its time; the world was not yet ready for its birth. It was not until 50 years later that people realized neural networks required digital input/output devices, massive computing power and storage, and ultimately, huge amounts of data. After all, the human brain consists of over 100 billion neurons. In 1958, humans could only simulate a few hundred millionths of the brain’s capacity.

Consequently, the enthusiasm generated by the Perceptron lasted less than a year before facing fierce criticism. Its biggest opponents were Marvin Minsky, one of the initiators of the Dartmouth AI workshop, and Seymour Papert, another mathematician and pioneer of computer science. In 1969, they published a book titled Perceptrons, criticizing the Perceptron for lacking a rigorous theoretical foundation. In fact, to this day, AI still struggles to find a solid mathematical cornerstone; in many ways, its behavior remains a black box—a trait surprisingly similar to the human brain.

The Perceptron was a data-driven machine learning model. After the Perceptron was severely criticized, the machine learning front had to remain silent for over a decade. During this period, knowledge engineering and expert systems took the spotlight. One of the most famous was a program called Internist-I, whose database contained descriptions of 500 diseases and 3,000 disease manifestations. However, because the real world is too complex, rule-based models handle retrieval well but appear rigid and superficial in reasoning. As rules multiply, compatibility decreases, and they soon fell out of favor.

Expert systems were humanity’s last hope for achieving AI at the time. Therefore, the failure of expert systems caused AI development to enter a low point, plunging into a frozen Cambrian period. However, just as Earth experienced the Cambrian period, evolution accelerated beneath the frozen ice, and revolutionary breakthroughs were imminent.

In 1986, Geoffrey Hinton[^hinton], David Rumelhart, and Ronald Williams published the backpropagation algorithm. This foundational work for deep learning made the training of multi-layer neural networks possible. Theoretically, this was a crucial step toward simulating a brain with 100 billion neurons. However, the era of deep learning did not arrive immediately; AI remained sealed under thick ice and snow.

Another 10+ years passed. In 1998, Yann LeCun proposed LeNet, the earliest convolutional neural network (CNN) and the first neural network in human history with practical utility. It was widely used on ATMs across the United States to read numbers on checks. Yann LeCun’s success sparked another AI wave, finally bringing the machine learning route back into the public eye.

However, Yann LeCun’s success could not yet generate a momentum-breaking, devastating offensive. Instead, after a brief period of excitement, AI research seemed to enter another dormancy. This time, however, it was only a short nap of four years. Soon, it would be awakened by AlexNet, followed by a torrential release of progress.

Behind this revival, two Chinese scientists—Jensen Huang and Fei-Fei Li—were the most important unsung heroes. The former provided powerful computing power for deep learning through NVIDIA’s graphics cards and the CUDA engine, while the latter provided the soil and nutrients for CNNs through the ImageNet dataset.

AlexNet is also a convolutional neural network, proposed by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. In the 2012 ImageNet competition, it achieved an 85% recognition accuracy, improving upon the previous year’s champion by a full 10%! While LeNet could only be applied to very small scenarios and datasets, AlexNet performed recognition on images with over 1,000 categories. Its success was clearly more realistic and groundbreaking. AlexNet crowned data-driven machine learning as the king. To this day, despite various shortcomings in machine learning and the unknown distance to general AI, there seems to be no voice questioning machine learning or advocating a return to rule-based expert systems.

The next breakthrough was machine vision surpassing human capabilities. According to research, the limit of human classification ability on the ImageNet dataset is a 5.1% error rate. Once machine vision’s error rate falls below this indicator, it surpasses humans, meaning AI applications in vision are fully mature.

This decisive victory was achieved by Chinese scientist Kaiming He in 2016. Through deep residual networks (ResNet), he increased the number of neural network layers to a staggering 152 (compared to the standards of the time). Through clever design, he allowed input data to bypass certain layers during the training phase, partially solving the gradient vanishing problem in deep networks.

Ultimately, ResNet reduced the recognition error rate to 4.5%, significantly surpassing human limits! There was no longer any reason to doubt or reject AI applications!

At this point, the machine learning route had achieved overwhelming victory, and AI research entered an acceleration era. Within just a few years, breakthroughs were even made in natural language understanding. Architectures represented by Transformers succeeded in multiple fields, including natural language understanding, text-to-image generation, text-to-video generation, and programming.

Those who have traveled 90 miles are only half-way done if they have not completed the last 10. It remains unknown when humanity’s ultimate vision—Artificial General Intelligence (AGI)—will be realized. AI should become humanity’s companion in the future, but at present, humanity’s Adam has not yet created his Eve. Moreover, although contemporary AI models are extremely powerful, their internal mechanisms still lack a solid theoretical foundation. Ethical issues raised by AI have only just begun to emerge. These are all hurdles we must overcome in the future.