As artificial intelligence and machine learning technologies have matured, they have moved from being a curiosity for researchers to tools used at industrial scale. Along with this, the methods used and the infrastructure required have started to change. These developments in artificial intelligence and machine learning create great opportunities for companies that want to engage with these topics to gain a competitive advantage in their industries, while also creating serious risks.
Only 10 years have passed since the remarkable team of Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton revolutionized deep learning with AlexNet, the model that won the ImageNet competition. This three-person team used a seven-layer convolutional neural network (CNN) to understand the content of photos and trained the model on a $700 graphics card designed for playing video games. Using a GPU to train a machine learning model for the first time not only let them beat their competitors by a wide margin in that competition, it also opened a new era in machine learning and deep learning.
With AlexNet and a few developments that followed, the world of machine learning changed forever. These advances caught the attention of large companies such as IBM, Google and Microsoft, which had been hesitant about the field. Pioneering start-ups began to be acquired by large companies. Academia also changed dramatically, as many experts moved extremely quickly from skepticism about artificial intelligence to wholeheartedly embracing deep learning.
Ten years have passed since AlexNet. Artificial neural networks have reached a level where they can solve both everyday problems and many industrial problems. Fundamentally, the neural networks used today are no different from the networks used in AlexNet 10 years ago. However, they have far more artificial neurons and far more definable connections. Models such as GPT-3, developed by OpenAI, were trained on supercomputer-scale infrastructure and began to be used to understand human language and generate human-like content. While AlexNet could be trained on a $700 GPU, training GPT-3 required thousands of GPUs, much, much more time and millions of dollars.
The emergence of these massive, high-cost foundation models brings opportunities, risks and limitations for newcomers and for those who want to innovate in artificial intelligence and machine learning. Although they cannot compete with Google, Facebook, Microsoft, Amazon or OpenAI at the cutting edge, smaller organizations can draw on the work of these giants, including their open-source ready-made models, to start building their own machine learning-powered applications.
Neural networks such as AlexNet were originally trained from scratch for each task. This approach works when a few weeks of training on a single piece of gaming hardware, a single GPU, is enough, but training becomes very difficult once neural network sizes, required processing power and training data volumes start scaling to meet goals. This led to the spread of an approach known as pretraining. With it, a neural network is first trained on a large general-purpose dataset using significant computing resources, and then fine-tuned for a specific task using a much smaller dataset.
As the industrialization of machine learning has taken over many fields (such as language or speech processing) and the amount of data available for training has grown significantly, the use of pretrained networks has exploded in recent years. Using pretrained networks allows a start-up, for example, to build a product with far less data and computing resources than it would need when starting from scratch. This approach is also becoming popular in academia, where researchers can quickly fine-tune a pretrained network for a new task and then publish the results.
Models such as BERT, GPT, DALL-E and CLIP, pretrained for specific task areas including understanding or generating written text, recognizing the content of photos or videos and processing audio, are pretrained on massive, general-purpose datasets (often on the order of billions of training examples) and released as open source by well-funded AI labs such as Google, Microsoft and OpenAI. The pace of innovation in commercialized machine learning applications and the democratizing effect of these foundation models cannot be underestimated. For practitioners in the field who do not have a spare supercomputer lying around, these open-source pretrained models have become a cure-all.
At PaperWork, we also fine-tune these open-source foundation models, trained on large datasets with huge budgets, for different use cases. Even this fine-tuning requires days or weeks of run time on several GPUs and dozens of retraining cycles. And producing models through purpose-specific fine-tuning does not end the work. Hardly a day goes by without new, massively funded models and approaches trained on new datasets being released, sending everyone back to the start. We train our AI models and make them available in our products. And at the end of every model project, we immediately start again to train new models. This is where the most important requirement for delivering AI-powered products lies: being able to create living models and stand firmly behind them.
Sources
https://www.ibm.com/think/topics/foundation-models
https://www.ibm.com/think/topics/large-language-models
https://dergipark.org.tr/tr/pub/insanvetoplum/article/1597520
Would you like to see PaperWork in your organization?
Request a Demo
