Sepp Hochreiter: "In AI, basic research and practice are very close together!"

AI researcher Sepp Hochreiter has achieved another remarkable breakthrough. A language model that works faster and more efficiently than ChatGPT and similar systems. However, the research lead that has been painstakingly built up in Austria is in danger of evaporating.

An algorithm that has the potential to put the currently heavily courted ChatGPT out of business. According to Sepp HochreiterDirector of the Institute for Machine Learning and the Linz Institute of Technology AI Lab, it is currently located on the servers of Johannes Keppler University (JKU) in Linz.

The German-born scientist, who has been conducting research in the field of machine learning in Austria for over ten years, laid the foundation for the development of today's AI systems with his work on Long Short-Term Memory (LSTM) in the late 1990s.

So far, however, Sepp Hochreiter has held back the algorithm, mainly due to a lack of funds for the necessary computing power to train the AI. But this could change in the foreseeable future, as Hochreiter explained at the second Deep Dive Meet Up of the industry magazine explained.

Enormous computing capacities required

But back to the beginning: the development of the algorithm was associated with a number of difficulties. Above all with a problem that many researchers in Austria criticise: For Basic research often lack the necessary small change.

This is not only indispensable when it comes to AI, as Prof Sepp Hochreiter explains in a Interview confirms: "Basic research and the Practical application go hand in hand in the field of artificial intelligence. A prime example of this are the so-called 'self-normalised networks'which we developed in Linz and which were implemented at Amazon just three months later, leading to significant increases in sales. The time span from the theoretical model that emerges from basic research to practical implementation can be extremely short in this field."

Hochreiter adds: "This can also be seen in Open AI, where the researchers hardly had time to bring developers into the team due to the speed of progress. Another striking feature of this area is that the newly developed tools are immediately adopted by large companies. In Austria, we are more used to a product being perfected extensively before it is launched on the market. But things are different in the AI sector, as the rapid market launch of ChatGPT shows."

Hochreiter's most recent achievement is based on its LSTM principle from the 1990s. In view of the increasing text length that programmes such as ChatGPT have to deal with, Sepp Hochreiter once again resorts to the LSTM concept back. However, this has been combined with Transformer technology, which is currently used in almost all large AI models. The result is an AI that is faster and can analyse much longer sentences.

However, training artificial intelligence requires enormous computing capacities and a lot of staff in the form of software developers - both of which are not feasible in Austria due to the limited budget, according to Hochreiter. In the Netherlands, for example, two billion euros are being invested in the research and development of AI. The University of Tübingen, where Hochreiter is a member of a committee, receives around 30 million euros for their AI centre.

Compared to the budgets of other countries for AI research and development, Austria is on a par with Uganda and Mexico, according to Hochreiter.

Funding requirement of around 500 million euros

For Basic AI research Prof Sepp Hochreiter sees a funding requirement in Austria of around 500 million euros: "I have looked at the Austrian funding measures again in detail and come up with an annual funding sum of around twelve million euros, which is actually earmarked for the field of AI - so we are a long way from the alleged 150 million euros that State Secretary Tursky mentioned. That annoys me because I can see what is currently being done in this area in Germany. There are AI centres that deserve the name."

Teaching the AI in particular requires a lot of computing power and, according to Hochreiter, is the most expensive factor in rolling out the algorithm. However, he now has investors from all over the world who want to invest in his technology. This would not be the first time that an intelligence developed by Hochreiter and his team has learnt to run at US companies: a few years ago, cutting-edge research by the Hochreiter team was tested at Amazon. This resulted in two billion more sales for the US company at the time. Nevertheless, Prof Sepp Hochreiter does not rule out the possibility that his language model will be further developed in Austria and brought to market maturity.

However, the AI pioneer is very sceptical about the location: "It's a disaster here! For six years, I've been criticising the fact that researchers and entrepreneurs in Austria have nowhere to dock. Industrial companies don't know what is possible with AI, and researchers don't know what industry needs."

Matthias Heschl

AI will play a central role