I note that the best short‑term strategy is to continue focused on generating libraries of human generated text with the help of domain experts across the world.
I record that this training data scarcity could be delayed by 1. data efficiency improvements 2. synthetic data 3. learning beyond currently common modality such as text, images, videos, audios.
Training data is going to be the biggest bottleneck for data hungry Gen AI!
Sooner we are entering into a phase where training data scarcity will become real similar to GPUs
A new study released Thursday by research group Epoch AI projects that tech companies will exhaust the supply of publicly available training data for AI language models by roughly the turn of the decade—sometime between 2026 and 2032.
In the short term, tech companies like ChatGPT-maker OpenAI and Google are tapping this by signing deals to tap into the steady flow of sentences coming out of Reddit forums and news media outlets.
In the longer term, there won’t be enough new blogs, news articles and social media commentary to sustain the current trajectory of AI development, putting pressure on companies to tap into sensitive data now considered private—such as emails or text messages—or relying on less‑reliable “synthetic data” spit out by the chatbots themselves.
These data‑scarcity insights complement the hardware advantage narrative in owning ai infrastructure advantage.
I concluded that the biggest bottleneck to AI progress is the availability of high‑quality, curated training data, which I call the "fossil fuel of AI."
On 2025‑09‑15 02:51 UTC I stated: "The fossil fuel of AI is the biggest bottleneck! This is called curated training data."