Many of them have been waging a war to protect their IP right to their news stories against what they see are rapacious marauders in the form of AI search bots.
To do that they have tried to block the bots from crawling their websites but it is a losing battle as the bots can still ignore their blockade and infiltrate their sites.
At the same time because of LLM search — particularly generative platforms like ChatGPT, Gemini, and Perplexity - website traffic at news portals all over the world has been dropping. Even in Indonesia Kompas.com reported a 30-60% drop in traffic since Google AI Overview had its core update in June this year.
The following is a survey by Maverick’s media outreach team on which Indonesian media is blocking which AI crawlers, and a conversation with the Executive Director of Indonesia;s Cyber Media Association, AMSI.
Indonesia’s Divided Media Landscape
A robots.txt audit of Indonesia’s top 100 online media outlets (SimilarWeb, September 2025) reveals a split in strategy. Robots.txt files act as “traffic lights” for web crawlers, instructing search engines and AI bots on whether they may access a site’s content. Yet, much like traffic lights, they are guidelines, not absolute barriers—some crawlers may still pass even when the signal says “stop.”
| Policy | Number of Outlets | Examples | Likely Motivation |
|---|---|---|---|
| Allow AI crawlers | 67 | Detik.com, Tribunnews.com, Katadata.id | Maintain visibility, maximize exposure |
| Disallow AI crawlers | 31 | Kompas.com, Tempo.co, Liputan6.com | Protect IP, safeguard editorial integrity |
| Unavailable | 2 | yummy.co, lektur.id | Sites could not be accessed |
Each AI system is treated differently, reflecting its reach and perceived risks:
- ChatGPT – A widely used AI by OpenAI for conversation, summaries, and recommendations; blocked by 29 publishers.
- CCBot – Common Crawl’s research-oriented crawler used for open datasets and AI training; blocked by 25 publishers.
- OpenAI – The broader infrastructure behind ChatGPT; blocked by 24 publishers.
- Perplexity – A search assistant synthesizing information from multiple sources; moderately restricted blocked by 14 publishers.
- Claude – Anthropic’s assistant designed for safer, reliable text generation; restricted by 8 publishers.
- Gemini – Google DeepMind’s generative model with reasoning and search integration; rarely blocked by 3 publishers.
- Google Extended – Extends Google Search with generative summaries; rarely blocked by 3 publishers.
Generative AI
Indonesian publishers are most wary of dominant platforms (ChatGPT, OpenAI, CCBot) while giving newer or less visible tools (Gemini, Google Extended) more leeway. This selective approach suggests protection is prioritized where exposure is highest.
Finding a Balanced Path
Elin Yunita Kristanti, Executive Director of AMSI, stresses that publishers cannot afford a “wait-and-see” approach:
“We cannot afford to make the mistake of being late and unprepared in the face of digital and social media disruption. Generative AI is revolutionizing the information ecosystem, and journalism must evolve to ensure it not only survives but thrives.”
Her perspective points to six priorities for Indonesian media:
-
Stay Ahead of Disruption
Media organizations should establish early warning systems to track emerging AI trends, act proactively before business models are disrupted, and use scenario planning to anticipate challenges such as traffic declines, algorithm changes, and licensing shifts -
Invest in Journalists and Journalism
To strengthen journalism, we must invest in people by training journalists in AI literacy, verification, data journalism, and ethics, build newsroom capacity with robust policies and quality controls, and focus on areas AI cannot replace such as investigative depth, contextual analysis, and local reporting.
According to the Journalism in the AI Era study by the Thomson Reuters Foundation, more than eight in ten journalists (81.7%) in Global South already use AI tools in their work, yet nearly four in five newsrooms (79.3%) have no formal AI policy, and more than half of users (57.6%) are self-taught -- underscoring the urgent need for training and governance -
Keep Humans in Control
Editorial judgment must remain a human responsibility, guided by ethics and accuracy, with AI serving only as a supporting tool -- never the final decision-maker, especially in sensitive or high-stakes reporting. -
Negotiate Fairly with AI Platforms
Publishers must secure licensing agreements to ensure fair compensation, push for transparent revenue-sharing models across ads, subscriptions, or hybrids, protect referral traffic against zero-click environments, and push regulators at the national and global levels to ensure a fair relationship between publishers and AI platforms. -
Diversify Revenue Streams
Media organization must reduce reliance on ad-driven traffic by building subscriptions, memberships, events, and branded content, explore direct-to-reader models, and leverage AI for personalization, while advocating for policies that enable sustainable innovation and reduce dependence on dominant platforms. -
Uphold Ethics and Transparency
In the AI era, editorials must uphold the highest ethical standards to safeguard credibility, with journalism earning trust through transparency about AI use, disclosure of training data and correction policies, strong safeguards against bias and misinformation, and advocacy for standards and regulations that protect information integrity.
Shaping the Future of Media in the AI Era
Generative AI is now part of the media landscape, influencing how audiences access and trust information. For Indonesian publishers, the challenge is not choosing between protection or presence, but designing strategies that balance both. Through Generative Engine Optimization (GEO), sound policies, and collective action, journalism can stay visible in AI ecosystems while safeguarding its integrity. The future belongs to publishers who adapt with resilience—and lead in shaping how trustworthy news lives in the AI era.
