Jump to content

Wikibase:Analytics/IABot: Difference between revisions

From wikibase
imported>Harej
Document the live IABot deployment-tracking model after the Airtable migration
Line 11: Line 11:


== Tracking deployments ==
== Tracking deployments ==
Goal: move out of Airtable. Track deployment information in Librarybase.
Deployment information has moved out of Airtable and into Librarybase. Each wiki InternetArchiveBot runs on, or might run on, has an item here. There are 789 wiki items as of November 2026.


Needed data fields:
Librarybase stores only what people decide or record: setup stage, approvals and the reasons behind them. Anything another system can answer more accurately is not stored here:
* '''Whether the bot is running''' comes from the InternetArchiveBot management interface. This includes who disabled it and why.
* '''Whether a wiki allows global bots''' comes from Meta's [[meta:Special:WikiSets/14|opted-out-of-global-bots wikiset]].
* '''Bot flag, blocks and last edit''' come from each wiki's API.
* '''Whether a wiki is closed''' comes from the site matrix.


* Wiki database code ([[Property:P34|P34]])
=== Wiki facts ===
* Wiki full name ([[Property:P35|P35 title]])
* Database code ([[Property:P34|P34]])
* Status ([[Property:P36|P36]])
* Title ([[Property:P35|P35]])
** Item type: [[Item:Q246911|bot deployment status (Q246911)]]
* Official website ([[Property:P41|P41]])
*** [[Item:Q246913|Defined in Management Interface (Q246913)]]
* Wikidata match ([[Property:P1|P1]], exact match)
*** [[Item:Q246914|Deployment Candidate (Q246914)]]
* Host ([[Property:P37|P37]]): the organization the wiki runs on
*** [[Item:Q246912|Pending Approval Request (Q246912)]]
** [[Item:Q246922|Wikimedia Foundation (Q246922)]]
*** [[Item:Q246915|Seeking Bot Approval (Q246915)]]
*** [[Item:Q246916|Stalled Request (Q246916)]]
*** [[Item:Q246917|Issues (Q246917)]]
*** [[Item:Q246918|Stalled Test (Q246918)]]
*** [[Item:Q246919|Test Operation (Q246919)]]
*** [[Item:Q246920|Ongoing Operation (Q246920)]]
*** [[Item:Q246921|Shelved (Q246921)]]
* Host ([[Property:P37|P37]])
** [[Item:Q246922|Wikimedia (Q246922)]]
** [[Item:Q246923|Miraheze (Q246923)]]
** [[Item:Q246923|Miraheze (Q246923)]]
** [[Item:Q246924|Weird Gloop (Q246924)]]
** [[Item:Q246924|Weird Gloop (Q246924)]]
** [[Item:Q246925|independent wiki (Q246925)]]
** [[Item:Q246925|independently hosted wiki (Q246925)]]
* Wiki family ([[Property:P38|P38]])
* Wiki family ([[Property:P38|P38]]): Wikimedia wikis only. Non-Wikimedia wikis are identified by host instead.
** [[Item:Q246926|Wikipedia (Q246926)]]
** [[Item:Q246926|Wikipedia (Q246926)]], [[Item:Q246927|Wiktionary (Q246927)]], [[Item:Q246928|Wikisource (Q246928)]], [[Item:Q246929|Wikinews (Q246929)]], [[Item:Q246930|Wikiquote (Q246930)]], [[Item:Q246931|Wikiversity (Q246931)]], [[Item:Q246932|Wikivoyage (Q246932)]], [[Item:Q246933|Wikibooks (Q246933)]]
** [[Item:Q246927|Wiktionary (Q246927)]]
** [[Item:Q249001|special Wikimedia wiki (Q249001)]]: Commons, Meta, MediaWiki.org, Incubator, Wikidata, Wikispecies, Outreach, Foundation Governance wiki
** [[Item:Q246928|Wikisource (Q246928)]]
** [[Item:Q246929|Wikinews (Q246929)]]
** [[Item:Q246930|Wikiquote (Q246930)]]
** [[Item:Q246931|Wikiversity (Q246931)]]
** [[Item:Q246932|Wikivoyage (Q246932)]]
** [[Item:Q246933|Wikibooks (Q246933)]]
* official website ([[Property:P41|P41]])
* InternetArchiveBot approval status ([[Property:P39|P39]])
** Item type: [[Item:Q246934|bot approval status (Q246934)]]
*** [[Item:Q246935|Local approval for fixing links (Q246935)]]
*** [[Item:Q246936|Local approval for linking to books (Q246936)]]
*** [[Item:Q246937|Global approval for fixing links (Q246937)]]
* InternetArchiveBot approval request link ([[Property:P40|P40]])
* Some way to document whether a wiki allows global bots or not
** I think that is implied with having a P39 of "global approval"


Updates:
=== Deployment status ([[Property:P36|P36]]) ===
P36 records the '''setup stage only'''. It says how far InternetArchiveBot's setup on the wiki has progressed, not whether the bot is running right now. A deployed wiki where the bot is stopped stays ''deployed''; the stop is recorded in the management interface. Item type: [[Item:Q246911|bot deployment status (Q246911)]].
* [[Item:Q246914|deployment candidate (Q246914)]]: we intend to deploy, and the approval process has not started
* [[Item:Q246915|seeking bot approval (Q246915)]]
* [[Item:Q246912|pending approval request (Q246912)]]: a request has been filed and not decided
* [[Item:Q246916|stalled approval request (Q246916)]]
* [[Item:Q246919|test operation (Q246919)]]: approved for, or running, a trial
* [[Item:Q246918|stalled test (Q246918)]]
* [[Item:Q246920|deployed (Q246920)]]: setup complete (formerly labelled "ongoing operation")
* [[Item:Q246921|shelved (Q246921)]]: we decided not to deploy
* [[Item:Q249002|wiki closed (Q249002)]]: the wiki is closed; this overrides any other status
* Retired: [[Item:Q246913|defined in management interface (Q246913)]], which the management interface itself shows, and [[Item:Q246917|stopped due to issues (Q246917)]], which is operational state. No item uses either.
* A wiki with no P36 statement has not been evaluated.
 
=== Approval status ([[Property:P39|P39]]) ===
Item type: [[Item:Q246934|bot approval status (Q246934)]]. Each statement is referenced to the approval discussion with [[Property:P40|P40]].
* [[Item:Q246935|local approval for fixing links (Q246935)]]
* [[Item:Q246936|local approval for linking to books (Q246936)]]
* [[Item:Q246937|global approval for fixing links (Q246937)]]: recorded only where the Airtable said the wiki relies on global bots '''and''' the wiki has not opted out of global bots on Meta. Opt-out membership alone is not enough, because some wikis (English, Russian, Swedish and Japanese Wikipedia, for example) are not opted out but still bar global bots from this task by local policy.
* On non-Wikimedia wikis, approval by the site operator is recorded as local approval.
 
Local approvals carry their history as qualifiers, taken from the wiki's rights log (which is added as a second P40 reference):
* Start time ([[Property:P45|P45]]): when the local bot flag was granted
* End time ([[Property:P46|P46]]): when the flag was removed. "Unknown value" means the approval ended at an unknown date.
* End cause ([[Property:P47|P47]]). Item type: [[Item:Q249473|approval end cause (Q249473)]].
** [[Item:Q249474|inactivity (Q249474)]]: for example, French Wikipedia (2026-02-28) and Polish Wikipedia (2025-03-15)
** [[Item:Q249475|revoked by community (Q249475)]]
** [[Item:Q249476|superseded by new approval (Q249476)]]
** [[Item:Q249477|withdrawn by operator (Q249477)]]
** [[Item:Q249478|wiki closed (Q249478)]]: for example, English, Dutch, Russian and Portuguese Wikinews
 
=== Updates ===
* Tracking Librarybase IDs in Airtable. Next task is to update existing records that already have Librarybase entries, and create new items for those that don't. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 03:35, 23 October 2025 (UTC)
* Tracking Librarybase IDs in Airtable. Next task is to update existing records that already have Librarybase entries, and create new items for those that don't. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 03:35, 23 October 2025 (UTC)
* Migrated all 789 rows of the Airtable "Wikis" table: updated 319 existing items, created 470 new ones, and added approval history from rights logs. Airtable's "Issues", "Resolved" and "Priority Area" columns were not migrated: issues live on Phabricator, and the priority areas are Phabricator tags. [[User:Harej with Claude (AI)|Harej with Claude (AI)]] 16:25, 3 November 2026 (UTC)


== Deployments ==
== Deployments ==

Revision as of 16:25, 3 November 2026

InternetArchiveBot has been operated by the Internet Archive since 2015, based on the earlier Cyberpower II. James Hare has been supporting the InternetArchiveBot effort since February 2020.

Versions

  • InternetArchiveBot 1: ?
  • InternetArchiveBot 2: The current version, flexibly supports many different kinds of wikis
    • Cycles through each wiki in alphabetical order
    • Supports all manner of wiki templates (or lack thereof)
  • InternetArchiveBot 3: The future
    • Event-centric workflow: link found dead, usage throughout wikis searched, replaced

Tracking deployments

Deployment information has moved out of Airtable and into Librarybase. Each wiki InternetArchiveBot runs on, or might run on, has an item here. There are 789 wiki items as of November 2026.

Librarybase stores only what people decide or record: setup stage, approvals and the reasons behind them. Anything another system can answer more accurately is not stored here:

  • Whether the bot is running comes from the InternetArchiveBot management interface. This includes who disabled it and why.
  • Whether a wiki allows global bots comes from Meta's opted-out-of-global-bots wikiset.
  • Bot flag, blocks and last edit come from each wiki's API.
  • Whether a wiki is closed comes from the site matrix.

Wiki facts

Deployment status (P36)

P36 records the setup stage only. It says how far InternetArchiveBot's setup on the wiki has progressed, not whether the bot is running right now. A deployed wiki where the bot is stopped stays deployed; the stop is recorded in the management interface. Item type: bot deployment status (Q246911).

Approval status (P39)

Item type: bot approval status (Q246934). Each statement is referenced to the approval discussion with P40.

Local approvals carry their history as qualifiers, taken from the wiki's rights log (which is added as a second P40 reference):

Updates

  • Tracking Librarybase IDs in Airtable. Next task is to update existing records that already have Librarybase entries, and create new items for those that don't. Harej (talk) 03:35, 23 October 2025 (UTC)
  • Migrated all 789 rows of the Airtable "Wikis" table: updated 319 existing items, created 470 new ones, and added approval history from rights logs. Airtable's "Issues", "Resolved" and "Priority Area" columns were not migrated: issues live on Phabricator, and the priority areas are Phabricator tags. Harej with Claude (AI) 16:25, 3 November 2026 (UTC)

Deployments

  • Wikimedia
    • Ongoing
  • Miraheze
    • On a handful of wikis, broken on a few wikis due to those wikis changing domain names
    • OAuth grants are broken as well
    • Idea: account creator script that uses a limited-scope bot password to log in as InternetArchiveBot and force local account creation for a specified wiki
  • Weird Gloop
    • Only four wikis
    • The reason we're not deployed there is because they want to be able to have dry runs that show the edits without actually making them