Jump to content

Wikibase:Analytics/IABot: Difference between revisions

Add links
From wikibase
← Older edit

Harej with Claude (AI) (talk | contribs)

1 user group
13,623 edits
VisualWikitext
imported>Harej
official website
Explain what Librarybase is and its role in tracking InternetArchiveBot
 
(4 intermediate revisions by 3 users not shown)
Line 1: Line 1:
'''InternetArchiveBot''' has been operated by the Internet Archive since 2015, based on the earlier Cyberpower II. James Hare has been supporting the InternetArchiveBot effort since February 2020.
'''InternetArchiveBot''' has been operated by the Internet Archive since 2015, based on the earlier Cyberpower II. James Hare has been supporting the InternetArchiveBot effort since February 2020.
'''[https://claude.ai/artifact/CivxuF86x8nUWuHbP6bACi?sk=LODA6onqh18saAQgeT69og Deployment Dashboard]'''
== How Librarybase tracks InternetArchiveBot ==
'''Librarybase''' is a structured database built on Wikibase, the same software that runs Wikidata. Instead of articles, it holds ''items'': one page per thing it describes, made of short statements such as "host: Wikimedia Foundation" or "deployment status: deployed". Each statement can carry a date range and a source, every change is kept in the page history, and anyone can look an item up or query across all of them.
For InternetArchiveBot, Librarybase is the bot's register of wikis. Every wiki the bot runs on, has run on, or might run on has an item here: all open public Wikimedia wikis, plus the non-Wikimedia wikis the bot works with. Each item records what the wiki is, how far InternetArchiveBot's setup there has got, and on what authority the bot runs there, with a link to the discussion where that permission was given. This replaced the Airtable spreadsheet that used to hold the same information, and it is meant to stay the reference until InternetArchiveBot can track these facts itself.
Librarybase keeps only what people decide or record: setup stages, approvals and the reasons behind them. Anything another system can answer more accurately is looked up there, not copied here:
* '''Whether the bot is running''' on a wiki, and who switched it off and why, comes from the InternetArchiveBot management interface (its <code>runpages</code> API).
* '''Whether a wiki allows global bots''' comes from Meta's [[meta:Special:WikiSets/14|opted-out-of-global-bots wikiset]].
* '''The bot flag, blocks and last edit''' come from each wiki's own API.
* '''Whether a wiki is closed''' comes from Wikimedia's site matrix.
The [https://claude.ai/artifact/CivxuF86x8nUWuHbP6bACi?sk=LODA6onqh18saAQgeT69og deployment dashboard] combines these sources into one view of where the bot is and isn't operating. The table below lists the properties used on wiki items and the values each can take.
=== Properties ===
{| class="wikitable"
! Property !! Name !! Potential values !! Comments
|-
| [[Property:P34|P34]] || database code || Text, for example <code>enwiki</code> || The wiki's database name. Every wiki item has one; tools match wikis on it. MD Wiki's code is <code>mdwikior_wiki</code>, but the management interface calls it <code>mdwiki_wiki</code>, which is kept as an alias on the item.
|-
| [[Property:P35|P35]] || title || Text (English) || The wiki's full name, for example "English Wikipedia".
|-
| [[Property:P41|P41]] || official website || URL || The wiki's address, for example <code>https://en.wikipedia.org</code>.
|-
| [[Property:P1|P1]] || exact match || URL of a Wikidata item || The wiki's Wikidata item, where Wikidata has one. Left off where Wikidata attaches the database code to an organization, not a wiki (some affiliate wikis).
|-
| [[Property:P37|P37]] || host || [[Item:Q246922|Wikimedia Foundation (Q246922)]]<br>[[Item:Q246923|Miraheze (Q246923)]]<br>[[Item:Q246924|Weird Gloop (Q246924)]]<br>[[Item:Q246925|independently hosted wiki (Q246925)]] || The organization the wiki runs on.
|-
| [[Property:P38|P38]] || wiki family || [[Item:Q246926|Wikipedia (Q246926)]]<br>[[Item:Q246927|Wiktionary (Q246927)]]<br>[[Item:Q246928|Wikisource (Q246928)]]<br>[[Item:Q246929|Wikinews (Q246929)]]<br>[[Item:Q246930|Wikiquote (Q246930)]]<br>[[Item:Q246931|Wikiversity (Q246931)]]<br>[[Item:Q246932|Wikivoyage (Q246932)]]<br>[[Item:Q246933|Wikibooks (Q246933)]]<br>[[Item:Q249001|special Wikimedia wiki (Q249001)]] || Wikimedia wikis only; non-Wikimedia wikis are identified by host instead. "Special Wikimedia wiki" covers the site matrix's specials: Commons, Meta, MediaWiki.org, Wikidata, Wikispecies, Incubator, test wikis and affiliate wikis.
|-
| [[Property:P36|P36]] || InternetArchiveBot deployment status || [[Item:Q246914|deployment candidate (Q246914)]]<br>[[Item:Q246915|seeking bot approval (Q246915)]]<br>[[Item:Q246912|pending approval request (Q246912)]]<br>[[Item:Q246916|stalled approval request (Q246916)]]<br>[[Item:Q246919|test operation (Q246919)]]<br>[[Item:Q246918|stalled test (Q246918)]]<br>[[Item:Q246920|deployed (Q246920)]]<br>[[Item:Q246921|shelved (Q246921)]]<br>[[Item:Q249002|wiki closed (Q249002)]] || Records the '''setup stage only''', not whether the bot is running now: a deployed wiki where the bot is switched off stays ''deployed'', and the switch-off is in the management interface. "Wiki closed" overrides every other value. A wiki with no P36 has not been evaluated, which includes every wiki the management interface doesn't list. Item type: [[Item:Q246911|bot deployment status (Q246911)]]. Retired: [[Item:Q246913|defined in management interface (Q246913)]], which the management interface itself shows, and [[Item:Q246917|stopped due to issues (Q246917)]], which is operational state; no item uses either. Q246920 was formerly labelled "ongoing operation".
|-
| [[Property:P39|P39]] || InternetArchiveBot approval status || [[Item:Q246935|local approval for fixing links (Q246935)]]<br>[[Item:Q246936|local approval for linking to books (Q246936)]]<br>[[Item:Q246937|global approval for fixing links (Q246937)]] || Item type: [[Item:Q246934|bot approval status (Q246934)]]. Global approval is recorded only where the Airtable said the wiki relies on global bots '''and''' the wiki hasn't opted out of global bots on Meta ([[meta:Special:WikiSets/14|wikiset 14]]). Opt-out membership alone isn't enough: some wikis (English, Russian, Swedish and Japanese Wikipedia, for example) aren't opted out but still bar global bots from this task by local policy. On non-Wikimedia wikis, approval by the site operator is recorded as local approval.
|-
| [[Property:P40|P40]] || InternetArchiveBot approval request URL || URL || Used as the '''reference''' on P39 statements, not as a main statement: first the approval discussion, then the wiki's rights log (<code>Special:Log?type=rights&page=User:InternetArchiveBot</code>) where the dates below come from it.
|-
| [[Property:P45|P45]] || start time || Date || Qualifier on P39 local approvals: when the local bot flag was granted, from the rights log. Where the flag was granted, removed and granted again, the latest grant.
|-
| [[Property:P46|P46]] || end time || Date, or "unknown value" || Qualifier on P39 local approvals: when the bot flag was removed. "Unknown value" means the approval ended on an unknown date.
|-
| [[Property:P47|P47]] || end cause || [[Item:Q249474|inactivity (Q249474)]]<br>[[Item:Q249475|revoked by community (Q249475)]]<br>[[Item:Q249476|superseded by new approval (Q249476)]]<br>[[Item:Q249477|withdrawn by operator (Q249477)]]<br>[[Item:Q249478|wiki closed (Q249478)]] || Qualifier on P39 local approvals, with P46. Item type: [[Item:Q249473|approval end cause (Q249473)]]. In use: inactivity (French Wikipedia 2026-02-28, Polish Wikipedia 2025-03-15) and wiki closed (English, Dutch, Russian and Portuguese Wikinews).
|}
=== Updates ===
* Tracking Librarybase IDs in Airtable. Next task is to update existing records that already have Librarybase entries, and create new items for those that don't. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 03:35, 23 October 2025 (UTC)
* Migrated all 789 rows of the Airtable "Wikis" table: updated 319 existing items, created 470 new ones, and added approval history from rights logs. Airtable's "Issues", "Resolved" and "Priority Area" columns were not migrated: issues live on Phabricator, and the priority areas are Phabricator tags. [[User:Harej with Claude (AI)|Harej with Claude (AI)]] 16:25, 3 November 2026 (UTC)


== Versions ==
== Versions ==
Line 9: Line 57:
* InternetArchiveBot 3: The future
* InternetArchiveBot 3: The future
** Event-centric workflow: link found dead, usage throughout wikis searched, replaced
** Event-centric workflow: link found dead, usage throughout wikis searched, replaced
== Tracking deployments ==
Goal: move out of Airtable. Track deployment information in Librarybase.
Needed data fields:
* Wiki database code ([[Property:P34|P34]])
* Wiki full name ([[Property:P35|P35 title]])
* Status ([[Property:P36|P36]])
** Item type: [[Item:Q246911|bot deployment status (Q246911)]]
*** [[Item:Q246913|Defined in Management Interface (Q246913)]]
*** [[Item:Q246914|Deployment Candidate (Q246914)]]
*** [[Item:Q246912|Pending Approval Request (Q246912)]]
*** [[Item:Q246915|Seeking Bot Approval (Q246915)]]
*** [[Item:Q246916|Stalled Request (Q246916)]]
*** [[Item:Q246917|Issues (Q246917)]]
*** [[Item:Q246918|Stalled Test (Q246918)]]
*** [[Item:Q246919|Test Operation (Q246919)]]
*** [[Item:Q246920|Ongoing Operation (Q246920)]]
*** [[Item:Q246921|Shelved (Q246921)]]
* Host ([[Property:P37|P37]])
** [[Item:Q246922|Wikimedia (Q246922)]]
** [[Item:Q246923|Miraheze (Q246923)]]
** [[Item:Q246924|Weird Gloop (Q246924)]]
** [[Item:Q246925|independent wiki (Q246925)]]
* Wiki family ([[Property:P38|P38]])
** [[Item:Q246926|Wikipedia (Q246926)]]
** [[Item:Q246927|Wiktionary (Q246927)]]
** [[Item:Q246928|Wikisource (Q246928)]]
** [[Item:Q246929|Wikinews (Q246929)]]
** [[Item:Q246930|Wikiquote (Q246930)]]
** [[Item:Q246931|Wikiversity (Q246931)]]
** [[Item:Q246932|Wikivoyage (Q246932)]]
** [[Item:Q246933|Wikibooks (Q246933)]]
* official website ([[Property:P41|P41]])
* InternetArchiveBot approval status ([[Property:P39|P39]])
** Item type: [[Item:Q246934|bot approval status (Q246934)]]
*** [[Item:Q246935|Local approval for fixing links (Q246935)]]
*** [[Item:Q246936|Local approval for linking to books (Q246936)]]
*** [[Item:Q246937|Global approval for fixing links (Q246937)]]
* InternetArchiveBot approval request link ([[Property:P40|P40]])
* Some way to document whether a wiki allows global bots or not
** I think that is implied with having a P39 of "global approval"
== Deployments ==
* Wikimedia
** Ongoing
* Miraheze
** On a handful of wikis, broken on a few wikis due to those wikis changing domain names
** OAuth grants are broken as well
** '''Idea:''' account creator script that uses a limited-scope bot password to log in as InternetArchiveBot and force local account creation for a specified wiki
* Weird Gloop
** Only four wikis
** The reason we're not deployed there is because they want to be able to have dry runs that show the edits without actually making them

Latest revision as of 17:43, 3 November 2026

InternetArchiveBot has been operated by the Internet Archive since 2015, based on the earlier Cyberpower II. James Hare has been supporting the InternetArchiveBot effort since February 2020.

Deployment Dashboard

How Librarybase tracks InternetArchiveBot

Librarybase is a structured database built on Wikibase, the same software that runs Wikidata. Instead of articles, it holds items: one page per thing it describes, made of short statements such as "host: Wikimedia Foundation" or "deployment status: deployed". Each statement can carry a date range and a source, every change is kept in the page history, and anyone can look an item up or query across all of them.

For InternetArchiveBot, Librarybase is the bot's register of wikis. Every wiki the bot runs on, has run on, or might run on has an item here: all open public Wikimedia wikis, plus the non-Wikimedia wikis the bot works with. Each item records what the wiki is, how far InternetArchiveBot's setup there has got, and on what authority the bot runs there, with a link to the discussion where that permission was given. This replaced the Airtable spreadsheet that used to hold the same information, and it is meant to stay the reference until InternetArchiveBot can track these facts itself.

Librarybase keeps only what people decide or record: setup stages, approvals and the reasons behind them. Anything another system can answer more accurately is looked up there, not copied here:

  • Whether the bot is running on a wiki, and who switched it off and why, comes from the InternetArchiveBot management interface (its runpages API).
  • Whether a wiki allows global bots comes from Meta's opted-out-of-global-bots wikiset.
  • The bot flag, blocks and last edit come from each wiki's own API.
  • Whether a wiki is closed comes from Wikimedia's site matrix.

The deployment dashboard combines these sources into one view of where the bot is and isn't operating. The table below lists the properties used on wiki items and the values each can take.

Properties

Property Name Potential values Comments
P34 database code Text, for example enwiki The wiki's database name. Every wiki item has one; tools match wikis on it. MD Wiki's code is mdwikior_wiki, but the management interface calls it mdwiki_wiki, which is kept as an alias on the item.
P35 title Text (English) The wiki's full name, for example "English Wikipedia".
P41 official website URL The wiki's address, for example https://en.wikipedia.org.
P1 exact match URL of a Wikidata item The wiki's Wikidata item, where Wikidata has one. Left off where Wikidata attaches the database code to an organization, not a wiki (some affiliate wikis).
P37 host Wikimedia Foundation (Q246922)
Miraheze (Q246923)
Weird Gloop (Q246924)
independently hosted wiki (Q246925)		 The organization the wiki runs on.
P38 wiki family Wikipedia (Q246926)
Wiktionary (Q246927)
Wikisource (Q246928)
Wikinews (Q246929)
Wikiquote (Q246930)
Wikiversity (Q246931)
Wikivoyage (Q246932)
Wikibooks (Q246933)
special Wikimedia wiki (Q249001)		 Wikimedia wikis only; non-Wikimedia wikis are identified by host instead. "Special Wikimedia wiki" covers the site matrix's specials: Commons, Meta, MediaWiki.org, Wikidata, Wikispecies, Incubator, test wikis and affiliate wikis.
P36 InternetArchiveBot deployment status deployment candidate (Q246914)
seeking bot approval (Q246915)
pending approval request (Q246912)
stalled approval request (Q246916)
test operation (Q246919)
stalled test (Q246918)
deployed (Q246920)
shelved (Q246921)
wiki closed (Q249002)		 Records the setup stage only, not whether the bot is running now: a deployed wiki where the bot is switched off stays deployed, and the switch-off is in the management interface. "Wiki closed" overrides every other value. A wiki with no P36 has not been evaluated, which includes every wiki the management interface doesn't list. Item type: bot deployment status (Q246911). Retired: defined in management interface (Q246913), which the management interface itself shows, and stopped due to issues (Q246917), which is operational state; no item uses either. Q246920 was formerly labelled "ongoing operation".
P39 InternetArchiveBot approval status local approval for fixing links (Q246935)
local approval for linking to books (Q246936)
global approval for fixing links (Q246937)		 Item type: bot approval status (Q246934). Global approval is recorded only where the Airtable said the wiki relies on global bots and the wiki hasn't opted out of global bots on Meta (wikiset 14). Opt-out membership alone isn't enough: some wikis (English, Russian, Swedish and Japanese Wikipedia, for example) aren't opted out but still bar global bots from this task by local policy. On non-Wikimedia wikis, approval by the site operator is recorded as local approval.
P40 InternetArchiveBot approval request URL URL Used as the reference on P39 statements, not as a main statement: first the approval discussion, then the wiki's rights log (Special:Log?type=rights&page=User:InternetArchiveBot) where the dates below come from it.
P45 start time Date Qualifier on P39 local approvals: when the local bot flag was granted, from the rights log. Where the flag was granted, removed and granted again, the latest grant.
P46 end time Date, or "unknown value" Qualifier on P39 local approvals: when the bot flag was removed. "Unknown value" means the approval ended on an unknown date.
P47 end cause inactivity (Q249474)
revoked by community (Q249475)
superseded by new approval (Q249476)
withdrawn by operator (Q249477)
wiki closed (Q249478)		 Qualifier on P39 local approvals, with P46. Item type: approval end cause (Q249473). In use: inactivity (French Wikipedia 2026-02-28, Polish Wikipedia 2025-03-15) and wiki closed (English, Dutch, Russian and Portuguese Wikinews).

Updates

  • Tracking Librarybase IDs in Airtable. Next task is to update existing records that already have Librarybase entries, and create new items for those that don't. Harej (talk) 03:35, 23 October 2025 (UTC)
  • Migrated all 789 rows of the Airtable "Wikis" table: updated 319 existing items, created 470 new ones, and added approval history from rights logs. Airtable's "Issues", "Resolved" and "Priority Area" columns were not migrated: issues live on Phabricator, and the priority areas are Phabricator tags. Harej with Claude (AI) 16:25, 3 November 2026 (UTC)

Versions

  • InternetArchiveBot 1: ?
  • InternetArchiveBot 2: The current version, flexibly supports many different kinds of wikis
    • Cycles through each wiki in alphabetical order
    • Supports all manner of wiki templates (or lack thereof)
  • InternetArchiveBot 3: The future
    • Event-centric workflow: link found dead, usage throughout wikis searched, replaced
Retrieved from "https://librarybase.org/w/index.php?title=Wikibase:Analytics/IABot&oldid=249162"