Librarybase - User contributions [en]

Project:Analytics/WCD

2026-05-20T19:42:16Z

Harej: Removing notes from abandoned build attempts

Project:Analytics/Orb

2026-05-20T17:25:21Z

Harej: one-third milestone

== 2025-10-25 Orb Blended ==
The virtual machine "Orb Blended" in my workstation cluster was created to experiment with blending the Wikidata Query Service with other datasets in a single Blazegraph instance. Work started earlier, but this is the first journal.

For the last few weeks it has been building the Wikidata Query Service. I have paused it at file 4470 since I am taking the system down. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 18:25, 25 October 2025 (UTC)

* Resuming build. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 03:49, 28 October 2025 (UTC)
* 2025-11-08: Rebuild took too long; have to start again. Downloading dump from November 3. If I can't complete the build by December 3, I am not sure Blazegraph is still viable.
* 2025-12-06: Rebuild of dataset is complete; local up-to-date copy of WDQS on dev1001.
* Started to gzip data.jnl in preparation to upload to fileserver and as a backup before experimenting with adding more data sources. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 02:00, 7 December 2025 (UTC)

== 2026-01 Rebuild attempt ==
Updater is running into numerous issues on both my workstation and the server, so I am downloading wikidata-20260112-all-BETA.ttl.gz to my workstation for a rebuild. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 22:33, 22 January 2026 (UTC)

Munge process produced 4940 bundles; now ingesting into Blazegraph. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 23:25, 24 January 2026 (UTC)

== 2026-04 Rebuild attempt ==
The last rebuild attempt, on my workstation, failed because it took too long to build the database. By the time it was finished, the data was more than 30 days old and could not be synced with recent changes. Trying again, directly on the production server (station1001), with wikidata-'''20260330'''-all-BETA.ttl.gz, only three days old as of writing.

* 2026-04-02: Direct rsync from WMF dump server to orb-wdqs
* 2026-04-02: Begin munge process
* 2026-04-04 15:23 (UTC): Begin loading of '''4971''' bundles
* 2026-04-09 22:28 (UTC): Completed 1657 (one-third of) files
* 2026-04-20 03:27 (UTC): Completed 3314 (two-thirds of) files
* Canceled at 5:52 PM local time on 2026-04-21 since it was not going to complete and more urgent hardware matters needed to be attended to

== 2026-05 Rebuild attempt ==
This time on the workstation again.

* 2026-05-16 at 10:55 AM PDT:
** docker compose exec wdqs /wdqs/munge.sh -c 25000 -f /sources/wikidata-'''20260511'''-all-BETA.ttl.gz -d /sources
* 2026-05-17 at 01:04 PM PDT (across 5,007 shards):
** /wdqs/loadData.sh -n wdq -d /sources && /wdqs/runUpdate.sh
** 2026-05-20 at 07:35:39 AM PDT: 1669 files (one-third) loaded

Project:Analytics/Orb

2026-05-17T20:04:47Z

Harej: Update

== 2025-10-25 Orb Blended ==
The virtual machine "Orb Blended" in my workstation cluster was created to experiment with blending the Wikidata Query Service with other datasets in a single Blazegraph instance. Work started earlier, but this is the first journal.

For the last few weeks it has been building the Wikidata Query Service. I have paused it at file 4470 since I am taking the system down. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 18:25, 25 October 2025 (UTC)

* Resuming build. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 03:49, 28 October 2025 (UTC)
* 2025-11-08: Rebuild took too long; have to start again. Downloading dump from November 3. If I can't complete the build by December 3, I am not sure Blazegraph is still viable.
* 2025-12-06: Rebuild of dataset is complete; local up-to-date copy of WDQS on dev1001.
* Started to gzip data.jnl in preparation to upload to fileserver and as a backup before experimenting with adding more data sources. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 02:00, 7 December 2025 (UTC)

== 2026-01 Rebuild attempt ==
Updater is running into numerous issues on both my workstation and the server, so I am downloading wikidata-20260112-all-BETA.ttl.gz to my workstation for a rebuild. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 22:33, 22 January 2026 (UTC)

Munge process produced 4940 bundles; now ingesting into Blazegraph. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 23:25, 24 January 2026 (UTC)

== 2026-04 Rebuild attempt ==
The last rebuild attempt, on my workstation, failed because it took too long to build the database. By the time it was finished, the data was more than 30 days old and could not be synced with recent changes. Trying again, directly on the production server (station1001), with wikidata-'''20260330'''-all-BETA.ttl.gz, only three days old as of writing.

* 2026-04-02: Direct rsync from WMF dump server to orb-wdqs
* 2026-04-02: Begin munge process
* 2026-04-04 15:23 (UTC): Begin loading of '''4971''' bundles
* 2026-04-09 22:28 (UTC): Completed 1657 (one-third of) files
* 2026-04-20 03:27 (UTC): Completed 3314 (two-thirds of) files
* Canceled at 5:52 PM local time on 2026-04-21 since it was not going to complete and more urgent hardware matters needed to be attended to

== 2026-05 Rebuild attempt ==
This time on the workstation again.

* 2026-05-16 at 10:55 AM PDT:
** docker compose exec wdqs /wdqs/munge.sh -c 25000 -f /sources/wikidata-'''20260511'''-all-BETA.ttl.gz -d /sources
* 2026-05-17 at 01:04 PM PDT (over 5,007 shards):
** /wdqs/loadData.sh -n wdq -d /sources && /wdqs/runUpdate.sh

Project:Analytics/Orb

2026-05-16T17:57:08Z

Harej: 2026-05 rebuild attempt

== 2025-10-25 Orb Blended ==
The virtual machine "Orb Blended" in my workstation cluster was created to experiment with blending the Wikidata Query Service with other datasets in a single Blazegraph instance. Work started earlier, but this is the first journal.

For the last few weeks it has been building the Wikidata Query Service. I have paused it at file 4470 since I am taking the system down. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 18:25, 25 October 2025 (UTC)

* Resuming build. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 03:49, 28 October 2025 (UTC)
* 2025-11-08: Rebuild took too long; have to start again. Downloading dump from November 3. If I can't complete the build by December 3, I am not sure Blazegraph is still viable.
* 2025-12-06: Rebuild of dataset is complete; local up-to-date copy of WDQS on dev1001.
* Started to gzip data.jnl in preparation to upload to fileserver and as a backup before experimenting with adding more data sources. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 02:00, 7 December 2025 (UTC)

== 2026-01 Rebuild attempt ==
Updater is running into numerous issues on both my workstation and the server, so I am downloading wikidata-20260112-all-BETA.ttl.gz to my workstation for a rebuild. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 22:33, 22 January 2026 (UTC)

Munge process produced 4940 bundles; now ingesting into Blazegraph. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 23:25, 24 January 2026 (UTC)

== 2026-04 Rebuild attempt ==
The last rebuild attempt, on my workstation, failed because it took too long to build the database. By the time it was finished, the data was more than 30 days old and could not be synced with recent changes. Trying again, directly on the production server (station1001), with wikidata-'''20260330'''-all-BETA.ttl.gz, only three days old as of writing.

* 2026-04-02: Direct rsync from WMF dump server to orb-wdqs
* 2026-04-02: Begin munge process
* 2026-04-04 15:23 (UTC): Begin loading of '''4971''' bundles
* 2026-04-09 22:28 (UTC): Completed 1657 (one-third of) files
* 2026-04-20 03:27 (UTC): Completed 3314 (two-thirds of) files
* Canceled at 5:52 PM local time on 2026-04-21 since it was not going to complete and more urgent hardware matters needed to be attended to

== 2026-05 Rebuild attempt ==
This time on the workstation again.

* 2026-05-16 at 10:55 AM PDT: docker compose exec wdqs /wdqs/munge.sh -c 25000 -f /sources/wikidata-20260511-all-BETA.ttl.gz -d /sources

Project:Analytics/Wikibases

2026-05-11T17:53:17Z

Harej: Migration complete

The "Cool Wikibase Network" includes Wikibases that James Hare created. Wikibases that were created by other people may be admitted into the network in the future.

These wikis are:

* <s>Open Miscellaneous Wikibase</s> (January 2023–January 2026)
* [https://librarybase.org Librarybase] (May 2023–present)
* [https://internetdomains.wiki Internet Domains Wikibase] (May 2023–present)
* [https://wikibase.world Wikibase World] (June 2023–present)
* [https://communities.miraheze.org Miraheze Communities] (September 2025–present)
Membership in the Cool Wikibase Network mainly means being included in the [[Template:Cwbn]] footer featured on the member wikis, a callback to the days of web rings. It otherwise confers no material advantage or special privilege.

==2026-01-16 Migration away from Wikibase.Cloud==
Wikibase Cloud is proposing new hosting requirements that would subject tenants to evaluation against arbitrary, shifting standards. To reduce operational uncertainty, I am moving critical systems off Wikibase.Cloud in favor of self-hosting.
* 2026-01-16: I migrated the majority of Open Miscellaneous Wikibase to Librarybase and then deleted the old wiki.
** InternetArchiveBot-related template tracking ([[Item:Q246948|example item]])
** [[Item:Q246951|MIT License modeling experiment]]
* 2026-01-18:
** Deleted the "credbot-daily" job on "vaccinesafety" Toolforge project in anticipation of migration.
** Posted [https://domains.wikibase.cloud/wiki/MediaWiki:Sitenotice Sitenotice on Internet Domains]
** Registered internetdomains.wiki
** [https://phabricator.wikimedia.org/T414893 Requested dump] of Internet Domains
*2026-04-28:
**Set up a VM on my server to turn import SQL dump and export XML dump for import into Miraheze; TTL dump for ingest into a query service
**Created "internetdomains.wiki" item to commemorate new wiki (and so that the query service updater doesn't complain about the data being too old)
**Ran commands:
***<code>php maintenance/run.php dumpBackup --full > /tmp/internetdomains-2026-03.xml</code>
***<code>php extensions/Wikibase/repo/maintenance/dumpRdf.php --format=ttl --server <nowiki>https://internetdomains.wiki</nowiki> > /tmp/internetdomains-2026-03.ttl</code>
Migration to Miraheze completed on May 10, 2026

Template:Cwbn

2026-05-11T00:30:41Z

Harej: Internet Domains new address

<div style="margin-top:2em"><hr style="width:5em; margin-bottom:0.75em;" />'''''Cool Wikibase Network'''''[https://wikibase.world Wikibase World][https://librarybase.org Librarybase][https://internetdomains.wiki Internet Domains][https://communities.miraheze.org Miraheze Communities]</div>

Project:Analytics/Wikibases

2026-04-29T05:36:24Z

Harej: Update

Project:Analytics/Orb

2026-04-26T06:58:39Z

Harej: Unfortunate update

== 2025-10-25 Orb Blended ==
The virtual machine "Orb Blended" in my workstation cluster was created to experiment with blending the Wikidata Query Service with other datasets in a single Blazegraph instance. Work started earlier, but this is the first journal.

For the last few weeks it has been building the Wikidata Query Service. I have paused it at file 4470 since I am taking the system down. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 18:25, 25 October 2025 (UTC)

* Resuming build. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 03:49, 28 October 2025 (UTC)
* 2025-11-08: Rebuild took too long; have to start again. Downloading dump from November 3. If I can't complete the build by December 3, I am not sure Blazegraph is still viable.
* 2025-12-06: Rebuild of dataset is complete; local up-to-date copy of WDQS on dev1001.
* Started to gzip data.jnl in preparation to upload to fileserver and as a backup before experimenting with adding more data sources. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 02:00, 7 December 2025 (UTC)

== 2026-01-22 Rebuild ==
Updater is running into numerous issues on both my workstation and the server, so I am downloading wikidata-20260112-all-BETA.ttl.gz to my workstation for a rebuild. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 22:33, 22 January 2026 (UTC)

Munge process produced 4940 bundles; now ingesting into Blazegraph. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 23:25, 24 January 2026 (UTC)

== 2026-04 Rebuild ==
The last rebuild attempt, on my workstation, failed because it took too long to build the database. By the time it was finished, the data was more than 30 days old and could not be synced with recent changes. Trying again, directly on the production server (station1001), with wikidata-'''20260330'''-all-BETA.ttl.gz, only three days old as of writing.

* 2026-04-02: Direct rsync from WMF dump server to orb-wdqs
* 2026-04-02: Begin munge process
* 2026-04-04 15:23 (UTC): Begin loading of '''4971''' bundles
* 2026-04-09 22:28 (UTC): Completed 1657 (one-third of) files
* 2026-04-20 03:27 (UTC): Completed 3314 (two-thirds of) files
* Canceled at 5:52 PM local time on 2026-04-21 since it was not going to complete and more urgent hardware matters needed to be attended to

Project:Analytics/Orb

2026-04-09T22:46:06Z

Harej: One-third milestone

== 2025-10-25 Orb Blended ==
The virtual machine "Orb Blended" in my workstation cluster was created to experiment with blending the Wikidata Query Service with other datasets in a single Blazegraph instance. Work started earlier, but this is the first journal.

For the last few weeks it has been building the Wikidata Query Service. I have paused it at file 4470 since I am taking the system down. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 18:25, 25 October 2025 (UTC)

* Resuming build. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 03:49, 28 October 2025 (UTC)
* 2025-11-08: Rebuild took too long; have to start again. Downloading dump from November 3. If I can't complete the build by December 3, I am not sure Blazegraph is still viable.
* 2025-12-06: Rebuild of dataset is complete; local up-to-date copy of WDQS on dev1001.
* Started to gzip data.jnl in preparation to upload to fileserver and as a backup before experimenting with adding more data sources. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 02:00, 7 December 2025 (UTC)

== 2026-01-22 Rebuild ==
Updater is running into numerous issues on both my workstation and the server, so I am downloading wikidata-20260112-all-BETA.ttl.gz to my workstation for a rebuild. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 22:33, 22 January 2026 (UTC)

Munge process produced 4940 bundles; now ingesting into Blazegraph. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 23:25, 24 January 2026 (UTC)

== 2026-04 Rebuild ==
The last rebuild attempt, on my workstation, failed because it took too long to build the database. By the time it was finished, the data was more than 30 days old and could not be synced with recent changes. Trying again, directly on the production server (station1001), with wikidata-'''20260330'''-all-BETA.ttl.gz, only three days old as of writing.

* 2026-04-02: Direct rsync from WMF dump server to orb-wdqs
* 2026-04-02: Begin munge process
* 2026-04-04 15:23 (UTC): Begin loading of '''4971''' bundles
* 2026-04-09 22:28 (UTC): Completed 1657 (one-third of) files

Project:Home

2026-04-08T16:20:54Z

Harej: Links to basic documentation pages

Welcome to '''Librarybase, the collaborative card catalog of everything'''.

Librarybase is a project to organize, refine, and distribute metadata, as a complement to other projects like [[Item:Q2|Wikidata]] or [[Item:Q253|OpenAlex]]. For more information about Librarybase:

* [[Project:Principles]]: The motivations behind the project
* [[Project:Scope]]: What is accepted for inclusion on this wiki
* [[Project:Consensus]]: How decisions are made here
{{cwbn}}

User:Harej

2026-04-08T16:11:52Z

Harej: IABot deployment status

Founder of Librarybase

Links:
* [https://meta.wikimedia.org/wiki/User:Harej Wikimedia user page]
* [https://scatter.red Scatter]
* [https://wikis.world/@hare @hare @wikis.world]
* [https://harej.co Personal website]

== Project ideas ==

* Mirror of WikiApiary, creating an item for each MediaWiki deployment and extension and service used by that deployment
* InternetArchiveBot deployment status
* An item for each Wikipedia article in each language, as part of Pageset
* Something with [https://obsoletemedia.org Museum of Obsolete Media]
* Historic patents
* Books written in the Odia language. Data pending import; [[Item:Q458|demonstration item]]
* Metadata items were created on works that may or may not be in the public domain, facilitating identification of these works. [[Item:Q509|Example item]]

== Classes and roles ==
Items are organized by class using [[Property:P19]], with [[Item:Q410]] as the root class item. See [https://query.librarybase.org/#prefix%20lb%3A%20%3Chttps%3A%2F%2Flibrarybase.org%2Fentity%2F%3E%0Aprefix%20lbt%3A%20%3Chttps%3A%2F%2Flibrarybase.org%2Fprop%2Fdirect%2F%3E%0Aselect%20%3Fitem%20%3FitemLabel%20where%20%7B%20%3Fitem%20lbt%3AP19%20lb%3AQ410%20.%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%2Cen%22.%20%7D%20%7D current list of class items on Librarybase Query Service]. This is meant to bring Librarybase closer in alignment with databases like Fatcat and OpenAlex.

This differs from Wikidata's "instance of" and "subclass of" properties in that items are assigned classes according to a controlled vocabulary. This can be used instead of or in conjunction with those properties. One major departure from Wikidata is the concept of the [[Item:Q413|single-edition work]], used to avoid creating separate work and edition items when there is no real reason to.

One subclass is "[[Item:Q422|role]]," which a member of a class can have. See [https://query.librarybase.org/#prefix%20lb%3A%20%3Chttps%3A%2F%2Flibrarybase.org%2Fentity%2F%3E%0Aprefix%20lbt%3A%20%3Chttps%3A%2F%2Flibrarybase.org%2Fprop%2Fdirect%2F%3E%0Aselect%20%3Fitem%20%3FitemLabel%20where%20%7B%20%3Fitem%20lbt%3AP19%20lb%3AQ422%20.%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%2Cen%22.%20%7D%20%7D current list of role-type items on Librarybase Query Service].

Project:Home

2026-04-08T03:27:27Z

Harej: Clearing projects, pending something else

Welcome to '''Librarybase, the collaborative card catalog of everything'''.

Librarybase is a project to organize, refine, and distribute metadata, as a complement to other projects like [[Item:Q2|Wikidata]] or [[Item:Q253|OpenAlex]]. The motivations behind the project are further described at [[Project:Principles]].

{{cwbn}}

User:Harej

2026-04-08T03:27:19Z

Harej: Moving from main page

Founder of Librarybase

Links:
* [https://meta.wikimedia.org/wiki/User:Harej Wikimedia user page]
* [https://scatter.red Scatter]
* [https://wikis.world/@hare @hare @wikis.world]
* [https://harej.co Personal website]

== Project ideas ==

* Fork of WikiApiary, creating an item for each MediaWiki deployment and extension and service used by that deployment
* An item for each Wikipedia article in each language, as part of Pageset
* Something with [https://obsoletemedia.org Museum of Obsolete Media]
* Historic patents
* Books written in the Odia language. Data pending import; [[Item:Q458|demonstration item]]
* Metadata items were created on works that may or may not be in the public domain, facilitating identification of these works. [[Item:Q509|Example item]]

== Classes and roles ==
Items are organized by class using [[Property:P19]], with [[Item:Q410]] as the root class item. See [https://query.librarybase.org/#prefix%20lb%3A%20%3Chttps%3A%2F%2Flibrarybase.org%2Fentity%2F%3E%0Aprefix%20lbt%3A%20%3Chttps%3A%2F%2Flibrarybase.org%2Fprop%2Fdirect%2F%3E%0Aselect%20%3Fitem%20%3FitemLabel%20where%20%7B%20%3Fitem%20lbt%3AP19%20lb%3AQ410%20.%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%2Cen%22.%20%7D%20%7D current list of class items on Librarybase Query Service]. This is meant to bring Librarybase closer in alignment with databases like Fatcat and OpenAlex.

This differs from Wikidata's "instance of" and "subclass of" properties in that items are assigned classes according to a controlled vocabulary. This can be used instead of or in conjunction with those properties. One major departure from Wikidata is the concept of the [[Item:Q413|single-edition work]], used to avoid creating separate work and edition items when there is no real reason to.

One subclass is "[[Item:Q422|role]]," which a member of a class can have. See [https://query.librarybase.org/#prefix%20lb%3A%20%3Chttps%3A%2F%2Flibrarybase.org%2Fentity%2F%3E%0Aprefix%20lbt%3A%20%3Chttps%3A%2F%2Flibrarybase.org%2Fprop%2Fdirect%2F%3E%0Aselect%20%3Fitem%20%3FitemLabel%20where%20%7B%20%3Fitem%20lbt%3AP19%20lb%3AQ422%20.%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22%5BAUTO_LANGUAGE%5D%2Cen%22.%20%7D%20%7D current list of role-type items on Librarybase Query Service].

Project:Analytics/Orb

2026-04-04T15:23:08Z

Harej:

== 2025-10-25 Orb Blended ==
The virtual machine "Orb Blended" in my workstation cluster was created to experiment with blending the Wikidata Query Service with other datasets in a single Blazegraph instance. Work started earlier, but this is the first journal.

For the last few weeks it has been building the Wikidata Query Service. I have paused it at file 4470 since I am taking the system down. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 18:25, 25 October 2025 (UTC)

* Resuming build. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 03:49, 28 October 2025 (UTC)
* 2025-11-08: Rebuild took too long; have to start again. Downloading dump from November 3. If I can't complete the build by December 3, I am not sure Blazegraph is still viable.
* 2025-12-06: Rebuild of dataset is complete; local up-to-date copy of WDQS on dev1001.
* Started to gzip data.jnl in preparation to upload to fileserver and as a backup before experimenting with adding more data sources. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 02:00, 7 December 2025 (UTC)

== 2026-01-22 Rebuild ==
Updater is running into numerous issues on both my workstation and the server, so I am downloading wikidata-20260112-all-BETA.ttl.gz to my workstation for a rebuild. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 22:33, 22 January 2026 (UTC)

Munge process produced 4940 bundles; now ingesting into Blazegraph. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 23:25, 24 January 2026 (UTC)

== 2026-04 Rebuild ==
The last rebuild attempt, on my workstation, failed because it took too long to build the database. By the time it was finished, the data was more than 30 days old and could not be synced with recent changes. Trying again, directly on the production server (station1001), with wikidata-'''20260330'''-all-BETA.ttl.gz, only three days old as of writing.

* 2026-04-02: Direct rsync from WMF dump server to orb-wdqs
* 2026-04-02: Begin munge process
* 15:23, 4 April 2026 (UTC): Begin loading of '''4971''' bundles

Project:Analytics/Orb

2026-04-03T15:25:21Z

Harej: Munge

== 2025-10-25 Orb Blended ==
The virtual machine "Orb Blended" in my workstation cluster was created to experiment with blending the Wikidata Query Service with other datasets in a single Blazegraph instance. Work started earlier, but this is the first journal.

For the last few weeks it has been building the Wikidata Query Service. I have paused it at file 4470 since I am taking the system down. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 18:25, 25 October 2025 (UTC)

* Resuming build. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 03:49, 28 October 2025 (UTC)
* 2025-11-08: Rebuild took too long; have to start again. Downloading dump from November 3. If I can't complete the build by December 3, I am not sure Blazegraph is still viable.
* 2025-12-06: Rebuild of dataset is complete; local up-to-date copy of WDQS on dev1001.
* Started to gzip data.jnl in preparation to upload to fileserver and as a backup before experimenting with adding more data sources. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 02:00, 7 December 2025 (UTC)

== 2026-01-22 Rebuild ==
Updater is running into numerous issues on both my workstation and the server, so I am downloading wikidata-20260112-all-BETA.ttl.gz to my workstation for a rebuild. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 22:33, 22 January 2026 (UTC)

Munge process produced 4940 bundles; now ingesting into Blazegraph. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 23:25, 24 January 2026 (UTC)

== 2026-04 Rebuild ==
The last rebuild attempt, on my workstation, failed because it took too long to build the database. By the time it was finished, the data was more than 30 days old and could not be synced with recent changes. Trying again, directly on the production server (station1001), with wikidata-'''20260330'''-all-BETA.ttl.gz, only three days old as of writing.

* 2026-04-02: Direct rsync from WMF dump server to orb-wdqs
* 2026-04-02: Begin munge process

Project:Analytics/Orb

2026-04-03T01:49:08Z

Harej: 2026-04 rebuild

== 2025-10-25 Orb Blended ==
The virtual machine "Orb Blended" in my workstation cluster was created to experiment with blending the Wikidata Query Service with other datasets in a single Blazegraph instance. Work started earlier, but this is the first journal.

For the last few weeks it has been building the Wikidata Query Service. I have paused it at file 4470 since I am taking the system down. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 18:25, 25 October 2025 (UTC)

* Resuming build. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 03:49, 28 October 2025 (UTC)
* 2025-11-08: Rebuild took too long; have to start again. Downloading dump from November 3. If I can't complete the build by December 3, I am not sure Blazegraph is still viable.
* 2025-12-06: Rebuild of dataset is complete; local up-to-date copy of WDQS on dev1001.
* Started to gzip data.jnl in preparation to upload to fileserver and as a backup before experimenting with adding more data sources. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 02:00, 7 December 2025 (UTC)

== 2026-01-22 Rebuild ==
Updater is running into numerous issues on both my workstation and the server, so I am downloading wikidata-20260112-all-BETA.ttl.gz to my workstation for a rebuild. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 22:33, 22 January 2026 (UTC)

Munge process produced 4940 bundles; now ingesting into Blazegraph. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 23:25, 24 January 2026 (UTC)

== 2026-04 Rebuild ==
The last rebuild attempt, on my workstation, failed because it took too long to build the database. By the time it was finished, the data was more than 30 days old and could not be synced with recent changes. Trying again, directly on the production server (station1001), with wikidata-'''20260330'''-all-BETA.ttl.gz, only three days old as of writing.

* 2026-04-02: Direct rsync from WMF dump server to orb-wdqs

Project:Reviews/Wikipedia citation extraction

2026-03-13T00:38:49Z

Harej: Start

== OpenAlex search: wikipedia citations ==

* Rahimi and Danesh 2023: [https://openalex.org/works/w4384659478 Analyzing Persian Wikipedia's citations to discover the effectiveness of Persian scientific papers: applied web mining techniques]
* Kokash, Natallia, Colavizza, Giovanni 2023: [https://openalex.org/works/w4393797324 A Comprehensive Dataset of Citations with Identifiers from English Wikipedia (2023)]
** [https://openalex.org/works/w4394028614 Same thing?]
* Baigutanova, Myung, Saez-Trumper, Chou, et al. 2023: [https://openalex.org/works/w4323925750 Longitudinal Assessment of Reference Quality on Wikipedia]
* Xiang and Chaoqun 2023: [https://openalex.org/works/w4321055481 Gender and Country Biases in Wikipedia Citations to Scholarly Publications]
** [https://openalex.org/works/w4321055481 Same thing?]
* Wight 2023: [https://openalex.org/works/w4394113280 Reference and map usage across Wikimedia wiki pages]
* Dunne, Halpin, Humphreys, McCormack et al. 2023: [https://openalex.org/works/w4367667587 Academia by Speculation: Debunking the Flawed Science behind the Claim that Wikipedia Influences Judges]
* Thompson, Luo, McKenzie, Richardson et al. 2023: [https://openalex.org/works/w4318612963 User-Generated Content Shapes Judicial Reasoning: Evidence From a Randomized Control Trial on Wikipedia]
* Zheng, Chen, Yan, Ni 2022: [https://openalex.org/works/w4308500239 Gender and country biases in Wikipedia citations to scholarly publications]
* Przybyła, Borkowski, Kaczyński 2022: [https://openalex.org/works/w4393821726 Wikipedia Complete Citation Corpus]
* Yang and Colavizza 2022: [https://openalex.org/works/w3210253694 A Map of Science in Wikipedia]
** [https://openalex.org/works/w4293024133 Same thing?]
* Benjakob, Aviram, Sobel 2023: [https://openalex.org/works/w4226283917 Citation needed? Wikipedia bibliometrics during the first wave of the COVID-19 pandemic]
* Czygan, Holzmann, Newbold 2021: [https://openalex.org/works/w3206003844 Refcat: The Internet Archive Scholar Citation Graph]
* Zhao and Strotmann 2021: [https://openalex.org/works/w3158340182 Mapping knowledge domains on Wikipedia: an author bibliographic coupling analysis of traditional Chinese medicine]
* Leva 2021: [https://openalex.org/works/w4393513762 English Wikipedia citations with possible SemanticScholar URLs as found by Unpaywall and OAbot]
** [https://openalex.org/works/w4393425517 Same thing?]
* Banasik-Jemielniak, Jemielniak, Wilamowski 2021: [https://openalex.org/works/w3130944461 Psychology and Wikipedia: Measuring Psychology Journals’ Impact by Wikipedia Citations]
* Sobel, Benjakob, Aviram 2021: [https://openalex.org/works/w3209525497 A meta analysis of Wikipedia's coronavirus sources during the COVID-19 pandemic]
** [https://openalex.org/works/w4393415590 Same thing?]
* '''Singh, West, Colavizza 2021: [https://openalex.org/works/w3113998794 Wikipedia citations: A comprehensive data set of citations with identifiers extracted from English Wikipedia]'''
* Piccardi, West, Redi, Colavizza 2020: [https://openalex.org/works/w3111097238 Quantifying Engagement with Citations on Wikipedia. (Part 2) (The translation and original text of the article are presented)]
* Piccardi, West, Redi, Colavizza 2020: [https://openalex.org/works/w3096844803 Quantifying Engagement with Citations on Wikipedia. (Part 1)]
* Colavizza 2020: [https://openalex.org/works/w3044875688 COVID-19 research in Wikipedia]
* '''Singh, West, Colavizza 2020: [https://openalex.org/works/w4393774949 Wikipedia Citations: A comprehensive dataset of citations with identifiers extracted from English Wikipedia]'''
* Agarwal, Redi, Sastry, Wood et al. 2020: [https://openalex.org/works/w3037750177 Wikipedia and Westminster: Quality and Dynamics of Wikipedia Pages about UK Politicians]
* Leva 2020: [https://openalex.org/works/w4393800038 Yearly pageviews of English Wikipedia articles with potential links to green open access scholarly articles]
* Piccardi, Redi, Colavizza, West 2020: [https://openalex.org/works/w3002073420 Quantifying Engagement with Citations on Wikipedia]
* Nicholson, Uppala, Sieber, Grabitz et al. 2020: [https://openalex.org/works/w3015681655 Measuring the quality of scientific references in Wikipedia: an analysis of more than 115M citations to over 800,000 scientific articles]
* Arroyo-Machado, Torres-Salinas, Herrera-Viedma, Romero-Frías 2020: [https://openalex.org/works/w4287868685 Science through Wikipedia: A novel representation of open knowledge through co-citation networks]
* '''Singh, West, Colavizza 2020: [https://openalex.org/works/w4394037276 A Comprehensive Dataset of Citations with Identifiers from English Wikipedia]'''
* Piccardi, Redi, Colavizza, West 2020: [https://openalex.org/works/w3021826561 Quantifying Engagement with Citations on Wikipedia]
* Joorabchi, Doherty, Dawson 2019: [https://openalex.org/works/w2998658522 ‘WP2Cochrane’, a tool linking Wikipedia to the Cochrane Library: Results of a bibliometric analysis evaluating article quality and importance]
* Redi, Fetahu, Morgan, Taraborelli 2019: [https://openalex.org/works/w2911641747 Citation Needed: A Taxonomy and Algorithmic Assessment of Wikipedia's Verifiability]
* Nazarian and Bassiliades 2018: [https://openalex.org/works/w2787826775 Towards Linking DBpedia’s Bibliographic References to Bibliographic Repositories]
* Lewoniewski, Härting, Węcel, Reichstein et al. 2018: [https://openalex.org/works/w2889055695 Application of SEO Metrics to Determine the Quality of Wikipedia Articles and Their Sources]
* Redi and Taraborelli 2018: [https://openalex.org/works/w4394434007 Accessibility and topics of citations with identifiers in Wikipedia]
** [https://openalex.org/works/w4394371398 Same thing?]
* Maggio, Willinsky, Steinberg, Mietchen et al.: [https://openalex.org/works/w2952239097 Wikipedia as a gateway to biomedical research: The relative distribution and use of citations in the English Wikipedia]
** [https://openalex.org/works/w2738894751 Same thing?]
* Leva 2017: [https://openalex.org/works/w3210630434 DOIs linked by the English Wikipedia which could be made available in green Open Access]
* Pooladian and Borrego 2017: [https://openalex.org/works/w2741737307 Methodological issues in measuring citations in Wikipedia: a case study in Library and Information Science]
* Serrano-López, Ingwersen, Sanz-Casado 2017: [https://openalex.org/works/w2724499206 Wind power research in Wikipedia: Does Wikipedia demonstrate direct influence of research publications and can it be used as adequate source in research evaluation?]
* Kousha and Thelwall 2017: [https://openalex.org/works/w2530523999 News stories as evidence for research? BBC citations from articles, Books, and Wikipedia]
* Kousha and Thelwall 2017: [https://openalex.org/works/w3035760182 Are wikipedia citations important evidence of the impact of scholarly articles and books]
* Halfaker, Kim, Forte, Taraborelli 2017: [https://openalex.org/works/w2772040480 Citations with contexts in Wikipedia]
** [https://openalex.org/works/w4394385802 Same thing?]
* Halfaker and Taraborelli 2017: [https://openalex.org/works/w4394533018 Scholarly article citations in Wikipedia]
* Deng and Liu 2017: [https://openalex.org/works/w2757374563 Evaluating the Citations of Information Systems Journals in Wikipedia]
*

Project:Reviews

2026-03-12T23:16:27Z

Harej: The first literature review on Librarybase

'''Literature reviews!''' Use Librarybase to catalog and synthesize works. Build metadata records while doing so.

By default, any given literature review is '''open for all''' to participate. If, for some reason, you do not want this, please clearly state so at the beginning of the page.

== Ongoing reviews ==

* [[Project:Reviews/Wikipedia citation extraction]]

Project:Analytics/Orb

2026-01-24T23:25:51Z

Harej: Now ingesting

Project:Analytics/Orb

2026-01-22T22:33:09Z

Harej: Rebuild

Project:Analytics/WCD

2026-01-22T03:50:11Z

Harej: Wikipedia Citations Now; longer term challenges

== Introduction ==
'''Wikipedia Citations Database''' is an effort supported by Internet Archive to build a comprehensive, historical database of each citation to appear on Wikipedia.

While methodologies of similar projects have focused on extracting standardized identifiers or easy-to-extract references, this project instead seeks to build a ''complete'' database, based on analyzing the structure.

== Generations ==

* WCD Generation 1: Wikibase instance on Wikibase Cloud. Importing data was painfully slow, and it was decided Wikibase was not a good format for the data we were trying to store. The dataset produced is lost and was not very interesting or useful.
* WCD Generation 2: [https://github.com/internetarchive/iari IARI]; Postgres-based, but was extremely difficult to work with. Parts of it are used to support the Internet Archive Reference Explorer.
* WCD Generation 3: The latest attempt, using [https://github.com/internetarchive/wiki-references-extractor wiki-references-extractor] and [https://github.com/internetarchive/wiki-references-db wiki-references-db] (WRDB). WRDB is a core component of the broader WCD project.
** Version 1: https://wikipediacitations.scatter.red
*** Partial build of English Wikipedia
*** Has issues dealing with broken wikitext
** Version 2:
*** Rather than build an entire database upfront, have it be possible to analyze a '''page URL''' at a '''point in time'''. This will allow us to prove the extraction mechanisms are working without resorting to building an entire database first
**** Available as [https://wikipediacitations-now.scatter.red Wikipedia Citations Now]
*** Be able to tell what part of the article a reference is from (in-line vs. endnote, etc.)
*** JSON representation of template parameters so you don't have to parse them out of the template
*** Use LLM to parse the article in general. Maybe have an option to select between classical parsing and LLM-based extraction.
**** Use basic mwparserfromhell extraction as first step. If a user comes across an entry and it looks broken, offer option to re-generate report with AI. User can then accept alternative.
*** In data model, associate references with revision id; then, associate revision IDs with timestamps in another table

== Longer term challenges ==

* Sometimes, Wikipedia includes statements from Wikidata, and those statements from Wikidata have citations, but they would not show up in revision text.
** Would this require cross-referencing with rendered HTML output?
** Or a "citations database plus" that includes the Wikidata item? (Wikipedia editors would probably like this for quickly comparing citations between Wikipedia article and Wikidata item.)

== 2025-10-16 WRDB migration ==
WRDB is the Postgres database of citation strings at the core of the WCD effort.

Migration away from station1001 is underway. This started sooner but I am documenting my progress as of today.

Successfully transferred between station1001 and the new VM, wrdb-gen3v1, and just now, verified via md5sum.
----
jh@wrdb-gen3v1:/bulk/downloads$ md5sum 2024-10-wrdb.sql.gz
'''1819'''ec7f76887e854903550ce1d'''30ac6''' 2024-10-wrdb.sql.gz

jh@station1001:/opt/librarybase/wikibase$ md5sum ~/2024-10-wrdb.sql.gz
'''1819'''ec7f76887e854903550ce1d'''30ac6''' /home/jh/2024-10-wrdb.sql.gz
----Downloading the English Wikipedia dump to my home workstation is going painfully slowly. I do not know why the download is only ~1.5MB/s. If I want to do a rebuild, my Plan B may be to instead set up a wrdb-gen3v2 on station1001 (rebuilt with Proxmox) and dedicate it to the rebuild. If I have a pre-process step that turns the dumps into a neat bundle of diffs, it should not require as much RAM to process (since I will no longer have to deal with XML in memory). I still want to do the WRDB rebuild on my workstation, but pre-processing on station1001 may give me a smaller file I can download. Once the pre-process is done I should have enough resources for a secondary copy of WDQS.

In the meantime, now that I have a database dump of WRDB on the wrdb-gen3v1 virtual machine, the next steps are:

# Re-import into Postgres
# Re-start web service
# Change proxy to point to new web service
# Create process for on-demand update:
## User requests data for a certain article. There is a notice saying to check back later if the data is not up to date.
## Article is added to the update queue
## Process works through the update queue, getting all revisions since the most recent one in the database
## Each revision has references extracted and the database is updated
## To prevent excessive updates, if an article is already in the queue, it can't be re-added to the queue. And if an article was refreshed in the last hour, it is sent to the back of the queue.
## Eventually add a low-priority queue that cycles through English Wikipedia in alphabetical order so that there is always passive updating.

On-demand updates for particular articles should tide me over until I have the opportunity to do a full rebuild.

'''Note:''' I can't take down the current instance on station1001 until the new one is set up. Once the new one is set up, that's the final service before I can begin setting up Proxmox on station1001.

== 2025-10-25 Downloading dump for Gen3V2 ==
Downloading paused for re-networking. To resume:
rsync -avP -e "ssh -J jh@154.29.79.171" jh@10.0.1.8:/bulk/public/wikimedia/enwiki/20250901/enwiki-20250901-pages-meta-history*.xml-p*.bz2 ./

Downloading resumed. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 03:47, 28 October 2025 (UTC)

Download came to a stop on the orb-blended VM I used because it filled up. So I created a new VM, wrdb-gen3v2, moved the files to that, and resumed the download there. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 00:16, 3 November 2025 (UTC)

Project:Analytics/WCD

2026-01-22T03:27:23Z

Harej: Wikipedia Citations Now

== Introduction ==
'''Wikipedia Citations Database''' is an effort supported by Internet Archive to build a comprehensive, historical database of each citation to appear on Wikipedia.

While methodologies of similar projects have focused on extracting standardized identifiers or easy-to-extract references, this project instead seeks to build a ''complete'' database, based on analyzing the structure.

== Generations ==

* WCD Generation 1: Wikibase instance on Wikibase Cloud. Importing data was painfully slow, and it was decided Wikibase was not a good format for the data we were trying to store. The dataset produced is lost and was not very interesting or useful.
* WCD Generation 2: [https://github.com/internetarchive/iari IARI]; Postgres-based, but was extremely difficult to work with. Parts of it are used to support the Internet Archive Reference Explorer.
* WCD Generation 3: The latest attempt, using [https://github.com/internetarchive/wiki-references-extractor wiki-references-extractor] and [https://github.com/internetarchive/wiki-references-db wiki-references-db] (WRDB). WRDB is a core component of the broader WCD project.
** Version 1: https://wikipediacitations.scatter.red
*** Partial build of English Wikipedia
*** Has issues dealing with broken wikitext
** Version 2:
*** Rather than build an entire database upfront, have it be possible to analyze a '''page URL''' at a '''point in time'''. This will allow us to prove the extraction mechanisms are working without resorting to building an entire database first
**** Available as [https://wikipediacitations-now.scatter.red Wikipedia Citations Now]
*** Be able to tell what part of the article a reference is from (in-line vs. endnote, etc.)
*** JSON representation of template parameters so you don't have to parse them out of the template
*** Use LLM to parse the article in general. Maybe have an option to select between classical parsing and LLM-based extraction.
**** Use basic mwparserfromhell extraction as first step. If a user comes across an entry and it looks broken, offer option to re-generate report with AI. User can then accept alternative.
*** In data model, associate references with revision id; then, associate revision IDs with timestamps in another table

== 2025-10-16 WRDB migration ==
WRDB is the Postgres database of citation strings at the core of the WCD effort.

Migration away from station1001 is underway. This started sooner but I am documenting my progress as of today.

Successfully transferred between station1001 and the new VM, wrdb-gen3v1, and just now, verified via md5sum.
----
jh@wrdb-gen3v1:/bulk/downloads$ md5sum 2024-10-wrdb.sql.gz
'''1819'''ec7f76887e854903550ce1d'''30ac6''' 2024-10-wrdb.sql.gz

jh@station1001:/opt/librarybase/wikibase$ md5sum ~/2024-10-wrdb.sql.gz
'''1819'''ec7f76887e854903550ce1d'''30ac6''' /home/jh/2024-10-wrdb.sql.gz
----Downloading the English Wikipedia dump to my home workstation is going painfully slowly. I do not know why the download is only ~1.5MB/s. If I want to do a rebuild, my Plan B may be to instead set up a wrdb-gen3v2 on station1001 (rebuilt with Proxmox) and dedicate it to the rebuild. If I have a pre-process step that turns the dumps into a neat bundle of diffs, it should not require as much RAM to process (since I will no longer have to deal with XML in memory). I still want to do the WRDB rebuild on my workstation, but pre-processing on station1001 may give me a smaller file I can download. Once the pre-process is done I should have enough resources for a secondary copy of WDQS.

In the meantime, now that I have a database dump of WRDB on the wrdb-gen3v1 virtual machine, the next steps are:

# Re-import into Postgres
# Re-start web service
# Change proxy to point to new web service
# Create process for on-demand update:
## User requests data for a certain article. There is a notice saying to check back later if the data is not up to date.
## Article is added to the update queue
## Process works through the update queue, getting all revisions since the most recent one in the database
## Each revision has references extracted and the database is updated
## To prevent excessive updates, if an article is already in the queue, it can't be re-added to the queue. And if an article was refreshed in the last hour, it is sent to the back of the queue.
## Eventually add a low-priority queue that cycles through English Wikipedia in alphabetical order so that there is always passive updating.

On-demand updates for particular articles should tide me over until I have the opportunity to do a full rebuild.

'''Note:''' I can't take down the current instance on station1001 until the new one is set up. Once the new one is set up, that's the final service before I can begin setting up Proxmox on station1001.

== 2025-10-25 Downloading dump for Gen3V2 ==
Downloading paused for re-networking. To resume:
rsync -avP -e "ssh -J jh@154.29.79.171" jh@10.0.1.8:/bulk/public/wikimedia/enwiki/20250901/enwiki-20250901-pages-meta-history*.xml-p*.bz2 ./

Downloading resumed. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 03:47, 28 October 2025 (UTC)

Download came to a stop on the orb-blended VM I used because it filled up. So I created a new VM, wrdb-gen3v2, moved the files to that, and resumed the download there. [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 00:16, 3 November 2025 (UTC)

Project:Analytics/PubPeer

2026-01-19T21:09:42Z

Harej: Observation

==API==
https://dashboards.pubpeer.com/docs/api#/operations/partner

Relevant parameters:
* <code>page</code>: start with <code>1</code> then iterate based on whether there are more results
* <code>per_page</code>: set at maximum value <code>300</code>
* <code>sort</code>:
* <s><code>published_at</code></s>: concerns when the document was ''published''; I only care about ''comments''

==Resources==
* Wikimedia Cloud Services
** Toolforge: project "pubpeer"
*** Don't think it's being used for anything
** Cloud VPS: project "wikicite"
*** VM wikicite-refsdb-proc-1.wikicite.eqiad1.wikimedia.cloud
*** Trove DB: ouqdvgrbzf3.svc.trove.eqiad1.wikimedia.cloud

==Process==
=== Data collection and indexing ===
; PubPeer Data (index_pubpeer.py)
* '''Initial Seed:'''
** Starts from 2000-01-01
** Uses an initial large date window (2000-01-01 to 2014-12-31) followed by 30-day increments
** API query handles pagination (per_page: 300) and automatically reduces the time window if the result set hits the 10,000-record limit to ensure no data is missed
* '''Subsequent Builds:'''
** Triggered via <code>python index_pubpeer.py --update</code>
** Identifies the latest_comment_date from the local database and starts fetching from that date to the present
* '''Database Updates:'''
** Updates the pubpeer_articles table
** Fields: id_pubpeer (URL), id_doi, id_pubmed, id_arxiv, title (truncated to 250 chars), and time_last_comment
; Wikipedia Citations (index_citations.py)
* '''Process:'''
** Pulls current external links from Wikimedia Cloud DB Replicas for DOI (org.doi.), PubMed (gov.nih.nlm.ncbi.pubmed.), and arXiv (org.arxiv.)
** Restricted to Main (0) and Draft (118) namespaces
** Matches these links against the local pubpeer_articles table
* '''Database Updates:'''
** '''wikipedia table:''' Stores language_code, mw_page_id, mw_page_title, and mw_talk_page_id. Page titles are refreshed during each run
** '''citations table:''' Maps id_pubpeer to id_wiki_page
** '''Stale Data:''' Automatically removes citations from the local database if the link has been removed from Wikipedia

=== Reporting and wiki updates ===
* '''Wiki Maintenance:'''
** Automatically handles page moves and deletions via sync_wikipedia_titles.py before updating reports
* '''Report Generation (report.py):'''
** '''Alerts Report:''' Lists new citations (time_last_updated_table IS NULL) or existing citations with new comments (time_last_comment > time_last_updated_table)
** '''Most Affected Report:''' Lists Wikipedia articles with the highest number of unique PubPeer-commented citations
** '''Article List Reports:''' Large table and alphabetical subpages (/By article/A, etc.) listing all matched citations
** '''Frequency Report:''' Aggregates by PubPeer article to show which research is most cited across Wikipedia
* '''User Interactions (Dismissals):''' (pending working implementation)
** The bot reads the current wiki report and compares it to its previous version
** If an editor removes a row from a wiki table, the bot marks that citation as dismissed = TRUE in the database and stops reporting it

=== Database Schema (schema.sql) ===
* '''wikipedia:''' id, language_code, mw_page_id, mw_page_title, mw_talk_page_id, librarybase_id
* '''pubpeer_articles:''' id_pubpeer, id_doi, id_pubmed, id_arxiv, title, time_last_comment
* '''citations:'''
** Links articles to wiki pages
** cited_id_type: 1 (DOI), 2 (PubMed), 3 (arXiv)
*** 0 is reserved for "other"
** time_last_updated_table: Tracks when the wiki report last included this citation
** time_last_talk_page_post: (Reserved for future talk page notifications)
** dismissed: Boolean flag for editor-driven dismissals

=== Post-Implementation Status ===
* '''Talk Page Notifications:''' Code includes fields for tracking (time_last_talk_page_post), but the active workflow currently focuses on centralized reports (Wikipedia:PubPeer/*) rather than automated talk page posting
* '''Frequency of Runs:''' Designed to be run periodically (e.g., via cron) using the --update flag for index_pubpeer.py
* '''Dismissals Not Recognized:''' The logic to recognize user removals of report entries does not work yet and has been disabled.

== 2026-01-19 Observation ==
Example edit: https://en.wikipedia.org/w/index.php?title=Wikipedia:PubPeer/By_article/Z&diff=prev&oldid=1333802277

Sometimes, an article will have multiple PubPeer IDs, with different sets of comments on each. This causes the report generation process to be confused. Should probably canonize on a different identifier.

Project:Analytics/Wikibases

2026-01-18T23:24:09Z

Harej: What membership entails

The "Cool Wikibase Network" includes Wikibases that James Hare created. Wikibases that were created by other people may be admitted into the network in the future.

These wikis are:

* <s>Open Miscellaneous Wikibase</s> (January 2023–January 2026)
* [https://librarybase.org Librarybase] (May 2023–present)
* [https://internetdomains.wiki Internet Domains Wikibase] (May 2023–present)
* [https://wikibase.world Wikibase World] (June 2023–present)
* [https://communities.miraheze.org Miraheze Communities] (September 2025–present)
Membership in the Cool Wikibase Network mainly means being included in the [[Template:Cwbn]] footer featured on the member wikis, a callback to the days of web rings. It otherwise confers no material advantage or special privilege.

==2026-01-16 Migration away from Wikibase.Cloud==
Wikibase Cloud is proposing new hosting requirements that would subject tenants to evaluation against arbitrary, shifting standards. To reduce operational uncertainty, I am moving critical systems off Wikibase.Cloud in favor of self-hosting.
* 2026-01-16: I migrated the majority of Open Miscellaneous Wikibase to Librarybase and then deleted the old wiki.
** InternetArchiveBot-related template tracking ([[Item:Q246948|example item]])
** [[Item:Q246951|MIT License modeling experiment]]
* 2026-01-18:
** Deleted the "credbot-daily" job on "vaccinesafety" Toolforge project in anticipation of migration.
** Posted [https://domains.wikibase.cloud/wiki/MediaWiki:Sitenotice Sitenotice on Internet Domains]
** Registered internetdomains.wiki
** [https://phabricator.wikimedia.org/T414893 Requested dump] of Internet Domains

[[User:Harej|Harej]] ([[User talk:Harej|talk]]) 20:04, 18 January 2026 (UTC)

Project:Analytics/Wikibases

2026-01-18T20:12:45Z

Harej: Linking to my Wikibases

Project:Analytics/Wikibases

2026-01-18T20:04:03Z

Harej: Created page with "==2026-01-16 Migration away from Wikibase.Cloud== Wikibase Cloud is proposing new hosting requirements that would subject tenants to evaluation against arbitrary, shifting standards. To reduce operational uncertainty, I am moving critical systems off Wikibase.Cloud in favor of self-hosting. * On 2026-01-16 I migrated the majority of Open Miscellaneous Wikibase to Librarybase and then deleted the old wiki. ** InternetArchiveBot-related template tracking (Item:Q246948|ex..."

==2026-01-16 Migration away from Wikibase.Cloud==
Wikibase Cloud is proposing new hosting requirements that would subject tenants to evaluation against arbitrary, shifting standards. To reduce operational uncertainty, I am moving critical systems off Wikibase.Cloud in favor of self-hosting.
* On 2026-01-16 I migrated the majority of Open Miscellaneous Wikibase to Librarybase and then deleted the old wiki.
** InternetArchiveBot-related template tracking ([[Item:Q246948|example item]])
** [[Item:Q246951|MIT License modeling experiment]]
* 2026-01-18: deleted the "credbot-daily" job on "vaccinesafety" Toolforge project in anticipation of migration.
* Posted [https://domains.wikibase.cloud/wiki/MediaWiki:Sitenotice Sitenotice on Internet Domains]
* Registered internetdomains.wiki
* [https://phabricator.wikimedia.org/T414893 Requested dump] of Internet Domains

[[User:Harej|Harej]] ([[User talk:Harej|talk]]) 20:04, 18 January 2026 (UTC)

Template:Cwbn

2026-01-16T16:24:38Z

Harej: Removing Open Misc

<div style="margin-top:2em"><hr style="width:5em; margin-bottom:0.75em;" />'''''Cool Wikibase Network'''''[https://wikibase.world Wikibase World][https://librarybase.org Librarybase][https://domains.wikibase.cloud Internet Domains][https://communities.miraheze.org Miraheze Communities]</div>

Item:Q246951

2026-01-16T16:14:45Z

Harej: /* wbsetclaim-create:2||1 */ Property:P44: MIT

Item:Q246951

2026-01-16T16:14:19Z

Harej: /* wbsetclaim-create:2||1 */ Property:P42: Item:Q246968

Item:Q246951

2026-01-16T16:13:55Z

Harej: /* wbsetclaim-create:2||1 */ Property:P42: Item:Q246964

Item:Q246951

2026-01-16T16:13:54Z

Harej: /* wbsetclaim-create:2||1 */ Property:P42: Item:Q246966

Item:Q246951

2026-01-16T16:13:52Z

Harej: /* wbsetclaim-create:2||1 */ Property:P42: Item:Q246967

Item:Q246951

2026-01-16T16:13:50Z

Harej: /* wbsetclaim-create:2||1 */ Property:P42: Item:Q246965

Property:P44

2026-01-16T16:11:23Z

Harej: /* wbeditentity-create:2|en */ SPDX identifier, license identifier

Item:Q246951

2026-01-16T16:09:46Z

Harej: /* wbsetclaim-update:2||1|5 */ Property:P42: Item:Q246953

Item:Q246951

2026-01-16T16:08:04Z

Harej: /* wbsetclaim-update:2||1|2 */ Property:P42: Item:Q246953

Item:Q246951

2026-01-16T16:07:08Z

Harej: /* wbsetclaim-create:2||1 */ Property:P42: Item:Q246953

Item:Q246968

2026-01-16T16:05:35Z