Project:Analytics/WCD - Revision history

Harej: Another report

2026-06-27T23:49:18Z

Another report

← Older revision		Revision as of 23:49, 27 June 2026
Line 28:		Line 28:
	** [https://citations.scatteranalytics.com/explorer/citation/1ab8b8026639f6959f94736a4ec83e777e63ef14/other-articles?page_id=3410 other articles] and [https://citations.scatteranalytics.com/explorer/template/1374/report?parameter_key=doi&parameter_value=10.1016/0006-3207%2891%2990031-4%20 doi parameter value]		** [https://citations.scatteranalytics.com/explorer/citation/1ab8b8026639f6959f94736a4ec83e777e63ef14/other-articles?page_id=3410 other articles] and [https://citations.scatteranalytics.com/explorer/template/1374/report?parameter_key=doi&parameter_value=10.1016/0006-3207%2891%2990031-4%20 doi parameter value]
	** [https://citations.scatteranalytics.com/explorer/citation/da6f699ca0a7f4d229011919ae1bd0b3c2b6f5e9/other-articles?page_id=1553396 other articles] and [https://citations.scatteranalytics.com/explorer/template/6385/report?parameter_key=isbn&parameter_value=978-0-19-958740-7 isbn parameter value]		** [https://citations.scatteranalytics.com/explorer/citation/da6f699ca0a7f4d229011919ae1bd0b3c2b6f5e9/other-articles?page_id=1553396 other articles] and [https://citations.scatteranalytics.com/explorer/template/6385/report?parameter_key=isbn&parameter_value=978-0-19-958740-7 isbn parameter value]
			*Only one page shows up on report when two should
			**[https://citations.scatteranalytics.com/explorer/citation/2ca3ae9d1d69ef6ba9336ce4df1581057a34ddd9/other-articles?page_id=30495845 This citation] appears on ''[https://citations.scatteranalytics.com/explorer/article?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FBoops_boops&follow_redirects=1 Boops boops]'' and ''[https://citations.scatteranalytics.com/explorer/article?url=https%3A//en.wikipedia.org/w/index.php%3Fcurid%3D60478829 Microcotyle isyebi].'' [https://citations.scatteranalytics.com/explorer/template/1374/report?parameter_key=doi&parameter_value=10.1007/s00436-019-06293-y Template parameter report] only shows one article even though the link on both resolve to the same URL (so not the issue of spaces at the parameter boundary)

	== Longer term challenges ==		== Longer term challenges ==

Harej: +finding

2026-06-27T19:52:25Z

+finding

← Older revision		Revision as of 19:52, 27 June 2026
Line 21:		Line 21:
	* [https://citations.scatteranalytics.com/explorer/template/6385/report?parameter_key=isbn&parameter_value=978-0-226-77142-7 Same ISBN in "Cite book" template used across 22 articles with varying citations]		* [https://citations.scatteranalytics.com/explorer/template/6385/report?parameter_key=isbn&parameter_value=978-0-226-77142-7 Same ISBN in "Cite book" template used across 22 articles with varying citations]
	* [https://citations.scatteranalytics.com/explorer/citation/bdbb085b4fb9a609b67d8732f076eab3d8d6a536/other-articles?page_id=3410 Two citations to the same book, but without ISBN]		* [https://citations.scatteranalytics.com/explorer/citation/bdbb085b4fb9a609b67d8732f076eab3d8d6a536/other-articles?page_id=3410 Two citations to the same book, but without ISBN]
			* [https://citations.scatteranalytics.com/explorer/template/6385/report?parameter_key=isbn&parameter_value=%201-873410-89-1%0A%20 The same book cited on five articles with different citations]

	Potential bugs:		Potential bugs:

Harej: New WCD

2026-06-26T21:31:13Z

New WCD

← Older revision		Revision as of 21:31, 26 June 2026
Line 8:		Line 8:
	* WCD Generation 1: Wikibase instance on Wikibase Cloud. Importing data was painfully slow, and it was decided Wikibase was not a good format for the data we were trying to store. The dataset produced is lost and was not very interesting or useful.		* WCD Generation 1: Wikibase instance on Wikibase Cloud. Importing data was painfully slow, and it was decided Wikibase was not a good format for the data we were trying to store. The dataset produced is lost and was not very interesting or useful.
	* WCD Generation 2: [https://github.com/internetarchive/iari IARI]; Postgres-based, but was extremely difficult to work with. Parts of it are used to support the Internet Archive Reference Explorer.		* WCD Generation 2: [https://github.com/internetarchive/iari IARI]; Postgres-based, but was extremely difficult to work with. Parts of it are used to support the Internet Archive Reference Explorer.
	* WCD Generation 3: The latest attempt, using [https://github.com/internetarchive/wiki-references-extractor wiki-references-extractor] and [https://github.com/internetarchive/wiki-references-db wiki-references-db] (WRDB). WRDB is a core component of the broader WCD project.		* WCD Generation 3: The latest attempt, using [https://github.com/internetarchive/wiki-references-extractor wiki-references-extractor] and [https://github.com/internetarchive/wiki-references-db wiki-references-db] (WRDB). WRDB is a core component of the broader WCD project. Version 2 adds [https://github.com/internetarchive/RevisionChest RevisionChest] as a pre-processing step.
	** Version 1: https://wikipediacitations.scatter.red		** Version 1: <s><nowiki>https://wikipediacitations.scatter.red</nowiki></s>
	*** Partial build of English Wikipedia		*** Partial build of English Wikipedia
	*** Has issues dealing with broken wikitext		*** Has issues dealing with broken wikitext
	** Version 2:		** Version 2: '''https://citations.scatteranalytics.com'''
	*** Rather than build an entire database upfront, have it be possible to analyze a '''~~page URL~~''' at a ~~'''point in time'''~~. ~~This will allow us to prove the extraction mechanisms~~ are ~~working without resorting to building an entire database first~~		*WCD Generation 4 (pending): To be built off of a combination of Generation 3 and [https://github.com/scatter-llc/UVAWikipediaCitationsDatabase UVAWikipediaCitationsDatabase] which adds mapping between extracted citations and underlying sources
	**** Available as [https://~~wikipediacitations-now~~.~~scatter~~.~~red Wikipedia Citations Now~~]
	*** Be able to ~~tell what part of~~ the ~~article a reference is from (in-line vs. endnote~~, ~~etc.)~~		== Interesting findings ==
	*** JSON representation of template ~~parameters so you don't have to parse them out of~~ the template		All are English Wikipedia articles
	*** Use LLM to parse the article in general. ~~Maybe have an option to select between classical parsing~~ and ~~LLM~~-~~based extraction.~~
	**** Use basic mwparserfromhell extraction as first step. ~~If a user comes across an entry~~ and ~~it looks broken, offer option to re-generate report with AI~~. ~~User can then accept alternative~~.		* [https://citations.scatteranalytics.com/explorer/citation/51c35f5a082e88d867dfb0872e4ab6ac7311a9e9/other-articles?page_id=12292404 Non-templated citation used on three different articles]
	*** In data model, associate references with revision id; then, associate revision IDs with timestamps in another table		* [https://citations.scatteranalytics.com/explorer/template/6385/report?parameter_key=isbn&parameter_value=978-0-226-77142-7 Same ISBN in "Cite book" template used across 22 articles with varying citations]
			* [https://citations.scatteranalytics.com/explorer/citation/bdbb085b4fb9a609b67d8732f076eab3d8d6a536/other-articles?page_id=3410 Two citations to the same book, but without ISBN]

			Potential bugs:

			* Discrepancy between same-citation report and matching template parameter (if two citations are an exact match then they should appear in the same template parameter use report)
			** [https://citations.scatteranalytics.com/explorer/citation/1ab8b8026639f6959f94736a4ec83e777e63ef14/other-articles?page_id=3410 other articles] and [https://citations.scatteranalytics.com/explorer/template/1374/report?parameter_key=doi&parameter_value=10.1016/0006-3207%2891%2990031-4%20 doi parameter value]
			** [https://citations.scatteranalytics.com/explorer/citation/da6f699ca0a7f4d229011919ae1bd0b3c2b6f5e9/other-articles?page_id=1553396 other articles] and [https://citations.scatteranalytics.com/explorer/template/6385/report?parameter_key=isbn&parameter_value=978-0-19-958740-7 isbn parameter value]

	== Longer term challenges ==		== Longer term challenges ==

Harej: Removing notes from abandoned build attempts

2026-05-20T19:42:16Z

Removing notes from abandoned build attempts

← Older revision		Revision as of 19:42, 20 May 2026
Line 26:		Line 26:
	** Would this require cross-referencing with rendered HTML output?		** Would this require cross-referencing with rendered HTML output?
	** Or a "citations database plus" that includes the Wikidata item? (Wikipedia editors would probably like this for quickly comparing citations between Wikipedia article and Wikidata item.)		** Or a "citations database plus" that includes the Wikidata item? (Wikipedia editors would probably like this for quickly comparing citations between Wikipedia article and Wikidata item.)

	~~== 2025-10-16 WRDB migration ==~~
	~~WRDB is the Postgres database of citation strings at the core of the WCD effort.~~

	~~Migration away from station1001 is underway. This started sooner but I am documenting my progress as of today.~~

	~~Successfully transferred between station1001 and the new VM, wrdb-gen3v1, and just now, verified via md5sum.~~
	~~----~~
	~~jh@wrdb-gen3v1:/bulk/downloads$ md5sum 2024-10-wrdb.sql.gz~~
	~~'''1819'''ec7f76887e854903550ce1d'''30ac6''' 2024-10-wrdb.sql.gz~~

	~~jh@station1001:/opt/librarybase/wikibase$ md5sum ~/2024-10-wrdb.sql.gz~~
	~~'''1819'''ec7f76887e854903550ce1d'''30ac6''' /home/jh/2024-10-wrdb.sql.gz~~
	----Downloading the English Wikipedia dump to my home workstation is going painfully slowly. I do not know why the download is only ~1.5MB/s. If I want to do a rebuild, my Plan B may be to instead set up a wrdb-gen3v2 on station1001 (rebuilt with Proxmox) and dedicate it to the rebuild. If I have a pre-process step that turns the dumps into a neat bundle of diffs, it should not require as much RAM to process (since I will no longer have to deal with XML in memory). I still want to do the WRDB rebuild on my workstation, but pre-processing on station1001 may give me a smaller file I can download. Once the pre-process is done I should have enough resources for a secondary copy of WDQS.

	~~In the meantime, now that I have a database dump of WRDB on the wrdb-gen3v1 virtual machine, the next steps are:~~

	~~# Re-import into Postgres~~
	~~# Re-start web service~~
	~~# Change proxy to point to new web service~~
	~~# Create process for on-demand update:~~
	~~## User requests data for a certain article. There is a notice saying to check back later if the data is not up to date.~~
	~~## Article is added to the update queue~~
	~~## Process works through the update queue, getting all revisions since the most recent one in the database~~
	~~## Each revision has references extracted and the database is updated~~
	~~## To prevent excessive updates, if an article is already in the queue, it can't be re-added to the queue. And if an article was refreshed in the last hour, it is sent to the back of the queue.~~
	~~## Eventually add a low-priority queue that cycles through English Wikipedia in alphabetical order so that there is always passive updating.~~

	~~On-demand updates for particular articles should tide me over until I have the opportunity to do a full rebuild.~~

	'''Note:''' I can't take down the current instance on station1001 until the new one is set up. Once the new one is set up, that's the final service before I can begin setting up Proxmox on station1001.

	~~== 2025-10-25 Downloading dump for Gen3V2 ==~~
	~~Downloading paused for re-networking. To resume:~~
	~~rsync -avP -e "ssh -J jh@154.29.79.171" jh@10.0.1.8:/bulk/public/wikimedia/enwiki/20250901/enwiki-20250901-pages-meta-history.xml-p.bz2 ./~~

	~~Downloading resumed. [[User:Harej\|Harej]] ([[User talk:Harej\|talk]]) 03:47, 28 October 2025 (UTC)~~

	Download came to a stop on the orb-blended VM I used because it filled up. So I created a new VM, wrdb-gen3v2, moved the files to that, and resumed the download there. [[User:Harej\|Harej]] ([[User talk:Harej\|talk]]) 00:16, 3 November 2025 (UTC)

Harej: Wikipedia Citations Now; longer term challenges

2026-01-22T03:50:11Z

Wikipedia Citations Now; longer term challenges

← Older revision		Revision as of 03:50, 22 January 2026
Line 20:		Line 20:
	**** Use basic mwparserfromhell extraction as first step. If a user comes across an entry and it looks broken, offer option to re-generate report with AI. User can then accept alternative.		**** Use basic mwparserfromhell extraction as first step. If a user comes across an entry and it looks broken, offer option to re-generate report with AI. User can then accept alternative.
	*** In data model, associate references with revision id; then, associate revision IDs with timestamps in another table		*** In data model, associate references with revision id; then, associate revision IDs with timestamps in another table

			== Longer term challenges ==

			* Sometimes, Wikipedia includes statements from Wikidata, and those statements from Wikidata have citations, but they would not show up in revision text.
			** Would this require cross-referencing with rendered HTML output?
			** Or a "citations database plus" that includes the Wikidata item? (Wikipedia editors would probably like this for quickly comparing citations between Wikipedia article and Wikidata item.)

	== 2025-10-16 WRDB migration ==		== 2025-10-16 WRDB migration ==

Harej: Wikipedia Citations Now

2026-01-22T03:27:23Z

Wikipedia Citations Now

← Older revision		Revision as of 03:27, 22 January 2026
Line 14:		Line 14:
	** Version 2:		** Version 2:
	*** Rather than build an entire database upfront, have it be possible to analyze a '''page URL''' at a '''point in time'''. This will allow us to prove the extraction mechanisms are working without resorting to building an entire database first		*** Rather than build an entire database upfront, have it be possible to analyze a '''page URL''' at a '''point in time'''. This will allow us to prove the extraction mechanisms are working without resorting to building an entire database first
			**** Available as [https://wikipediacitations-now.scatter.red Wikipedia Citations Now]
	*** Be able to tell what part of the article a reference is from (in-line vs. endnote, etc.)		*** Be able to tell what part of the article a reference is from (in-line vs. endnote, etc.)
	*** JSON representation of template parameters so you don't have to parse them out of the template		*** JSON representation of template parameters so you don't have to parse them out of the template

Harej: Download done

2025-11-03T00:16:46Z

Download done

← Older revision		Revision as of 00:16, 3 November 2025
Line 56:		Line 56:

	Downloading resumed. [[User:Harej\|Harej]] ([[User talk:Harej\|talk]]) 03:47, 28 October 2025 (UTC)		Downloading resumed. [[User:Harej\|Harej]] ([[User talk:Harej\|talk]]) 03:47, 28 October 2025 (UTC)

			Download came to a stop on the orb-blended VM I used because it filled up. So I created a new VM, wrdb-gen3v2, moved the files to that, and resumed the download there. [[User:Harej\|Harej]] ([[User talk:Harej\|talk]]) 00:16, 3 November 2025 (UTC)

Harej: Downloading resumed

2025-10-28T03:47:23Z

Downloading resumed

← Older revision		Revision as of 03:47, 28 October 2025
Line 54:		Line 54:
	Downloading paused for re-networking. To resume:		Downloading paused for re-networking. To resume:
	rsync -avP -e "ssh -J jh@154.29.79.171" jh@10.0.1.8:/bulk/public/wikimedia/enwiki/20250901/enwiki-20250901-pages-meta-history.xml-p.bz2 ./		rsync -avP -e "ssh -J jh@154.29.79.171" jh@10.0.1.8:/bulk/public/wikimedia/enwiki/20250901/enwiki-20250901-pages-meta-history.xml-p.bz2 ./

			Downloading resumed. [[User:Harej\|Harej]] ([[User talk:Harej\|talk]]) 03:47, 28 October 2025 (UTC)

Harej: To resume download

2025-10-25T18:18:55Z

To resume download

← Older revision		Revision as of 18:18, 25 October 2025
Line 50:		Line 50:

	'''Note:''' I can't take down the current instance on station1001 until the new one is set up. Once the new one is set up, that's the final service before I can begin setting up Proxmox on station1001.		'''Note:''' I can't take down the current instance on station1001 until the new one is set up. Once the new one is set up, that's the final service before I can begin setting up Proxmox on station1001.

			== 2025-10-25 Downloading dump for Gen3V2 ==
			Downloading paused for re-networking. To resume:
			rsync -avP -e "ssh -J jh@154.29.79.171" jh@10.0.1.8:/bulk/public/wikimedia/enwiki/20250901/enwiki-20250901-pages-meta-history.xml-p.bz2 ./

Harej: /* 2025-10-16 WRDB migration */ Note

2025-10-17T03:43:05Z

2025-10-16 WRDB migration: Note

← Older revision		Revision as of 03:43, 17 October 2025
Line 48:		Line 48:

	On-demand updates for particular articles should tide me over until I have the opportunity to do a full rebuild.		On-demand updates for particular articles should tide me over until I have the opportunity to do a full rebuild.

			'''Note:''' I can't take down the current instance on station1001 until the new one is set up. Once the new one is set up, that's the final service before I can begin setting up Proxmox on station1001.