Project:Analytics/WCD
Introduction
Wikipedia Citations Database is an effort supported by Internet Archive to build a comprehensive, historical database of each citation to appear on Wikipedia.
While methodologies of similar projects have focused on extracting standardized identifiers or easy-to-extract references, this project instead seeks to build a complete database, based on analyzing the structure.
Generations
- WCD Generation 1: Wikibase instance on Wikibase Cloud. Importing data was painfully slow, and it was decided Wikibase was not a good format for the data we were trying to store. The dataset produced is lost and was not very interesting or useful.
- WCD Generation 2: IARI; Postgres-based, but was extremely difficult to work with. Parts of it are used to support the Internet Archive Reference Explorer.
- WCD Generation 3: The latest attempt, using wiki-references-extractor and wiki-references-db (WRDB). WRDB is a core component of the broader WCD project. Version 2 adds RevisionChest as a pre-processing step.
- Version 1:
https://wikipediacitations.scatter.red- Partial build of English Wikipedia
- Has issues dealing with broken wikitext
- Version 2: https://citations.scatteranalytics.com
- Version 1:
- WCD Generation 4 (pending): To be built off of a combination of Generation 3 and UVAWikipediaCitationsDatabase which adds mapping between extracted citations and underlying sources
Interesting findings
All are English Wikipedia articles
- Non-templated citation used on three different articles
- Same ISBN in "Cite book" template used across 22 articles with varying citations
- Two citations to the same book, but without ISBN
Potential bugs:
- Discrepancy between same-citation report and matching template parameter (if two citations are an exact match then they should appear in the same template parameter use report)
Longer term challenges
- Sometimes, Wikipedia includes statements from Wikidata, and those statements from Wikidata have citations, but they would not show up in revision text.
- Would this require cross-referencing with rendered HTML output?
- Or a "citations database plus" that includes the Wikidata item? (Wikipedia editors would probably like this for quickly comparing citations between Wikipedia article and Wikidata item.)