Project:Analytics/WCD

Introduction

Wikipedia Citations Database is an effort supported by Internet Archive to build a comprehensive, historical database of each citation to appear on Wikipedia.

While methodologies of similar projects have focused on extracting standardized identifiers or easy-to-extract references, this project instead seeks to build a complete database, based on analyzing the structure.

Generations

• WCD Generation 1: Wikibase instance on Wikibase Cloud. Importing data was painfully slow, and it was decided Wikibase was not a good format for the data we were trying to store. The dataset produced is lost and was not very interesting or useful.
• WCD Generation 2: IARI; Postgres-based, but was extremely difficult to work with. Parts of it are used to support the Internet Archive Reference Explorer.
• WCD Generation 3: The latest attempt, using wiki-references-extractor and wiki-references-db (WRDB). WRDB is a core component of the broader WCD project. Version 2 adds RevisionChest as a pre-processing step.
  • Version 1: https://wikipediacitations.scatter.red
    • Partial build of English Wikipedia
    • Has issues dealing with broken wikitext
  • Version 2: https://citations.scatteranalytics.com
• WCD Generation 4 (pending): To be built off of a combination of Generation 3 and UVAWikipediaCitationsDatabase which adds mapping between extracted citations and underlying sources

Interesting findings

All are English Wikipedia articles

• Non-templated citation used on three different articles
• Same ISBN in "Cite book" template used across 22 articles with varying citations
• Two citations to the same book, but without ISBN
• The same book cited on five articles with different citations

Potential bugs:

• Discrepancy between same-citation report and matching template parameter (if two citations are an exact match then they should appear in the same template parameter use report)
  • other articles and doi parameter value
  • other articles and isbn parameter value
• Only one page shows up on report when two should
  • This citation appears on Boops boops and Microcotyle isyebi. Template parameter report only shows one article even though the link on both resolve to the same URL (so not the issue of spaces at the parameter boundary)

Longer term challenges

• Sometimes, Wikipedia includes statements from Wikidata, and those statements from Wikidata have citations, but they would not show up in revision text.
  • Would this require cross-referencing with rendered HTML output?
  • Or a "citations database plus" that includes the Wikidata item? (Wikipedia editors would probably like this for quickly comparing citations between Wikipedia article and Wikidata item.)