Peer-Reviewed Research and Publications / Findings

We utilize examinations to evaluate the caliber of each student’s individual ability to perform paleographic interpretation. The paleography exams used five quantitatively-evaluated machine-graded instruments to gauge student proficiency in terms of:

• Ability to distinguish individual letters from one another,
• Ability to recognize individual letters within words,
• Ability to identify abbreviations, and
• Ability to transcribe small sections of texts.

Exams #1 and #2 were introductory thirteenth-century Spanish paleography, Exam #3 was intermediate fifteenth-century paleography, and Exams #4 and #5 were advanced fifteenth-century paleography. (The figure below presents sample manuscript images from the paleography exams.)

Our evaluation of the forty-eight citizen scholars’ performance indicates that on average students recognized 74.8% of medieval Spanish handwriting (thirteenth and fifteenth-century scripts) when tested using five machine-graded paleographic exams. At the introductory level, or thirteenth-century script, students scored very high on their exams (79.7% and 77.7%). At the intermediate level, which was a fairly readable fifteenth-century handwriting that students specifically studied in practice lessons, exam scores remained high (79.9%). However, when presented with an entirely novel fifteenth-century hand, what we considered advanced paleography, students average scores dropped by about 10 percentage points (67.3% and 69.4%). Given students only engaged in two weeks of paleographical training, we consider this level of comprehension and accuracy to be quite good for non-specialists.

MARTINEZ-DAVILA ET. AL. 2018, 21-26.

Forty-five (45) hours of introductory historical and paleographical training creates highly-accurate paleographers. Our evaluation of forty-eight randomly-selected citizen scholars’ transcription performance indicates that on average they identified 87.9% of all the letter-forms, numbers, and abbreviations from fifteenth-century manuscripts. Or put another way, our assessment of student transcriptions revealed a 12.1% average error rate.

To make this assessment, we performed an expert review of their individual transcriptions of four blocks of text from a cathedral manuscript. Our analysis of each of the individual student transcriptions of manuscript required a character-by-character appraisal of three sections of the student’s work. That is, we checked each student’s first line of transcription, last line of transcription, and a machine-selected random line of transcription. The types of errors we noted were incorrectly identified or missing letters, numbers, abbreviations, and spaces. Students were given the option of spelling out abbreviations or denoting them with an asterisk. To calculate an error rate, we first counted the number of characters (letters, numbers, and spaces) in each line and created a model transcription. Subsequently, we counted the number of student errors or deviations from the model transcription.

For example, the model first line transcription of Folio 136 verso without abbreviations was recorded as including eighty-seven characters. (See table below.)

However, if the student chose to use asterisks where abbreviations appeared in the manuscript, then the model first line with abbreviations was recorded as including seventy-six characters. As capitalization of letters in medieval manuscripts can be haphazard, students were not evaluated using this metric. To earn a 0.0% error rate a student would be required to transcribe the first line with no errors. Unsurprisingly, no student generated a perfect transcription. For example, Student #30’s transcription included four errors and therefore corresponded eighty-three correct characters out of eighty-seven model characters. The error rate is therefore calculated with the following formula: Error rate = | (Student Transcription – Model Transcription without Abbreviations) / Model Transcription without Abbreviations | = | (83 – 87) / 87 | = 4.6%.

MARTINEZ-DAVILA ET. AL. 2018, 26-28.

When queried about their perspective on the value of “crowdsourced research”, students in our Deciphering Secrets Massive Open Online Courses (MOOCs) and transcription project resoundingly supported it. Specifically, 82.5% of our students believe it is very important for academic researchers to incorporate the public into scholarly investigations. A full 100% believe these scholarly-public collaborations are very or somewhat important. When asked if they wanted to participate in other types of research:

• 85% indicated they wanted to contribute to additional paleography transcription projects,
• 65% stated they wanted to assist with editing and creating final versions of transcribed manuscripts, and
• 75% wanted to help index the contents of manuscripts that have not been abstracted.

After completing our course, 77.5% of our students stated they were very likely to return and take additional Deciphering Secrets MOOCs. Therefore, it appears that this student body is highly motivated and is an important, untapped cooperative resource for scholars.

MARTINEZ-DAVILA ET. AL. 2018, 6-7.

One of the principal questions that often arises when seeking to advance citizen science is: How can scholars capture the attention of the public and convince them to participate in research initiatives? In my experience, the answer is in seeding the deep intellectual and cultural curiosity of the university-educated public. Humans are naturally cooperative, collaborative, and curious. We care about the humanities – history, the arts, philosophy, and language – because they embody the noblest goals of our globalized culture and our yearning to serve the higher purpose of making positive democratic, egalitarian, and socially beneficial contributions to our world. Yet, as technological change and internet-connectivity have advanced, we recognize that the essence of our humanity is being swallowed whole as our lives become less focused on people and more on technological devices like the Apple iPhone. A short trip on the Metro Madrid, lunching in a cafeteria, or even sitting in our own flats, reveals the pervasiveness of this phenomenon where humans stare intently into little video screens and do not acknowledge or talk to each other. Who is serving whom in this world – do we serve technology, or does it serve us?

The key to drawing the digitally connected public to citizen science projects is a fundamental recognition that humanity is in desperate search for individual meaning and purpose, as well as opportunities to share their personal realizations through social connection. Meaning, purpose, and sociality cannot be satisfied through the canned whimsies of devices. Today’s global world, at least prior to the COVID-19 pandemic, is a panorama of human cultures in dialogue with one another. Naturally, cultural differences and distinctiveness evoke powerful emotional responses that range from love, amity, indifference, confusion, and hostility. Through the lens of medieval Iberian history, the public can explore and engage with each other and scholars via the safety of the temporal distance of the Middle Ages. This faraway place provides a safe intellectual and emotional arena for the public to evaluate and reflect on contemporary intercultural issues through the medium of medieval Spanish Jewish, Christian, and Muslim relations. In effect, our 2020-selves inhabit the history of medieval Spain so that we can contemplate the best and worst of human relations.

MARTINEZ-DAVILA, ROGER L., 2020. “The Space Between Jews, Christians, and Muslims in Medieval Spain: MOOCs, Citizen Science, and Digital Manuscript Collections.” Proceedings of Dark Archives Conference 2019, Medium Ævum Journal. (forthcoming)