Automated "quality checks" by databases of incoming records fsck a whole bunch of data up. Heckuva job, programmers. I bet y'all didn't think you needed human review.

An exhaustive reference to problems seen in real-world data along with suggestions on how to resolve them. - GitHub - Quartz/bad-data-guide: An exhaustive reference to problems seen in real-world d...

Wherein I reiterate what others have said and express frustration with The Powers That Be  Of all of the problems in the world of data sharing, the most frustrating to me is the data that are shared but are completely unusable to others. As I've argued before (and of course, I'm not alone in this ar

OpenRefine, formerly Google Refine, bills itself as “a free, open source, powerful tool for working with messy data.”  As someone who works with messy data almost every day, I can’t recommend it enough.  While Open Refine is a great tool for cleaning up “grid-shaped data” (spreadsheets), it’s a bit more challenging to use when your source data is … Continue reading Adventures in metadata hygiene: using Open Refine, XSLT, and Excel to dedup and reconcile name and subject headings in EAD →

D-Lib Magazine

Thomas Herndon is definitely getting an A this semester.

Big-data boondoggles and brain-inspired chips are just two of the things we’re really getting wrong

Amid the pressure and enthusiasm for researchers to share data, a rapidly growing number of tools and services have emerged. What do we know about the quality of these data? Why does quality matter? And who should be responsible for data quality? We believe an essential measure of data quality is the ability to engage in informed reuse, which requires that data are independently understandable. In practice, this means that data must undergo quality review, a process whereby data and associated files are assessed and required actions are taken to ensure files are independently understandable for informed reuse. This paper explains what we mean by data quality review, what measures can be applied to it, and how it is practiced in three domain-specific archives. We explore a selection of other data repositories in the research data ecosystem, as well as the roles of researchers, academic libraries, and scholarly journals in regard to their application of data quality measures in practice. We end with thoughts about the need to commit to data quality and who might be able to take on those tasks.

An investigation into how patient outcomes are assessed in clinical trials has revealed a worrying lack of consistency, raising concerns about funding being wasted on the acquisition of poor quality data.

This paper presents the results of a survey of quality assurance (QA) practices within the field of web archiving. To understand current QA practices, the authors surveyed 54 institutions engaged in web archiving, which included national libraries, colleges and universities, and museums and art libraries.

The analysis of giant data sets promises unique business insights, but much manual effort is still required to prepare the information for parsing.

The Leek group guide to data sharing . Contribute to jtleek/datasharing development by creating an account on GitHub.

Don't forget to include century data!

Open data initiatives may hold much promise and value, but more attention is needed on how these projects are developing as complex socio-technical systems. Rob Kitchin elaborates on four specific …