Download as PDF, PPTX
Uploaded byMark Wilkinson
Introducing the fair evaluator
The document provides an overview of a fair evaluation tool designed for assessing compliance with various fairness metrics through a web-based interface. Users can register new metrics, execute evaluations, and recover previous results while the tool undergoes peer review for stability. The system emphasizes automated testing to encourage continuous improvement in fairness standards.
More Related Content
Similar to Introducing the fair evaluator
More from Mark Wilkinson
Introducing the fair evaluator
- 1. The FAIR Evaluator Automated,Objective, Aspirational! “Strawman” code by Mark Wilkinson Sneak-preview for Bio IT World 2018
- 2. Overview - Every Metricis associated with a Web-based interface that can evaluate compliance with that Metric - New Metrics can be registered simply by pointing to the URL for the interface of their evaluation software - Collection of Metrics can be assembled by anyone, to represent the aspects of FAIRness that they care about (e.g. a journal vs. funding agency vs researcher) - You can execute an evaluation by providing an IRI to be tested, and a collection of Metrics to be applied - Evaluations can be recovered (see previous input data) and/or re-executed, either through the Web interface, or by direct connection to the Evaluator from software (i.e. the Web page is only for people)
- 3. The Evaluator willsoon be peer-reviewed so we are NOT inviting you to try it yourself ...yet! We need it to be stable and consistent :-)
- 4.
- 5.
- 6.
- 7. Browse Metrics What principledoes this Metric test?
- 8. Browse Metrics What isthe address of the testing interface?
- 9. Browse Metrics Browse toone of them….
- 10. Interface Definition (Swagger 2.0- a globally popular standard)
- 11.
- 12. Metrics Collections (at themoment, I have only collected the individual FAIR facets together - anyone can create a collection)
- 13.
- 14.
- 15.
- 16. Explore one evaluation Seethe result!
- 17.
- 18. Want to re-execute? Seethe raw input data
- 19. Raw Input Data (copy/edit/pasteand HTTP POST to the Evaluator)
- 20. Web-based Execution ofan Evaluation
- 21. Web-based Execution ofan Evaluation What is being tested?
- 22. Web-based Execution ofan Evaluation This questionnaire field is automatically created by examining the Swagger definition for the Metric Tester Swagger Doc.
- 23.
- 24. Verification of Answer(s)to Questionnaire The Evaluator Metric Tester FAIRSharing.org F1: DOI DOI - OK?
- 25. Verification of Answer(s)to Questionnaire The Evaluator Metric Tester FAIRSharing.org Standard! Is this really a DOI?
- 26. Verification of Answer(s)to Questionnaire The Evaluator Metric Tester FAIRSharing.org Yes, This is a DOI
- 27. Automated, Objective, Aspirational! It’sHARD to pass the automated tests! (much harder than you would think!) They are unforgiving, because they are automated They force you to think about your interface from the perspective of a machine-visitor
- 28. Automated, Objective, Aspirational! Ifyou ever start to score 100% on the Metrics… ...we will just create harder Metrics!! FAIR is ASPIRATIONAL! You can always do better!
- 29. Contact: Mark Wilkinson mark.wilkinson@upm.es Ministerio deEconomía y Competitividad grant number TIN2014-55993-RM