EQIPD - User contributions [en]

2.2.3 Documentation of the experiment and deviations

2021-03-23T19:42:10Z

2A02:908:182:F8E0:F44A:43E3:950D:CFDA:

== A. Background & Definitions ==

Experimental record: for definition please check the item [[3.1.2.1 Traceability of data and any person having impact on data]]

Integrity of the experimental records: for details please check the item [[3.1.2 Procedures for how and when to record data]]

Witness: for definition please see [[3.1.2.2 Process for witnessing of records]]

== B. Guidance & Expectations ==
The following documents should be created before the study execution and in place during the course of the study:
* study protocol (and amendments, if any)
* required allowance to perform the study (e.g. animal use allowance)
* instructions for handling research material or other materials with unknown or established risk to human health and environment (if applicable)
* certificates of analysis of investigational products (if applicable)
* description of processes to identify, report and record changes in routinely performed assays (if applicable)

The following documents should be in place to describe what was done during the course of the study:
* all activities related to the study execution must be recorded, dated, initialed (or signed) and saved
* all activities should be recorded at the time they are generated or observed (i.e. recorded without a significant delay)
* any changes related to the study execution must be documented - why, when and by whom (when documents are amended, one should make sure not to over-write the original version)

'''PLEASE DO NOT FORGET'''
* To implement measures securing data integrity (see section [[3.1.2 Procedures for how and when to record data| 3.1.2]] for details)
* To consider implementing a witnessing if generated results are intended to support intellectual property claims

== C. Resources ==
The following two videos provide a short description on data integrity and the guiding principles of ALCOA:
{{#ev:youtube|https://www.youtube.com/watch?v=AdY9usIFLT8}}
{{#ev:youtube|https://www.youtube.com/watch?v=Xh-lt4pALq0}}

----------------
back to [[Toolbox]]

Next item: [[2.3.1 Generation, recording, handling and archiving of raw data]]

2.3.2 Primary analysis and evaluation of raw data

2021-03-23T19:41:53Z

2A02:908:182:F8E0:F44A:43E3:950D:CFDA:

== A. Background & Definitions ==
Primary analysis of raw data is the data processing required in order to derive (secondary) data that will be shared, presented and/or subjected to statistical analysis.

Information about primary analysis of raw data is critical for establishing a connection between raw data and reported results and is therefore an essential part of data traceability (see item [[3.1.2.1 Traceability of data and any person having impact on data]]).

== B. Guidance & Expectations ==
Primary analysis of raw data should:
* be performed blinded (e.g. by an experimenter unaware of pharmacological treatment)
** NOTE that, for knowledge-claiming research ([[2.1.4 Purpose of research]], this is a requirement
* maintain the original randomization scheme (if applicable)
* follow a pre-specified analysis plan that may be a part of the study protocol
** NOTE that, for knowledge-claiming research ([[2.1.4 Purpose of research]], this is a requirement
* include data verification (even in case of data produced by automatic systems there are generally additional data which are manually produced. Examples may be body weight, volume of drugs administered, unplanned observations performed during an experiment such as aberrant behavior)
* include a data validity check i.e. with respect to acceptance criteria pre-defined in the study protocol

Data generated via primary analysis of raw data should be securely stored (see item [[3.1.1 Platform to record data]]). Alternatively, one may store tools, algorithms, scripts and related analysis-related information that would be sufficient to reconstitute the analysis. If the latter approach is taken, two requirements apply:
* Repetition of the analysis should be possible for any researcher with the necessary skills
* One should ensure technical feasibility of such re-analysis for the entire period during which raw data are stored (e.g. ability to re-analyze should not be affected by updates in software or readability of guiding information)

'''PLEASE DO NOT FORGET'''
* To consider adding this subject to a training program for new employees or refresher training
* To label and store all primary analysis files in such a way that it ensures data traceability (for details see item [[3.1.2.1 Traceability of data and any person having impact on data]])
* Outside the pre-specified criteria, exclusion of data points and observations is only possible as long as primary analysis is conducted blind (i.e. before unblinding)
* All decisions to exclude data MUST be transparent (e.g. if necessary, recorded and reported)

== C. Resources ==
to be added

----------------
back to [[Toolbox]]

Next item: [[2.3.3 Statistical analysis]]

2.3.3 Statistical analysis

2021-03-23T19:41:22Z

2A02:908:182:F8E0:F44A:43E3:950D:CFDA:

== A. Background & Definitions ==

P-hacking: P-hacking means that analytical decisions are made after the results are known and data are analyzed in many different ways until the wanted results are reached or acquired. This can include e.g. use of an alternative statistical test and the post-hoc use of normalization A post-hoc increase in sample size/number of experiments also constitutes p-hacking.
Examples of various forms of P-hacking can be found in [https://www.ncbi.nlm.nih.gov/pubmed/25692012 Motulsky 2015]

"HARKing" means Hypothesizing After the Results are Known: a hypothesis derived based on the interpretation of the data is presented as having existed before the data were obtained.

== B. Guidance & Expectations ==
The following recommendations are based on [https://www.ncbi.nlm.nih.gov/pubmed/25692012 Motulsky 2015]

Statistical analysis should be performed exactly as described in the study protocol.

Any changes (e.g. in steps used to process and analyze the data or changes to study hypothesis) must be documented; the reason for a change must be explained and the study conclusion may need to be labeled as “preliminary”.

As the p-value provides no information about the actual size of the observed effect, it is recommended to calculate, document and present the effect size as difference, percent difference, ratio, or correlation coefficient along with its confidence interval.

It is strongly recommended to report statistical hypothesis testing (and place significance asterisks on figures) only if a decision is to be based on that one analysis.

It is strongly advised against the use of the word “significant” in a report or a publication; in plain English "significant" means "relevant" or "important", but a p-value provides no basis for the importance of a finding. If statistical hypothesis testing is used to make a decision, it is recommended to state the p-value, a preset p-value threshold (statistical alpha), and the decision.

Once the statistical analysis is conducted, it is recommended to plot figures that show the distribution of data (scatter plot; box & whiskers; violin plot). However, if the data have to be presented as a mean (e.g. in a table), display results as a mean and the standard deviation (mean ± SD or median with inter-quartile ranges if normal distribution is not assumed) (for more information please check the item [[2.3.4 Data visualization]]).

It is recommended not to plot the mean with error bars that represent the standard error (mean ± SEM) because SEM is not an indicator of variability but of precision and as such less informative than confidence intervals.

It is strongly recommended to report all details when describing statistical methods (for more information please check the item [[2.3.4 Data visualization]]).

'''PLEASE DO NOT FORGET'''

To preregister studies as it helps to reduce P-hacking and “HARKing” (for details please check the item [[2.1.11 Preregistration]]).

== C. Resources ==
Guidelines on reporting of statistical analysis (in vivo research):
[[ARRIVE Essential - Statistical methods]]

Essential reading:
* Motulsky HJ (2015) Common misconceptions about data analysis and statistics. Pharmacol Res Perspect. 3(1). [https://www.ncbi.nlm.nih.gov/pubmed/25692012]

Tools:
* [https://www.jamovi.org/features.html JAMOVI - free and open statistical software to bridge the gap between researcher and statistician - built on top of the R statistical language]
* [https://f.hubspotusercontent00.net/hubfs/4627953/GraphPad-Flowchart-Choose-the-Right-Statistical-Tool.pdf How to choose the right statistical tool]

----------------

back to [[Toolbox]]

Next item: [[2.3.4 Data visualization]]

2.3.4 Data visualization

2021-03-23T19:41:01Z

2A02:908:182:F8E0:F44A:43E3:950D:CFDA:

== A. Background & Definitions ==
Data visualization serves to enhance readers' understanding of the data and to provide support for appraisal of the data. In the scientific publications, authors frequently use the static graphical forms to show the data that support key findings ([https://www.ncbi.nlm.nih.gov/pubmed/25901488 Weissgerber et al. 2015]). Technically more challenging as it requires specific tools and skills, interactive graphics may improve data presentation in scientific publications by allowing readers viewing different types of graphs plotted based on the same data sets, improve readers' understanding of the data and allowing for adequate evaluation of data ([https://www.ncbi.nlm.nih.gov/pubmed/27332507 Weissgerber et al. 2016] and [https://www.ncbi.nlm.nih.gov/pubmed/28974579 Weissgerber et al. 2017]).

It should be remembered that many of the expectations about the best practices of graphical representation of the data do apply to tabular and other forms as well.

== B. Guidance & Expectations ==

Data analysis should be performed exactly as described in the study protocol and figures, which represent data and should provide sufficient information allowing to understand and rigorously interpret data (the following recommendations are based on [https://www.ncbi.nlm.nih.gov/pubmed/25692012 Motulsky 2015], [https://www.ncbi.nlm.nih.gov/pubmed/25901488 Weissgerber et al. 2015]):
* For scientific publications, it is recommended to use figures that show the distribution of continuous data, i.e. univariate scatterplots, box plots, or histograms.
* For studies with small sample sizes, it is recommended to use univariate scatterplots, which is the most informative graphical presentation.
* Bar and line graphs are not recommended for the use in studies with small sample size. Such graphs do not show data distribution as the same bar or line graph can be plotted based on substantially different datasets.
* If data distribution cannot be presented, it is recommended to plot mean with error bars that represent the standard deviations (mean ± SD), or means with error bars showing the 95 % confidence interval (CI) of the mean.
* Researchers are encouraged to provide information if and where raw data are available.

As the data visualization cannot be separated from data analysis, it is strongly recommended to report all details when describing data processing and statistical methods ([https://www.ncbi.nlm.nih.gov/pubmed/25692012 Motulsky 2015]):
* it must be clear what experimental unit is (e.g. biological replicates, technical replicates, etc.);
* if any observations were excluded, it must be clearly stated how many were eliminated, the rule used to identify them, and a statement whether this rule was chosen before collecting data;
* if data were subjected to normalization, please explain exactly why and how you defined 100 and 0 %;
* explain the details of the statistical methods you used. For example, if you fit a curve using nonlinear regression, explain precisely which model you fit to the data and whether (and how) data were weighted.

Finally, researchers are strongly encouraged to visualize the data in a way that supports reporting of the effect sizes with expressions of uncertainty ([https://www.eneuro.org/content/6/4/ENEURO.0205-19.2019 Calin-Jageman and Cumming 2019]).

'''PLEASE DO NOT FORGET'''
* To consider adding this subject to a training program for new employees or refresher training (if appropriate)
* To apply the same principles and standards to graphical and non-graphical forms of data visualization

== C. Resources ==
* Motulsky HJ (2015) Common misconceptions about data analysis and statistics. Pharmacol Res Perspect. 3(1). [https://www.ncbi.nlm.nih.gov/pubmed/25692012]
* Weissgerber TL, Milic NM, Winham SJ, Garovic VD (2015) Beyond Bar and Line Graphs: Time for a New Data Presentation Paradigm. PLoS Biol 13(4): e1002128. [https://www.ncbi.nlm.nih.gov/pubmed/25901488]
* Weissgerber TL Savic M, Winham SJ et al. (2017) Data visualization, bar naked: A free tool for creating interactive graphics. J Biol Chem. 292(50):20592-20598. [https://www.ncbi.nlm.nih.gov/pubmed/28974579]
* Weissgerber TL, Garovic VD, Savic M et al, (2016) From Static to Interactive: Transforming Data Visualization to Improve Transparency. PLoS Biol. 14(6). [https://www.ncbi.nlm.nih.gov/pubmed/27332507]
* Ellis DA, Merdian HL (2015) Thinking Outside the Box: Developing Dynamic Data Visualizations for Psychology with Shiny. Front Psychol. 6:1782. [https://www.ncbi.nlm.nih.gov/pubmed/?term=Thinking+Outside+the+Box:+Developing+Dynamic+Data+Visualizations+for+Psychology+with+Shiny]
* Calin-Jageman RJ, Cumming G (2019) Estimation for Better Inference in Neuroscience. eNeuro 1 August 2019, 6 (4) ENEURO.0205-19.2019. [https://www.eneuro.org/content/6/4/ENEURO.0205-19.2019]

Journal guidelines:
* PLOS Biology (2016) Submission guidelines: data presentation in graphs [https://journals.plos.org/plosbiology/s/submission-guidelines#loc-data-presentation-in-graphs]
* ASPET journals' instructions for authors (detailed explanations with examples) [https://doi.org/10.1124/jpet.119.264143.]

Online tools for creating interactive graphics:
* Allows to create interactive line graphs [http://statistika.mfub.bg.ac.rs/interactive-graph/]
* Allows to create customized interactive graphics such as univariate scatterplots, box plots, violin plots and, only for educational purposes, bar graphs [http://statistika.mfub.bg.ac.rs/interactive-dotplot/]
* To create interactive graphics from data obtained with repeated independent experiments - [http://statistika.mfub.bg.ac.rs/interactive-repeated-experimentsdotplot/]

----

back to [[Toolbox]]

Next item: [[2.4 Reporting]]

4.2.2 Error and incident management

2021-03-23T19:40:37Z

2A02:908:182:F8E0:F44A:43E3:950D:CFDA:

== A. Background & Definitions ==
This item refers to one of the [[Core Requirements]] (Core Requirement 16 - "Critical incidents and errors during study conduct must be analyzed and appropriately managed") and is, therefore, considered essential.

Critical Incident: An unexpected deviation from the intention (e.g. study protocol) with a larger or longlasting impact on the experimental outcome or on the research unit (RU).



== B. Guidance & Expectations ==
The EQIPD QS recogizes two levels of error and incident management - mandatory (to meet the EQIPD Core Requirement) and voluntary (for systematic error management).

The EQIPD Core Requirement on error and incident reporting

Basic reporting of errors and incidents is important to create transparency of the experimental work performed in a RU. The following is expected:
* Error documentation
** All errors affecting the outcome of experiments must be documented during study conduct within the experimental records
* Impact assessment
** The researcher must assess the incident and decide whether the error has to be communicated (click here for a decision tree)
** This is an informal step and no documentation is required
* Mitigation approach
** Depending on the impact and scope, the mitigation approach must either be developed by the researcher themselves, by the lab supervisor or in a dedicated team. It could also be discussed in a lab meeting if most group members are affected.
** The mitigation approach should be documented (e.g. with a formal template, as a note with the experimental record, as an Email trail in a dedicated mailbox or any other way)
** If applicable, measures should be taken to avoid this error in the future

To fulfill this Core Requirement, it is expected that awareness about the error culture is created (e.g. during lab meetings).

Voluntary approaches providing a systematic error management

To report and document errors at this level the following approach may be applied:
* Identify a person responsible for collecting errors and incidents
* Report errors via a document (please see below for a template) or electronic tool (e.g. [https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.2000705 LabCIRS])
* The person overseeing errors and incidents is informed (anonymously)
* Countermeasures shall be discussed and established to prevent reoccurence
* The error and the countermeasures shall be presented and discussed with the whole team (e.g. during lab meetings)

'''PLEASE DO NOT FORGET'''

To facilitate a positive and transparent error culture. This can be done with short trainings when onboarding new team members or during an annual training. The presentation created by the EQIPD team can provide a starting point (LINK).

'''RISK ASSESSMENT'''
Admitting and handling errors and incidents is a sensitive topic as it may be perceived as embarrassing and shameful. Hence, special care has to be applied while discussing, documenting and dealing with erros and incidents.

== C. Resources ==
Template for documenting errors and critical incidents - 4.2.2 Error Reporting.docx

LabCIRS:
* software [https://github.com/major-s/labcirs]
* publication [https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.2000705]

Powerpoint presentation with decision tree on Error Management and creating awareness: [https://paasp.sharepoint.com/:p:/s/EQIPD/Ee8vZYriaw1OguV7kUtUXfMBVlZPFBjitJVEvbeL6-ivDA?e=dmdOqq EQIPD training slides on Error Management 200331.pptx]

Other publications:
* Errors and error management in biomedical research [https://doi.org/10.5281/zenodo.1406922]

----

back to [[Toolbox]]

Next item: [[4.2.3 Responsible conduct of research]]

FAQ

2021-03-23T19:40:04Z

2A02:908:182:F8E0:F44A:43E3:950D:CFDA:

== About the quality system in general ==

===Can I appoint my student or postdoc to be the Process Owner for the EQIPD Quality System in my lab (group, department, institute)?===

As explained in the corresponding Toolbox section, the Process owner should be competent and should have sufficient control of resources. Certain tasks and functions can be delegated by the Process Owner to appropriate people in the research unit but the overall responsibility should stay with the competent (senior) person with relevant knowledge and access to resources (person that will likely remain in the research unit for a long time).

===Contract Research Organizations (CROs) often do not know the purpose of research. How can they state in the study protocol whether a study is to inform a formal knowledge claim or not?===
If sponsor of a study does not provide such information or is not willing to have this information to be included in the study protocol, then the "knowledge-claiming research" box should not be ticked. It is nevertheless expected that the study protocol includes information about any and all measures taken to protect against the risks of bias. For all research, EQIPD recommends to apply maximal possible rigor.

===If a study involves a series of independently performed experimental procedures, assessments and analyses, should they all be conducted with the same level of research rigor?===
There is no such requirement. The only expectation is that, for every individual experiment, procedure or analysis, risks of bias are assessed and protective measures are transparently reported.

===How could small organizations (especially those operating virtually) fulfill the core requirement on Responsible conduct of research?===
It may indeed be not possible in some cases to identify a person of trust or set up an anonymous mailbox or a confidential electronic "hotline" within the organization. from where concerns are triaged to a dedicated person. In such situations, EQIPD advises to look for alternative solutions - a person of trust in the neighboring research unit or organization; a member of the advisory board; a representative of the funders or investors; a member of the IACUC or animal welfare body reviewing the in vivo work in the research unit. Even small biotech or CROs do not operate in complete isolation and one may only be advised to look around and identify individual(s) who would be willing to support research integrity efforts of the research unit and who would be in a position to deal with violations and allegations of misconduct in agreement with the national or institutional guidelines.

===In our laboratory, hand-written notes taken during the experiments are scanned and PDFs are then stored electronically. We treat these scanned copies as raw data and discard the paper originals. Is such practice acceptable?===
If it is not possible to maintain both paper-based and electronic archive (laboratory notebook), then such practice is acceptable. Of course, such copies should be complete and of good quality.

===We have a number of internal documents (templates, policies, procedures) written in a non-English language. Should these be translated?===
No, there is no need to translate. You can make references to these internal documents, for example, in the Documentation Plan, etc. If you decide to conduct an external assessment of the Quality System performance (e.g. for accreditation purposes), EQIPD experts will read in the original language or have it translated into English.

===We have a number of internal documents that, according to an internal policy, cannot be sent or taken outside of our organization. Will that prevent us from getting an EQIPD Quality System accreditation?===
No, albeit this creates some inconvenience, this is not a roadblock. Should these documents be seen by EQIPD experts as essential, this increases a probability that your research unit is chosen for onsite visit (rather being assessed remotely).

== Specifically about knowledge-claiming research ==

===Contract Research Organizations (CROs) often do not know the purpose of research. How can they state in the study protocol whether a study is to inform a formal knowledge claim or not?===
If sponsor of a study does not provide such information or is not willing to have this information to be included in the study protocol, then the "knowledge-claiming research" box should not be ticked. It is nevertheless expected that the study protocol includes information about any and all measures taken to protect against the risks of bias. For all research, EQIPD recommends to apply maximal possible rigor.

===Are PK, bioanalysis and other “quantitative” studies in pharmacology informing a formal knowledge claim?===
Such studies often lack a “hypothesis” (scientific or statistical) but a “hypothesis” is not a requirement for “knowledge claim” studies. Thus, such studies may indeed be declared as research to inform a knowledge claim if the scientists decide so and apply maximal possible rigor.

===Would EQIPD external or internal assessment experts need to judge which study belongs to the “knowledge claim” category and which not?===
No, definitely NOT. EQIPD requests that the scientists themselves declare before doing a study whether it is intended to inform a knowledge claim or not. And it is the scientists themselves who conduct a risk assessment and decide what research rigor measures should be applied. The job of internal or external assessors is only to confirm that experimental process and reporting are transparent.
Let us assume I have no a priori intention for a knowledge claim when I plan my work. Nevertheless, I do the study(ies) under full rigor. Can I still make a knowledge claim after the studies are completed?EQIPD does recognize this example as highly relevant because there are indeed situations when, for example, a need to use the research resulrs to justify a certain decision is not evident when a study is planned. There are two ways to look at such situation.
On the one hand, in the end of the day, it is the research rigor that matters and not just the name of “knowledge-claiming research”. Therefore, in principle, all studies conducted under maximal possible rigor can support decision making, build confidence, etc., whether "knonwledge claim" was prespecified or not. However, if prespecified, studies must be conducted under maximal rigor or no such claim is possible. If, for some reason, there is a deviation from the pre-specified research rigor conditions during the course of the study, it needs to be clarified in the study report and the “knowledge claim” may no longer be valid.
On the other hand, EQIPD stongly advises all EQIPD QS-compliant research units to declare the "knowledge claiming research" in ALL cases when studies are planned with maximum possible rigor (i.e. all conditions for a knowledge-claiming research are met). Such practice will help avoid any questions and doubts about value of the study. Also, one should note that one does not need to know exactly if and how the study results can be used (e.g. for what kind of decisions) in order to declare a study to be "knowledge-claiming".

----

go to [[Toolbox]]

1.4.1.2 Data sharing

2021-03-23T19:38:59Z

2A02:908:182:F8E0:F44A:43E3:950D:CFDA:

==A. Background & Definitions==
Open Science: From the development to the dissemination of knowledge, ‘Open Science’ (OS) aims to make scientific research, data and their dissemination available to any interested person, from professionals to citizens - with the ultimate goal to make it easier to publish and communicate scientific knowledge. OS fosters sharing and collaboration, introducing a systemic change to the way scientific research is done.

Publication bias: Studies with neutral and null results are more likely to end up in the file drawer than studies with statistically significant findings. This phenomenon (defined as “publication bias”) leaves scientists, funding agencies and clinicians with a distorted view of the scientific evidence, which can lead to poor decisions about what research directions are most promising and should be funded or what medical treatments should be recommended to patients.

Preregistration: A time-stamped, read-only version of the study protocol and data analysis plan created before the study (see [[2.1.11 Preregistration]]).

==B. Guidance & Expectations==
Open Science:
* Open Data: It is advisable to make data underlying reported results openly available - to the greatest extent permissible by legal and ethical constraints.
* Open Materials: It is advisable to make research materials or analytical code available for others to use and reuse.
* TOP Guidelines: Open Science can be improved by increasing transparency of the research process and products. Here, the Transparency and Openness Promotion (TOP) Guidelines provide guidance on how to enhance transparency in the science that journals publish: 8 standards within the TOP guidelines move scientific communication toward greater openness. Moreover, the guidelines are sensitive to openness barriers by articulating a process for exceptions to sharing because of ethical issues, intellectual property concerns, or availability of necessary resources.

Multi-partner projects:
* For each multi-partner project, especially for Academia-Industry-collaborations, it should be agreed on - before project start - which data sets can be published and when: A dedicated plan already in place before/when creating the data can be helpful here.
* For Academia-Industry collaborations and to provide PhD students with the possibility to defend and publish their PhD thesis, it is recommended that the academic institution establishes conditions so that students are able to at least submit and defend their PhD theses under certain secrecy conditions.
* For provide a perspective for early career researchers at the beginning of a collaboration project, it is recommended to agree and checked which data from the project are non-IP relevant and can be published and/or uploaded to data repositories (ideally within an acceptable embargo timeframe).
* Consider generating a ‘shadow publication’ including IP-relevant data sets and uploading to the private, non-public areas of repositories like the OSF. Later, after e.g. relevant patents are granted, the publication can be submitted in no time simply by pushing a button.

==C. Resources==
* fiddle: a tool to combat publication bias by getting research out of the file drawer and into the scientific community [https://portlandpress.com/clinsci/article/134/20/2729/226790/fiddle-a-tool-to-combat-publication-bias-by]

* Data repositories, for example the Open Science Framework [https://osf.io/]

* TOP Guidelines [https://www.cos.io/initiatives/top-guidelines]