Rigor Table:

In the rigor table (Table 1), SciScore highlights sentences that include various elements of rigor as described by Hackam and Redelmeier in 2006, and by van der Warp and colleagues in 2010. SciScore was trained using sentences from thousands of published papers that were tagged by expert curators to indicate that the sentence described a rigor criterion such as blinding (either during the experiment or during data analysis). SciScore uses conditional logic when scoring your paper. This means that if, for instance, a cell line is detected, then SciScore ‘expects’ to find cell line authentication criteria within the paper. If a cell line is not detected, these criteria (cell line authentication and contamination) will not be included in scoring. In cases where a criterion is not detected by SciScore but is expected, the statement “not detected” is displayed. In cases where a criterion is neither detected nor expected, "not required" is shown. It is possible that a criterion is not necessary for a particular manuscript or that SciScore, an automated tool, makes a mistake. If SciScore makes substantial mistakes with your manuscript, please contact us to help us learn from our mistakes.

Please note there are currently some bugs impacting this table and our scoring. We intended to have all criteria in Table 1 impact scoring (either conditionally or otherwise), however, protocol identifiers, data information, and code information have not been implemented into our scoring algorithm as of yet. We are diligently working on this and apologize for any inconvenience this may cause.

The rigor items detected in this version of SciScore include the following:

• Ethics Statement: a statement addressing ethical approval for experimental research (or why ethical approval is not required)

Institutional Review Board (IRB) - when research involves human subjects
The trial was approved by the NRES Committee London—South East.
Statement of participant consent
All infants were enrolled with informed parental permission under a protocol that was reviewed and approved by the Institutional Review Boards of the respective study sites.
Institutional Animal Care and Use Committee (IACUC) - when research involves vertebrate animals
All animal experiments were performed in accordance with relevant guidelines and regulations and were approved by the University of Pennsylvania Institutional Animal Care and Use Committee (IACUC).
Field Sample Permit - when research involves field sampling
Permission to conduct field surveys on each location was given by the individual landowners concerned, and by the regulatory authority (Natural England) in those situations where the field site was afforded protected status (i.e. Site of Special Scientific Interest).

• Euthanasia: a statement discussing the act of inducing the humane death of an animal in an effort to minimize pain, discomfort, or stress

General Euthanasia
Mice injected with CGG-NP23 were boosted on day 21 with the same inoculum and killed on day 28.
Euthanasia Agent or Method
Twelve hours after the final doses, the animals were euthanized by cervical dislocation.

• Group Selection: a statement addressing participant selection and removal over the course of a study

Inclusion and Exclusion Criteria - when there are characteristics that qualify or disqualify a prospective subject
Exclusion criteria were pregnancy, severe medical conditions, abnormal laboratory baseline values, unstable psychiatric features (e.g., suicidal), a history of alcoholism or drug abuse, epilepsy, brain trauma with loss of consciousness, neurological illness, or a concomitant Axis I psychiatric disorder.
Attrition - when there is dropout or loss of samples and subjects from a study
One participant withdrew from the yoga exercise group due to personal consideration.

• Subject Demographics: a statement describing the study of a population based on certain characteristics

Sex as a biological variable - when human subjects, organisms, or cell lines are used
Male and female mice were first considered separately in hindpaw nocifensive response experiments.
Age - when human subjects or organisms are used, including embryonic and instar stages
All mice were 8–16 weeks of age.
Weight - when human subjects or organisms are used
One Thoroughbred healthy adult horse (540 kg body mass) from the Royal Veterinary College (RVC) participated in the study.

• Core Reporting Standards: a statement addressing core reporting standards as set forth by the NIH (adapted from Landis et al.)

Randomization of subjects into groups
The strain-mixture randomly assigned organisms as low or high abundance.
Blinding of investigators in conduct or analysis
The experimenter was blind to the treatment groups.
Power analysis to determine sample or group size
Sample size was based on estimations by power analysis with a level of significance of 0.05 and a power of 0.9.

• Replication of the entire study or parts of the study: a statement describing the number of times an experiment was performed with enough detail to determine the type of replication

Type of replication - technical or biological
Each real-time PCR experiment included technical replicates, in a final volume of 15 µL.
Number of times that replication occurred
The experiment was replicated four times.

• Cell Line Authenticity: a sentence describing the steps taken to verify a cell line's identity and prove it's contaminant free

Authentication of cell lines - cell line identity verification, usually by STR profiling
All cells were authenticated by STR profiling and flow cytometry.
Mycoplasma contamination detection
All cell lines were obtained from ATCC and tested negative for mycoplasma contamination.

• Protocol Identifiers:

Unique identifier, URL or DOI - Protocols.io, Protocol Exchange, STAR Protocols, JoVE, Bio-protocol, MethodsX, Nature Protocols, Spring Protocols, BioTechniques, PROSPERO, ClinicalTrials.gov, and EUClinicalTrials Register
To study the effect of dutasteride on Abi metabolism, serum samples were collected from patients treated on a phase II clinical trial at Dana-Farber Cancer Institute (NCT01393730).

• Code Information:

Code availability statement
Image analysis was performed with ImageJ software macro (code available upon request).
Code identifiers - URL from github, google code, bitbucket
All scripts used for the analyses in this paper are available at the Github repository (https://github.com/vplagnol/recursive_splicing).

• Data Information:

Data availability statement
All other relevant data that support the conclusions of the study are available from the authors on request.
Data identifiers - dbSNP, dbVar, Sequence Read Archive, BioProject, Protein Circular Dichroism Data Bank, ArrayExpress, GEO, European Genome-phenome Archive, Japanese Genotype-phenotype Archive, MassIVE, MetaboLights, PeptideAtlas, ProteomeXchange, FlowRepository, Image Data Resource, European Nucleotide Archive, UniProt, dbGaP, Biostudies, and ClinVar
The complete results are uploaded in NCBI GEO as GSE75387.

Scoring for Rigor Table (5 total points):

The rigor table makes up 5 points of the total score. Those five points are split evenly among the expected rigor criteria. Scores are rounded to the nearest whole number. For each sentence that describes an expected rigor criterion, e.g. blinding, SciScore adds the fractional number of points for that criterion, and if it is unable to find a statement on blinding then this section is labeled "Not Detected" and receives a score of 0. To improve detection, please make sure that your language is clear and written in standard English.

Conditional criteria such as cell line authenticity are only included in the expected list if cell lines are detected in the Key Resources table (Table 2). Likewise, an IACUC statement is expected if an appropriate animal is detected in the Key Resources table. Currently, the field sample permit will be detected but never expected.

When organisms or human participants are detected, it is expected that blinding, group selection criteria such as randomization and inclusion/exclusion, attrition, and demographic information such as sex or gender will be present. Biological variables such as sex should inform subject and group selection.

SciScore attempts to classify papers based on the paper type to reduce the burden of requiring all criteria where it may be irrelevant, however, we tend to err on the side of caution, expecting criteria where SciScore is unsure. We do this because SciScore is primarily a tool that assists peer review by bringing attention to something that may have been omitted. Protocol, code, and data identifiers refer to persistent identifiers (either a DOI such as DOI:10.17504/protocols.io.9gbh3sn, a URL such as https://github.com/tophat, or an accession number in a repository such as GSE145917). SciScore will then try to authenticate these. For accession numbers, SciScore will check for the identifiers’ existence in their source database. For DOIs and URLs, SciScore will check to see if these resolve. Identifiers that are validated will be displayed in blue, while dead links will be shown in red such as DOI:10.17504/protocols.io.9gbh3snr. This is intended to quickly alert the author or reviewer to potential problems with a website or a typo in an accession number. SciScore does not check the relevance of the cited identifiers, only their existence. In rare cases, a typo may still result in a valid identifier. Consequently, we wish to remind users that SciScore is not a substitute for expert review. Rather, SciScore should be used in concert with reviewers for the best results.

How to get a better score on this section:

• Ensure that each criterion that is expected is addressed in your manuscript (refer to what is expected in the rigor items list above). In general though, adding more rigor criteria will increase points for this section.
• Pro Tip: If SciScore expects that a criterion should be filled, but you do not believe that this is relevant, address it using a negative statement. Examples:
• No subjects were excluded from our study.
• We did not assess whether subjects were male or female because embryos were not genotyped.
• Experimental subjects were not randomized into groups because this was deemed irrelevant to this study.
• Experimenters were not blinded to the subject's genotype because knockout mice were visibly different from controls.
• We did not check for sample sizes using a power analysis because our study does not report statistics between groups or within group variables.
• No technical replication was completed because the Sasquatch was visible only once.
• Possible Problems: SciScore does not recognize my sentence as fulfilling the criterion. In some cases such as power analysis, there are a surprisingly small number of example sentences in the published literature. This is a serious problem for science, but also for SciScore because text mining analysis depends on seeing lots of syntax patterns. Take a look at the sentences above, these syntax patterns were tested and should be recognized. Writing similar sentences should enable SciScore to recognize your sentence.
• Tip for reviewers: If you see the word Sasquatch in the manuscript, consider rejecting the paper.

Key Resources Table:

The key resources table (Table 2), contains:

• Sentences that “should” have RRIDs

• Key biological resources detected (names of cell lines or antibodies)

• RRIDs - if detected, these are checked and validated by SciScore

RRIDs are unique identifiers for reagents and other resources that largely overlap with the resource types that have been labeled as particularly problematic by the National Institutes of Health in recent changes to their grant review criteria, please see "key biological resources", e.g., antibodies, cell lines and transgenic organisms. The RRID initiative is led by community repositories that provide persistent, unique identifiers to their resources, such as transgenic mice, salamanders, antibodies, cell lines, plasmids and software projects such as statistical software. RRIDs are described on the rrids.org website and in a primer by Bandrowski and Martone in 2016. RRIDs are unique numbers that resolve to a particular database record, for example, the RRID:CVCL_0063 resolves to this record for a cell line (Cellosaurus community repository).

How does it work: The information in the Cellosaurus database (https://web.expasy.org/cellosaurus/) is structured and curated by Cellosaurus staff, the authority for cell lines (all RRIDs have an authority specific for the resource type). If authors use this RRID, then SciScore will ask the database about that particular identifier. In cases where a RRID fails to resolve (i.e. database has no record of that identifier, most likely due to a typo), SciScore will display an “unresolved” error message in red. If an RRID was recently submitted to the authority by authors, it often takes a week or more to become available in the database, thus exercising caution in the interpretation of the SciScore report in cases of newly minted RRIDs is advisable.

Sentences that ‘should have RRIDs’ are detected by SciScore using patterns in sentences that are similar to how each resource is commonly described in published papers. A sentence that describes one or more antibodies may be detected by SciScore and this will be placed into the table without a corresponding RRID. SciScore will then attempt to find the name, catalog number, and vendor of the resource. In cases where the tool is relatively confident, it will suggest an RRID (this will contain the word “suggestion” and be in gray), as a courtesy. A link is provided, so authors can quickly verify whether the correct RRID was suggested.

Note of caution: Please verify all RRID suggestions, only the author can know whether suggestions are correct.

The Key Resources types detected in this version of SciScore include the following:

• Antibodies
• Authority: Antibody Registry (ABR), Homepage, Submit Data

• Cell Lines
• Authority: Cellosaurus, Homepage, Contact Page

• Organisms
• Mice, authority: Mouse Genome Informatics (MGI), Homepage, Submit Data, Strains
• Rats, authority: Rat Genome Database (RGD), Homepage, Submit Data, Strains
• Worm, authority: Wormbase, Homepage, Submit Data, Submit worms
• Fly, authority: Flybase, Homepage, Submissions, BDSC, Kyoto
• Zebrafish, authority: Zebrafish Information Network (ZFIN), Homepage, Submit Data, ZIRC Submissions
• Xiphophorus, authority: XGSC, Homepage, Contact Page
• Frog (Xenopus), authority: Xenbase, Homepage, Contact, NXR Frog Repository
• Salamander (Ambystoma), authority: AGSC, Homepage, Stock Center
• Pig, authority: NSRRC, Homepage, Contact Page
• Tetrahymena, authority: Homepage, Contact Page

• Plasmids
• Authority: Addgene, Homepage, Submit Plasmids, Contact Page

• Software Tools
• Authority: SciCrunch Registry (SCR), Homepage, Submit Data

Scoring for Resources Table (5 total points):

The total for the entire Key Resources table is 5 points with scores rounded to the nearest whole number. Each resource that is detected in this section is included in the score. For each valid RRID detected with matching metadata (e.g. catalog number or name), full points are awarded. Because a single resource can often be described in a variety of ways, SciScore utilizes fuzzy matching to correctly link resources with their corresponding RRIDs. In cases where multiple resources and RRIDs are listed out in a single sentence, authors should verify that the resources and RRIDs are correctly matched as SciScore is not perfect. Partial points are awarded if SciScore detects resources where a suggestion can be made, or if an RRID does not resolve properly. Therefore, the way to maximize the points from this section is to add RRIDs and proper citations that include vendor names, catalog numbers, lot and version numbers into the methods section of the manuscript for every key resource used.

How to get a better score on this section:

• Ensure that each antibody, mouse, cell line, etc. has an accompanying RRID. SciScore will point out sentences or parts of a table where these items are located. Adding more RRIDs for additional antibodies that SciScore did not find does not hurt your score, it improves it.
• Pro Tip: Use the catalog number and vendor to improve the probability that the antibody name and RRID will be recognized as the same item.
• Common Problem: Antibody name shows up in one sentence the next “sentence” contains the RRID so SciScore thinks that my RRID is alone and there is an unidentified antibody. This can happen when a sentence is broken in your document, a return character or another invisible symbol may be at fault. Check your manuscript.
• Common Problem: Two antibodies, e.g., anti-mouse-nAChR antibody and nAChR antibody, are the same, but SciScore puts both on the list. To help SciScore, use the same name throughout the methods section. There should be no need to use the same RRID twice.

Other Entities Table (not included in scoring):

The other entities table (Table 3) contains:

• Statistical tests

• Oligonucleotides

• Additional problems, if found

Sentences containing entities of interest are shown in the leftmost column, while the specific statistical tests and oligonucleotides detected are displayed in the right column. Again, none of the criteria in the “other entities table” impact the overall score.

General notes on interpretation of text mining results:

Incorrect sentences: SciScore is a machine learning, text analysis tool, and it is therefore susceptible to making two types of errors: false positives and false negatives.

False negatives: The most common error occurs when the algorithm fails to detect a sentence that contains a rigor criterion or a resource, such as an antibody. False negatives generally occur either because the sentence is complex or in a less common syntax pattern. Generally, simple sentences in clear standard English are simpler to process and result in fewer false negatives. If a truly complex sentence structure is required to describe reagents, a table may help not only SciScore, but also human readers. If an RRID is detected in a sentence, SciScore will be triggered to take a look at the sentence, which may have been skipped otherwise.

False positives: This type of error occurs when SciScore falsely detects something including cases where a sentence does not contain an antibody, but the algorithm asserts that this sentence does have an antibody. If many resources are used and all have RRIDs, a single false positive will not reduce the score substantially, if at all. But if only 1-2 resources are used or if the false positive is in the cell line or organism category, it will trigger scoring for cell line authentication and other rigor criteria, which can reduce your SciScore needlessly. False positives are most often seen in the tools portion of table 2, as the algorithm detects company names, where it should not. We try to minimize these false positives using several strategies, however, they still occur in roughly 3-5% of cases. If this impacts your score, please contact our team (http://sciscore.com) and include the sentence where SciScore made the error. While we can't fix the score, SciScore can certainly learn from its mistakes for improved performance next time around.

Known Issues:

As mentioned before, SciScore is not perfect. Below we have provided a list of problems we are working to fix. If you notice any other persistent problems, please contact us.

• Equations and figures, go figure. If you have been unable to get your SciScore report back or are noticing a significant number of errors, equations and figures can sometimes be the culprit. Try submitting without them.

• We are still trying to iron out all of the kinks in our new identifers sections (for protocols, code IDs and data IDs). Sometimes identifiers will still be shown in red even though they resolve (and therefore should be blue). By clicking on the links provided in your SciScore report, you can check this out for yourself to double check SciScore. If you are redirected to a specific data record, then the identifier resolves! If not, then there could be an issue with the identifer (typo, etc.), the network (i.e. we reach out to a database and it does not respond), or the identifer is so new that our records haven't been updated to match.

• Issues with replication. Issues with replication. But for real, our replication section is still a little buggy. When SciScore detects both the type of replication and the number of replications, SciScore sometimes only shows one sentence even if entites are present in separate sentences. This is only an issue with the reporting table, and it will not impact your score.