OSF design registration (draft)¶

Draft text for the first registration, using the Open-Ended Registration template. Submit it by November 10, 2026, after both annotators have uploaded their DAGs. Items marked [check] need Matt's decision or confirmation before submission. The full analysis plan is registered separately on December 4, before the test set is touched.

Title¶

From Literature to Living DAGs: design registration for an edge-evidence schema and low-cost extraction of causal claims from cohort studies

Summary¶

We will build a gold corpus of 150 PubMed abstracts on cardiovascular and other non-communicable diseases, annotated in a new edge-evidence schema for causal diagrams. We will test whether low-cost typed classification and extraction models can fill that schema, and we will replay the literature in publication order for two prespecified exposure–outcome pairs. This registration fixes the pairs, the question each pair addresses, the sampling of abstracts, the annotation and DAG-drawing procedures, and our expectations, before any pair abstract is retrieved or any model is run. A second registration on December 4, 2026 will fix the analysis plan, the development/test split and the decision thresholds.

Status of the study at the time of registration¶

30 pilot abstracts were retrieved from PubMed and read by the lead annotator (MS) while drafting the annotation guideline. 15 of them form the calibration set; none was retrieved by the pair searches below. One pilot abstract (PMID 41997799) reports on LDL cholesterol control and cardiovascular disease and will be excluded from the LDL pair sample if the search returns it. [check]
The annotation tool and guideline (version 0.1) are complete. The 100 general abstracts and the 50 pair abstracts have not been retrieved.
No model has been run on any abstract.
Both annotators' DAGs for the two pairs are attached to this registration (component "Expert DAGs"), drawn before either annotator read any pair abstract.

Research questions¶

RQ1. Can two trained annotators fill the claim record with Cohen's κ ≥ 0.60 on claim strength, causal intent, polarity, direction and design?
RQ2. Do rule-based edge grades agree with grades assigned independently by the second annotator?
RQ3. What share of claims is worded as associational but coded as causal intent, and what share of the evidence (including precise null findings) would exclusion rules like those of PubMedCausal and EpiKG2DAG discard?
RQ4. How do accuracy, calibration and cost per claim differ between typed decision and extraction models, generative large language models and fine-tuned encoders, and can a confidence-based cascade match the strongest model at lower cost?
RQ5. How closely do the final literature-derived DAGs match the two annotators' DAGs, relative to how closely those two DAGs match each other?
RQ6. When do edge grades and minimal adjustment sets change as evidence accrues, and does the automated pipeline find the same change points as the human annotations?

The two prespecified pairs¶

	LDL cholesterol → ASCVD	Educational attainment → ASCVD
Role	Positive control: convergent evidence from cohorts, Mendelian randomisation and trials	Contested social pathway: no agreed DAG
Exposure	LDL cholesterol concentration measured in adulthood (higher vs lower)	Educational attainment fixed by early adulthood (highest level completed or years of schooling)
Outcome	Incident ASCVD: coronary heart disease, ischaemic stroke, peripheral artery disease	Same
Population	Adults free of ASCVD at baseline in population cohorts	Same
Time zero	Cohort baseline	Cohort baseline in adulthood
Target	Total effect	Total effect

The two pairs share the outcome, and LDL cholesterol is a plausible mediator in the education DAG, so the two DAGs share nodes and the LDL → ASCVD edge.

Corpus and sampling¶

General set (100 abstracts). Prospective cohort studies of cardiovascular and other non-communicable diseases published 2015–2026, stratified by exposure family (lipids, blood pressure, glycaemia, adiposity indices, lifestyle and diet, environment, social determinants) and publication period, and sampled at random within strata.
Pair sets (25 abstracts each). Original human studies reporting an estimate for the pair's exposure and outcome, of any design (cohort, nested case–control, Mendelian randomisation, sibling comparison, randomised trial), published 1990–2026. Reviews, meta-analyses, comments and cross-sectional studies are excluded. Abstracts are sampled at random within three publication periods (1990–2004, 2005–2014, 2015–2026), with 8, 8 and 9 abstracts per period. [check periods and allocation]
Draft PubMed searches [check]:
LDL: ("LDL cholesterol"[tiab] OR "low-density lipoprotein cholesterol"[tiab] OR "LDL-C"[tiab]) AND ("coronary"[tiab] OR "cardiovascular"[tiab] OR "ischemic stroke"[tiab] OR "ischaemic stroke"[tiab] OR "atherosclerotic"[tiab]) AND ("cohort"[tiab] OR "prospective"[tiab] OR "mendelian randomization"[tiab] OR "randomized"[tiab]) AND humans[mh] AND 1990:2026[dp]
Education: ("educational attainment"[tiab] OR "education level"[tiab] OR "educational level"[tiab] OR "years of schooling"[tiab] OR "years of education"[tiab]) AND ("coronary"[tiab] OR "cardiovascular"[tiab] OR "stroke"[tiab] OR "atherosclerotic"[tiab]) AND ("cohort"[tiab] OR "prospective"[tiab] OR "mendelian randomization"[tiab] OR "sibling"[tiab]) AND humans[mh] AND 1990:2026[dp]
The random seed for all sampling is 20261118. Screening for eligibility is done by MS on titles and abstracts, and replacements are drawn from the same stratum.

Annotation¶

Unit: the primary exposure–outcome finding of each abstract (first finding stated in the conclusions).
MS labels all 150 abstracts on all fields. The second annotator (A), from outside the project, labels the 15 calibration abstracts and a random 30% of the main set (45 abstracts, seed 20261118) on the core fields: finding present, claim strength, causal intent, polarity, direction, design.
Neither annotator sees the other's labels on a record until MS shares it for comparison; the tool enforces this. After a calibration call on the 15 calibration abstracts, the guideline is frozen as version 1.0 and posted to this project.
Disagreements are resolved by consensus between the two annotators. Agreement (RQ1) is reported on the labels before consensus. Consensus on the main set is completed before MS looks at any test-set model output.

DAGs¶

Both annotators draw a DAG in dagitty syntax for each pair, independently, before reading any abstract of that pair or any model output. A draws both before reading any abstract at all.
Each DAG is submitted in the annotation tool and uploaded by its author to this project. Neither annotator sees the other's DAG for a pair until both have submitted it.
Because no correct DAG exists for the contested pair, RQ5 compares the literature-derived DAG with each annotator's DAG and sets this against the agreement between the two annotators' DAGs. Disagreements are classified by type (edge presence, variable role as confounder or mediator, unmeasured common causes) and by whether the literature can resolve them.

Prespecified expectations¶

The LDL → ASCVD edge reaches a moderate or high grade early in the replay and stays stable.
The education → ASCVD edge is graded lower than the LDL edge, and its grade changes when Mendelian randomisation or sibling-comparison studies enter the replay.
The two annotators' DAGs agree more for LDL → ASCVD than for education → ASCVD.
Claims worded as associational but coded as causal intent are common in both the general set and the education set (RQ3); we make no numerical prediction.

Registered later (December 4, 2026)¶

The development/test split of the 150 abstracts, the models and versions, the cascade thresholds, the metrics and decision rules for each research question, and the handling of rare categories.

Deviations¶

Any change to the pairs, searches, sampling or procedures above after this registration will be reported, with its date and reason, in the preprint and in this project.

Roles and conflicts of interest¶

MS designed the study, built the annotation tool and the extraction pipeline, and develops DAGpedia, the open DAG repository the schema is designed for. Because MS is also the lead annotator and one of the two DAG authors, his labels and DAGs are not independent of the pipeline; this is reported as a limitation. [check: add funding and competing interests]