Skip to content

Annotation Kit: CVD Claim Pilot

Matt Sakamaki · version of November 4, 2026

At a glance

One collaborator (A) and Matt build the gold labels for a medRxiv preprint (submission December 18, 2026) that tests whether low-cost System One models can extract causal claims from CVD cohort abstracts into DAGpedia. All work happens on one web page, and neither of you sees the other’s labels until Matt shares a record for comparison.

Step Who What Time When
1. DAGs A and Matt, separately One DAG for each of two pairs, before reading any abstract about 3 h by Nov 10
2. Calibration A and Matt 15 abstracts each, then a calibration call about 2.5 h Nov 11–17
3. Main sample A Core fields on 45 abstracts (random 30% of the main set) about 4.5 h Nov 18 – Dec 4
4. Consensus A and Matt One call to agree on every disagreement about 1 h Dec 5–7
5. Edge grades A Evidence grades for a handful of edges about 1.5 h Dec 8–11
— Matt All fields on all 150 abstracts — Nov 11 – Dec 4

In total about 13 hours for A, including 30 minutes to read this kit.

Links:

Using the annotation page

Matt sends you the page link. It runs on Cloudflare and needs no Claude account. The page saves every label to its own database as you go, so you can stop and resume on any device.

  1. Open the link and enter the email address Matt registered for you. Cloudflare emails you a one-time code; type it in to sign in.
  2. Check the role chip at the top: it should say Second annotator (A). If it says you are not on the team list, tell Matt which email you used.
  3. Start in DAGs. Annotate, Compare and agree, and Edge grades stay locked until you submit both DAGs.
  4. In Annotate, work through your list on the left. A filled dot means done, a half dot means draft. Press Save on each record: a record with every required field is saved as done and the page moves to the next one.
  5. Press Refresh to pick up anything Matt has shared since you opened the page.

Who can see what:

Data Matt A
DAGs for the two pairs own; A’s shown for a pair once both have submitted it own; Matt’s shown for a pair once both have submitted it
Abstracts all; the 50 pair abstracts unlock after his two DAGs calibration set and your main-set sample, after your two DAGs
A’s labels read own; locked once shared
Matt’s labels own core fields of shared records only
Consensus values read and write read and write, shared records only
Model outputs not on this page never

The server enforces these rules, so the page cannot show you more even by mistake. Sign-in lasts about a day; if a save fails with a sign-in message, reload the page and sign in again. Unsaved changes on the open record are lost on reload, so save before you step away.

Guide for A: coding an abstract

Draw both DAGs first (next section); your abstracts unlock once you submit them. After that you code one finding per abstract, the primary one, on six fields. Expect about 5–6 minutes per abstract once you are used to it; read the whole abstract, then code from the conclusions.

Which finding. Take the first exposure–outcome result stated in the Conclusions (or the last results sentence if there is no Conclusions section). Write its exposure and outcome in a few words in the two boxes at the top, so Matt can check you both coded the same finding. If the conclusions lead with several exposures, take the one named in the title.

Field Choices Rule Watch for
Exposure–outcome finding yes / no Yes when the abstract reports its own estimate or result linking an exposure to an outcome; a meta-analysis counts. Comments, narrative reviews and prevalence-only reports are No. With No, only Design is coded.
Claim strength direct causal / conditional causal / associational / no relation The strongest causal wording used about the primary finding in the conclusions. Direct: causes, leads to, reduces, protective effect. Conditional: may cause, could contribute, might lower. Associational: associated with, linked to, predicts, related to. No relation: not associated, no association. “Risk factor” or “predictor” alone is associational. Judge the words, not the design.
Causal intent causal question / descriptive or predictive / unclear What the study is trying to learn, whatever the wording. Causal: the effect of the exposure, shown by confounder adjustment aimed at that effect, causal methods (MSM, g-methods, MR, target trial emulation), or prevention or treatment recommendations. Descriptive or predictive: prevalence, disparities, risk prediction, discrimination (AUC, C-index, NRI) as the aim. This field is coded independently of strength. An associational wording with causal intent is the pattern the study measures, so code what you see. Use Unclear only when the signals truly conflict.
Polarity asserted / negated / hedged Asserted: stated as found. Negated: stated as absent. Hedged: stated with uncertainty (may, might, possibly, suggests, appears). A hedge about usefulness (“may serve as a marker”) does not hedge the finding itself.
Direction increases / decreases / null Outcome risk at higher or present exposure. Protective is Decreases. Use the estimate when one is given: a 95% CI that includes 1 (0 for β) is Null, whatever the wording. For a contrast (Q5 vs Q1, high vs low), follow the authors’ comparison.
Study design prospective cohort / retrospective cohort / nested case–control / other The design of the analysis that produced the primary finding. A cross-sectional analysis drawn from a cohort is Other. Registry or claims data followed backward in time is Retrospective.

Worked examples from the pilot set (these records are not in your calibration set):

PMID Primary finding Strength Intent Polarity Direction Design Why
42554187 Healthy lifestyle → CVD mortality in CKM syndrome associational causal asserted decreases prospective “Significantly associated with lower”; the call for “timely and targeted lifestyle interventions” signals causal intent.
42325531 CTI-based adiposity indices → incident CVD associational descriptive or predictive asserted increases prospective AUC, NRI and “supplementary risk marker” make prediction the aim.
42786666 SPISE index → incident CVD direct causal causal asserted decreases prospective “Inversely associated”, then “this protective effect”: the strongest wording is an unhedged effect. Hard case; flag it if you disagree.
42078342 Race and ethnicity → CVD after breast cancer associational descriptive or predictive asserted increases prospective “Disparities” describes differences between groups; no intervention on the exposure is proposed.
42333849 Ceramide scores (review) — — — — other Narrative review that quotes other studies’ HRs: finding = No.

A null illustration (invented sentence): “Coffee intake was not associated with incident stroke (HR 1.02, 95% CI 0.94–1.10).” → no relation, negated, null.

When unsure, choose your best answer, tick Flag for discussion and write one line in Note. Flags are discussed at the calibration call; they never count against you. Do not look up the full text, and do not discuss specific records with Matt until both of you have finished the calibration set.

DAGs, consensus and edge grades

You and Matt each draw a DAG for two prespecified pairs, independently. For the contested pair nobody has a correct DAG, so the study compares the literature-derived DAG with both of yours and with how far your two DAGs agree with each other (RQ5). Draw before you read any abstract, and you never see model output at any stage.

1. DAGs for both pairs (about 3 h, by November 10, before anything else)

Each pair comes with a fixed question, so that differences between your DAG and Matt’s reflect causal structure rather than different questions:

LDL cholesterol → ASCVD (positive control) Educational attainment → ASCVD (contested social pathway)
Exposure LDL cholesterol concentration measured in adulthood (higher vs lower) Educational attainment fixed by early adulthood (highest level completed or years of schooling)
Outcome Incident ASCVD: coronary heart disease, ischaemic stroke, peripheral artery disease Same
Population Adults free of ASCVD at baseline in population cohorts Same
Time zero Cohort baseline Cohort baseline in adulthood
Target Total effect of LDL cholesterol on ASCVD Total effect of educational attainment on ASCVD
  • Draw from your own knowledge. Do not read any pair abstract or ask Matt what the pipeline found, and do not discuss your DAGs with Matt until both of you have submitted.
  • Keep the exposure and outcome marked as in each template. Include confounders, mediators you think matter (for education, for example income, occupation, health behaviours, blood pressure, LDL cholesterol) and unmeasured common causes as latent nodes (for example childhood socioeconomic position or cognitive ability, if you think they matter).
  • Use short plain node names: age, sex, smoking, diabetes, SBP, BMI, statin use, Lp(a), parental education, income.
  • Draw on dagitty.net if you like, then copy the text under Model code into the DAGs tab; switch between the two pairs at the top. The tab shows a preview and counts nodes and edges.
  • Add a few lines in Notes on key assumptions and any node you were unsure about.
  • Press Submit DAG (twice, to confirm) for each pair, then upload the same code to your folder in the OSF project, so the time is recorded outside the page. You see Matt’s DAG for a pair once both of you have submitted it.

2. Consensus (calibration call Nov 17; main-set call Dec 5–7, about 1 h)

When Matt shares records, they appear in Compare and agree with both sets of labels side by side and the disagreements highlighted. Your labels on those records are then locked, so the originals stay as they were for the agreement statistics.

  • Go through the disagreements together on a video call; one of you saves the agreed value for each field.
  • Decide from the guideline (Guide for A, above), not from who is more senior. The agreed value can differ from both labels.
  • If you coded different findings (check the exposure and outcome rows), agree first on which finding the guideline’s rule selects.
  • Write one line in Reason whenever the guideline was unclear; those lines become guideline revisions.
  • The main-set call closes before Matt looks at any test-set model output.

3. Edge grades (about 1.5 h, December 8–11)

After consensus, Matt lists a subset of edges from both pairs in the Edge grades tab, each with the human-coded claims behind it. Grade each edge as you would for a DAG you trust; RQ2 compares your grades with the rule-based ones using weighted κ.

Grade Meaning
High Consistent results from trials or designs with strong identification (g-methods, MR, target trial emulation)
Moderate Two or more adjusted cohort studies, consistent direction, causal intent
Low A single study, or studies without confounder adjustment
Conflicting Precise results in opposite directions, or precise nulls alongside positive results

Also give each edge a status: present, absent (only if you would remove the arrow) or undetermined. A null counts as evidence against an arrow only when its 95% CI lies within HR or RR 0.90–1.11.

Schedule, blinding and credit

Two dates cannot move: your two DAGs by Nov 10, 2026, before the calibration round, and your main-set labels by Dec 4, 2026, the day the analysis plan is registered on OSF.

  • Sign in, submit both DAGs and upload them to OSF — by Nov 10, 2026
  • A and Matt code the 15 calibration abstracts — Nov 11–16
  • Calibration call, 30–45 min on video; guideline frozen afterwards — Nov 17, 2026
  • A codes the 45-abstract main-set sample — Nov 18 – Dec 4, 2026
  • Consensus call on the main-set disagreements — by Dec 7, 2026
  • A grades the listed edges — by Dec 11, 2026
  • A reads the preprint draft and confirms consent to post — by Dec 16, 2026

Blinding rules:

  • Both DAGs come before any abstract, and you see Matt’s DAG for a pair only after both of you have submitted it; the page enforces both.
  • Neither of you sees the other’s labels on a record until Matt shares it; the page enforces this too.
  • A never sees model output, and consensus on the main set closes before Matt looks at test-set output.
  • Outside the two calls, do not discuss specific records.

Credit: Matt will confirm before the calibration call whether you are offered co-authorship (CRediT roles: data curation, validation and methodology) or an acknowledgment. medRxiv requires every author’s consent to post, so a co-author also reviews the draft by December 16.