The whole program,
laid out end to end

Students earn points (“XP”) for work done, and each day has a goal. The experiment designer calls this a blocker, because every threshold in the system is set in points. Under today’s values a beginner reaches the daily goal in 4–6 minutes of real work and an advanced student in 16–20. If students stop at the goal, 18 lessons could add up to as little as 90 minutes of actual teaching, with more than half of it spent on testing. The intended mix of new learning, review and writing cannot fit inside the goal.

The latest experiment design drops the idea of giving both comparison groups identical practice time and instead gives them identical software settings, fingerprinted every lesson. That also narrows what we can claim afterwards, from “teaching in skill order works, at equal dose” to “this whole policy, as deployed, works”. Everything downstream — what counts as the trial running correctly, the pre-registered plan — waits on this.

Open from the start, and more important now that the two groups will spend different amounts of time. One document says the 28-minute placement test and the checking-up minutes count; the experiment’s analysis plan excludes them. Students are treated as idle after 60 seconds of inactivity in one document and 90 in the other. Both are in force.

Eight essays in the grader training set (S07, S16, S20, S21, S26, S30, S32, S36) where both AI graders disagreed with the official answer in the same way. They are held out of training until a person decides who is right. This holds up the training set, which holds up training the graders, which holds up the grading study, which holds up everything.

The founding plan still says the AI scorer must agree with human graders at 0.80 on each separate quality. Measurement shows two trained humans manage about 0.5; one of our own reports says 0.80 is unreachable by these methods; the database documentation still cites 0.80 as the rule. Four documents, three positions. Proposed replacement: keep 0.80 for the overall grade against the average of two humans; for each quality, require the AI to match humans at least as well as humans match each other; and require it to hold up under attack.

“Write down the success measure and the analysis plan before any student is enrolled” appears in two places as a hard requirement. Nobody has signed it. It must include follow-up writing checks at 30, 60 and 90 days after the month ends — inside a one-month module, the “spaced” reviews come round after only about a day and a half on average, which tells us nothing about lasting learning.

Recording whether a student planned before drafting, or named a goal, needs new screens — and that changes the writing part of the lesson, which is about to be frozen for the trial. So it has to be decided before the freeze, not after. The research side’s own view is that counting revisions and timing the first sentence is enough for the first trial.

“Skill A must come before skill B” is really three separate claims — you can’t do B without A; teaching A first is faster; knowing A predicts B — and each needs reporting on its own. No format for that exists, so results about skill order have nowhere to be recorded. Related: two teachers were asked to review the links and have not replied, and the six links they flag are supposed to replace the six the research plan is currently built around.

Students in the placement-test group finish the main material sooner and get extension work. Either cap how much extension work they get, or say openly that the trial tests the whole package (order plus extras), not order alone. Either way, no extension work exists in the software yet.

The paper answer sheet prints the student’s ID code in a box, and a photograph of it can contain a name the student wrote there. The tool that strips names before anything reaches the AI cannot read inside a photo. Either someone types up handwritten work (and strips names) first, or handwriting stays out of the AI path until it can be checked.

Background: from 2027 a federal tax credit rewards donations to approved scholarship organisations, which could fund families’ places. Closed: the foundation we work with cannot be one of those organisations as it is set up — it is a private operating foundation, and “operating” is a subtype, not an exemption. Converting it is advised against. Open: Route A (register VPA as an eligible school — free, do it regardless), Route B (partner with ACE Scholarships, an existing Denver organisation — one phone call, but they choose who gets funded), Route C (a Vail Valley community scholarship fund — only once Treasury finalises its rules on related parties).

Four things interact: VPA’s nonprofit status, bridge loans from private backers, a software platform that could have investor-style returns, and a related scholarship body that might fund VPA’s own students. The recommendation is one session with a nonprofit tax attorney covering all four together rather than four separate conversations.

The academic labs we want to hire graders from may well answer a request for paid hours with an offer to collaborate and co-write a paper. If we want paid contract hours, say so explicitly in the first email. The people we are targeting are exactly the ones who would want to publish about this — which is also why the contractor agreement needs a clear transfer of intellectual property and a no-outside-publication clause.

Three different jobs, and mixing them up is the mistake to avoid. $30–40 an hour for essay graders is generous — the big testing companies pay $15–25. But $30–50 is below market for expert advisers: budget $50–75 an hour, or flat fees of $1,500–5,000. The outreach emails still have placeholder brackets where the rate should be.

$500–2,000 per student per semester fits inside state education-savings-account awards and inside a scholarship pool that pays out in $1,700 units, and needs no purchase order or public tender. The open question worth real money: does an accredited course that earns credit qualify for a better-funded bucket than “tutoring”?

Federal student-privacy law (FERPA) lets a school share student records with people doing work for the school — but the school designates them; we cannot designate our own contractors. Removing names is enough for grading. But to compare a student’s first essay with their essay weeks later — which the success measure requires — we need the law’s formal research exception, granted by the school. Start now; it will take longer than hiring. Contacts: Cat, Jared, Meg.

It names who counts as working for the school, what the data may be used for, bans passing it on or using it to train anyone’s AI, and sets how long it is kept and how it is deleted. Until it exists, no student account should be switched on. Alongside it: the published privacy notice with the retention periods word for word, and the written parent-consent process.

He built the three essay datasets our whole scoring plan rests on. His lab’s jobs page openly offers paid student positions with no formal application; the lab recently moved from Georgia State and is rebuilding, so there is less competition for student hours. It also publishes on bias in AI writing feedback. scott.crossley@vanderbilt.edu

The best source of graders in Colorado: she runs English Education and the master’s program whose students are current and future ninth-grade English teachers — exactly the grader we want. Two hours from Edwards, so training sessions can be in person. stacy.bailey@unco.edu

Co-created one of the first automated essay scorers and ran large-scale automated scoring at Pearson. The most credentialed person in Colorado on this exact problem, with the industry background to understand a paid consulting request without a research grant attached. Ask him about making the scorer hard to fool. peter.foltz@colorado.edu

Directs a doctoral program that trains people in exactly what we need: grader agreement, whether a test measures what it claims, and fairness across groups of students. One email to the director reaches the whole cohort. Also there: William Penuel, who studies how schools and researchers work together. derek.briggs@colorado.edu

Over $3.5M in Gates, NSF and Spencer funding to study whether automated writing feedback actually improves writing. His research question is our biggest risk. Small group, fast personal replies. joshwils@udel.edu

She built Writing Pal, the closest thing to this product that has ever existed, including its own hard lessons about drills that did not carry over into real writing. We have already rebuilt 28 of its measurements and found where they fail. Also at ASU: Maria Goldshtein, a postdoc in fair and inclusive language analysis — a near-exact match for our fairness-reporting requirement.

Four things to get: the cost trade-off between a purpose-trained scoring model and calling a large AI model each time, at our volume (our open question, and he has solved it for a consumer product at scale); whether two people he knows would review our scoring design — we have no machine-learning adviser and fair, human-matching scoring is a real machine-learning problem; what he learned from 100+ interviews about why people don’t write; and how to reach customers. Be clear that his measure of success is how much people use the app, and ours can never be — an investor will mix the two up.

Denver-based, already running in Colorado, and set up to receive the tax-credited donations. Supporters donate and take the credit; VPA families apply. No new legal entity, no spending rules to police, no annual state certification. The trade is that ACE decides who gets funded. Must happen in 2026, before the credit starts on 1 January 2027.

The foundation cannot become a scholarship organisation as it is set up, and converting it is an expensive way to get a worse version of the ACE partnership. This needs to go back to them as a finding, not sit in a document. Also confirm whether it files the private-foundation tax return (Form 990-PF) — the definitive check. It could donate to a future scholarship body, but a private foundation giving to an organisation it effectively controls raises self-dealing questions.

Nonprofit tax: legal structure, bridge loans, investor-style platform economics and a related scholarship body, covered together. Privacy and education: review the database design and privacy document, rule on whether Colorado’s student-data act adds anything beyond the federal laws, review the contractor agreement once so it can be reused, and confirm the dataset licences before anything ships.

The AI provider does not train on customer data by default, but whether we qualify for “keep nothing” data handling is unconfirmed. Get it in writing and attach it to the data agreement as the record of who processes student data on our behalf, before the trial.

Still outstanding. Six of the “this skill comes before that one” links are flagged as doubtful and undecided, and the six the research plan currently tests were chosen by looking at the map’s shape, not by anyone who teaches ninth grade. The form exists and is ready to hand over.

The tool is live with 24 sample essays: teachers grade blind, then see the AI’s grade, then say whether the AI’s feedback helps, should wait for a teacher, or should be shown to the student. Teacher-vs-AI and teacher-vs-teacher agreement are computed automatically. Grows to 50–100 teachers. This is the only data we will ever have on whether teachers find the feedback appropriate.

Confidentiality that respects student-privacy law, a clear transfer of intellectual property, no outside publication, and safeguards for treating people as contractors rather than employees. A lawyer reviews it once; reuse it forever. Three things that will bite: international students on F-1 visas need written permission from their university before signing, funded research assistants have caps on outside work, and a tax form (W-9) is needed before the first payment.

The organisation that now looks after two of our main datasets, and the source of the licence discrepancy we found. Worth a conversation regardless. Perpetual Baffour there works on bias in automated essay scoring. info@the-learning-agency.com