On November 15, 2026, the top scorer on South Korea’s tax accountant qualifying exam admitted using generative AI during the test. He had missed just one of the 160 questions in the first round. The next day, the National Tax Service began proceedings to invalidate his result and suspend his eligibility to sit the exam. His first-round score was 397.5 out of 400. It told us exactly how many questions he had answered correctly. It could not tell us which answers he had worked out himself. Reporting on the case · National Tax Service announcement
Why now? A public experiment using South Korea’s College Scholastic Ability Test (CSAT), held the same week, offers one clue. Two generations of AI took the same 507 questions from the 2027 admissions exam, released on November 19. GPT-5, a 2025 model, answered 436 correctly (86.0%). GPT-6.2 Astra, a 2026 model, answered all 507 (100%). That is a difference of 71 questions, or 14.0 percentage points. In Physics I, the older model answered 9 of 20 correctly and the newer model all 20. The researcher supplied 26 sets of questions as text and images, covering the Korean language and mathematics electives, English, Korean history, social studies, and science. Both models used the high reasoning setting and had no access to search or a calculator. Item-by-item scores released November 20 · Experiment design
A perfect mathematics score was already possible a year earlier. In a study conducted just after the November 2025 CSAT, models in the GPT-5 family answered all 46 core and elective mathematics questions correctly under certain input conditions. What stood out this year was the breadth of the result. A single model earned full marks across subjects from Korean to science. When AI answers were often wrong, choosing which ones to use still required judgment from the student. As errors become rare, an answer sheet reveals less about the student’s own ability. The Ministry of Education recorded 111 exam violations involving electronic devices in this year’s CSAT. Five involved using a device during the exam. Those figures do not tell us how many students submitted AI-generated answers. Study of the November 2025 CSAT mathematics exam · Ministry of Education figures on exam violations
From Turing’s imitation game to professional exams
In 1950, Alan Turing proposed exploring whether machines could think through a conversation experiment. In what is now known as the Turing test, a judge exchanges written messages with two participants whose identities are hidden. One is human and the other is a machine. All the judge sees is their replies. Turing’s 1950 paper
When IBM’s Deep Blue played world chess champion Garry Kasparov in 1997, the rules of chess determined the winner. In the Turing test, a person read the conversation and judged the identity of the participants. In chess, the match result showed who played better. IBM’s account of the Deep Blue match
As language models became more widely used, researchers needed a way to test their knowledge across many fields at once. MMLU, introduced in 2020, used questions with four answer choices in 57 subjects, including mathematics, law, and medicine. The model chose one answer without consulting outside sources or asking the evaluator a follow-up question. Reading an unfamiliar question and marking one of the options resembles the multiple-choice sections of the CSAT and professional qualifying exams. Scoring one response to a question in this way is often called single-turn evaluation. MMLU research paper
Professional exams show a similar shift. Law&Company tested its legal AI service, SuperLawyer, on the multiple-choice section of the Korean bar exam. It answered 123 of 150 questions correctly (82%) on the 14th exam in 2025 and all 150 (100%) on the 15th in 2026. Its score of 375 on the latter section exceeded the 360 earned by the candidate who ranked first overall that year. These were the developer’s tests of a service that combines a model with legal resources, using two different editions of the exam. A separate issue came before the Ministry of Justice in November. It discussed a possible investigation into suspected AI use in written exam answers from the previous five years. SuperLawyer announcement · Reporting on the Ministry of Justice discussions
In an exam room, it matters whether the candidate did the work. At a company, the question changes. If AI can handle some tasks as well as a specialist, where should we use it, and which decisions should a person remain responsible for? OpenAI’s GDPval examined actual work products from 44 occupations. Specialists compared work by humans and AI without knowing who had produced each submission. In results released in March 2026, they rated GPT-5.4’s work as good as or better than the professional’s work in 83.0% of comparisons. The evaluation covered a single submission against a defined brief. A company that wants AI to revise the brief after talking with a client, or decide how to handle an exception at final approval, needs to test those tasks itself. GDPval methodology · GPT-5.4 results
When an agent reports success, is the work done?
An agent uses a model’s responses to read files, run programs, and decide what to do next based on the results. SWE-bench gives an agent an issue reported in a real software repository, then runs tests against its code changes. Terminal-Bench 2.0 provides an execution environment and automated checks for 89 tasks performed in a terminal. If the agent says it fixed a problem and the same error appears on the next run, the problem is still there. SWE-bench · Terminal-Bench research paper
We measured this distinction in our own company. We created 30 synthetic cases across five tasks, including aggregating data and reconciling files. We ran each case three times with the same model, Qwen3.5 9B, and the same agent, OpenCode.
| What we checked | Passed / total | Rate |
|---|---|---|
| Runs in which the program exited normally | 88 / 90 runs | 97.8% |
| Runs that passed every check on reading the input, the values in the output file, and the format of the final response | 72 / 90 runs | 80.0% |
| Cases that passed on all three runs | 23 / 30 cases | 76.7% |
Aggregating data from one CSV file passed on all 18 runs. Matching identifiers across two files passed on only 4 of 18. In some runs, the agent read both files but left out identifiers or changed the order in which they appeared in the source.
This is where human review becomes difficult. AI can finish the CSAT faster than a person, and a single answer key is still enough to mark it. Finding one missing identifier, however, can mean that a reviewer has to reconcile the original files all over again. As tasks grow longer, reviewing the result can take as much effort as doing the work from scratch. A program can check every value and its position. A separate evaluator agent can read the source and the execution log to flag important exceptions that the written explanation left out. A human reviewer then looks for mistakes that the evaluators missed. Reviewing a sample of automatic passes is how we estimate how often an incorrect result was accepted. Evaluating an agent’s outputs and actions
With the same model, a practitioner’s experience can change the result
One instruction in our skill for reviewing user interfaces says to change a setting, open the activity log before saving, then return to the settings tab. The screen may look fine when it first opens, yet this sequence can make an unsaved value disappear. Someone who has seen the bug knows where to click the next time they review the interface. Once they record the steps and the expected result, an agent can repeat the check. A skill packages reusable procedures like this with the tools needed to carry them out.
A skill’s author and the way it is written can also affect performance. SkillsBench researchers tested the same Claude Code agent and Opus 4.7 model on 87 tasks. The pass rate was 43.0% without skills, 61.2% with expert-written skills, and 34.9% when the agent wrote its own skills. With the same model and expert-written skills, a different agent, OpenHands, achieved 53.1%. Having the agent write its own procedures reduced the pass rate by 8.1 percentage points compared with running without skills. Two people using the same model can get different results depending on which failures they recognize and whether they capture that experience in a procedure the agent can repeat. SkillsBench paper, Tables 1 and 6
What should we measure on company data?
We are preparing an evaluation system to compare language model (LLM) × agent × skill combinations. We want to choose models suited to our work and test whether employees can use the results in practice. Our file experiment used synthetic data and one combination. The next step is to collect requests that our company has handled, the original materials, and the final versions corrected by staff, then give the same work to several combinations. For a monthly aggregation task, for example, we can compare a general-purpose model with one tailored to our work, switch agents for each model, and run each pairing with and without an employee’s skill. We need to set aside data from a month the skill author has never seen to learn whether the improvement holds on fresh material.
| What we want to know | What we would measure at work |
|---|---|
| Was the task completed accurately? | The share of all requests for which the figures match the source and every required item was delivered. Count missing items and incorrect values separately. |
| Will it pass again? | The share of cases that pass all three runs when each case is repeated three times. Calculate this for each type of work to find the tasks that fail most often. |
| Did it save employees work? | The share of results employees used as delivered and the time they spent making corrections. Compare with the time needed for the same type of work without AI. |
| Did it stop when it should? | The share of requests requiring approval that stopped before execution, and the number of actions that exceeded the agent’s permissions. Also measure how often the agent unnecessarily stopped an authorized request. |
| What does one successful task cost? | Add the costs of failed runs and retries, then divide the total by the number of passes. Measure elapsed time from the request until a person can use the result. |
These measurements should determine which permissions an agent gets. If totals in a summary table match the source, we can automate the draft. If a reconciliation task repeatedly drops identifiers, we can add a dedicated checker and keep a person reviewing the results until the combination passes the same cases again. For an action that requires approval, such as sending material outside the company, the tool that performs the action should check for approval before executing it. A combination that bypasses approval even once during testing should not receive that permission. This gives us a way to decide both which model to buy and what work to entrust to it.
A score on 507 CSAT questions cannot tell me whether a model will spot a missing row in one of our summary tables. The person who knows whose work that missing row will affect is the employee who uses the table today. I think this is where differences in the performance of personalized AI will grow. A company that captures its employees’ experience of failure in skills and tests those skills on the next month’s unfamiliar data can decide for itself what else to entrust to each new model.
