The Top Scorer on Korea’s Tax Accountant Exam Used AI. How Should We Judge AI at Work?

On November 15, 2026, the top scorer on South Korea’s tax accountant qualifying exam admitted using generative AI during the test. He had missed just one of the 160 questions in the first round. The next day, the National Tax Service began proceedings to invalidate his result and suspend his eligibility to sit the exam. His first-round score was 397.5 out of 400. It told us exactly how many questions he had answered correctly. It could not tell us which answers he had worked out himself. Reporting on the case · National Tax Service announcement

Why now? A public experiment using South Korea’s College Scholastic Ability Test (CSAT), held the same week, offers one clue. Two generations of AI took the same 507 questions from the 2027 admissions exam, released on November 19. GPT-5, a 2025 model, answered 436 correctly (86.0%). GPT-6.2 Astra, a 2026 model, answered all 507 (100%). That is a difference of 71 questions, or 14.0 percentage points. In Physics I, the older model answered 9 of 20 correctly and the newer model all 20. The researcher supplied 26 sets of questions as text and images, covering the Korean language and mathematics electives, English, Korean history, social studies, and science. Both models used the high reasoning setting and had no access to search or a calculator. Item-by-item scores released November 20 · Experiment design

Two models take the same CSATOf the same 507 questions, GPT-5 answered 436 correctly and GPT-6.2 Astra answered all 507 correctly.Two models take the same CSATAccuracy on 507 questions from the 2027 CSATGPT-5436 / 50786.0%GPT-6.2 Astra507 / 507100%0%50%100%

A perfect mathematics score was already possible a year earlier. In a study conducted just after the November 2025 CSAT, models in the GPT-5 family answered all 46 core and elective mathematics questions correctly under certain input conditions. What stood out this year was the breadth of the result. A single model earned full marks across subjects from Korean to science. When AI answers were often wrong, choosing which ones to use still required judgment from the student. As errors become rare, an answer sheet reveals less about the student’s own ability. The Ministry of Education recorded 111 exam violations involving electronic devices in this year’s CSAT. Five involved using a device during the exam. Those figures do not tell us how many students submitted AI-generated answers. Study of the November 2025 CSAT mathematics exam · Ministry of Education figures on exam violations

From Turing’s imitation game to professional exams

In 1950, Alan Turing proposed exploring whether machines could think through a conversation experiment. In what is now known as the Turing test, a judge exchanges written messages with two participants whose identities are hidden. One is human and the other is a machine. All the judge sees is their replies. Turing’s 1950 paper

The Turing test A judge exchanges written messages with a human and a machine without knowing which is which. The judge compares their replies to identify the human. The Turing test One is human. One is a machine. Player A Player B Ask in writing Reply in writing Judge Compares replies to identify the human.

When IBM’s Deep Blue played world chess champion Garry Kasparov in 1997, the rules of chess determined the winner. In the Turing test, a person read the conversation and judged the identity of the participants. In chess, the match result showed who played better. IBM’s account of the Deep Blue match

As language models became more widely used, researchers needed a way to test their knowledge across many fields at once. MMLU, introduced in 2020, used questions with four answer choices in 57 subjects, including mathematics, law, and medicine. The model chose one answer without consulting outside sources or asking the evaluator a follow-up question. Reading an unfamiliar question and marking one of the options resembles the multiple-choice sections of the CSAT and professional qualifying exams. Scoring one response to a question in this way is often called single-turn evaluation. MMLU research paper

Professional exams show a similar shift. Law&Company tested its legal AI service, SuperLawyer, on the multiple-choice section of the Korean bar exam. It answered 123 of 150 questions correctly (82%) on the 14th exam in 2025 and all 150 (100%) on the 15th in 2026. Its score of 375 on the latter section exceeded the 360 earned by the candidate who ranked first overall that year. These were the developer’s tests of a service that combines a model with legal resources, using two different editions of the exam. A separate issue came before the Ministry of Justice in November. It discussed a possible investigation into suspected AI use in written exam answers from the previous five years. SuperLawyer announcement · Reporting on the Ministry of Justice discussions

In an exam room, it matters whether the candidate did the work. At a company, the question changes. If AI can handle some tasks as well as a specialist, where should we use it, and which decisions should a person remain responsible for? OpenAI’s GDPval examined actual work products from 44 occupations. Specialists compared work by humans and AI without knowing who had produced each submission. In results released in March 2026, they rated GPT-5.4’s work as good as or better than the professional’s work in 83.0% of comparisons. The evaluation covered a single submission against a defined brief. A company that wants AI to revise the brief after talking with a client, or decide how to handle an exception at final approval, needs to test those tasks itself. GDPval methodology · GPT-5.4 results

When an agent reports success, is the work done?

An agent uses a model’s responses to read files, run programs, and decide what to do next based on the results. SWE-bench gives an agent an issue reported in a real software repository, then runs tests against its code changes. Terminal-Bench 2.0 provides an execution environment and automated checks for 89 tasks performed in a terminal. If the agent says it fixed a problem and the same error appears on the next run, the problem is still there. SWE-bench · Terminal-Bench research paper

We measured this distinction in our own company. We created 30 synthetic cases across five tasks, including aggregating data and reconciling files. We ran each case three times with the same model, Qwen3.5 9B, and the same agent, OpenCode.

What we checkedPassed / totalRate
Runs in which the program exited normally88 / 90 runs97.8%
Runs that passed every check on reading the input, the values in the output file, and the format of the final response72 / 90 runs80.0%
Cases that passed on all three runs23 / 30 cases76.7%

Aggregating data from one CSV file passed on all 18 runs. Matching identifiers across two files passed on only 4 of 18. In some runs, the agent read both files but left out identifiers or changed the order in which they appeared in the source.

This is where human review becomes difficult. AI can finish the CSAT faster than a person, and a single answer key is still enough to mark it. Finding one missing identifier, however, can mean that a reviewer has to reconcile the original files all over again. As tasks grow longer, reviewing the result can take as much effort as doing the work from scratch. A program can check every value and its position. A separate evaluator agent can read the source and the execution log to flag important exceptions that the written explanation left out. A human reviewer then looks for mistakes that the evaluators missed. Reviewing a sample of automatic passes is how we estimate how often an incorrect result was accepted. Evaluating an agent’s outputs and actions

Evaluating a file reconciliation task An agent matches identifiers across two files and produces an output file and an execution log. A program compares the output with the source files to find missing or duplicate identifiers. An evaluator agent examines how exceptions were handled. A person reviews all cases where the checks disagree and a sample of automatic passes. How we check an agent’s work Match IDs across two files The agent runs the tools It saves the output and log Check the output Check against sourcefor gaps or repeats Check the log An evaluator checksexception handling Human review All conflicts and a sample of passes
An example of how to evaluate an agent that reconciles two files.

With the same model, a practitioner’s experience can change the result

One instruction in our skill for reviewing user interfaces says to change a setting, open the activity log before saving, then return to the settings tab. The screen may look fine when it first opens, yet this sequence can make an unsaved value disappear. Someone who has seen the bug knows where to click the next time they review the interface. Once they record the steps and the expected result, an agent can repeat the check. A skill packages reusable procedures like this with the tools needed to carry them out.

Turning experience into a repeatable test A user changes a retention setting to 30 days and visits the log tab before saving. On returning, the field is empty. When this experience is recorded as a skill, an agent can repeatedly enter a value, switch tabs and back, and check whether the original value remains. The screen and value are illustrative. Turning experience into a test After editing Back from the log Setup Log Setup Log Retention period Retention period 30 d Empty Steps saved in the skill Enter valueSwitch tabsCheck value Check the value is still 30 days.
In this example, the check passes only if “30 days” is still in the field after returning from the log tab.

A skill’s author and the way it is written can also affect performance. SkillsBench researchers tested the same Claude Code agent and Opus 4.7 model on 87 tasks. The pass rate was 43.0% without skills, 61.2% with expert-written skills, and 34.9% when the agent wrote its own skills. With the same model and expert-written skills, a different agent, OpenHands, achieved 53.1%. Having the agent write its own procedures reduced the pass rate by 8.1 percentage points compared with running without skills. Two people using the same model can get different results depending on which failures they recognize and whether they capture that experience in a procedure the agent can repeat. SkillsBench paper, Tables 1 and 6

Task pass rates by skill setupWith the same model and agent, the task pass rate was 43.0% without skills, 61.2% with expert-written skills, and 34.9% with agent-written skills.Task pass rates by skill setupSkillsBench · Claude Code + Opus 4.7No skills43.0%Expert-written skills61.2%Agent-written skills34.9%0%50%100%

What should we measure on company data?

We are preparing an evaluation system to compare language model (LLM) × agent × skill combinations. We want to choose models suited to our work and test whether employees can use the results in practice. Our file experiment used synthetic data and one combination. The next step is to collect requests that our company has handled, the original materials, and the final versions corrected by staff, then give the same work to several combinations. For a monthly aggregation task, for example, we can compare a general-purpose model with one tailored to our work, switch agents for each model, and run each pairing with and without an employee’s skill. We need to set aside data from a month the skill author has never seen to learn whether the improvement holds on fresh material.

What we want to knowWhat we would measure at work
Was the task completed accurately?The share of all requests for which the figures match the source and every required item was delivered. Count missing items and incorrect values separately.
Will it pass again?The share of cases that pass all three runs when each case is repeated three times. Calculate this for each type of work to find the tasks that fail most often.
Did it save employees work?The share of results employees used as delivered and the time they spent making corrections. Compare with the time needed for the same type of work without AI.
Did it stop when it should?The share of requests requiring approval that stopped before execution, and the number of actions that exceeded the agent’s permissions. Also measure how often the agent unnecessarily stopped an authorized request.
What does one successful task cost?Add the costs of failed runs and retries, then divide the total by the number of passes. Measure elapsed time from the request until a person can use the result.

These measurements should determine which permissions an agent gets. If totals in a summary table match the source, we can automate the draft. If a reconciliation task repeatedly drops identifiers, we can add a dedicated checker and keep a person reviewing the results until the combination passes the same cases again. For an action that requires approval, such as sending material outside the company, the tool that performs the action should check for approval before executing it. A combination that bypasses approval even once during testing should not receive that permission. This gives us a way to decide both which model to buy and what work to entrust to it.

A score on 507 CSAT questions cannot tell me whether a model will spot a missing row in one of our summary tables. The person who knows whose work that missing row will affect is the employee who uses the table today. I think this is where differences in the performance of personalized AI will grow. A company that captures its employees’ experience of failure in skills and tests those skills on the next month’s unfamiliar data can decide for itself what else to entrust to each new model.

답글 남기기

이 사이트는 Akismet을 사용하여 스팸을 줄입니다. 댓글 데이터가 어떻게 처리되는지 알아보세요.