I was curious how well the latest LLMs could handle customs classification, so I made my own benchmark called BrokerBench. It was a fun project that taught me quite a bit about building AI evals.
It seems to me that the public needs more domain-specific evals—not only to choose models and weigh the tradeoffs among accuracy, cost, and speed, but also to decide when the models have become good enough to use credibly for a particular scenario.
A customs brokerage hobbyist
What got me interested in this?
I’ve worked in logistics since I was a baby, and although my current employer doesn’t do customs brokerage, that’s how I got my start.
When I got out of high school, I needed a part-time summer job. Living under the flight path of LAX, I stumbled into a driver job for a local company that turned out to be a customs broker. After driving for a few months, they figured out I wasn’t a dumb kid, and suddenly I found myself entry writing.
A few years later, I actually took the customs broker exam, which is notoriously difficult. I was a few points short of the passing score of 75 percent. Friends told me to challenge it and said they would probably give me a pass. But by then I had already decided to pivot into building software—a career that would last the next 34 years.
Most of that career was freight-related: TMS, WMS, and visibility apps. But there were a few projects that kept the customs dream alive. I’ve built software for US CBP AES, AMS and ISF, along with integrations with various customs SaaS products, most of which have been since acquired by Descartes.
So I guess you could call me a customs brokerage hobbyist.
Fast-forward to AI.
Why classification is hard
Classification has always been the black art of licensed customs brokers and trade attorneys. Determining the correct classification for a product sounds simple until you actually look at a page of the tariff schedule.
Here’s an example HTSUS page that applies to a men’s suit:
Seem a bit complicated? I hate to tell you but this is only the first of 27 pages of classifications for men’s suits. You would have to sort through all 27 pages of characteristics about material types, natural or synthetic fibers, or even whether the suit contains 15% or more by weight of down or water fowl plumage. I’ve heard that the apparel and textile classifications are so complex that some brokers specialize in classifying just those.
The headings in the HTS schedule cannot be read in isolation. Classification is governed by the General Rules of Interpretation and the legally controlling section, chapter, and subheading notes, while general notes and special provisions may further affect tariff treatment. Then there is the whole legal angle, where importers can request binding rulings from CBP about which HTS classification applies in a specific scenario.
Attorneys often get involved to make arguments, which means case law can also create subtle interpretations that matter to classification.
Even with that complexity, AI has come a long way. On the surface, it seems like this should be doable with the latest models, such as Claude Fable 5 or the GPT-5.6 family.
AI cannot be the legally accountable party for an HTS classification. The importer of record remains ultimately responsible for exercising reasonable care and for the correctness of the entry, even when a licensed customs broker assists. The broker also has independent professional and regulatory obligations. But AI should be able to dramatically streamline the work and provide the information necessary to make that determination.
That, my friends, is why I built a customs eval.
Building the benchmark
Running prompts against a collection of models and applying deterministic scoring is a fairly well-solved engineering problem. Designing a trustworthy benchmark is the hard part, and coding agents are pretty savvy about helping with the implementation. GPT-5.6 was my friend and did much of the heavy lifting.
I used OpenRouter as my gateway to whatever models I wanted to evaluate. Given that many model providers expose OpenAI-compatible APIs, it isn’t all that difficult to call model APIs directly and avoid OpenRouter’s credit-purchase fee. OpenRouter passes through the underlying model prices without a token markup, but currently charges a 5.5 percent fee when credits are purchased. At the moment, I’m not sure how many models I intend to include, so OpenRouter was a quick path to flexibility.
The cost of running the evals was a concern because I have to pay for the tokens. I was a bit worried about running the eval against frontier models. But that concern didn’t pan out because the eval turned out to be quite token-light.
Creating the ground truth
Creating the ground truth for the test questions is the real work for any benchmark. This is still in flux, and I don’t want to reveal too much about my methods because I don’t want the models to simply memorize the answers if they scrape this article or other content I publish before future testing runs.
Generally, I implemented a mix of multiple-choice questions, structured-answer questions, and HTS classification questions. I included broad coverage across domains that are typical topics in customs broker courses and on the official exams.
Because the Customs Broker License Examination answers are published after each test, I repurposed some of those questions with variations. I intend to run future exam questions and answers against the models so I can compare each model’s exam score.
I also created 15 classification questions based on the July 1, 2026, Revision 11 of the HTSUS. These questions provide fixed evidence from chapter notes and tariff text, then require the models to interpret that evidence and return an exact 10-digit HTSUS classification. This lets me test reasoning against recently published tariff material without depending on whatever may or may not have appeared in the models’ training data.
The benchmark is still a draft
The draft results and a collection of public samples are available for review. I’m keeping the complete scored corpus and answer keys private while I continue validating the benchmark. I say “draft” because I’m still reviewing it. I’m also reaching out to friends who are licensed brokers to have them validate the evals.
These are preliminary design-review results, not a definitive leaderboard. Each model has currently been run once against the corpus, and the questions have not yet completed independent licensed-broker review or formal psychometric validation.
Ultimately, I would love to have every eval question and answer vetted by a licensed broker to ensure accuracy. If you are a broker and would like to help, let me know at info@brokerbench.ai.
A few implementation lessons
The maximum output-token setting had a big impact on model performance. Some models were far more token-hungry and couldn’t really do well when the limit was set below 4,000 tokens.
Because my evals require structured output, bad JSON counted as a failed eval. I made a few tweaks to reduce this problem, and increasing the maximum token count helped, but it still happens.
I also noticed that sometimes the eval runs encounter infrastructure errors. In my initial version, a technical error on one or two questions would cause the entire model to be left out of the rankings. That forced me to build a function to rerun failed evals in isolation.
One thing that surprised me was how slow gpt-oss-120b was in this particular run. OpenRouter routed the published run through DekaLLM. It averaged about 17 seconds per question and ranked seventh out of the eight models, despite being extraordinarily inexpensive. That may say as much about the particular hosted endpoint as it does about the underlying model, so I wouldn’t generalize too much from one run.
Finally, I found that the Chinese open-source models did not perform well, including the new Kimi K3. I’m not sure whether that is because these models use less training data from U.S. government websites or because they generally don’t perform well on this type of task. Given that this is a U.S.-centric task anyway, I decided to focus on U.S.-based models, with the exception of the 32B Qwen model. That seems to be one of the popular models for fine-tuning with LoRA, so I was interested in seeing its performance as a baseline in case I want to experiment with fine-tuning at some point.