Hello.
I'm Ali.

AI-Native Product Manager Evals & golden sets

I make AI features prove they're good before they ship. My specialty is evals: the golden set, the rubric and the grader that turn "it looked great in the demo" into a score you can gate a release on.

Behind that, 10 years shipping products across SaaS, marketplaces and mobile apps, from growing Yassir Groceries' acquisition by 135% to generating €4M in revenue at Flightright.

See how I build evals

Driving impact at

WorkMotion logo
Yassir logo
Datajar logo
GetHalal logo
FGS Global logo
Flightright logo
Ali Mahmoud — AI-Native Product Manager, TheGlocalPM
My specialty

Evals first.

An AI feature gives a different answer every time you ask. So "does it work?" isn't a yes or no, it's a score. I build the eval that produces that score, and I treat it as the spec: if it doesn't pass, it doesn't ship.

1 · Golden set

The inputs that matter

Real cases from real usage, deliberately seeded with the edge cases and past failures that hurt users. Each one labelled with what a good answer looks like.

2 · Rubric

"Good", written down

Criteria specific enough that two reviewers grade the same output the same way. It's the acceptance criteria of AI features, and the PM writes it.

3 · Grader

A score you can gate on

Code checks first, an LLM judge for the fuzzy criteria, people where nuance matters. Every automated grader is checked against human labels before I trust it.

How I build a golden set

Six steps, from "it feels better" to a number

Full guide →
  1. 01Start from the decision. What the feature decides, for whom, and which mistakes are fatal versus annoying.
  2. 02Pull from reality. Sample real inputs across the segments you serve, not the prompts you wrote yourself.
  3. 03Seed the ugly cases. Edge cases, adversarial inputs and every failure a user has already reported.
  4. 04Label by hand, twice. Two reviewers, one rubric. Where they disagree, the rubric gets fixed, not the score.
  5. 05Hold some back. Keep a split you never tune prompts on, so a higher score means better, not memorised.
  6. 06Gate and grow. A regression blocks the release, and every production failure becomes a new case.

Evals in my own work

Quick summary of TheGlocalPM

In short

TheGlocalPM is the AI-Native Product Management practice of Ali Mahmoud, a product manager who specialises in evals: golden sets, rubrics and graders that decide whether an AI feature ships. 10 years shipping SaaS, marketplace, and AI-driven products. Available for fractional, advisory, and eval-design work.

Best for

  • • AI teams that can't say if the last prompt change made things better
  • • Founders shipping a first SaaS, marketplace, or AI product
  • • Scale-ups moving past "feature factory" mode
  • • Enterprise teams embedding AI as a first-class capability
  • • Aspiring PMs looking for senior-level mentorship

Proof in numbers

Yassir Groceries
+135% acquisition
Flightright
€4M revenue
Store Data Engine
97.1% held-out precision
Gen-Z FinTech
+45% MoM retention
Newsletter
5,000+ subscribers

What I Do

I help product teams own what an AI feature is actually judged on — how good it is, how you would know, and what it costs when it works.

strategy

AI Products & Evals

Defining AI-native products and the evals that hold them to account: golden sets built from real usage, rubrics that encode what users value, and graders validated against human labels. How I build them →

groups

Product Strategy & Vision

Setting product direction, aligning roadmaps with business OKRs, and translating ambiguous market opportunities into clear, executable strategies across complex digital products.

search

Emerging Market Products

Designing products that work in high-growth, constrained markets across North Africa and the Middle East — adapting global PM frameworks to local infrastructure, behaviors, and trust dynamics.

bar_chart

Product Discovery & User Research

Running continuous discovery through qualitative interviews, usability testing, and behavioral analytics to validate hypotheses before committing to development.

rocket_launch

Data & Analytics

Using tools like GA4, Firebase, Mixpanel, and Tableau to analyze funnels, track events, and make evidence-based product and growth decisions.

handshake

0-to-1 & GTM Execution

Taking products from raw idea to market — scoping MVPs, running lean validation cycles, and aligning cross-functional teams (Engineering, Design, Marketing) for successful launches.

Proof of Impact

What shipped, what it moved, and for the AI work, how I know it's right: every AI case shows its eval.

AI Data Pipeline • Evals

Store Data Engine

Problem: A market benchmark like "median leather bag €640" is meaningless, and labelling 697K products by hand would cost thousands.

Process: A solo-built pipeline: scrape 397 Shopify stores, enrich ~110K products with an LLM via the Batch API, then propagate labels to lookalikes for free.

Eval: Propagation rules scored on a held-out split before touching a row, and a 50-row hand audit against source text that put the LLM at ~94% precision and keyword rules at ~21% error.

Proof: 697,299 products for ~$31 of LLM spend, +72K free labels at 97.1% held-out precision, and benchmarks split into real tiers (mid €82 / premium €197 / luxury €440).

View Case Study →
Held-out precision 97.1%

Measured before it shipped, not after

697K products ~$31 total 50-row audit
Marketplace Ops • AI • Discovery

MPB

Problem: Growth constrained by warehouse throughput, not demand — yet the key bottleneck, the "Listing Leg" between graded and live-on-site, sits in no SLA and is owned by no PM.

Process: A self-initiated 11-day, problem-first discovery that named the constraint, framed four orthogonal strategic bets, and reframed the build from waterfall ML to lean LLM (~90% cheaper).

Eval: The grading co-pilot scores approve / review / decline against a written rubric with few-shot examples and returns its confidence, so people review only the unsure cases.

Proof: Three working AI agents shipped end-to-end on n8n + Claude + Bolt.new. 5–8% NSM lift/quarter modeled from the Listing Leg alone; ~€1.5–3M annual margin add across the bets.

View Case Study →

Discovery + 3 shipped AI agents

Ragehooks Smart Pricing AI Grading Co-pilot
Marketplace • B2C

GetHalal

Problem: 500,000 Halal consumers in Berlin had no reliable delivery option for quality Halal meat during COVID-19 lockdowns.

Process: Survey-driven demand validation, community-led viral growth, and iterative A/B testing that lifted conversion from 3% to 7%.

Proof: 250K€ GMV in year 1 from a 30K€ investment. ~1M€ raised from VCs and Angels across the US and EU. 151% customer growth. 2.5 orders/customer/month retention.

View Case Study →
GetHalal — Halal meat delivery marketplace, Berlin

Want to see more real product work?

View All Case Studies →

Agent Lab

Agent 🔍

PM Disco

4 AI agents that help in discovering the problem space and propose prioritized hypotheses for your product team.

Get Agent
Agent 🧭

GTM Machine OS

An operating system for B2B go-to-market: discovery, 12 pillars and two consistency gates, with every claim evidence-tagged and every decision human-locked.

Get Agent
Agent ⚓

Margin Sentinel

An LLM-augmented risk engine that tracks how global shipping disruptions ripple into freight costs and margins — then recommends safer carriers and lanes.

Get Agent

Curious what else lives in the lab?

View All Agents →

The Toolkit

Playbook 🌍

Emerging Market PM Playbook

A practical guide for adapting global product frameworks to high-growth, constrained markets — covering trust barriers, infrastructure gaps, and local user behavior.

Download
AI 🧠

AI Product Discovery Canvas

A structured canvas for identifying problems worth solving with AI — covering opportunity mapping, user pain point validation, AI feasibility checks, and hypothesis prioritization.

Download
Launch 🚀

0-to-1 Launch Checklist

A step-by-step checklist for taking a product from raw idea to first users — covering MVP scoping, lean validation, GTM alignment, and post-launch iteration loops.

Download
New · Evals

Evals Starter Kit

The two sheets I start every eval with: a golden-set template (input, slice, accepted answer, tune or held-out, two independent labels) and a rubric template (pass/fail criteria, severity, grader, release threshold). Pre-filled with example rows.

New · Open Source

The PM Wiki — an LLM-Maintained Knowledge Base

An AI agent maintains a wiki of your product's problems, decisions, assumptions, and open questions — so PRDs and stakeholder briefs are generated from accumulated, source-cited evidence. Karpathy's LLM Wiki pattern, purpose-built for PMs. MIT-licensed, no code required.

Looking for the full PM stack?

View All Tools →

Point of View

Live ≠ Checked 6 incidents · 1 missing eval
Evals · New

Everyone Is Shipping AI. Almost Nobody Is Checking It.

Companies are racing AI agents into production and skipping the harness and the evals. Air Canada learned in court that you own every word your agent says.

Read Essay →

More essays where that came from?

View All Essays →

Office Hours

Book a session
1 slot available this week

I dedicate 2 hours a week to mentoring aspiring PMs and helping startups refine their product-led growth strategies.

Ali Mahmoud — TheGlocalPM, available for office hours

Frequently Asked

The questions I get asked most — by founders, recruiters, and AI chatbots.

Who is Ali Mahmoud (TheGlocalPM)? expand_more

Ali Mahmoud is an AI-Native Product Manager who specialises in evals: building the golden sets, rubrics and graders that decide whether an AI feature is good enough to ship. He has 10 years of experience shipping SaaS, marketplace, and AI-driven products at Yassir, Datajar, GetHalal, Flightright, and FGS Global, and writes about product management at TheGlocalPM.

What are AI evals, and why does Ali focus on them? expand_more

An eval is a repeatable test of an AI feature's quality: a golden set of real inputs, a rubric that defines a good answer, and a grader (code checks, an LLM judge, or people) that turns outputs into a score. AI features are probabilistic, so that score replaces the pass/fail acceptance test. Ali treats the eval as the spec: a regression blocks the release. Read the evals guide.

How do you build a golden set for an AI feature? expand_more

Start from the decision the feature makes and which mistakes are fatal. Sample real inputs across the segments you serve, seed the edge cases and every reported failure, label each case by hand with two reviewers against one rubric, hold back a split you never tune prompts on, then gate releases on the score and add every production failure as a new case. The step-by-step guide has a worked example.

What kinds of products has Ali shipped? expand_more

SaaS platforms, mobile apps, and two-sided marketplaces. Highlights include growing Yassir Groceries' acquisition by 135%, generating €4M in revenue at Flightright, and launching AI-driven product workflows at Datajar. See the full case studies.

Does Ali work with startups or enterprises? expand_more

Both. Ali advises early-stage founders on product-market fit and discovery, and partners with scale-ups and enterprises on product strategy, AI integration, and team operating models. Read his point of view or skim the six core skills he brings to every engagement.

How do I book a session with Ali? expand_more

Book a coffee chat or strategy call at cal.com/alim-datajar/coffeechat. Office hours are limited to two hours per week.

What is Ali's approach to product management? expand_more

Build with chaos, lead with logic — pair human-centered discovery and ethnographic research with hard metrics, AI-augmented workflows, and tight cross-functional execution. The toolkit and agent lab show how that plays out in practice.

Get my weekly PM logic.

No fluff. Just hard data, human psychology, and product strategies that actually work.