We took on a migration in early 2025 where a publisher was moving roughly 31,000 URLs off a Drupal 7 install onto a static build. The old URLs were slugs like /node/48219. The new ones were readable. There was no mapping table, the editorial team who built the taxonomy had left, and the client’s budget for the redirect work was about three days. Hand-mapping 31,000 URLs is not a three day job.

That project is the clearest case of technical SEO automation actually earning its money that we’ve shipped. Embeddings got us to a reviewable candidate list in under two hours of compute and about eleven dollars of API spend. It also taught us where the modern crop of ai seo tools quietly falls apart: the moment a task requires a judgement call with business consequences, the automation becomes a very confident liability.

So this is the split. Not “AI is good” or “AI is hype”, but which specific technical SEO jobs survive automation and which ones you should keep on a human’s desk.

Sort the work into three buckets first

Every technical SEO task falls into one of three categories, and the category tells you whether to automate it.

• Classification at scale. “Which of these 9,000 pages is a thin tag archive?” “Which old URL is about the same thing as this new URL?” Models are genuinely good at this, and the cost of a wrong answer is one item in a review queue.
• Generation under constraints. Draft meta descriptions, JSON-LD scaffolds, alt text, hreflang stubs. Workable, but only behind schema validation and length limits. The output is never shipped unvalidated.
• Decisions with site-wide blast radius. Which URL is canonical. Whether to noindex a faceted parameter. Whether to allow GPTBot. Automating these is how you end up deindexing a product category on a Friday afternoon.

Most disappointment with seo with ai comes from treating bucket three as if it were bucket one. The model will answer confidently either way. That’s the trap.

Redirect mapping: the best use of embeddings in SEO

The trick that makes this work is what you embed. Our first attempt used full page text, and near-duplicate articles collapsed into each other: every “best running shoes” variant scored 0.95 against every other. The fix was to embed a short identity string instead, the slug words, the <title> and the first <h1>, joined with a space. Accuracy on manual spot checks jumped from unusable to good enough to trust with a threshold.

import OpenAI from "openai";
const openai = new OpenAI();

// Identity string, not body text. Body text makes near-duplicates indistinguishable.
const identity = (p) => ${p.slug.replace(/[-/]/g, " ")} ${p.title} ${p.h1}.trim();

async function embedAll(pages) {
  const out = [];
  for (let i = 0; i  d.embedding));
  }
  return out;
}

const cos = (a, b) => {
  let dot = 0, na = 0, nb = 0;
  for (let i = 0; i < a.length; i++) { dot += a[i] * b[i]; na += a[i]  2; nb += b[i]  2; }
  return dot / Math.sqrt(na * nb);
};

// Top 3 candidates per old URL, so a human reviews options rather than re-doing the work.
function mapUrls(oldVecs, newVecs, newPages) {
  return oldVecs.map(v =>
    newVecs
      .map((w, i) => ({ to: newPages[i].url, score: cos(v, w) }))
      .sort((a, b) => b.score - a.score)
      .slice(0, 3)
  );
}

Then set bands, and be honest about them. Above 0.82 we auto-accepted and spot-checked 2 percent. Between 0.65 and 0.82 went into a spreadsheet for review, roughly 6,000 rows, which two people cleared in a day and a half. Below 0.65 got redirected to the nearest surviving category page, and anything with no sensible parent returned 410 Gone rather than a soft 404 or a redirect to the homepage. Homepage redirects for dead content are still the most common migration mistake we inherit.

One caveat: weight your review queue by traffic, not by count. Pull the last 16 months of impressions from Search Console and sort the review list descending. The 200 URLs that carry 80 percent of the organic entry traffic deserve human eyes regardless of their similarity score.

Where technical SEO automation pays, and where scripts beat models

Crawl budget work is the area where people reach for an LLM and shouldn’t. You want reproducible counts, not a narrative summary. A one-liner over gzipped access logs will tell you where Googlebot is spending its time:

# Googlebot hits by top-level path, last 7 days of rotated logs
zcat access.log.{1..7}.gz | grep -i 'Googlebot/2.1' \
  | awk '{print $7}' | cut -d'/' -f2 | sort | uniq -c | sort -rn | head -20

Status code distribution for Googlebot only: the number that actually matters
zcat access.log.{1..7}.gz | grep -i 'Googlebot/2.1' \
  | awk '{print $9}' | sort | uniq -c | sort -rn

Find out where your logs live before you promise this to a client. On Netlify and Vercel you don’t get raw origin access logs in the shape above without the higher tiers or a log drain, which changes the whole approach; we covered the differences in our static site hosting comparison. Cloudflare Logpush into R2 plus DuckDB is a decent substitute and costs almost nothing.

Where AI does help here is the last mile: taking the output of your deterministic pipeline and clustering 4,000 unique 404 paths into 30 named patterns such as “old WooCommerce variation query strings” or “author archive pagination beyond page 12”. That’s classification over text you already trust. Verify bot identity by reverse DNS before you draw conclusions; a meaningful share of Googlebot user agents in raw logs are fake.

Structured data: generating is trivial, validating is the job

Ask any 2026-era model for Product JSON-LD and you get valid-looking output in seconds. You will also, eventually, get an aggregateRating for a product with no reviews, or a priceValidUntil that was never in the source data. Google treats invented review markup as a manual action risk. The generation step is cheap; the gate is the deliverable.

<?php
/**

Emit Product JSON-LD only when every required field exists in real data.
Fail closed: no markup beats fabricated markup.
 */
function scwproductjsonld( array $d ): string {
    foreach ( [ 'name', 'image', 'sku', 'price', 'currency', 'availability' ] as $k ) {
        if ( empty( $d[ $k ] ) ) {
            return '';
        }
    }

    $schema = [
        '@context' => 'https://schema.org',
        '@type'    => 'Product',
        'name'     => $d['name'],
        'sku'      => $d['sku'],
        'image'    => array_values( (array) $d['image'] ),
        'offers'   => [
            '@type'         => 'Offer',
            'price'         => number_format( (float) $d['price'], 2, '.', '' ),
            'priceCurrency' => $d['currency'],
            'availability'  => 'https://schema.org/' . $d['availability'],
        ],
    ];

    // Ratings only when the store actually has them. No reviews, no property.
    if ( ! empty( $d['reviewcount'] ) && (int) $d['reviewcount'] > 0 ) {
        $schema['aggregateRating'] = [
            '@type'       => 'AggregateRating',
            'ratingValue' => (string) $d['rating'],
            'reviewCount' => (int) $d['review_count'],
        ];
    }

    return '<script type="application/ld+json">'
        . wpjsonencode( $schema, JSONUNESCAPEDSLASHES | JSONUNESCAPEDUNICODE )
        . '</script>';
}

Use the model to write the mapping function once, from your real field names to schema.org types. Then never let it near the runtime path. Run the Rich Results Test and the schema.org validator in CI against ten representative URLs per template; a diff in the output is a build failure.

Internal linking suggestions that survive review

Internal linking is the second-best AI task after redirects, and it fails for a predictable reason: ask a model for link suggestions and it invents both the target URL and anchor text that sounds plausible. Both need to be grounded. Build an index of your real published URLs with their headings, retrieve the top candidates for each paragraph, and constrain anchor text to strings that appear verbatim in the target page’s <h2> set or title. If you’ve not built a retrieval step before, our plain English guide to RAG covers the shape of it.

Two rules we apply on every client run. Cap suggestions at three per 1,000 words, because an accepted-everything pipeline produces pages with 40 links and a wrecked reading experience. And reject any suggestion where the target already links back, unless the pages are genuinely a pair. Reciprocal link soup looks like exactly what it is.

Four things that keep failing when automated

Core Web Vitals diagnosis. Models will confidently tell you to lazy-load an image that is the LCP element. They cannot see your waterfall. Use the Chrome UX Report and a real trace; the model can help you read the trace afterwards.

Canonical decisions. Which of two near-identical pages should survive depends on backlinks, conversion rate and internal politics. None of that is in the HTML.

Hreflang clusters. Automated hreflang generation produces non-reciprocal tags the moment one locale is missing a page. Generate the clusters from your CMS relations, not from a model’s inference.

AI crawler policy. This is a commercial decision and it belongs to the client. Write it explicitly rather than inheriting whatever a plugin ships:

# robots.txt
User-agent: Google-Extended     # Gemini training, does not affect Search indexing
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot       # surfaces you in ChatGPT search results; usually keep this one
Allow: /

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap_index.xml

Note the asymmetry: blocking training crawlers while allowing retrieval crawlers is the position most publishers we work with land on. Decide it deliberately.

The pipeline we actually run

For retainer clients the whole thing is four cron jobs and a review inbox. A nightly crawl with Screaming Frog in list mode or a headless Playwright script dumps status codes, titles, canonicals and word counts into Postgres. A daily Search Console API pull lands impressions and clicks per URL. A SQL layer flags the deterministic problems: orphans, redirect chains longer than one hop, duplicate titles, pages with impressions but no internal links. Only the flagged rows go to a model, and only for clustering and drafting.

Cost on a 20,000 URL site runs about 4 to 7 dollars a month in API spend, because the deterministic layer filters out 95 percent of the rows before anything hits a token budget. If your bill is larger than that, you’re sending whole pages where you should be sending a row. Our notes on context windows and token cost explain why that difference compounds, and if the pipeline runs unattended you’ll want the tracing and regression setup too. A silently degraded prompt that starts returning empty arrays looks identical to a clean site.