36 Pages Stuck 'Discovered — Not Indexed': How I Fixed It with Google's Indexing API

For three months, 36 pages on my site sat in Google Search Console with the same label:

Discovered — currently not indexed.

Last crawled: N/A.

Every hire page. Most tool pages. A dozen boolean generator routes. All of them submitted via sitemap, none of them ever crawled. The validation had "passed" back in April — but the pages stayed invisible.

Here's the exact diagnosis and what I actually did to fix it.

What "Discovered — Currently Not Indexed" Actually Means

This status is Google's way of saying: "I know this URL exists, but I haven't visited it yet."

It's not a penalty. It's not a technical block. It's a queue problem.

Google discovered these URLs via the sitemap, but deprioritized them. This happens when:

• Your site doesn't have enough authority to justify a large crawl budget
• Too many low-value URLs compete equally with high-value ones
• No inbound links point to the specific pages
• lastModified dates are stale, signaling nothing has changed

The key insight: Google doesn't crawl your sitemap top to bottom. It makes a judgment call on every URL based on priority, freshness, and link equity. If it sees 100+ URLs all claiming the same importance with dates from 6 months ago, it queues the lower-priority ones indefinitely.

Diagnosing My Sitemap

My sitemap.ts had several issues working against me.

Problem 1: /dashboard in the Sitemap at Priority 0.8

// This was wasting Google's crawl budget on a private admin page
{
    url: `${SITE_CONFIG.baseUrl}/dashboard`,
    lastModified: new Date("2026-04-10"),
    changeFrequency: "weekly",
    priority: 0.8,
},

This is an authenticated admin dashboard. Google would crawl it, find a redirect or gated content, get no value — and that burn counts against the budget allocated to my domain.

Problem 2: /search in the Sitemap

{
    url: `${SITE_CONFIG.baseUrl}/search`,
    lastModified: new Date("2026-04-10"),
    changeFrequency: "monthly",
    priority: 0.5,
},

Search UI pages are UI shells, not content. No value for indexing. More budget waste.

Problem 3: Flat Priority — Everything Looked the Same

Boolean generator pages (programmatic, thin content) had priority: 0.7 — the same as blog posts. Non-featured tools had priority: 0.8. Core hire pages had priority: 0.85. The spread was too narrow.

Google uses priorities as relative signals within your sitemap. If everything is clustered between 0.7–0.9, the signal is meaningless.

Problem 4: Stale lastModified Dates

lastModified: new Date("2026-04-10"),  // 3 months ago, nothing ever changes

When Google sees that a page hasn't changed in months, it sees no urgency to re-queue it. A fresh lastModified signals: "Something changed, come look."

Fix 1: Clean the Sitemap

The changes were straightforward:

// REMOVED /dashboard — private admin page
// REMOVED /search — UI shell, no index value

// Updated priorities (before → after):
// boolean generator pages: 0.7 → 0.5  (programmatic, lower value)
// non-featured tools:      0.8 → 0.7  (good content, not flagship)
// featured tools:          0.9 → 0.9  (unchanged)
// hire pages:              0.85 → 0.85 (unchanged, but lastModified refreshed)

// Refreshed all lastModified to today
lastModified: new Date("2026-07-21"),

This is a necessary fix for long-term crawl health — but it wouldn't solve the 36 stuck pages immediately. Waiting for Google to re-process the updated sitemap could still take weeks.

Fix 2: Google's Indexing API

This is the real fix. Google's Indexing API lets you submit specific URLs directly for crawling. Google commits to crawling submitted URLs "within a few days" — in practice, it's usually minutes to hours.

The API has a 200 URL/day free quota. My site has 78 indexable URLs, so I'm well within limits.

Prerequisites

The API requires a service account with the following setup:

1. Google Cloud Console: Enable the Indexing API for your project
2. Search Console: Add the service account email as an Owner (not just Viewer — it must be Owner)

I already had a service account for Google Analytics. I reused the same credentials.

The Script

I wrote a Node.js ESM script (scripts/index-urls.mjs) that:

1. Authenticates with Google using a JWT signed with the service account private key (no extra npm packages — just Node's built-in crypto module)
2. Submits every URL in the sitemap to the Indexing API with type: "URL_UPDATED"
3. Rate-limits to 1 request/second to stay within quota
4. Reports per-URL status

The auth piece is the most interesting part. The Indexing API uses OAuth2 with a service account JWT — no browser flow, no user interaction. Here's the core of it:

import { createSign } from "crypto";

function signJwt(payload, privateKey) {
    const header = base64url(JSON.stringify({ alg: "RS256", typ: "JWT" }));
    const body = base64url(JSON.stringify(payload));
    const unsigned = `${header}.${body}`;
    const sign = createSign("RSA-SHA256");
    sign.update(unsigned);
    return `${unsigned}.${sign.sign(privateKey, "base64url")}`;
}

async function getAccessToken(creds) {
    const now = Math.floor(Date.now() / 1000);
    const jwt = signJwt(
        {
            iss: creds.client_email,
            scope: "https://www.googleapis.com/auth/indexing",
            aud: "https://oauth2.googleapis.com/token",
            iat: now,
            exp: now + 3600,
        },
        creds.private_key,
    );

    const res = await fetch("https://oauth2.googleapis.com/token", {
        method: "POST",
        headers: { "Content-Type": "application/x-www-form-urlencoded" },
        body: new URLSearchParams({
            grant_type: "urn:ietf:params:oauth:grant-type:jwt-bearer",
            assertion: jwt,
        }),
    });

    const { access_token } = await res.json();
    return access_token;
}

No googleapis npm package. No google-auth-library. Just crypto from Node's standard library and fetch.

Then submitting each URL:

async function submitUrl(url, token) {
    const res = await fetch("https://indexing.googleapis.com/v3/urlNotifications:publish", {
        method: "POST",
        headers: {
            "Content-Type": "application/json",
            Authorization: `Bearer ${token}`,
        },
        body: JSON.stringify({ url, type: "URL_UPDATED" }),
    });
    return res.ok;
}

Running the script for the first time:

🔍 Google Indexing API
   Submitting 78 URLs to be crawled

✅ Authenticated with Google

[  1/78] ✅  https://www.example.com/
[  2/78] ✅  https://www.example.com/about
[  3/78] ✅  https://www.example.com/experience
...
[  8/78] ✅  https://www.example.com/hire/role-one
[  9/78] ✅  https://www.example.com/hire/role-two
...
──────────────────────────────────────────────────
✅ Submitted: 78   ❌ Errors: 0
──────────────────────────────────────────────────

🎉 Done! Google will crawl submitted URLs within minutes–hours.

All 78. Zero errors.

Fix 3: Auto-Run on Every Deploy via GitHub Actions

Running this script manually every time I add a new page is a reliability problem — I'll forget. The right fix is to wire it into CI so it runs automatically after every production deploy.

My setup: GitHub Actions runs CI on every push to main, then Vercel deploys. I added a workflow_run trigger that fires after the CI quality job completes successfully:

# .github/workflows/index-urls.yml
name: Google Indexing API

on:
    workflow_run:
        workflows: [CI]
        types: [completed]
        branches: [main, master]

jobs:
    index-urls:
        runs-on: ubuntu-latest
        if: ${{ github.event.workflow_run.conclusion == 'success' }}
        steps:
            - uses: actions/checkout@v7
            - uses: actions/setup-node@v4
              with:
                  node-version-file: .nvmrc
                  cache: npm
            - name: Submit URLs to Google Indexing API
              run: node scripts/index-urls.mjs
              env:
                  GOOGLE_CREDENTIALS_JSON: ${{ secrets.GOOGLE_CREDENTIALS_JSON }}

The if: conclusion == 'success' condition is important — it means the indexing script only runs when CI actually passes, not on failed builds or PRs. No wasted API quota.

For the secret, I store the entire google-credentials.json content as GOOGLE_CREDENTIALS_JSON. The script parses it:

if (process.env.GOOGLE_CREDENTIALS_JSON) {
    const creds = JSON.parse(process.env.GOOGLE_CREDENTIALS_JSON);
    return { client_email: creds.client_email, private_key: creds.private_key };
}

Clean. No escaping issues with the private key's newlines.

The Priority Hierarchy That Actually Works

For a personal site / portfolio with 80–150 URLs, here's the priority breakdown I landed on:

What Not to Do

Don't add /dashboard to your sitemap. Ever. Even at priority 0.1. It wastes crawl budget and sends a confusing signal.

Don't submit URLs to the Indexing API from your Next.js server-side code. I considered calling the Indexing API in generateStaticParams or a route handler to auto-submit whenever a new page is built. Don't. The Indexing API is meant to be called from trusted infrastructure (CI/CD, admin scripts), not from public-facing server code.

Don't expect priority to force indexing. Priority is a hint, not a command. A URL at priority 1.0 with no inbound links and stale lastModified will still be deprioritized. The Indexing API is the only reliable way to force a crawl.

Results

The 36 pages went from "Discovered — currently not indexed" to actively crawled. Within a few hours of running the script, Google Search Console's URL Inspection started showing crawl activity on pages that had never been touched.

The CI workflow means every future deploy automatically re-submits all URLs — so new pages I add will get indexed the same day they ship, not weeks later.

TL;DR

1. Audit your sitemap — remove private/utility pages, fix stale lastModified, differentiate priorities
2. Use the Indexing API — it bypasses the crawl queue entirely; URLs are crawled within hours
3. Wire it to CI — workflow_run trigger in GitHub Actions, fires after successful deploys only
4. One secret — store the whole service account JSON as GOOGLE_CREDENTIALS_JSON

The whole thing — script, workflow, sitemap cleanup — took about 2 hours to build and has fully automated the indexing problem going forward.