Node.js LinkedIn Scraper

At its core, any scraper does three things: fetch a page, extract structured data from the markup, and store it. In Node there are two fundamentally different ways to do the fetch, and the choice dictates everything else.

The naïve HTTP approach works on simple sites. On LinkedIn it mostly returns an auth wall or a login redirect, because the valuable content sits behind a session and is rendered client-side. That pushes most people toward a headless browser — which is where the cost, fragility, and ban risk all spike. Understanding that fork up front saves you from building the wrong thing twice.

You'll need Node 18+ (for built-in fetch) and a couple of packages. Cheerio parses HTML server-side; Playwright is here so you can see what a headless approach costs you in the next steps.

mkdir linkedin-data && cd linkedin-data
npm init -y
npm install cheerio playwright
npx playwright install chromium     # downloads a real browser

# keep your config out of source control
echo "LINKFINDER_API_KEY=lf_live_your_key_here" > .env
echo ".env" >> .gitignore

{
  "name": "linkedin-data",
  "type": "module",
  "engines": { "node": ">=18" },
  "dependencies": {
    "cheerio": "^1.0.0",
    "playwright": "^1.44.0"
  }
}

Here's the canonical "scraper" everyone starts with: fetch a public, logged-out URL and pull structured fields out of the HTML with Cheerio. This is the legitimate, low-risk shape of scraping — reading public markup, identifying yourself honestly, and parsing what's returned.

import * as cheerio from "cheerio";

async function fetchPublic(url) {
  const resp = await fetch(url, {
    headers: {
      // Identify yourself honestly; don't impersonate a logged-in user.
      "User-Agent": "MyResearchBot/1.0 (+https://example.com/bot)",
      "Accept": "text/html",
    },
  });

  if (resp.status === 999 || resp.redirected) {
    // 999 = LinkedIn's anti-bot block; a redirect usually means an auth wall.
    throw new Error(`Blocked or gated (status ${resp.status}). Stop here.`);
  }
  return resp.text();
}

function parseOpenGraph(html) {
  const $ = cheerio.load(html);
  // Public pages expose basic Open Graph tags — the safe, intended surface.
  return {
    title: $('meta[property="og:title"]').attr("content") || null,
    description: $('meta[property="og:description"]').attr("content") || null,
    url: $('meta[property="og:url"]').attr("content") || null,
  };
}

const html = await fetchPublic("https://www.linkedin.com/company/example");
console.log(parseOpenGraph(html));

Run this and you'll quickly meet the wall: on most profile and people URLs, LinkedIn returns a login gate or a 999 status instead of the data. That's not a bug to engineer around — it's the platform telling you that surface isn't open to automation. The moment your answer to that wall is "log in and drive a headless browser through it," you've crossed from reading public data into ToS-violating territory, and into the failure modes in the next step.

The reason "Node.js LinkedIn scraper" tutorials don't survive contact with production is that the fetch was never the hard part. Here's what actually consumes the time and burns the accounts:

• Auth walls. The valuable data requires a logged-in session — which means automating a real account, which violates the ToS and puts that account at risk.
• Fingerprinting. Headless Chromium leaks automation flags (navigator.webdriver), unusual TLS handshakes, and missing browser quirks. Detection is independent of your IP.
• Bans & checkpoints. CAPTCHAs, 999 blocks, and account restrictions escalate quickly once you're flagged. Pushing through a CAPTCHA is how a warning becomes permanent.
• Layout churn. Selectors break whenever the front-end changes. A scraper that worked Monday silently returns empty fields Friday.
• Infrastructure. Proxies, browser pools, retry queues, and monitoring — you end up maintaining a small distributed system to fetch some text.

Add it up and a "simple scraper" becomes an arms race against a company that invests heavily in stopping exactly what you're doing. The cost isn't the code; it's the perpetual maintenance and the accounts you lose along the way.

Two separate things get conflated constantly, and the distinction matters for what you build:

• Scraping public data — In the U.S., the hiQ v. LinkedIn litigation signaled that scraping publicly accessible data is unlikely to violate the anti-hacking statute (the CFAA). That's about criminal hacking liability, not permission.
• Breaching the contract — LinkedIn's User Agreement separately prohibits automated collection. Scraping while logged in can be a breach of that contract and grounds for a ban or civil claim, even if it isn't "hacking."
• Privacy law — If you collect personal data on EU/UK or California residents, GDPR and CCPA/CPRA apply regardless of whether the data was public. Public ≠ unregulated.

Here's the punchline most engineers reach after the third banned account: if you want the data, you don't need a scraper at all. A licensed data API gives you structured records from the same fetch() you already wrote — no Playwright, no proxies, no fingerprints, no disposable accounts, no broken selectors.

const API_KEY = process.env.LINKFINDER_API_KEY;
const BASE_URL = "https://api.linkfinderai.com";

async function call(type, input_data, extra = {}) {
  const resp = await fetch(BASE_URL, {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ type, input_data, ...extra }),
  });
  const data = await resp.json();
  return data.status === "success" ? data.result : null;
}

// Resolve a profile, then append the fields you need.
const url = await call("lead_full_name_to_linkedin_url", "Sarah Mitchell CloudCore");
const profile = url && await call("linkedin_profile_to_linkedin_info", url);
const email   = url && await call("linkedin_profile_to_email", url);

console.log({ url, email, profile });

curl -X POST "https://api.linkfinderai.com" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $LINKFINDER_API_KEY" \
     -d '{
       "type": "linkedin_profile_to_linkedin_info",
       "input_data": "https://linkedin.com/in/sarah-mitchell-sales"
     }'

Same language, same fetch, a tiny fraction of the code — and none of the moving parts that break. Each request is a flat 1 credit, including misses, so cost is predictable instead of "however many accounts we burned this month."

There's a narrow case for rolling your own and a much wider case for not. Be honest about which one you're in:

Maybe build it yourself if…

• You only ever touch genuinely public, logged-out pages and stay within robots.txt and the ToS.
• The volume is tiny and one-off, and a broken run costs you nothing.
• Scraping is the product and you're prepared to staff the arms race.

Buy / use an API if…

• You want the data, reliably, and don't care how it arrives.
• You need volume, freshness, or structured output you can build on.
• You don't want your accounts, your IPs, or your legal posture exposed.

For the overwhelming majority of sales, marketing, and recruiting use cases, the second list wins. The scraper is a means to an end, and the API delivers that end without the maintenance tax or the ban risk.