Use Hacker News Story, Comment & Karma Scraper with Python

Pull Hacker News front-page, newest, best, Ask HN, Show HN, or job stories, look up specific items or user profiles, or run a full-text search across HN history. Filter by score, domain, date range, and comment count or author.

1. Choose the data you need

Review the published README for supported collection modes, examples, and limits. The recipe below uses the current published input definition.

2. Prepare a source-based input

This JSON contains published prefills and required-field defaults. Review every value for your task before running it; it is not a promise of successful collection.

{
  "commentAuthorFilter": [],
  "commentSort": "rank",
  "contentTypes": [
    "story"
  ],
  "domainAllowlist": [],
  "domainBlocklist": [],
  "enableCommentHierarchy": false,
  "excludeDeadOrDeleted": true,
  "exhaustiveSearch": false,
  "hiringKeywords": [],
  "hiringMonths": 1,
  "hiringThreadIds": [],
  "includeAuthorProfiles": false,
  "includeSubmissions": false,
  "itemIds": [],
  "maxComments": 0,
  "maxDepth": 5,
  "maxItems": 20,
  "mode": "topStories",
  "remoteOnly": false,
  "searchQueries": [],
  "searchSort": "relevance",
  "startUrls": [],
  "submissionTypes": "all",
  "usernames": []
}

Download input.example.json and save your reviewed copy as input.json.

Field	Type	Required	Description
`mode`	string	required	What to scrape. Feeds walk the live Hacker News lists. `item` looks up specific IDs, `user` loads profiles, `search` queries Hacker News history, `frontPage` reads a historical day, and `whoIsHiring` parses monthly hiring threads.
`itemIds`	array	optional	Hacker News item IDs to look up, as numbers or numeric strings. Stories, comments, jobs, polls, and poll options are all accepted.
`usernames`	array	optional	Hacker News usernames whose profiles and, optionally, submissions you want.
`searchQueries`	array	optional	One or more keyword queries. Every query runs independently and the results are merged without duplicates. Leave empty to browse by tags, author, or story instead.
`searchSort`	string	optional	`relevance` ranks by keyword match. `date` returns newest first and is more reproducible.
`contentTypes`	array	optional	Which Hacker News record types search should return. Selecting several types returns any of them. `front_page` covers only the current front page.
`author`	string	optional	Only return content written by this Hacker News username.
`storyId`	string	optional	Only return content attached to this story ID. Useful with `contentTypes` set to `comment` to search inside one thread.
`exhaustiveSearch`	boolean	optional	Hacker News search exposes at most 1,000 hits per query. When enabled, the actor splits the requested date range into smaller windows so deeper result sets can be collected. Relevance order then follows each date window rather than one global ranking.
`dateRangeFrom`	string	optional	Drop items posted before this date (UTC, `YYYY-MM-DD`). Empty means no lower bound.
`dateRangeTo`	string	optional	Drop items posted after this date (UTC, `YYYY-MM-DD`). Empty means no upper bound.
`minScore`	integer	optional	Drop stories, jobs, and polls below this point score. Comments are unaffected because Hacker News does not publish comment scores.
`minCommentCount`	integer	optional	Drop stories and polls with fewer than this many comments.
`domainAllowlist`	array	optional	Keep only stories whose link host contains one of these text fragments. For example, `github.com` or `arxiv.org`.
`domainBlocklist`	array	optional	Drop stories whose link host contains one of these text fragments. For example, `twitter.com` or `x.com`.
`excludeDeadOrDeleted`	boolean	optional	Drop items marked dead or deleted by Hacker News. Turning this off uses the slower exact comment path so those records can be included.
`commentAuthorFilter`	array	optional	Only keep comments written by these usernames. Matching is case-insensitive and accepts partial names.
`commentSort`	string	optional	`rank` keeps Hacker News display order. `newest` orders sibling replies newest first.
`maxComments`	integer	optional	How many comments to collect for each story. 0 skips comment collection. This is applied after a story passes your filters.
`maxDepth`	integer	optional	Deepest reply level to keep when comments are collected. Root comments are depth 0.
`enableCommentHierarchy`	boolean	optional	When true, each story carries a nested `comments` tree. When false, comments are separate rows and never contain duplicate reply rows.
`includeSubmissions`	boolean	optional	For mode `user`, also emit the user's recent stories and comments as normal records after the profile.
`submissionTypes`	string	optional	Which submissions to emit when `includeSubmissions` is enabled.
`frontPageDate`	string	optional	Historical day to read in mode `frontPage`, as `YYYY-MM-DD`. Empty uses yesterday (UTC).
`hiringMonths`	integer	optional	How many recent monthly Who is hiring? threads to parse. Ignored when specific thread IDs are supplied.
`hiringThreadIds`	array	optional	Specific Who is hiring? thread IDs to parse. Overrides the automatic monthly selection.
`hiringKeywords`	array	optional	Keep only job posts whose text, company, role, location, or tech stack contains one of these terms.
`remoteOnly`	boolean	optional	Keep only hiring posts that explicitly mention remote work.
`maxItems`	integer	optional	Hard cap on emitted records across all types. Nested comments also count toward this cap.
`includeAuthorProfiles`	boolean	optional	Add `authorKarma` and `authorCreatedAt` to stories and comments. This costs one extra lookup per unique author.
`startUrls`	array	optional	Hacker News links to read directly: item, user, historical front-page, and hiring-thread URLs. IDs and URLs are merged with the mode-specific lists.

3. Start an Actor run

Set APIFY_TOKEN in your environment. Python 3.10 or newer is sufficient; this recipe uses no third-party packages.

Execution may incur charges. Review current pricing on Apify. The request asks for a 120-second timeout and a charge limit; these settings are subject to Apify’s pricing and do not guarantee an all-inclusive spending cap.

import json
import os
import urllib.request

# Save the schema-derived example as input.json; review and edit it first.
with open("input.json", encoding="utf-8") as stream:
    actor_input = json.load(stream)

token = os.environ["APIFY_TOKEN"]
request = urllib.request.Request(
    "https://api.apify.com/v2/acts/shC9wAKWFYbt1zZYs/runs?timeout=120&maxTotalChargeUsd=0.50",
    data=json.dumps(actor_input).encode("utf-8"),
    headers={"Authorization": "Bearer " + token, "Content-Type": "application/json"},
    method="POST",
)
# This starts a potentially billable run. Do not automatically retry this POST.
with urllib.request.urlopen(request, timeout=40) as response:
    run = json.load(response)["data"]
print("Run ID:", run["id"])
print("Status:", run["status"])
# Inspect and download results in Apify Console once the run succeeds.

4. Inspect the actual results

Open the run in Apify Console and wait for a successful completion. Use the published output schema to locate results and the dataset schema to understand declared views. Missing or optional record fields must not be treated as guaranteed values.

Stories, comments, jobs, polls, poll options, user profiles, and parsed Who is hiring? posts from Hacker News.

Questions before you run

Which inputs are required?

The published required fields are mode

What does this guide verify?

The fields, prefills, and schema links are rendered from the published build. This is a usage recipe, not a paid live extraction test. Source availability and collection results can vary.

What if the run request times out?

Check Apify Console before retrying. The first request may already have created a run. Do not automatically repeat a run-creation POST.

Open on Apify Full Actor reference

Generated from source build ajANmalEbfWPogs9n and its README and schemas. Source changes regenerate this guide; no third-party editorial claims are inferred.