What's the Best Way to Scrape Reddit Posts and Comments for an LLM? Use an API, Then Trim the Thread

The most reliable way to get Reddit posts and comments into an LLM is an API that returns them as JSON, then a compact text format before the prompt. Scavio's Reddit API gives you a subreddit's feed with /api/v1/reddit/subreddit/posts, each post's body with /post, and the full comment tree with nested replies with /post/comments, every call 1 credit (about $0.002 on the Growth plan billed yearly) with no Reddit app or approval. Flattened to one indented line per comment, two r/homelab threads went from 8,796 tokens to 2,443.

A developer on r/ClaudeCode asked exactly this: how to collect posts and comments from niche subreddits about setups and builds, structure them by title, upvotes and theme, and feed the result into Claude. They asked three things: how to access Reddit's API now that restrictions have changed, whether scraping tools or third-party APIs are better, and how to clean the data first. One reply was blunt: "You wont get access to the API." On r/webscraping, a thread that ranks on Google for this question, another developer wanted the top 50 posts of 15 subreddits a month with their comments and was weighing the .json URLs, old.reddit and a headless browser. Here are the three answers.

1. What Reddit's own API allows now

Reddit's Responsible Builder Policy, updated October 5, 2026, says you "must request access and get explicit approval before accessing any Reddit data through our API". Commercial use needs explicit written approval, and so does using the data to train machine learning or AI models. Its Data API Wiki gives approved free apps 100 queries per minute per OAuth client id and says traffic without OAuth or login credentials "will be blocked", which rules out the .json shortcut for anything you depend on. If your project is non-commercial and gets approved, the official API with PRAW is free. If it is commercial, or you cannot wait on approval, use a third-party API.

2. Pull the subreddit, then the threads worth reading

Start with the feed. For r/homelab:

curl -s https://api.scavio.dev/api/v1/reddit/subreddit/posts \
  -H "Authorization: Bearer $SCAVIO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"subreddit": "homelab", "sort": "HOT"}'

It returned 6 posts in 1.3 s and a next_cursor. One row, as captured (avatar URL trimmed):

{
  "post_id": "t3_1xglqv4",
  "title": "One year later… I should’ve listened to you guys.",
  "author": "DoneD9",
  "subreddit": "homelab",
  "created_at": "2026-10-27T07:10:01.656000+0000",
  "score": 173,
  "num_comments": 35,
  "share_count": 8,
  "url": "https://redd.it/1xglqv4"
}

score and num_comments are on every row, so filter here, before you spend calls on comments. Feed rows have no body; /api/v1/reddit/post adds it. For this post the body was two words, "Please help", and the rest was a photo gallery that comes back as a URL and thumbnail, not image content. That is common in build and setup subreddits, and it is why the comments matter: the 35 comments here are where people say what is in the photo and what to change.

curl -s https://api.scavio.dev/api/v1/reddit/post/comments \
  -H "Authorization: Bearer $SCAVIO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"post_id": "t3_1xglqv4"}'

Comments arrive in tree order with a depth field; pass next_cursor while has_more is true, and expand any comment that carries a reply_cursor with /post/comments/replies. This thread took 6 calls and the other r/homelab thread we pulled, 39 comments, took 5. sort on a subreddit feed accepts HOT, NEW, TOP, BEST, RISING and CONTROVERSIAL, but TOP is all-time, so for "top 50 this month" page NEW back through the month and rank by score yourself.

3. Clean it before the prompt

Raw JSON wastes most of the context window on keys, ids and cursors. Keep the title, the body and one line per comment, indented by depth, with the score in brackets:

Those lines are from the captured thread, with the comments between them left out. Counted with the o200k_base tokenizer, the two r/homelab threads were 8,796 tokens as JSON with every field and 2,443 as compact text, 72% fewer, with every comment kept. The scores carry the upvote signal the r/ClaudeCode poster wanted, and indentation keeps who answered whom. For pattern-finding across many threads, ask the model for themes with the score totals behind each, and keep the post URL on every thread so you can check its claims. The Python that fetches a tree and writes this format is in how to get Reddit data into an LLM, and in Claude Code you can skip it: add Scavio's MCP server with claude mcp add --transport http scavio https://mcp.scavio.dev/mcp and ask Claude to read the subreddit itself.

What it costs

Take 5 niche subreddits and the top 50 posts of each every week. The feeds are about 40 calls, the post bodies 250, and the comments about 800 at the 3.2 calls per thread we measured across 14 threads, so roughly 1,090 credits a week and 4,400 a month. That fits the 7,000 credits of the $30 Project plan, or about $8.70 at the Growth-yearly rate of $2 per 1,000 credits. Bigger threads cost more: a 60-comment thread took 10 calls. For the r/webscraping job, 15 subreddits at 50 posts a month, the posts and comments come to about 3,200 credits a month, plus the feed pages to walk back through the month.

Try the Reddit API free, 50 credits, no card, or run a subreddit in the playground. For the full vendor comparison, including Apify actors and Reddit's own API, see the best Reddit data APIs for LLMs.

Reference: Reddit API overview · subreddit posts docs · post comments docs · scraping Reddit without the official API