trader.ideas.scraper module

Fetch one web page and ask a local LLM what this app could build from it.

The model never fetches anything itself. Issue #75’s investigation gave a large “thinking” model the tool-calling wheel for fetching via ironclaw (a separate agentic framework) and hit a real wall: first-token latency on a tool-heavy request blew past the framework’s fixed client timeout and produced an infinite retry loop, not a clean failure. The fix, proven end to end in a throwaway prototype before this module existed, is the same shape LlmStrategy already uses: fetching happens in plain code, the model only ever sees one finished prompt and writes text back. synthesize_ideas below sends no tools field, ever.

Stdlib `urllib` on purpose, same reasoning as trader.llm.ollama: adding httpx to the runtime dependency list for one GET would be a new dependency this project does not otherwise carry.

class trader.ideas.scraper.ExtractedPage(title: str, text: str)[source]

Bases: object

Parameters:

  • title (str)

  • text (str)

title: str
text: str
exception trader.ideas.scraper.IdeasScraperError[source]

Bases: TraderError

Fetching the page, extracting its text, or synthesizing ideas failed.

trader.ideas.scraper.extract_text(html, *, max_page_chars)[source]

Strip markup and return the page’s title and prose, truncated to max_page_chars.

Raises:

IdeasScraperError – nothing readable remains after stripping markup.

Parameters:

  • html (str)

  • max_page_chars (int)

Return type:

ExtractedPage

trader.ideas.scraper.fetch_page(url, *, timeout_seconds)[source]

Fetch url and return its raw HTML.

Raises:

IdeasScraperError – the request failed, timed out, or did not return HTML.

Parameters:

  • url (str)

  • timeout_seconds (int)

Return type:

str

trader.ideas.scraper.scrape_ideas(url, *, output_dir, fetch_timeout_seconds, max_page_chars, llm)[source]

Fetch url, synthesize product-feature ideas, and write them to output_dir. Returns the path written.

Raises:

  • IdeasScraperError – fetching, extraction, or synthesis produced nothing usable.

  • LlmError – the model call itself failed (also a TraderError).

Parameters:

  • url (str)

  • output_dir (Path)

  • fetch_timeout_seconds (int)

  • max_page_chars (int)

  • llm (LlmProvider)

Return type:

Path

trader.ideas.scraper.synthesize_ideas(llm, *, url, title, text)[source]

Ask the model for product-feature ideas grounded in text.

Raises:
Parameters:
Return type:

str

trader.ideas.scraper.write_ideas_markdown(output_dir, *, url, title, model, markdown_body, now=None)[source]

Write the synthesized ideas to a new markdown file under output_dir.

Never overwrites an existing file for the same slug — a _2, _3, … suffix is appended instead, so re-running against the same URL accumulates rather than silently discards a prior write this is meant to be a backlog of, per issue #75’s stage-1/stage-2 split.

Parameters:

  • output_dir (Path)

  • url (str)

  • title (str)

  • model (str)

  • markdown_body (str)

  • now (datetime | None)

Return type:

Path