Show HN: Pulpie – Models for Cleaning the Web
Hey HN, I'm Shreyash, founder of Feyn. We built Pulpie, a family of Pareto optimal models for cleaning the web. Pulpie strips boilerplate (ads, footers, sidebars) from raw HTML and returns just the main content as HTML or Markdown. We match SOTA extraction quality while being 20x cheaper. Cleaning 1 billion webpages costs $7,900 with Pulpie versus $159,000 with Dripper, the current leading extractor. The gains come from architecture. Today's leading extractors are decoders that generate output one token at a time. Each step reads the full model from memory to produce a single token. Conversely, Pulpie models are encoders. They run one forward pass over the full input HTML and label each block as boilerplate or content. As a result, Pulpie is compute-bound while decoders are memory-bound. Cheaper GPUs have relatively more compute than memory bandwidth. This makes Pulpie easy to run optimally. Here's Pulpie and Dripper cleaning the same pages side by side: https://www.youtube.com/watch?v=ibd-tIiQECo . You can try a side-by-side comparison yourself: https://huggingface.co/spaces/feyninc/pulpie Our motivation for Pulpie came from building a deep research harness. Every search API returns noisy content that contains ads, nav elements, and sidebars. In one instance, an ad for "Gemini on Pixel" slipped into our search results, got passed into LLM context, and ended up in the final answer served to the user. Pretty embarrassing moment for us but it helped us realize how bad data kills model intelligence. We built Pulpie to get clean data for cheap. All models are open source on Hugging Face. You can read about our training process and how to use Pulpie here: https://usefeyn.com/blog/pulpie-pareto-optimal-models-for-cl... Happy to answer any questions!
Discussion Highlights (11 comments)
lnenad
Very nice! Thank you for building this.
kocamaz
It's good looking, and I liked it. The trial page accessed from the hugging face website is a very inefficient experience when I use Mozilla and the dark theme, FYI.
esafak
Why does the 'Quality vs Cost of Web Content Extraction' chart not have zero cost at the origin? Up to the right does not have to mean better; we can read.
andrethegiant
Why not use a plain old html → markdown converter? You can easily strip out ads using CSS /jQuery-like selectors. That would cost zero dollars.
zaptheimpaler
So this is tailored towards kind of a "reader view" for models right? Can it handle images, tables, shadow DOMs too? Like there are 3 use cases I have now - one is a simple text view for models to understand it, one is a "web clip" mode which would ideally preserve images and media, and one is to extract tabular data from web pages. Which ones is this good at?
tyzoid
How does this work on pages that require JavaScript in order to render?
cpill
I did some research on this about 10 years ago. I spent 2 days hand labelling data from scraped news sites. Then built a good old fashioned Random Forest model to classify html nodes based on some feature engineering. turns out the P tag and the number-of-words threshold get you 90% of the way there, on news sites anyway. Great thing about RF models is they tell you which features are the most important. fun little project (apart from the 2 days of data labelling).
geniium
Amazing I was just looking for something like this to be able to import web page content into Whisperit
wiradikusuma
Does it work with ecommerce for product scraping? E.g. Amazon, or Shopee (big in SEA)
emblemapp
This looks really cool
vishalkundar
Very interesting