Scraping 241 UK council planning portals – 2.6M decisions so far
I've been scraping 241 UK council planning portals – 2.6M decisions so far UK planning data is technically public. In practice it's locked behind 400+ different council portals, some still running bespoke ASP.NET that looks like it dates from 2004, some behind AWS WAF, all with subtly different schemas. I've spent four months scraping them. I'm now at 241 councils and 2.6 million decisions across England, Scotland and Wales. The scraping problem Most UK councils run one of a handful of portal systems, Idox being the most common. In theory this makes things easy. In practice every council has configured theirs differently, some block non-browser requests via TLS fingerprinting, some have rate limits that will get you banned inside 10 minutes, and a handful are running the aforementioned bespoke ASP.NET. I ended up writing several scrapers: a standard requests-based one, a Playwright-based one for councils that block anything that doesn't look like a real browser, and a curl_cffi one for TLS fingerprinting. Some councils I still can't get. Liverpool's portal sits behind AWS WAF with a JavaScript challenge. I have a working Playwright-based scraper that solves the challenge once and reuses cookies, but the WAF rate-limits the IP after about 10 requests and then blocks me for a day. So I have 60k Liverpool decisions from an old scrape and no easy way to add more. What I found The approval rate stuff is what most people come for. Nationally it's around 88%, but it varies wildly by ward within a council, not just between councils. The more interesting finding came from the time-to-decision data. Across 119 English and Welsh councils, 36.5% of home extension applications missed the statutory 8-week target in 2025, up from 27.9% in 2019. Guildford is the worst at scale: 66% of decisions over target, averaging 13.3 weeks. What it is now A postcode checker (free) and paid PDF reports (£19/£79). Zero paying customers so far, which is fine. I've been heads down on data quality and coverage. Site is planninglens.co.uk if you want to poke around. AMA on the scraping side – that's where the interesting problems are.
Discussion Highlights (20 comments)
CJefferson
So, this sounds exciting to me, but the postcode checker really feels like a spam as a user. All it tells me is 'Mixed results'. I could make a website that prints 'mixed results', I bet most results are 'mixed'! I understand wanting to get money, but honestly, there is no way I would give money to this website in it's current state, you are giving me far too little info before asking me to hand over a credit card. Then , if someone gives you £19, a crazy amount of money honestly, the last page of the report is an advert to give them 4 times more!
beatthatflight
Worth trying claude/gemini to see if they'll do some scraping for you. I've found some paywall sites only too happy to allow Gemini past the wall.
ashish-alex
Working on similar problem in another domain. I found agentic direction powerful with browser use plugged into a multimodal (strong agentic capability) llm like gpt 5.4 mini working in a loop with orchestrator evaluator/judge.
sublimefire
Send a message to infoshareplus.com They might be interested in your data because they operate a business around local govs.
dabeeeenster
Have you tried using Browserless/similar to scrape around tricky hosts?
efaref
Great site. This data should really be more accessible. Planning in the UK is a total crapshoot, subject to the whims of the planning authorities. In our case, a simple rear extension and dormer loft conversion, similar to hundreds of thousands across the country, we ended up having to appeal which added 2 years and tens of thousands of pounds in costs to our extension project. Our area shows up as a high refusal area, which tracks. It would be good to add appeal data in (also a public gateway) to show which councils are just being unreasonable. I personally think the planning regulations in this country are the cause of many ills, including the housing shortage. It just costs so much to get through planning these days, it is often just not worth it. Data like this could help us get that changed.
imdsm
How long did the scraping take you to build?
ferngodfather
Your terms: > You may not use automated tools to scrape, copy, or bulk-download data from our service. Pot kettle, huh.
safehuss
This is awesome! Worked on something similar albeit a different industry. For the more challenging scrapes, would highly recommend using the Chrome Devtools MCP to be able to attach the network requests, being made by the browser to the site, as context for your agent/LLM chat - this approach really helped me to write a solid API-based scraper (also using curl_cffi) and bypassed the old tedious playwright-based approach I used to rely on.
vr46
Amazing! It’s so bloody hard to access this information or even to know what there is. Careful not to expose the councils too publicly before they shut you off
pbhjpbhj
Have you spoken to any planners, a quick search for similar applications in other LAs might be a useful thing for them. There's a Royal Institute of Town Planners, they probably have a magazine you could advertise in (but equally that might get you blocked, idk). RICS people could probably use the data too? I guess it's useful house-buyer info; houses in the vicinity had successful loft conversions, say. On the data side - it's something of a moat for you now, but I could see you being successful with FOI requests. An MP might be interested in championing open data access.
pbhjpbhj
Is any of the data on Gov.uk - any scrapping tips there? I've tried scraping some patent tribunal data but haven't been successful (just using Python (copying in session data), I guess Playwright might be useful there).
doublesocket
It's the most ridiculous situation with council technology that they all use different providers for what are fundamentally the same functions. It's the same for council tax and a host of other services as it is for planning. Consequently, at least from the various portals I've used, they all do it badly. This absolutely could and should be done by a single, well funded central team.
edent
Have you tried using FoI to get the data? I've had some success with data requests - often getting dumps in CSV or similar. I appreciate that won't necessarily capture live / recent data. But it might be quicker than waiting for rate-limits to reset.
notarobot123
It looks like this kind of data will start to be more open in the future. New legislation introduces mandatory data standards in England: https://mhclgdigital.blog.gov.uk/2026/04/22/data-standards-l...
niffydroid
Ace, I can see how this could actually be quite useful for house conveyancing. You've put a lot of effort into this. How are you affected by the upcoming changes to local government? They'll no doubt be some rationalisation at some point.
morkee
I hate to be a downer but... > UK planning data is technically public. it's public, but still copyrighted by those who submitted it the councils also have database rights over their database, unless you've obtained explicit permission from them directly https://en.wikipedia.org/wiki/Database_right#United_Kingdom > I ended up writing several scrapers: a standard requests-based one, a Playwright-based one for councils that block anything that doesn't look like a real browser, and a curl_cffi one for TLS fingerprinting. so they're explicitly trying to stop you doing this, and ... you're openly admitting to bypassing their technical measures to try and stop you? have you heard of the Computer Misuse Act? I doubt the 240 councils are going to be happy once they find out you've done this, especially if you're selling it on for profit
simonjuk
I work with public data, and I'd love to get access to this data, but I suspect that although you have scraped the data from public websites, there are licensing and copyright implications for actually using it. See also the open addresses project by Data Adaptive [1] which is using Freedom of Information requests to publish public council tax address data. The problem they have run into there is that their address datasets are derived from proprietary Ordnance Survey data. It looks like data.gov.uk is in the process of standardising the planning application process, and publishing them under OGL [2]. [1]: https://www.owenboswarva.com/blog/post-addr44.htm [2]: https://www.planning.data.gov.uk/dataset/planning-applicatio...
codeulike
I'd be careful because even though its 'public' data, scraping it might not be legal due to TOS of the various sites. I did a search for my postcode and got given results for a different area and council miles away
lifeisstillgood
Some thoughts 1. Brilliant! Governments (and corps) treat public data like it’s theirs not ours. Information yearns to be free. 2. Having said that, you are likely violating T&Cs by scraping at all. 3. It is a lot easier to defend your position if you are making it free and public yourself. 4. But paying for food is nice 5. I suggest the business model here is providing architects and lawyers with strong evidence of prior planning decisions nationally Most people applying for (difficult) planning have experience locally. But the planning system is a mess because it is not coherent nationally or regionally. The win here is not providing a copy of your data (that has legal issues) but providing pointers to decisions that support the case of the person paying you. So I want to turn an old pub into tasteful housing and a cafe for the local village. The local planning team don’t like it, I could spend money bribing them and the councillors (see how much I understand British democracy) or I could get from you the fifteen pub to housing conversion decisions from around the country and use that to help my bribed councillors defend their u-turn Everyone wins :-)