Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: London Borough of Lewisham 403 error #3079

Open
6 of 7 tasks
theawesomestrob opened this issue Nov 23, 2024 · 0 comments
Open
6 of 7 tasks

[Bug]: London Borough of Lewisham 403 error #3079

theawesomestrob opened this issue Nov 23, 2024 · 0 comments

Comments

@theawesomestrob
Copy link

I Have A Problem With:

A specific source

What's Your Problem

Trying to configure the source with UPRN, I get this error:

The source returned an invalid response: "403 Client Error: Forbidden for url: https://lewisham.gov.uk/myservices/recycling-and-rubbish/your-bins/collection". Please check the provided arguments and try again.

Pasting that URL into a browser works fine.
Trying postcode+number instead of UPRN, I get the same error with a different URL:

The source returned an invalid response: "403 Client Error: Forbidden for url: https://lewisham.gov.uk/api/AddressFinder?postcodeOrStreet=SE231LB". Please check the provided arguments and try again.

Pasting that into a browser gives an error saying "The page you are looking for cannot be displayed because an invalid method (HTTP verb) is being used."

Trying either URL via curl I get a 403, eg:

$ curl -i 'https://lewisham.gov.uk/myservices/recycling-and-rubbish/your-bins/collection'
HTTP/1.1 403 Forbidden
Server: Microsoft-Azure-Application-Gateway/v2
Date: Sat, 23 Nov 2024 18:16:00 GMT
Content-Type: text/html
Content-Length: 179
Connection: keep-alive

<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>Microsoft-Azure-Application-Gateway/v2</center>
</body>
</html>

Looks like they're doing stupid user-agent filtering, as passing a different user-agent header returns a 200 response.

Seems any user-agent string with 'curl' or 'python' (or 'ruby', even) in it results in a 403:

$ printf '%-32s %s\n' user-agent response; for agent in web-enabled-hair-{straighteners,curlers} 'some script written in '{ruby,python,bash} ; do printf '%-32s ' "${agent}"; curl -s -o /dev/null -w '%{http_code}\n' -A "$agent" 'https://lewisham.gov.uk/myservices/recycling-and-rubbish/your-bins/collection' ; d
one
user-agent                       response
web-enabled-hair-straighteners   200
web-enabled-hair-curlers         403
some script written in ruby      403
some script written in python    403
some script written in bash      200

Source (if relevant)

lewisham_gov_uk

Logs

no relevant logs

Relevant Configuration

No response

Checklist Source Error

  • Use the example parameters for your source (often available in the documentation) (don't forget to restart Home Assistant after changing the configuration)
  • Checked that the website of your service provider is still working
  • Tested my attributes on the service provider website (if possible)
  • I have tested with the latest version of the integration (master) (for HACS in the 3 dot menu of the integration click on "Redownload" and choose master as version)

Checklist Sensor Error

  • Checked in the Home Assistant Calendar tab if the event names match the types names (if types argument is used)

Required

  • I have searched past (closed AND opened) issues to see if this bug has already been reported, and it hasn't been.
  • I understand that people give their precious time for free, and thus I've done my very best to make this problem as easy as possible to investigate.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@theawesomestrob