OTA update via platformio not working and Preferences.h questions #72

PPKGroup · 2022-01-20T03:54:51Z

Hi,

I'm using platformio (on VSC) and Win10 for programming of my device.
In my platformio.ini file I have placed the following code:
[env:WiFi]
upload_protocol = espota
upload_port = 10.100.8.60

When I try to build and upload, I get the following error:

Uploading .pio\build\WiFi\firmware.bin
14:18:41 [DEBUG]: Options: {'esp_ip': '10.100.8.60', 'host_ip': '0.0.0.0', 'esp_port': 3232, 'host_port': 48565, 'auth': '', 'image': '.pio\build\WiFi\firmware.bin', 'spiffs': False, 'debug': True, 'progress': True, 'timeout': 10}
14:18:41 [INFO]: Starting on 0.0.0.0:48565
14:18:41 [INFO]: Upload size: 1284432
Sending invitation to 10.100.8.60 ..........
14:18:41 [ERROR]: No response from the ESP
*** [upload] Error 1

If I browse to the device e.g. http://10.100.8.60/update, the ElegantOTA web page is displayed.

Any ideas on how to get this to work through the platformio environment?

I read somewhere on the RNT website that ElegantOTA overwrites the spiffs and Preferences (NVS) flash storage.
In platformio is there a way of limiting where ElegantOTA writes to?

Thanks,
Neil.

viktak · 2023-09-13T10:18:41Z

I would also like to have this feature implemented... It was sooo easy to use it before Asynwebserver...

normen · 2023-09-22T22:01:48Z

For me it works like this with the extra script, note the URL:

extra_scripts = platformio_upload.py
upload_protocol = custom
upload_url = http://192.168.123.123/ota/upload

Edit: Wait, no. It just gives an "OK" but the firmware isn't written 😕

mindsuru · 2023-10-19T16:46:01Z

ive had the same problem. For me it helped to use these platformio.ini configurations

extra_scripts = platformio_upload.py
upload_protocol = custom
upload_url = http://[IP or hostname] ; eg http://192.168.178.20

and i had to modify the platformio_upload.py file, and excuse me, i haven't been programming very long. It's not pretty but it works for me.
It looks like something was changed in the ElegantOTA library because of the GET and POST requests, so the script encounters problems. So I didn't do more than mimic the behavior of the browser. And thats it:


# Allows PlatformIO to upload directly to ElegantOTA
#
# To use:
# - copy this script into the same folder as your platformio.ini
# - set the following for your project in platformio.ini:
#
# extra_scripts = platformio_upload.py
# upload_protocol = custom
# upload_url = <your upload URL>
# 
# An example of an upload URL:
# upload_URL = http://192.168.1.123

import requests
import hashlib
from urllib.parse import urlparse
Import("env")

try:
    from requests_toolbelt import MultipartEncoder, MultipartEncoderMonitor
    from tqdm import tqdm
except ImportError:
    env.Execute("$PYTHONEXE -m pip install requests_toolbelt")
    env.Execute("$PYTHONEXE -m pip install tqdm")
    from requests_toolbelt import MultipartEncoder, MultipartEncoderMonitor
    from tqdm import tqdm

def on_upload(source, target, env):
    firmware_path = str(source[0])
    upload_url = env.GetProjectOption('upload_url')

    with open(firmware_path, 'rb') as firmware:
        md5 = hashlib.md5(firmware.read()).hexdigest()

        parsed_url = urlparse(upload_url)
        host_ip = parsed_url.netloc

        # Führe die GET-Anfrage aus
        start_url = f"{upload_url}/ota/start?mode=fr&hash={md5}"
        # start_response = requests.get(start_url)
        start_headers = {
            'Host': host_ip,
            'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/118.0',
            'Accept': '*/*',
            'Accept-Language': 'de,en-US;q=0.7,en;q=0.3',
            'Accept-Encoding': 'gzip, deflate',
            'Referer': f'{upload_url}/update',
            'Connection': 'keep-alive'
            }

        start_response = requests.get(start_url, headers=start_headers)
        
        # Drucke Anfrage- und Antwortkopfzeilen für die GET-Anfrage
        # print("GET Anfragekopfzeilen:", start_response.request.headers)
        # print("GET Antwortkopfzeilen:", start_response.headers)

        if start_response.status_code != 200:
            print("Start-Request fehlgeschlagen " + str(start_response.status_code))
            return

        firmware.seek(0)
        encoder = MultipartEncoder(fields={
            'MD5': md5,
            'firmware': ('firmware', firmware, 'application/octet-stream')}
        )

        bar = tqdm(desc='Upload Progress',
                   total=encoder.len,
                   dynamic_ncols=True,
                   unit='B',
                   unit_scale=True,
                   unit_divisor=1024
                   )

        monitor = MultipartEncoderMonitor(encoder, lambda monitor: bar.update(monitor.bytes_read - bar.n))

        post_headers = {
            'Host': host_ip,
            'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/118.0',
            'Accept': '*/*',
            'Accept-Language': 'de,en-US;q=0.7,en;q=0.3',
            'Accept-Encoding': 'gzip, deflate',
            'Referer': f'{upload_url}/update',
            'Connection': 'keep-alive',
            'Content-Type': monitor.content_type,
            'Content-Length': str(monitor.len),
            'Origin': f'{upload_url}'
        }


        response = requests.post(f"{upload_url}/ota/upload", data=monitor, headers=post_headers)
        
        # Drucke Anfrage- und Antwortkopfzeilen für die POST-Anfrage
        # print("POST Anfragekopfzeilen:", response.request.headers)
        # print("POST Antwortkopfzeilen:", response.headers)

        bar.close()

        if response.status_code != 200:
            print("Upload fehlgeschlagen")
        else:
            print("Upload erfolgreich", response.text)

            
env.Replace(UPLOADCMD=on_upload)

ayushsharma82 · 2023-10-19T17:03:28Z

@mindsuru The internal API of ElegantOTA indeed got changed to provide a much better & stable experience. Your fix looks promising! It was a community PR that added it in past that's why it wasn't updated with V3.

If you don't mind, can you open a PR that fixes the platformio_upload.py script?

mindsuru · 2023-10-19T22:06:07Z

Thanks,
and okay, I'll give it a try. I'm not that familiar with GitHub yet, but it should work out somehow. I'll read up on it.

I changed the python file slightly. Now you can use it with the old configuration in your platformio.ini

viktak · 2023-10-30T14:28:37Z

@mindsuru For me the version you have here does not work. It gives the following error, then quits:

....
Uploading .pio\build\esp32\firmware.bin
Start-Request fehlgeschlagen 401

Seems to me that it cannot find the OTA page... This is the relevant part of my platformio.ini:

extra_scripts = 
    pre:../../scripts/preIncrementBuildNumber.py
    platformio_upload.py
upload_protocol = custom
upload_url = http://192.168.123.137

As you can see I have an extra script running, but I don't suppose that would be an issue as the behavior is that same when I remove it.
If I can help testing any new version of your script, don't hold back :)

bartoszzajac · 2023-11-21T17:59:41Z

@mindsuru @ayushsharma82 When can we expect PR and merge?

mindsuru · 2023-11-22T16:40:34Z

@viktak is your device reachable with this ip you‘ve choosen in your platformio.ini file?

ayushsharma82 · 2023-11-22T16:54:54Z

@mindsuru @ayushsharma82 When can we expect PR and merge?

It has been merged but yet to be released.

ayushsharma82 · 2023-11-22T16:56:14Z

@viktak is your device reachable with this ip you‘ve choosen in your platformio.ini file?

I guess he's using authentication that's why response code was '401' which means 'Unauthorized'. The script might need adjustments to work with authentication enabled.

viktak · 2023-11-22T16:56:38Z

Yes, the device is reachable, all other functions of the asyncwebserver and elegantota are all working.

viktak · 2023-11-22T16:57:31Z

@viktak is your device reachable with this ip you‘ve choosen in your platformio.ini file?

I guess he's using authentication that's why response code was '401' which means 'Unauthorized'. The script might need adjustments to work with authentication enabled.

Yessss, I definitely use authentication.

But the authentication is added on a per page basis, i.e. for each of the protected pages (not all of them are) I add this line:

                if(!request->authenticate(ADMIN_USERNAME, appSettings.adminPassword))
                    return request->requestAuthentication();

So the pages that don't use authentication should work just fine. In fact, those pages of mine that do not require authentication work fine.

mindsuru · 2023-11-23T07:35:16Z

Okay good to know. I will have a look on it. But unfortunatly i‘m ill right now. It may take a bit.

viktak · 2023-11-23T07:54:58Z

Okay good to know. I will have a look on it. But unfortunatly i‘m ill right now. It may take a bit.

Get well soon, that's the priority!

mindsuru · 2023-11-24T10:51:03Z

i think thats it. now both upload methods should work. regardless of whether you use the encrypted or non-encrypted method.
i opened a pull request for that change

viktak · 2023-11-24T11:07:06Z

@mindsuru Yes, it is now working.
I just checked the version in the pull request, and at first it failed with a message that there was no username defined, but once I defined all the credentials the upload worked flawlessly.
Here are all the relevant additions in platformio.ini I had to make it work:

extra_scripts = 
    platformio_upload.py

build_flags =
    '-DELEGANTOTA_USE_ASYNC_WEBSERVER = 1'

lib_deps =
    ottowinter/ESPAsyncWebServer-esphome @ ^3.1.0
    esphome/AsyncTCP-esphome @ ^2.0.1
    ayushsharma82/ElegantOTA @ ^3.1.0

upload_protocol = custom
upload_url = http://w.x.y.z
username = username 
password = password

viktak · 2023-11-24T17:06:54Z

One more request, perhaps it is possible to rename the expected variables from this:

upload_url = http://w.x.y.z
username = username 
password = password

to this:

custom_upload_url = http://w.x.y.z
custom_username = username 
custom_password = password

This is to get rid of those pesky PlatformIO warning:

Warning! Ignore unknown configuration option `upload_url` in section [env:esp32]
Warning! Ignore unknown configuration option `username` in section [env:esp32]
Warning! Ignore unknown configuration option `password` in section [env:esp32]

mindsuru · 2023-11-24T17:21:36Z

sure and thanks for this information

viktak · 2023-12-29T10:16:59Z

I have been using the updated script for a couple of months now on various projects. Here are my observations:

Doing an update from the /update page works all the time, every time, be it the firmware or the spiffs.
From PlatformIO, uploading the code works perfectly all the time, every time.
However, uploading the spiffs from PlatformIO is a hit and miss for me, and I haven't figured out what the problem might be. I haven't even noticed a pattern, to be honest. Sometimes it works, sometimes it fails during upload, sometimes it goes fine to 100%, then it says:

Upload Progress: 100%|██████████| 1.50M/1.50M [00:07<00:00, 205kB/s]

Upload faild.
Server response: Failed to write chunked data to free space

or it complains about the magic byte (I can't remember the exact message...)
4. Some typos in the console: "faild" *3 and "successfull" *1

github-actions · 2024-07-02T06:39:01Z