Need a sanity check on EOF error

[RusBus]

I'm woefully under educated on golang, so forgive me as I learn and tinker.

I'd like to learn how uTLS works in order to integrate it in other projects. My first step is to take and test a few snippets of bare minimum code from examples.go, starting with the HttpGetDefault function. I'd like to run a simple GET request to different websites, examine the client hello on wireshark, and move on from there. I've commented out lines with the endpoints to test requestHostname and requestAddr and added my own below it. Commenting/uncommenting those and running a quick go run . returns an expected HTTP response code (302, 404 etc) when using the default endpoint of facebook.com or even testing against google.com.

WHAT IS EXPECTED: When modifying the values hardcoded in requestHostname and requestAddr I should get an http response with at least something like 302, 200, or 404.
PROBLEM: However, I get the following error when switching that to other websites: #> HttpGetDefault failed: unexpected EOF

When changing the var requestHostname to a different address, it returns an EOF. I've searched everywhere I could but keep reading suggestions to use http.Close() = true but this also doesn't seem to work, or I don't understand where to place it. I think the defer.tlsConn.Close() performs this task but again I don't have a thorough enough understanding of this library yet.

Full code:

package main

import (
	"bufio"
	//"encoding/hex"
	"fmt"
	"net"
	"net/http"
	"net/http/httputil"
	"net/url"
	"time"

	"github.com/refraction-networking/utls"
	"golang.org/x/net/http2"
)

var (
	dialTimeout   = time.Duration(15) * time.Second
	sessionTicket = []uint8(`Here goes phony session ticket: phony enough to get into ASCII range
Ticket could be of any length, but for camouflage purposes it's better to use uniformly random contents
and common length. See https://tlsfingerprint.io/session-tickets`)
)

//  ***DEFAULT IN uTLS.  This works normally:
//var requestHostname = "facebook.com" // speaks http2 and TLS 1.3
//var requestAddr = "31.13.72.36:443"

//  ***MAIN PROBLEM: Using this returns an EOF:
var requestHostname = "tools.scrapfly.io" // speaks http2 and TLS 1.3
var requestAddr = "34.145.9.240:443"

//  *** Also added this for a sanity check: 
//var requestHostname = "google.com" // speaks http2 and TLS 1.3
//var requestAddr = "8.8.8.8:443"

func HttpGetDefault(hostname string, addr string) (*http.Response, error) {
	config := tls.Config{ServerName: hostname, InsecureSkipVerify: true,}
	fmt.Printf("#########\n")
	fmt.Printf("TLS Config ServerName: %v\n", config.ServerName)

	dialConn, err := net.DialTimeout("tcp", addr, dialTimeout)
	fmt.Printf("dialConn Info: %v\n", dialConn)
	if err != nil {
		return nil, fmt.Errorf("net.DialTimeout error: %+v", err)
	}
	tlsConn := tls.Client(dialConn, &config)
	fmt.Printf("tlsConn Info: %v\n", tlsConn)
	fmt.Printf("tlsConn Info: %v\n", tlsConn.ConnectionState())
	fmt.Printf("tlsConn Info: %v\n", tlsConn.ConnectionState().NegotiatedProtocol)
	
	defer tlsConn.Close()
	fmt.Printf("tlsConn Info: %v\n", tlsConn.ConnectionState())
	return httpGetOverConn(tlsConn, tlsConn.ConnectionState().NegotiatedProtocol)
}

func httpGetOverConn(conn net.Conn, alpn string) (*http.Response, error) {
	req := &http.Request{
		Method: "GET",
		//URL:    &url.URL{Host: "www." + requestHostname + "/"},
		//URL:    &url.URL{Host: "" + requestHostname + "/api/tls"},
        URL:    &url.URL{Host: "" + requestHostname + "/"},
		Header: make(http.Header),
		//Host:   "www." + requestHostname,
		Host:   "" + requestHostname,		
        //Host:   "" + requestHostname,
	}

	switch alpn {
	case "h2":
		req.Proto = "HTTP/2.0"
		req.ProtoMajor = 2
		req.ProtoMinor = 0

		tr := http2.Transport{}
		cConn, err := tr.NewClientConn(conn)
		if err != nil {
			return nil, err
		}
		return cConn.RoundTrip(req)
	case "http/1.1", "":
		req.Proto = "HTTP/1.1"
		req.ProtoMajor = 1
		req.ProtoMinor = 1

		err := req.Write(conn)
		if err != nil {
			return nil, err
		}
		return http.ReadResponse(bufio.NewReader(conn), req)
	default:
		return nil, fmt.Errorf("unsupported ALPN: %v", alpn)
	}
}

func main() {
	var response *http.Response
	var err error
	
	response, err = HttpGetDefault(requestHostname, requestAddr)
	fmt.Printf("#> RESPONSE: %+v\n", response)
	if response == nil {
		fmt.Printf("RESPONSE IS NIL!!: %+v\n", response)
		
	}

	if err != nil {
		fmt.Printf("#> HttpGetDefault failed: %+v\n", err)
		fmt.Printf("Hostname info: %v\n", requestHostname)
		fmt.Printf("requestAddr info: %v\n", requestAddr)
		fmt.Errorf("# ERROR INFO: %w\n", err)
		
	} else {
		fmt.Printf("#> HttpGetDefault response: %+s\n", dumpResponseNoBody(response))
		fmt.Printf("Hostname info: %v\n", requestHostname)
		fmt.Printf("requestAddr info: %v\n", requestAddr)
		fmt.Printf("#> HttpGetDefault FULL response: %+s\n", response)
	}

}

func dumpResponseNoBody(response *http.Response) string {
	resp, err := httputil.DumpResponse(response, false)
	if err != nil {
		return fmt.Sprintf("failed to dump response: %v", err)
	}
	return string(resp)
}

[gaukas]

Interesting observation. While my experience with Go's http library is limited and could not give a conclusive explanation, I would suggest the following steps to take to investigate.

1. Enable the key logger in Config
2. Use Wireshark to inspect the complete TLS handshake AND the following HTTP communication (you will need the key recorded by the key logger)
3. Compare to the behavior with when it is considered "normal".

Usually if you can observe the TLS handshake being completed correctly (and not followed by secured renegotiation), uTLS's job is done and the problem is more likely in how you used the HTTP libraries.

One last thing to note, that io.ErrUnexpectedEOF is considered not equivalent to io.EOF.