* header.go ContentEncoding() getter and setters
For Response the CE header is stored into a separate field because compressed responses are often used.
But for the Request let's just peek and store it from headers map
* http.go: New BodyUncompressed() method for request and responses
The new method returns a body and uncompress if it's gzipped
* Response.ContentEncoding(): store as field
The CE is not so often used for plain APIs responses and even not so often used for static files and on the fly compression.
But still it should be checked each time.
Also having a dedicated field getter and setter simplifies code
* header.go Use shorter Response.setNonSpecial() and Request.setNonSpecial() methods instead of SetCanonical()
The change should improve performance because the setSpecialHeader() call is omitted.
As a downside on adding a new basic header field all putHeader() must be replaced with a direct getter and setter.
* Read response when client closes connection #1232
* Fix edge case were client responds with invalid header
* Follow linter suggestions for tests
* Changes after review
* Reafactor error check after review
* Handle connection reset on windows
* Remove format string from test where not needed
* Run connection reset tests not on Windows
* feature: add request and response body size limit, it prevents the large body from slowly stretching the memory of the entire service
* fix: http.go fmt
* refact: optimize code naming
* Update http.go
Co-authored-by: Erik Dubbelboer <erik@dubbelboer.com>
* Update http.go
Co-authored-by: Erik Dubbelboer <erik@dubbelboer.com>
* Update http.go
Co-authored-by: Erik Dubbelboer <erik@dubbelboer.com>
* Update http.go
Co-authored-by: Erik Dubbelboer <erik@dubbelboer.com>
* Update http.go
Co-authored-by: Erik Dubbelboer <erik@dubbelboer.com>
Co-authored-by: Erik Dubbelboer <erik@dubbelboer.com>
* Allow to set Host header for Client
* Allow to change Host header without tests violation
* Rename AllowToChangeHostHeader and add tests.
* Allow to use empty uri.Host() when req.Header.Host() does not empty
Currently, the only way to set URI for a request is to call SetRequestURI(string).
Then when a request performed the string will be parsed into a fasthttp.URI struct.
If there are many requests with the same URI then we'll waste CPU for a parsing of the same URI string.
With the new SetURI(*URI) method we can once parse a URI string into a fasthttp.URI struct and then reuse it for many requests.
Unfortunately the URI will be copied because may be modified inside the request.
But anyway this will be more lightweight than parsing.
* Adding ConvertHTTPRequest and renaming ConvertRequest to ConvertFastRequest
* Removing forServer boolean from ConvertHTTPRequest
* Preparing for PR
* Reverting adaptor changes
* Fixing godoc, adding req.ReadBody function as well
* Update comment to be more clear
As per @erikdubbelboer suggestion
Co-authored-by: Erik Dubbelboer <erik@dubbelboer.com>
Co-authored-by: Erik Dubbelboer <erik@dubbelboer.com>
* http: refactor out crlf reading function
Make it a bit simpler and make it reusable.
* streaming: fix chunked stream test
This test is supposed to check for stream unchunking, but it's not really
effective with that because chunks created by createChunkedBody() get wrapped
into another chunk by writeBodyStream(), so we end up with chunkedBody in
request handler although what we really want is plain body.
Deduplicate test and benchmark also.
* streaming: fix Read interface
It wasn't actually compatible with io.Reader as io.Reader _never_ returns n >
len(p) while this function easily did that for chunked payloads confusing its
users:
panic: runtime error: slice bounds out of range [:528] with capacity 512
goroutine 562 [running]:
io.ReadAll(0x9f4380, 0xc0003be1a0, 0xc0004fcd80, 0x0, 0x0, 0xc00086bc30, 0x46f99b)
/usr/lib64/go/1.16/src/io/io.go:634 +0x205
io/ioutil.ReadAll(...)
/usr/lib64/go/1.16/src/io/ioutil/ioutil.go:27
github.com/valyala/fasthttp.getChunkedTestEnv.func1(0xc001fdc680)
/home/rik/dev/fasthttp/streaming_test.go:108 +0x6c
github.com/valyala/fasthttp.(*Server).serveConn(0xc000416d80, 0xa034e8, 0xc0004da880, 0x0, 0x0)
/home/rik/dev/fasthttp/server.go:2219 +0x12ee
github.com/valyala/fasthttp.(*workerPool).workerFunc(0xc000148960, 0xc0003be160)
/home/rik/dev/fasthttp/workerpool.go:223 +0xba
github.com/valyala/fasthttp.(*workerPool).getCh.func1(0xc000148960, 0xc0003be160, 0x8b4ec0, 0xc0003be160)
/home/rik/dev/fasthttp/workerpool.go:195 +0x35
created by github.com/valyala/fasthttp.(*workerPool).getCh
/home/rik/dev/fasthttp/workerpool.go:194 +0x11f
It also returned len(p) in some cases where it read less than that.
- Allow DisablePreParseMultipartForm in combination with
StreamRequestBody.
- Support streaming into MultipartForm instead of reading the whole body
first.
- Support calling ctx.PostBody() when streaming is enabled.
New Functions:
CompressHandlerBrotliLevel(h RequestHandler, brotliLevel, otherLevel int) RequestHandler
Request.BodyUnbrotli() ([]byte, error)
Response.BodyUnbrotli() ([]byte, error)
AppendBrotliBytesLevel(dst, src []byte, level int) []byte
WriteBrotliLevel(w io.Writer, p []byte, level int) (int, error)
WriteBrotli(w io.Writer, p []byte) (int, error)
AppendBrotliBytes(dst, src []byte) []byte
WriteUnbrotli(w io.Writer, p []byte) (int, error)
AppendUnbrotliBytes(dst, src []byte) ([]byte, error)
New Constants:
CompressBrotliNoCompression
CompressBrotliBestSpeed
CompressBrotliBestCompression
CompressBrotliDefaultCompression
Brotli compression levels are different from gzip/flate. Because of this we have separate level constants and CompressHandlerBrotliLevel takes 2 levels.
I didn't add Brotli support to CompressHandler as this could cause a spike in CPU usage when users upgrade fasthttp.
fasthttp.CompressBrotliDefaultCompression is not the same as
brotli.DefaultCompression. brotli.DefaultCompression is more than twice
as slow as fasthttp.CompressBrotliDefaultCompression which I thought was
unreasonable as default.
* feat: make client to wait when no free connections
* feat: make client to wait when no free connections
use AcquireTimer to do timeout instead of using context
* feat: make client to wait when no free connections
Add BenchmarkClientGetEndToEndWaitConn* to test heap allocation
in waiting for free connection situation
* feat: make client to wait when no free connections
Add BenchmarkHTTPClientGetEndToEndWaitConn* to test heap allocation
in waiting for free connection situation
* feat: make client to wait when no free connections
fix bug in BenchmarkHTTPClientGetEndToEndWaitConn*
* feat: make client to wait when no free connections
fix bug in TestHostClientMaxConnWaitTimeoutSuccess make it wait
longer to avoid ErrNoFreeConns on travis-ci
* feat: make client to wait when no free connections
fix do not compile benchmark(NetHTTP?)ClientGetEndToEndWaitConn
if go version < 1.11.x
* feat: make client to wait when no free connections
fix the bug that if deadline is earlier than MaxConnWaitTimeout,
still wait MaxConnWaitTimeout which later than deadline.
* feat: make client to wait when no free connections
fix race condition in TestHostClientMaxConnWaitTimeoutError
* feat: make client to wait when no free connections
fix bug in TestHostClientMaxConnWaitTimeoutWithEarlierDeadline
This means we can't skip parsing headers for GET requests anymore. This
can be seen as good as it also allows us to reject malformed GET
requests, something we didn't do before this. Performance also isn't
affect much:
benchmark old ns/op new ns/op delta
BenchmarkClientGetEndToEnd1Inmemory-16 640 641 +0.16%
BenchmarkClientGetEndToEnd10Inmemory-16 713 710 -0.42%
BenchmarkClientGetEndToEnd100Inmemory-16 732 749 +2.32%
BenchmarkClientGetEndToEnd1000Inmemory-16 759 774 +1.98%
BenchmarkClientGetEndToEnd10KInmemory-16 785 808 +2.93%
BenchmarkNetHTTPClientGetEndToEnd1Inmemory-16 5045 4954 -1.80%
BenchmarkNetHTTPClientGetEndToEnd10Inmemory-16 5806 6225 +7.22%
BenchmarkNetHTTPClientGetEndToEnd100Inmemory-16 7877 7998 +1.54%
BenchmarkNetHTTPClientGetEndToEnd1000Inmemory-16 16603 16559 -0.27%
- Some tests can't be run in parallel.
- `URI` had a pointer to `RequestHeader` which was updated with
`RequestHeader.CopyTo` which resulted in the URI pointing to the wrong
`RequestHeader` causing bugs and race conditions.
The only reason `URI` contained a pointer to `RequestHeader` was to delay the
call to `RequestHeader.Host()` until really needed. But these days instead
of parsing all headers, `RequestHeader.Host()` uses
`RequestHeader.peekRawHeader()` which is rather fast. So we can remove the
pointer in `URI` and completely decouple the two structs improving code
quality and fixing the bug.
For some reason this results in faster code on average as well:
benchmark old ns/op new ns/op delta
BenchmarkClientGetEndToEnd1Inmemory-8 1189 1369 +15.14%
BenchmarkClientGetEndToEnd10Inmemory-8 1143 1161 +1.57%
BenchmarkClientGetEndToEnd100Inmemory-8 1228 1236 +0.65%
BenchmarkClientGetEndToEnd1000Inmemory-8 1213 1213 +0.00%
BenchmarkClientGetEndToEnd10KInmemory-8 1362 1350 -0.88%
BenchmarkClientEndToEndBigResponse1Inmemory-8 139967 130070 -7.07%
BenchmarkClientEndToEndBigResponse10Inmemory-8 142233 131809 -7.33%
BenchmarkServerGet1ReqPerConn-8 1726 1593 -7.71%
BenchmarkServerGet2ReqPerConn-8 882 927 +5.10%
BenchmarkServerGet10ReqPerConn-8 440 436 -0.91%
BenchmarkServerGet10KReqPerConn-8 341 339 -0.59%
BenchmarkServerPost1ReqPerConn-8 1728 1706 -1.27%
BenchmarkServerPost2ReqPerConn-8 968 963 -0.52%
BenchmarkServerPost10ReqPerConn-8 506 505 -0.20%
BenchmarkServerPost10KReqPerConn-8 424 420 -0.94%
BenchmarkServerGet1ReqPerConn10KClients-8 1117 1051 -5.91%
BenchmarkServerGet2ReqPerConn10KClients-8 565 514 -9.03%
BenchmarkServerGet10ReqPerConn10KClients-8 390 387 -0.77%
BenchmarkServerGet100ReqPerConn10KClients-8 355 348 -1.97%
BenchmarkServerHijack-8 339 348 +2.65%
BenchmarkServerMaxConnsPerIP-8 326 325 -0.31%
BenchmarkServerTimeoutError-8 24355 24180 -0.72%
* Add `Response.SetBodyRaw` method that serves a `[]byte` slice without touching it (as an alternative to `SetBody`)
* Update various response related functions that are impacted after the incoduction of `Response.bodyRaw`
* Add a few test-cases in relation to `Response.SetBodyRaw`
Not doing this means people using Client can still inspect the response
headers when reading the body fails (for example when it is too big).
fixes#456
Sergey Ponomarev
Erik Dubbelboer
zzzzwc
ArminBTVS
Rennbon
Tolyar
ichx
Shivansh Vij
Evgenii
Meng
MoreFreeze
Roman Khimov
Daniel Firsht
Vladimir Shteinman
Kirill Danshin
kiyon
Mohammad Alian
So-chiru
Markmerc
Daniel Qian
alexDango
Marcelo Pires
Ciprian Dorin Craciun
xuecai
Berezhnoy Pavel
Gabriel Pérez S