Differences

This shows you the differences between two versions of the page.

Link to this comparison view

irc:1461880800 [2017/05/27 13:44] (current)
Line 1: Line 1:
 +[00:14:30] <​testn>​ Hi there!
 +
 +[08:36:48] *** ChanServ sets mode: +o temporal_
 +
 +[08:54:55] <​cescoffier>​ ChicagoJohn:​ response -> {} ?
 +
 +[10:23:17] *** ChanServ sets mode: +o temporalfox
 +
 +[15:08:47] <​ranie>​ hey guys, anyone experiencing this issue when using an http client? java.lang.IllegalStateException:​ No response handler at io.vertx.core.http.impl.ClientConnection.handleResponse(ClientConnection.java:​275) at io.vertx.core.http.impl.HttpClientImpl$ClientHandler.doMessageReceived(HttpClientImpl.java:​869) at io.vertx.core.http.impl.HttpClientImpl$ClientHandler.doMessageReceived(HttpClientImpl.java:​847) at io.vertx.core.http.impl.Ver
 +
 +[15:49:12] <​cescoffier>​ ranie : before calling end(), you need to attach a responseHandler on the request: client.get().responseHandler(response -> {....}).end()
 +
 +[15:57:38] <​ranie>​ already did
 +
 +[15:57:42] <​ranie>​ found the problem
 +
 +[15:57:53] <​ranie>​ there was a 301 redirect to https
 +
 +[16:41:52] <​ChicagoJohn>​ anyone alive in here?
 +
 +[16:42:36] <​ChicagoJohn>​ ive got an implementation question
 +
 +[16:44:47] <​temporalfox>​ ChicagoJohn hi
 +
 +[16:44:55] <​temporalfox>​ maybe I can help you
 +
 +[16:45:48] <​ChicagoJohn>​ cool.  so ive created an endpoint that accepts a file upload. ​ that part is working.
 +
 +[16:46:37] <​ChicagoJohn>​ as the file comes in, each chunk slowly fills up a buffer. ​ when that buffer is full, it opens a connection to another remote server and sends that buffer.
 +
 +[16:46:52] <​ChicagoJohn>​ i have all of this working sorta.
 +
 +[16:47:43] <​ChicagoJohn>​ the issue i am facing is that vertx completes the file upload before all of the additional connections sending out the buffers complete
 +
 +[16:48:13] <​ChicagoJohn>​ my guess is that i need to create a worker verticle for the secondary connections to be run within?
 +
 +[16:50:41] <​ChicagoJohn>​ i believe the problem lies in the fact that all new client requests live inside the routerContext.request().uploadHandler() callback. ​ and once the routerContext.response().end() is called, it collapses all of the internal client requests and therfore the client responses have no where to return to
 +
 +[17:01:29] <​temporalfox>​ hum
 +
 +[17:01:48] <​ChicagoJohn>​ right....
 +
 +[17:01:55] <​temporalfox>​ I'​m ​ reading
 +
 +[17:02:03] <​ChicagoJohn>​ no worries
 +
 +[17:02:21] <​temporalfox>​ so you receive a file upload
 +
 +[17:02:30] <​temporalfox>​ and you wan t to transform the bytes to another server
 +
 +[17:02:33] <​temporalfox>​ using a connection
 +
 +[17:02:56] <​temporalfox>​ "to transfer"​
 +
 +[17:05:39] <​ChicagoJohn>​ yes.  my vertx app is merely a middle man at this point but its growing into a full app.
 +
 +[17:07:22] <​ChicagoJohn>​ I could always wait for the file upload to complete, and then build a scheduler to push the files to the other server later. ​ with this method i could inspect the files if needed. ​ but then i would need more storage space on my vertx side i guess
 +
 +[17:07:57] <​temporalfox>​ you should use a Pump for this
 +
 +[17:08:06] <​temporalfox>​ basically when you recieve the file upload
 +
 +[17:08:09] <​temporalfox>​ you pause() the request
 +
 +[17:08:14] <​temporalfox>​ then you connect to this other server
 +
 +[17:08:19] <​temporalfox>​ when you get the connection
 +
 +[17:08:40] <​temporalfox>​ you create a Pump between the HttpFileUpload buffer and the HttpClientRequest you connect to
 +
 +[17:08:46] <​temporalfox>​ and you resume() the initial request
 +
 +[17:08:55] <​temporalfox>​ look at Pump in vertx core doc
 +
 +[17:09:34] <​ChicagoJohn>​ will do.  thanks
 +
 +[17:09:49] <​ChicagoJohn>​ you guys are all generally '​overseas'​ right? ​ im in the US
 +
 +[17:12:01] <​temporalfox>​ we are on CET timezone
 +
 +[17:14:53] <​ChicagoJohn>​ yep.  thats why this channel is dead when im at work.  8am here.  thanks for the help ill jump on that pump stuff
 +
 +[17:48:18] <​temporalfox>​ not dead
 +
 +[17:48:30] <​temporalfox>​ I can be there sometimes in the evening
 +
 +[17:48:39] <​temporalfox>​ otherwise you have the vertx google group
 +
 +[19:07:12] <​ChicagoJohn>​ temporalfox,​ if you are still here i have a follow up question regarding pump
 +
 +[19:24:37] <​temporalfox>​ yes
 +
 +[19:24:46] <​temporalfox>​ I'm there for a couple of minutes ChicagoJohn
 +
 +[19:26:00] <​ChicagoJohn>​ so the server i have to send my data to places a limit on how much data it can receive.
 +
 +[19:26:18] <​ChicagoJohn>​ once that happens, i need to create a new request to a modified endpoint
 +
 +[19:26:48] <​ChicagoJohn>​ starting upload is /<​uploadId>/<​offset>/ ​ which is id/0/
 +
 +[19:27:20] <​ChicagoJohn>​ after the first send, the new url is the same pattern but with an updated offset. ​ which is why i was having to create so many requests in the first place
 +
 +[19:27:58] <​temporalfox>​ ChicagoJohn ok
 +
 +[19:28:08] <​temporalfox>​ then you would need to anticipate this
 +
 +[19:28:23] <​temporalfox>​ does the file uplaod sets a content-lenght header ?
 +
 +[19:28:25] <​ChicagoJohn>​ with the pump, i can only set one read stream and one write stream. ​ can i recreate the pump per request using the original upload request, or would i need to build a new object to handle this
 +
 +[19:28:29] <​temporalfox>​ if yes you can anticipate it
 +
 +[19:28:37] <​temporalfox>​ you can stop the pump
 +
 +[19:28:46] <​temporalfox>​ and create a new one I htink
 +
 +[19:29:06] <​temporalfox>​ the pump itself is quite easy code
 +
 +[19:29:16] <​temporalfox>​ you could fork it and adapt it to your needs
 +
 +[19:29:22] <​temporalfox>​ if you can't make it work
 +
 +[19:29:32] <​temporalfox>​ but if you get the idea of how the pump works, it's the most important
 +
 +[19:30:01] <​temporalfox>​ perhaps one thing you could do is
 +
 +[19:30:06] <​temporalfox>​ create a WriteStream
 +
 +[19:30:10] <​temporalfox>​ implementation
 +
 +[19:30:20] <​temporalfox>​ that creates the many files
 +
 +[19:30:29] <​temporalfox>​ when a file is full
 +
 +[19:30:31] <​temporalfox>​ it stop the request
 +
 +[19:30:35] <​temporalfox>​ creates a new request
 +
 +[19:30:45] <​temporalfox>​ and until the request is created
 +
 +[19:30:48] <​temporalfox>​ you buffer the buffers
 +
 +[19:30:58] <​temporalfox>​ and you return true in writeQueueFull
 +
 +[19:31:02] <​temporalfox>​ when the request is created
 +
 +[19:31:09] <​temporalfox>​ you consume the buffers
 +
 +[19:31:13] <​temporalfox>​ and you call the drainHandler
 +
 +[19:31:20] <​temporalfox>​ so you can Pump this with the HttpServerRequest
 +
 +[19:31:48] <​temporalfox>​ also another altenative
 +
 +[19:31:55] <​temporalfox>​ is to buffer everything on the file system
 +
 +[19:32:02] <​temporalfox>​ but I think it depends on your use case
 +
 +[19:49:23] <​ChicagoJohn>​ my internet disconnected. ​ did i miss your response?
 +
 +[19:56:01] <​ChicagoJohn>​ temporalfox just checking if you responded
 +
 +[20:03:20] <​testn>​ I have a problem with netSocket() where it leaks a connection https://​github.com/​eclipse/​vert.x/​issues/​1396
 +
 +[20:04:04] <​testn>​ I actually prefer that Http{Client/​Server}{Request/​Response} provides a mechanism to terminate the request/​response early
 +
 +[20:04:39] <​testn>​ we are trying to implement a good http proxy with vert.x but runs into a problem with large content either PUT/GET
 +
 +[20:05:05] <​testn>​ if the connection on one end aborts early, it's difficult to do that the same on the other side of the connection
 +
 +[20:17:58] <​temporalfox>​ ChicagoJohn you can look at the log for the response
 +
 +[20:18:25] <​temporalfox>​ testn I find netSocket() not reliable
 +
 +[20:18:35] <​temporalfox>​ what do you mean by early ?
 +
 +[20:18:46] <​temporalfox>​ and "​terminate"​ :-)
 +
 +[20:19:45] <​temporalfox>​ testn what do you support ? HTTP/1.1 ? keep-alive and/or pipelining ?
 +
 +[20:21:37] <​testn>​ 1.1 with keepalive
 +
 +[20:21:40] <​testn>​ no pipelining
 +
 +[20:22:04] <​testn>​ i hate netSocket... i find that it rarely works
 +
 +[20:22:17] <​testn>​ I would prefer to have a reliable way to close a connection
 +
 +[20:23:10] <​testn>​ regarding my case, I have a proxy to front GET/PUT traffic but when under load, the GET request can slow down and the client may not wait for the response
 +
 +[20:23:40] <​testn>​ so it kills the connection... if I don't close the backend connection, it will keep streaming useless data to a blackhole
 +
 +[20:24:00] <​testn>​ and that kills the bandwidth on my end as user will only keep retrying and make the situation worse
 +
 +[20:24:39] <​testn>​ for the PUT request, I faced another challenge. If the connection is closed in the middle, I don't have a good way to abort a connection
 +
 +[20:25:13] <​testn>​ if I call end() early for chunked http request, it will assume that the request is completed successfully where it does not
 +
 +[20:27:59] <​temporalfox>​ for PUT by "​close"​ you mean your client request or the request you make to the server ?
 +
 +[20:28:21] <​temporalfox>​ ah you mean your client and you want to end() the client request
 +
 +[20:29:07] <​temporalfox>​ so in both case you want to close the connection to the server ?
 +
 +[20:29:43] <​temporalfox>​ testn
 +
 +[20:30:16] <​testn>​ correct
 +
 +[20:30:55] <​temporalfox>​ and what is missing is a HttpClientRequest#​close method
 +
 +[20:31:18] <​testn>​ no such method in 3.2.1
 +
 +[20:31:28] <​temporalfox>​ yes it does not exists
 +
 +[20:31:57] <​testn>​ no HttpClientResponse.close() for the client to close the stream earlier either
 +
 +[20:32:07] <​temporalfox>​ in 3.3 there is reset() method for HTTP/2
 +
 +[20:32:10] <​temporalfox>​ which achieve that
 +
 +[20:32:20] <​temporalfox>​ perhaps we can implement it also for HTTP/1.1 with a simple close
 +
 +[20:32:30] <​temporalfox>​ (instead of adding a new close() method)
 +
 +[20:32:53] <​testn>​ that'd be nice
 +
 +[20:33:02] <​testn>​ how close are we on 3.3?
 +
 +[20:33:15] <​temporalfox>​ it would be in June
 +
 +[20:33:24] <​temporalfox>​ maybe you can try to set a time out ?
 +
 +[20:33:26] <​testn>​ that's still far away
 +
 +[20:33:28] <​temporalfox>​ very small
 +
 +[20:33:45] <​temporalfox>​ not far away according to my perspective and schedule :-)
 +
 +[20:33:48] <​testn>​ timeout cannot be changed after the connection is made
 +
 +[20:33:55] <​temporalfox>​ ok
 +
 +[20:34:44] <​temporalfox>​ maybe you can use one HttpClient per connection and close the client as work around ?
 +
 +[20:34:55] <​temporalfox>​ (I don't think the overhead would be too much)
 +
 +[20:34:55] <​testn>​ righ now timeout i set it to 60 seconds to handle customers with low/​unstable network
 +
 +[20:35:31] <​testn>​ the problem with that is that we cannot reuse the connection
 +
 +[20:35:46] <​testn>​ which is also bad in term of socket in use (especially with CLOSE_WAIT)
 +
 +[20:36:12] <​testn>​ I wish you guys can do a minor release much more often like what netty does
 +
 +[20:36:46] <​temporalfox>​ testn that's something we should do I agree
 +
 +[20:37:17] <​testn>​ all these little bugs here and there kinda put me off from using it production
 +
 +[20:37:24] <​temporalfox>​ testn that's a problem of HTTP : you cannot make a clean abort of a request, specially with keep-alive
 +
 +[20:37:36] <​temporalfox>​ http/2 supports stream reset()
 +
 +[20:37:53] <​testn>​ you can just close it... and thus the connection cannot be reused anymore
 +
 +[20:37:57] <​temporalfox>​ so the best is to close the connection
 +
 +[20:38:14] <​testn>​ the problem is more on the server side actually for keep alive
 +
 +[20:38:26] <​temporalfox>​ wdym ?
 +
 +[20:38:38] <​testn>​ basically, let's say that you make a request and the server responds and then try to close the connection.
 +
 +[20:39:04] <​testn>​ as soon as you read the response from the server, you will put the connection to be reused again only to know that the server has closed the connection
 +
 +[20:39:22] <​testn>​ we found that happened a lot for our busy service
 +
 +[20:39:30] <​temporalfox>​ but the client checks that the connection is not closed before using it
 +
 +[20:39:41] <​temporalfox>​ I agree that the connection might be closed when doing the connection
 +
 +[20:39:42] <​testn>​ so we work around that by sending "​Connection:​ close" back to the client and let the client to close it
 +
 +[20:39:55] <​temporalfox>​ yes that's how an http/1.1 server should behave
 +
 +[20:40:07] <​temporalfox>​ a server that does not send Connection: close should not close the connection
 +
 +[20:40:14] <​testn>​ yup
 +
 +[20:40:22] <​temporalfox>​ I need to take care of my daughter :-)
 +
 +[20:40:35] <​temporalfox>​ can you open an issue for HttpClientRequest reset() for HTTP/1.1 at least ?
 +
 +[20:40:59] <​testn>​ for the server close, it should not close the connection either
 +
 +[20:41:05] <​temporalfox>​ right :-)
 +
 +[20:41:36] <​testn>​ for example, if the request is long or too large or too slow, I can send a response early to tell the client that i'm not interested in serving the request anymore
 +
 +[20:42:22] <​testn>​ if i close the connection, the client will get a socket reset from the sending side before it can read the response.
 +
 +[20:43:07] <​testn>​ regarding HttpClientRequest.reset() it is a weird problem to solve
 +
 +[20:43:28] <​testn>​ HttpClientRequest does not require to have an associated connection
 +
 +[20:43:44] <​testn>​ so when you call reset(), there can be either
 +
 +[20:43:57] <​testn>​ 1.it has not yet established a connection yet
 +
 +[20:44:25] <​testn>​ 2. the connection has already been reused by someone else (in the case that the response comes early)
 +
 +[20:48:40] <​testn>​ @temporalfox what do you think about https://​github.com/​eclipse/​vert.x/​issues/​1369
 +
 +[20:48:47] <​testn>​ do you think it is a valid bug?
 +
 +[21:36:59] <​temporalfox>​ can you provide a reproducer testn  ?
 +
 +[21:37:48] <​temporalfox>​ testn reset should only be called when the client request is associated with the ClientCOnnection otherwise throw an IllegalStateException
 +
 +[21:38:12] <​temporalfox>​ now there is sendHead(Handler<​Void>​) in HttpClientRequest
 +
 +[21:38:29] <​temporalfox>​ actually Handler<​HttpVersion>​
 +
 +[21:38:56] <​temporalfox>​ that is called when the client has connected
 +
 +[21:39:08] <​testn>​ i'd prefer an explicit callback for the connection
 +
 +[21:39:15] <​temporalfox>​ there is one actually now
 +
 +[21:39:25] <​temporalfox>​ but it only works for Http/2
 +
 +[21:39:29] <​testn>​ :(
 +
 +[21:39:34] <​temporalfox>​ HttpClientRequest connectionHandler(@Nullable Handler<​HttpConnection>​ handler);
 +
 +[21:39:41] <​temporalfox>​ it could be extended I think to HTTP/1.1
 +
 +[21:39:54] <​temporalfox>​ but in this case there would be HttpConnection and Http2Connection
 +
 +[21:39:58] <​temporalfox>​ maybe
 +
 +[21:40:11] <​temporalfox>​ because HttpConnection has method that don't make sense for HTTP/1.1
 +
 +[21:41:49] <​temporalfox>​ or they would innoperate
 +
 +[21:41:55] <​temporalfox>​ but it would be confusing to explain
 +
 +[21:47:28] <​testn>​ what is the key differences between HttpConnection/​HttpConnection2?​
 +
 +[21:47:38] <​temporalfox>​ HttpConnection2 does not exists
 +
 +[21:47:39] <​testn>​ the method should be mostly interoperable isn't it?
 +
 +[21:47:50] <​testn>​ Http2Connection i mean
 +
 +[21:48:06] <​temporalfox>​ at the connection level you can change HTTP2 settings
 +
 +[21:48:19] <​temporalfox>​ send GO_AWAY frames
 +
 +[21:48:22] <​temporalfox>​ or handle them
 +
 +[21:49:03] <​temporalfox>​ these would only work for http/2
 +
 +[21:54:22] <​testn>​ I see
 +
 +[21:55:03] <​testn>​ for the connection close issue, I think I can probably work around it for now using reflection while waiting for 3.3
 +
 +[21:56:20] <​testn>​ what do you think of netsocket going forward? it seems to be very bug ridden
 +
 +[22:08:54] <​temporalfox>​ testn I think it's only usage today is for the CONNECT support
 +
 +[22:09:06] <​temporalfox>​ that's what it was designed for
 +
 +[22:09:18] <​temporalfox>​ given that with HTTP/2 netSOcket() returns a "​fake"​ socket
 +
 +[22:09:35] <​temporalfox>​ that uses an HTTP/2 stream with Data frames
 +
 +[22:09:54] <​temporalfox>​ so it's more a pair of ReadStream/​WriteStream
 +
 +[22:10:23] <​temporalfox>​ testn so you would like to have an HttpConnection that works for HTTP/1.1 in 3.3 ?
 +
 +[22:19:15] <​testn>​ i think it's much more compatible
 +
 +[22:19:29] <​testn>​ user don't need to treat them differently