Support streaming uploads

Does feign support streaming as opposed to byte[] in memory? Saw a former co-worker ask on SO.

pasting what I said on SO:

Feign was designed for control plane apis, which often don't benefit by streaming upwards. Streaming downwards is supported, though.

I've no concern with being more efficient with regards to how buffering works (ex. alternative to byte array). Just bear in mind that most of feign's design revolves around templating forms (json or xml) and reusing these as much as possible (ex. on retransmit, buffered + fixed length is easy and predictable).

I think I'd be most happy about a "streaming" design if it were coupled to the http client. IOTW, a subtype that addresses streaming in a way that makes sense in the transport. For example, InputStream for regular java, OkIo buffer for OkHttp, Netty ByteBuff for Netty, etc.

ps thanks.. I wasn't watching the netflix-feign tag on SO!

FWIW, what I'm thinking is that the body param would address streaming somehow.
Ex. when okhttp is installed, simply passing an okio buffer as a parameter would magically work.

    @RequestLine("POST /")
    void body(okio.BufferedSource source);

    @RequestLine("POST /")
    void body(okio.Buffer source);

The tricky part is implicitly getting a correct length or determining chunking. This is where things get complex, and an area I've hoped to avoid as dealing with control apis is complicated enough :)

from an extension POV, a feign Client could register the types it handles natively. These would override the encoder types and skip processing somehow.

ex.

List<Type> transportTypes();

Client.Default might return byte[], InputStream.
OkHttpClient might return com.squareup.okhttp.Request, okio.Buffer, okio.ByteString etc.

just thoughts..

@adriancole

@adriancole thanks for the response. I've ended up pushing feign beyond control plane apis :-) I like the idea of customizing the Client. Maybe the Type mapped to some kind of Handler to actually the processing.

yeah that might work. It could also be the case that we parameterize Client and Request with whatever the ideal string body is. This helps avoid the byte array dumping which was only there to support binary payloads. IOTW when we can handle binary parts (buffered or streaming), we can go back and fix the base case of just addressing strings.

@NiteshKant

Also, there will be a known set of profiles that work together as they all share types. We could try out making ideally integrated stacks. It would likely be less complex in the long run than a lot of extension points.

okhttp (http) + okio (buffers) + moshi (json)
rxnetty (http + buffer) + ?? (json that works well w/ByteBuff)

wdyt? cc @NiteshKant @jdamick

I like the integrated stacks idea as that is what I would want for spring.

yeah stacks would be good, i'm doing something similar to support multipart encoding or not, but it's not all that elegant at the moment.

Any movement on this issue? Need this for a project I'm working on. Would be happy to look at contributing.

@jared2501

I think there are various requests coming in for aspects around buffering, non-blocking, streaming, and asynchronous invocation, but a solution that addresses them all needs to be chosen.

For example, RxNetty could, and choosing one stack to try this with reduces the work involved, plus risk of premature or over abstraction.

@jared2501 would RxNetty work for you? If not, what would?

@uschi2000

Hmm, I think @uschi2000 / @markelliot might have better opinions here than me on our use case.

@adriancole

@adriancole, you said that

Streaming downwards is supported, though.

Can you provide an example? How does one currently enable streamed output, especially under JAXRSConstract?

java.io.InputStream is supported (ex by delaring a method returns feign.Response). Im not sure if there is any special casing for jaxrs

…

On 9 Mar 2017 12:27, "Ivan Sopin" ***@***.***> wrote: @adriancole <https://github.com/adriancole>, you said that Streaming downwards is supported, though. Can you provide an example? How does one currently enable streamed output, especially under JAXRSConstract? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#220 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAD6178E044jEZy4EiD2TkyFbqOOaexEks5rj9QcgaJpZM4D_d6b> .

@adriancole

@adriancole I came upon this thread looking to upload 5 GB+ files using Spring Cloud Feign. You mentioned okio.BufferedSource in a comment above, but it seems to be exact opposite of what I need. I don't want to buffer anything, just stream the InputStream to the destination. Looking at the source code of RequestTemplate, looks like the body has to be a byte[]. Any suggestions?

IMO, mixing up streaming support with async IO is delaying a solution; instead of gaining support for blocking but streaming upload, we have nothing even after 2 years 3 months now.

BufferedSource doesn't imply buffering the entire message. As documented, it allows for read-ahead and other means to be more efficient when putting things on the wire. https://github.com/square/okio/blob/master/okio/src/main/java/okio/BufferedSource.java At the moment, there's no workaround as the api wasn't designed for large uploads. This means none of the plugins would work for that out of the box. Progressing this would imply choosing a model to support things. InputStream is one option, but in reality it is more likely a callback to an outputstream or similar is what would be needed on the output side.

Hello,
I submitted a new feature to support java8 streams in #586.
Please vote if it solves your issue.

@phymbert

@phymbert Java 8 stream and streaming upload are completely different things :(

@asarkar

@asarkar For sure, this is about streaming response as the original issue description, isn't

I've re-titled the issue. Please refrain from arguing about things that don't impact this issue. Remember there are a couple hundred watchers on this repo and this type of conversation spams them.

Sunbelt Computer Software

PL/B Language Development and Support

Navigation Menu

Repository navigation

Support streaming uploads #220

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Sunbelt Computer Software

PL/B Language Development and Support

Support streaming uploads #220

Description

Activity

codefromthecrypt commented on Apr 13, 2015

codefromthecrypt commented on Apr 13, 2015

codefromthecrypt commented on Apr 13, 2015

codefromthecrypt commented on Apr 13, 2015

spencergibb commented on Apr 13, 2015

codefromthecrypt commented on Apr 13, 2015

codefromthecrypt commented on Apr 13, 2015

spencergibb commented on Apr 13, 2015

jdamick commented on Apr 14, 2015

jared2501 commented on Mar 17, 2016

codefromthecrypt commented on Mar 17, 2016

jared2501 commented on Mar 17, 2016

ivansopin commented on Mar 9, 2017

codefromthecrypt commented on Mar 9, 2017

asarkar commented on Jul 20, 2017

codefromthecrypt commented on Jul 21, 2017

phymbert commented on Aug 21, 2017

asarkar commented on Aug 21, 2017

phymbert commented on Aug 21, 2017

codefromthecrypt commented on Aug 22, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions