Asynchronous HTTP Client - Clojure - Documentation

When you do:

(http/GET <client> url)

Result will be a map of clojure.core/promises, and represents response.

Following HTTP methods have been covered so far:

• GET
• POST
• PUT
• DELETE
• HEAD
• OPTIONS

For detailed description see HTTP methods.

You can submit options to HTTP methods as keyworded arguments, like this:

(http/GET <client> url :query {:key "value"})

Following options are supported:

:query

query parameters

:headers

custom headers to be sent out

:body

body to be sent, allowed only with PUT/POST

:cookies

cookies to be sent

:proxy

proxy to be used

For detailed usage of options see Request options.

Response map contains following keys:

:status

promise of lazy map of status fields

:code

response code

:msg

response message

:protocol

protocol with version

:major

major version of protocol

:minor

minor version of protocol

:headers

promise of lazy map of headers where header names are keyworded, like :server for example

:body

promise of response body, this is ByteArrayOutputStream, but you have convenience functions to convert it for example to string:

(http/string (http/GET <client> <url>))

:done

promise that is delivered once response receiving is done

:error

promise, if there was an error you will find Throwable here

For consuming HTTP streams use:

(http/stream-seq <client> :get url)

Response here is same as in Asynchronous operations but :body will be lazy sequence of ByteArrayOutputStreams.

You can still use convenience functions like http/string for body, but remember that you are dealing now with seq.

For more details please see Lazy sequence.

This allows you to provide callbacks that will get triggered on HTTP response events like:

• received status line,
• received headers,
• received body part,
• completed request,
• handle error.

All callbacks are expected to return tuple with first element been a value to be delivered for given response processing phase, second element is controlling execution and if you make it :abort than processing response is going to be terminated.

For detailed information on how to use this mode please see Low level.

Most basic invocation of http.async.client/GET is only with url you want to get. Extended invocation includes options that can be any options accepted by http.async.client.request/prepare-request [:headers :query ..].

Simple invocation:

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/GET client "<your url>")
        status (http/status resp)
        headers (http/headers resp)]
    (println (:code status))
    (http/await resp)
    (println (http/string resp))))

Invocation with query parameters:

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/GET client "<your url>" :query {:param-name "some-value"})
        status (http/status resp)
        headers (http/headers resp)]
    (println (:code status))
    (http/await resp)
    (println (http/string resp))))

Invocation with proxy:

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/GET client "<your url>"
                       :query {:param-name "some-value"}
                       :proxy {:host host :port port})
        status (http/status resp)]
    (println (:code status))
    (http/await resp)
    (println (http/string resp))))

Invocation with cookies:

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/GET client "http://localhost:8123/cookie"
                       ;; Send cookie
                       :cookies #{{:domain "http://localhost:8123/"
                                   :name "sample-name"
                                   :value "sample-value"
                                   :path "/cookie"
                                   :max-age 10
                                   :secure false}})]
    (doseq [cookie (http/cookies resp)] ; Read cookies from server response
      (println "name:" (:name cookie) ", value:" (:value cookie)))))

Notice http.async.client/cookies function extracts cookies from response headers, so to start processing it you don't need to wait for whole response to arrive.

http.async.client/PUT/http.async.client/POST work the same way as GET but they also accept :body argument.

:body can be:

• String,
• map, sent as form parameters,
• vector, sent as multipart message,
• input stream,
• java.io.File, this will be sent using zero byte copy.

To call http.async.client/DELETE on a resource:

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/DELETE "<your url>")]
    ;; do something with resp
    ))

To call http.async.client/HEAD on a resource:

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/HEAD "<your url>")]
    ;; do something with resp
    ))

To call http.async.client/OPTIONS on a resource:

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/OPTIONS "<your url>")]
    ;; do something with resp
    ))

Query parameters is a map of keywords and their values. You use it like so:

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/GET client url :query {:key1 "value1" :key2 "value2"})]
    (http/await resp)
    (http/string resp)))

Custom headers can be submitted same way as :query:

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/GET client url :headers {:header-name1 "value1"
                                            :header-name2 "value2"})]
    (http/await resp)
    (http/string resp)))

Body can be provided with a message only with PUT/POST, it doesn't make sense to have body with other HTTP methods.

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/PUT client url :body "sample body")]
    (http/await resp)
    (http/string resp)))

:body can be:

• String,
• map, sent as form parameters,
• vector, sent as multipart message,
• input stream,
• java.io.File, this will be sent using zero byte copy.

Please see PUT/POST for more documentation.

Cookies can be provided to request as follows:

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/GET client "http://localhost:8123/cookie"
                       :cookies #{{:domain "http://localhost:8123/"
                                   :name "sample-name"
                                   :value "sample-value"
                                   :path "/cookie"
                                   :max-age 10
                                   :secure false}})]
    (http/await resp)
    (http/string resp)))

:cookies option takes sequence of cookie maps, in this example a hash set. Cookie map consist of:

:domain

Domain that cookie has been installed

:name

Cookie name

:value

Cookie value, note that there is no additional processing so you should encode it yourself if needed.

:path

Path on with cookie has been installed

:max-age

Max age that cookie was configured to live

:secure

If cookie is secure cookie

Cookie reading is described in Reading cookies.

Proxy can be configured per request basis as follows:

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/GET client url :proxy {:host h :port p})]
    (http/await resp)
    (http/string resp)))

Proxy expects a map with following keys:

:host

proxy host

:port

proxy port

:protocol

optional protocol to communicate with proxy. Can be :http (default) or :https.

:user

optional user name to use for proxy authentication. Must be provided with :password.

:password

optional password to use for proxy authentication. Must be provided with :user.

Authentication can be configured per request basis. For now BASIC and DIGEST methods are supported.

Basic method is default, so you don't have to specify it:

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/GET client url :auth {:user u :password p})]
    ;; Check if response is not 401 or so and process response
    ))

Though you can:

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/GET client url :auth {:type :basic :user u :password p})]
    ;; Check if response is not 401 or so and process response
    ))

And for digest method you will need realm as well:

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/GET client url
                       :auth {:type :digest :user u :password p :realm r})]
    ;; Check if response is not 401 or so and process response
    ))

Controlling preemptive authentication behavior is also possible:

(with-open [client (http/create-client)]
  (let [resp (http/GET client url :auth {:user u :password p :preemptive true})]
    ;; process response
    ))

Response timeout can be configured per request as well. Timeout value is time in milliseconds in which response has to be received. There is special value -1 that indicates infinite timeout.

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/GET client url :timeout -1)]
    (http/await resp)
    ;; process response
    ))

Sample above will wait until response is fully received, as long as it takes (-1 timeout).

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/GET client url :timeout 100)]
    (http/await resp)
    (if (http/failed? resp)
      ;; did not get response in configured timeout
      ;; process response
      )))

Example above configures timeout to 100ms, so await will only wait for 100ms, after that response is done. Which doesn't necessarily mean that it was delivered to client successfully, because it was restricted by timeout, that is why example contains check if response has failed.

You can get HTTP Stream as lazy sequence of it's body. This is very convenient method as seq is native type of Clojure so you can apply all mapping, filtering or any other standard function that you like to it.

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/stream-seq client :get url)]
    (doseq [s (http/string resp)]
      (println s))))

stream-seq arguments:

http method

any of supported methods can be used, though it makes sense only to use :get, :put and :post

url

URL of HTTP resource

options

same as normal Request options.

It is important to understand that seqs returned by body or string (which in turn calls body) are backed by queue. One of consequences of it is that once you consumed some body parts they will not be available anymore. Let's see code speak for itself.

(let [resp (http/stream-seq :get url)]
  (println "1: " (first (http/string resp)))
  (println "2: " (first (http/string resp))))

Assuming that part1 is first chunk and part2 is second. This code will print following:

1: part1
2: part2

Second consequence of been directly backed by queue is that you can have multiple consumers of same response and non of them will get same body part.

And finally this implementation is not holding to it's head.

Consuming HTTP Stream with call-back is quite straight forward with http.async.client. You will need to know what HTTP Method you will call, what URL and provide a call back function to handle body parts been received.

(with-open [client (http/create-client)] ; Create client
  (let [parts (ref #{})
        resp (http/request-stream client :get url
                                  (fn [state body]
                                    (dosync (alter parts conj (string body)))
                                    [body :continue]))]
    ;; do something to @parts
    ))

Few notes on implementing body part callback:

• state is a map with :status and :headers as promises, at stage when you get called for body part, both of them should be in place already, though it is advised to use convenience methods to read them, see Reading status line and Reading headers,
• call-back has to follow guidelines described in Body part,
• some streams are not meant to be finished, in that case don't collect body parts, as for sure you will run out of available resources,
• try not to do any heavy lifting in this callback, better send it to agent.

If you call any of Asynchronous operations, Streaming or Raw mode you actually asynchronously execute HTTP request. Some times you might need to wait for response processing to be done before proceeding, in order to do so you call http.async.client/await. It takes only one argument, that is response and returns once receiving has finished.

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/GET client url)]
    (http/await resp)))

Sample above will behave like synchronous HTTP operation. For convenience it returns same response so you can use it further, for example like that:

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/GET client url)]
    (http/string (http/await resp))))

http.async.client/status returns status lazy map of response. It will wait until HTTP Status has been received.

(with-open [client (http/create-client)] ; Create client
  (let [resp   (http/GET client url)
        status (http/status resp)]
    (:code status)))

Sample above will return HTTP response status code, notice that after this returns headers and body, might not been delivered yet.

http.async.client/redirect? checks if response is redirect.

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/await (http/GET client url))]
    (when (http/redirect? resp)
      (println "Response is redirect."))))

http.async.client/headers returns headers lazy map of response. It will wait until HTTP Headers are received.

(with-open [client (http/create-client)] ; Create client
  (let [resp    (http/GET client url)
        headers (http/headers resp)]
    (:server headers)))

Again, like in case of status, body might not have been delivered yet after this returns.

http.async.client/content-type returns value of Content-Type header.

(with-open [client (http/create-client)] ; Create client
  (let [resp         (http/GET client url)
        content-type (http/content-type resp)]
    (println "Content-Type of response:" content-type)))

http.async.client/location will return redirect target if response is redirect.

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/GET client url)
        loc  (http/location resp)]
    (println "Location of redirect:" loc)))

http.async.client/cookies returns seq of maps representing cookies. It will wait until HTTP Headers are received.

(with-open [client (http/create-client)] ; Create client
  (let [resp    (http/GET client url)
        cookies (http/cookies resp)]
    (map :name cookies)))

Sample above will return sequence of cookie names that server has set.

http.async.client/body returns either ByteArrayOutputStream or seq of it, depending if you used Asynchronous operations or Streaming respectively. It will not wait for response to be finished, it will return as soon as first chunk of HTTP response body is received.

http.async.client/string returns either string or seq of strings, again depending if you used Asynchronous operations or Streaming respectively. It will not wait for response to be finished, it will return as soon as first chunk of HTTP response body is received.

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/GET client url)]
    (http/string (http/await resp))))

Sample above will return string of response body. http.async.client/string is lazy so you can use it in case of streams as well.

(with-open [client (http/create-client)] ; Create client
  (let [resp    (http/stream-seq client :get url)
        strings (http/string resp)]
    (doseq [part strings]
      (println part))))

Sample above will print parts as they are received, and will return once response receiving is finished.

http.async.client/error will return Throwable that was cause of request failure iff request failed, else nil.

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/GET client url)]
    (http/await resp)
    (when-let [err (http/error resp)]
      (println "failed processing request: " err))))

At any given time of processing HTTP Response you can cancel it by calling http.async.client/cancel.

(with-open [client (http/create-client)] ; Create client
  (let [resp (http/GET client url)]
    (http/cancel resp)))

Please see canceling-request test.

You can also check status of request.

If you need to get URL of response at hand, there are two associations in it that you will find useful:

:url

encoded url,

:raw-url

not encoded url.

And two convenience functions:

• http.async.client/url,
• http.async.client/raw-url.

http.async.client can be configured with User-Agent. To do so you can use http.async.client/create-client and remember to close created client yourself, best is to use it within macro like with-open, though make sure that body of it will wait for whole response to finish.

(with-open [client (http/create-client :user-agent "Your User Agent/1.0")]
  (let [resp (http/GET client url)]
    ;; do stuff with resp
    ))

http.async.client can be configured to allow, or not, HTTP compression.

(with-open [client (http/create-client :compression-enabled true)]
  (let [resp (http/GET client url)]
    ;; do stuff with resp
    ))

Enabling HTTP redirects following.

(with-open [client  (http/create-client :follow-redirects true)]
  (let [resp (http/GET client url)]
    ;; do stuff with resp
    ))

Query parameters are not passed on to any redirect that is followed (by default parameters are removed).

(with-open [client (http/create-client :remove-params-on-redirect true)]
  (let [resp (http/GET client url)]
    ;; do stuff with resp
    ))

Keep Alive is enabled by default. This implies using pool for connections.

(with-open [client (http/create-client :keep-alive true)]
  (let [resp (http/GET client url)]
    ;; do stuff with resp
    ))

Maximum number of connections to be cached per host. Above this number connections will still be created but will not be kept alive.

(with-open [client (http/create-client :max-conns-per-host 10)]
  (let [resp (http/GET client url)]
    ;; do stuff with resp
    ))

Maximum number of total connections opened, submitting new request while all allowed connections are active, will result in rejection.

(with-open [client (http/create-client :max-conns-total 100)]
  (let [resp (http/GET client url)]
    ;; do stuff with resp
    ))

Maximum number of redirects to follow.

(with-open [client (http/create-client :max-redirects 3)]
  (let [resp (http/GET client url)]
    ;; do stuff with resp
    ))

With http.async.client apart from per connection :timeout you can globally configure connection, request and idle timeouts. All timeout values are in milliseconds and magic value -1 is interpreted as infinite wait. idle connection in pool timeout works only on connections in pool, connections idle, for configured time, in pool will be closed.

(with-open [client (http/create-client :connection-timeout 10
                                       :request-timeout 1000
                                       :idle-in-pool-timeout 100)]
  (let [resp (http/GET client url)]
    ;; request processing
    ))

Example above will timeout connection if it was not established in 10ms, request if it was not received in 1sec, or connection when it was idling in pool for more than 100ms.

Client can be also configured with global HTTP Proxy settings.

(with-open [client (http/create-client :proxy {:host h :port p})]
  (let [resp (http/GET client url)]
    ;; do stuff with resp
    ))

Proxy expects a map with following keys:

:host

proxy host

:port

proxy port

:protocol

optional protocol to communicate with proxy. Can be :http (default) or :https.

:user

optional user name to use for proxy authentication. Must be provided with :password.

:password

optional password to use for proxy authentication. Must be provided with :user.

Default authentication realm to be used globally can be configured. For now BASIC and DIGEST methods are supported.

Basic method is default, so you don't have to specify it:

(with-open [client (http/create-client :auth {:user u :password p})] 
  (let [resp (http/GET client url)]
    ;; Check if response is not 401 or so and process response
    ))

Though you can:

(with-open [client (http/create-client :auth {:type :basic :user u :password p})]
  (let [resp (http/GET client url)]
    ;; Check if response is not 401 or so and process response
    ))

And for digest method you will need realm as well:

(with-open [client (http/create-client :auth {:type :digest :user u :password p :realm r})]
  (let [resp (http/GET client url)]
    ;; Check if response is not 401 or so and process response
    ))

Preemptive authentication can be enabled or disabled globally per client:

(with-open [client (http/create-client :auth {:type :basic :user u :passowrd p :preemptive true})]
  (let [resp (http/GET client url)]
    ;; process response
    ))

Since v0.4.2 it is possible to use SSL certificates with client.

For client to use SSL you need to provide it with ssl-context.

(require [http.async.client.cert :as cert])
(let [ctx (cert/ssl-context :keystore-file ks-file
                            :keystore-password password
                            :certificate-file cert-file
                            :certificate-alias other-cert-alias)]
  (with-open [client (http/create-client :ssl-context ctx)]
    (let [resp (http/GET client url)]
      ;; process response
      )))

For more documentation please consult docstring of http.async.client.cert/ssl-context and tests.

Whenever you've created http.async.client via http.async.client/create-client you will need to close it. To do so you call http.async.client/close.

(let [client (http/create-client)]
  (try
    (let [resp (http/GET client url)]
      ;; process response
      )
    (finally
     (http/close client))))

Status line callback gets called after status line has been received with arguments:

• response map
• Status map has following keys:
  • :code status code (200, 404, ..)
  • :msg status message ("OK", ..)
  • :protocol protocol with version ("HTTP/1.1")
  • :major major protocol version (1)
  • :minor minor protocol version (0, 1)

Sample code to illustrate how to use status callback:

(with-open [client (http/create-client)] ; create client
  (let [request (request/prepare-request :get "http://example.com") ; create request
        status (promise)                ; status promise that will be delivered by callback
        response (request/execute-request
                  client request        ; execute *request*
                  :status               ; status callback
                  (fn [res st]          ; *res* is response map, same as one returned by *execute-request*
                                        ; *st* is status map, as described above
                    (deliver status st) ; deliver status promise
                    [st :abort]))]      ; return status to be delivered to response map and abort further processing of response.
    (println @status)))                 ; await status to be delivered and print it.

Headers callback gets called after headers have been received with arguments:

• response map
• lazy map of headers. Keys in that map are (keyword (.toLowerCase <header name>)), so "Server" headers is :server and so on.

Sample code to illustrate how to use headers callback:

(with-open [client (http/create-client)] ; create client
  (let [request (request/prepare-request :get "http://example.com") ; create request
        headers (promise)               ; headers promise that will be delivered by callback
        response (request/execute-request
                  client request        ; execute *request*
                  :header               ; header callback
                  (fn [res hds]         ; *res* is response map, same as one returned by *execute-request*
                                        ; *hds* is headers map, as described above
                    (deliver headers st) ; deliver headers promise
                    [hds :abort]))]     ; return headers to be delivered to response map and abort further processing of response.
    (println @headers)))

Body part callback gets called after each part of body has been received with arguments:

• response map
• ByteArrayOutputStream that contains body part received.

Following code sample will show how to count number of body parts received:

(with-open [client (http/create-client)] ; create client
  (let [cnt (atom 0)                    ; body parts counter
        request (request/prepare-request :get "http://localhost:8123/stream") ; create request
        resp (execute-request
client* request          ; execute *request*
              :part                     ; body part callback
              (fn [_ p]                 ; ignore response map,
                                        ; *p* is body part
                (swap! cnt inc)         ; increment body part counter
                [p :continue]))]        ; return part to be delivered to response map and continue processing of response.
    (http/await resp)                   ; wait for response to finish
    (println @cnt)))                    ; print body parts counter

Next sample will collect body parts:

(with-open [client (http/create-client)] ; create client
  (let [parts (atom #{})                ; body parts collector
        request (request/prepare-request :get "http://localhost:8123/stream") ; create request
        resp (execute-request
client* request          ; execute *request*
              :part                     ; body part callback
              (fn [_ ^ByteArrayOutputStream p] ; ignore response map,
                                        ; *p* is body part
                (let [p (.toString part "UTF-8")] ; extract text
                  (swap! parts conj p)  ; collect body part
                  [p :continue])))]     ; return part to be delivered to response map and continue processing of response.
    (http/await resp)                   ; wait for response to finish
    (println @parts)))                  ; print collected body parts

This callback gets called when receiving of response body has finished with only one argument, i.e. response map.

Following snippet shows how to measure execution time:

(with-open [client (http/create-client)] ; create client
  (let [request (request/prepare-request :get "http://localhost:8123/") ; create request
        finished (promise)               ; execution time will be stored here
        start (System/currentTimeMillis) ; start time
        resp (execute-request
client* request           ; execute *request*
              :completed                 ; completed callback
              (fn [_]                    ; ignore response map,
                (deliver finished        ; deliver execution time
                         (- (System/currentTimeMillis) start))))]
    (println @finished)))                ; print execution time

You have to be aware of fact that :completed callback is called only on successful response. Next snippet will show that :completed callback is not called when request errors:

(with-open [client (http/create-client)] ; create client
  (let [request (request/prepare-request :get "http://not-existing-host/") ; create request
        finished (promise)               ; would get delivered if :completed callback would get executed
        resp (execute-request
client* request           ; execute *request*
              :completed                 ; completed callback, will not fire in this example
              (fn [_]                    ; ignore response map,
                (deliver finished true)))] ; deliver finished promise
    (http/await resp)                    ; wait for response
    (false? (realized? finished))))      ; finished promise never got delivered

Error callback gets called when error while processing has been encountered with arguments

• response map
• Throwable that was a cause of failure

Next code snippet shows error callback in use:

(with-open [client (http/create-client)] ; create client
  (let [request (request/prepare-request :get "http://not-existing-host/") ; create request
        errored (promise)                ; will store exception
        resp (execute-request
client* request           ; execute *request*
              :error                     ; error callback
              (fn [_ e]                  ; ignore response map, *e* is exception
                (deliver errored e)))]   ; deliver errored promise
    (http/await resp)                    ; wait for response
    (println @errored)))                 ; print exception from callback