Low-level bindings to write data from a URL into memory, disk or a callback
function. These are mainly intended for httr, most users will be better
off using the curl or curl_download function, or the
http specific wrappers in the httr package.
curl_fetch_memory(url, handle = new_handle()) curl_fetch_disk(url, path, handle = new_handle()) curl_fetch_stream(url, fun, handle = new_handle()) curl_fetch_multi( url, done = NULL, fail = NULL, pool = NULL, data = NULL, handle = new_handle() ) curl_fetch_echo(url, handle = new_handle())
| url | A character string naming the URL of a resource to be downloaded. |
|---|---|
| handle | a curl handle object |
| path | Path to save results |
| fun | Callback function. Should have one argument, which will be a raw vector. |
| done | callback function for completed request. Single argument with response data in same structure as curl_fetch_memory. |
| fail | callback function called on failed request. Argument contains error message. |
| pool | a multi handle created by new_pool. Default uses a global pool. |
| data | (advanced) callback function, file path, or connection object for writing
incoming data. This callback should only be used for streaming applications,
where small pieces of incoming data get written before the request has completed. The
signature for the callback function is |
The curl_fetch functions automatically raise an error upon protocol problems (network, disk, ssl) but do not implement application logic. For example for you need to check the status code of http requests yourself in the response, and deal with it accordingly.
Both curl_fetch_memory and curl_fetch_disk have a blocking and
non-blocking C implementation. The latter is slightly slower but allows for
interrupting the download prematurely (using e.g. CTRL+C or ESC). Interrupting
is enabled when R runs in interactive mode or when
getOption("curl_interrupt") == TRUE.
The curl_fetch_multi function is the asynchronous equivalent of
curl_fetch_memory. It wraps multi_add to schedule requests which
are executed concurrently when calling multi_run. For each successful
request the done callback is triggered with response data. For failed
requests (when curl_fetch_memory would raise an error), the fail
function is triggered with the error message.
# Load in memory res <- curl_fetch_memory("http://httpbin.org/cookies/set?foo=123&bar=ftw") res$content#> [1] 7b 0a 20 20 22 63 6f 6f 6b 69 65 73 22 3a 20 7b 0a 20 20 20 20 22 62 61 72 #> [26] 22 3a 20 22 66 74 77 22 2c 20 0a 20 20 20 20 22 66 6f 6f 22 3a 20 22 31 32 #> [51] 33 22 0a 20 20 7d 0a 7d 0a#> [1] "/var/folders/hp/fg4q7m353d91v9mgm4d1stvh0000gn/T//RtmpY7qhky/file92e45013adbd"#> [1] "{\"url\": \"https://httpbin.org/stream/10\", \"args\": {}, \"headers\": {\"Host\": \"httpbin.org\", \"Accept\": \"*/*\", \"Accept-Encoding\": \"deflate, gzip\", \"User-Agent\": \"R (3.6.1 x86_64-apple-darwin15.6.0 x86_64 darwin15.6.0)\"}, \"origin\": \"145.136.143.132, 145.136.143.132\", \"id\": 0}" #> [2] "{\"url\": \"https://httpbin.org/stream/10\", \"args\": {}, \"headers\": {\"Host\": \"httpbin.org\", \"Accept\": \"*/*\", \"Accept-Encoding\": \"deflate, gzip\", \"User-Agent\": \"R (3.6.1 x86_64-apple-darwin15.6.0 x86_64 darwin15.6.0)\"}, \"origin\": \"145.136.143.132, 145.136.143.132\", \"id\": 1}" #> [3] "{\"url\": \"https://httpbin.org/stream/10\", \"args\": {}, \"headers\": {\"Host\": \"httpbin.org\", \"Accept\": \"*/*\", \"Accept-Encoding\": \"deflate, gzip\", \"User-Agent\": \"R (3.6.1 x86_64-apple-darwin15.6.0 x86_64 darwin15.6.0)\"}, \"origin\": \"145.136.143.132, 145.136.143.132\", \"id\": 2}" #> [4] "{\"url\": \"https://httpbin.org/stream/10\", \"args\": {}, \"headers\": {\"Host\": \"httpbin.org\", \"Accept\": \"*/*\", \"Accept-Encoding\": \"deflate, gzip\", \"User-Agent\": \"R (3.6.1 x86_64-apple-darwin15.6.0 x86_64 darwin15.6.0)\"}, \"origin\": \"145.136.143.132, 145.136.143.132\", \"id\": 3}" #> [5] "{\"url\": \"https://httpbin.org/stream/10\", \"args\": {}, \"headers\": {\"Host\": \"httpbin.org\", \"Accept\": \"*/*\", \"Accept-Encoding\": \"deflate, gzip\", \"User-Agent\": \"R (3.6.1 x86_64-apple-darwin15.6.0 x86_64 darwin15.6.0)\"}, \"origin\": \"145.136.143.132, 145.136.143.132\", \"id\": 4}" #> [6] "{\"url\": \"https://httpbin.org/stream/10\", \"args\": {}, \"headers\": {\"Host\": \"httpbin.org\", \"Accept\": \"*/*\", \"Accept-Encoding\": \"deflate, gzip\", \"User-Agent\": \"R (3.6.1 x86_64-apple-darwin15.6.0 x86_64 darwin15.6.0)\"}, \"origin\": \"145.136.143.132, 145.136.143.132\", \"id\": 5}" #> [7] "{\"url\": \"https://httpbin.org/stream/10\", \"args\": {}, \"headers\": {\"Host\": \"httpbin.org\", \"Accept\": \"*/*\", \"Accept-Encoding\": \"deflate, gzip\", \"User-Agent\": \"R (3.6.1 x86_64-apple-darwin15.6.0 x86_64 darwin15.6.0)\"}, \"origin\": \"145.136.143.132, 145.136.143.132\", \"id\": 6}" #> [8] "{\"url\": \"https://httpbin.org/stream/10\", \"args\": {}, \"headers\": {\"Host\": \"httpbin.org\", \"Accept\": \"*/*\", \"Accept-Encoding\": \"deflate, gzip\", \"User-Agent\": \"R (3.6.1 x86_64-apple-darwin15.6.0 x86_64 darwin15.6.0)\"}, \"origin\": \"145.136.143.132, 145.136.143.132\", \"id\": 7}" #> [9] "{\"url\": \"https://httpbin.org/stream/10\", \"args\": {}, \"headers\": {\"Host\": \"httpbin.org\", \"Accept\": \"*/*\", \"Accept-Encoding\": \"deflate, gzip\", \"User-Agent\": \"R (3.6.1 x86_64-apple-darwin15.6.0 x86_64 darwin15.6.0)\"}, \"origin\": \"145.136.143.132, 145.136.143.132\", \"id\": 8}" #> [10] "{\"url\": \"https://httpbin.org/stream/10\", \"args\": {}, \"headers\": {\"Host\": \"httpbin.org\", \"Accept\": \"*/*\", \"Accept-Encoding\": \"deflate, gzip\", \"User-Agent\": \"R (3.6.1 x86_64-apple-darwin15.6.0 x86_64 darwin15.6.0)\"}, \"origin\": \"145.136.143.132, 145.136.143.132\", \"id\": 9}"# Stream with callback res <- curl_fetch_stream("http://www.httpbin.org/drip?duration=3&numbytes=15&code=200", function(x){ cat(rawToChar(x)) })#> ***************# Async API data <- list() success <- function(res){ cat("Request done! Status:", res$status, "\n") data <<- c(data, list(res)) } failure <- function(msg){ cat("Oh noes! Request failed!", msg, "\n") } curl_fetch_multi("http://httpbin.org/get", success, failure) curl_fetch_multi("http://httpbin.org/status/418", success, failure) curl_fetch_multi("https://urldoesnotexist.xyz", success, failure) multi_run()#> Oh noes! Request failed! Could not resolve host: urldoesnotexist.xyz #> Request done! Status: 200 #> Request done! Status: 418#> $success #> [1] 2 #> #> $error #> [1] 1 #> #> $pending #> [1] 0 #>str(data)#> List of 2 #> $ :List of 7 #> ..$ url : chr "http://httpbin.org/get" #> ..$ status_code: int 200 #> ..$ type : chr "application/json" #> ..$ headers : raw [1:378] 48 54 54 50 ... #> ..$ modified : POSIXct[1:1], format: NA #> ..$ times : Named num [1:6] 0 0.000884 0.108057 0.108139 0.214733 ... #> .. ..- attr(*, "names")= chr [1:6] "redirect" "namelookup" "connect" "pretransfer" ... #> ..$ content : raw [1:291] 7b 0a 20 20 ... #> $ :List of 7 #> ..$ url : chr "http://httpbin.org/status/418" #> ..$ status_code: int 418 #> ..$ type : chr NA #> ..$ headers : raw [1:381] 48 54 54 50 ... #> ..$ modified : POSIXct[1:1], format: NA #> ..$ times : Named num [1:6] 0 0.000921 0.108114 0.108153 0.21482 ... #> .. ..- attr(*, "names")= chr [1:6] "redirect" "namelookup" "connect" "pretransfer" ... #> ..$ content : raw [1:135] 0a 20 20 20 ...