1/* Part of SWI-Prolog 2 3 Author: Jan Wielemaker 4 E-mail: J.Wielemaker@vu.nl 5 WWW: http://www.swi-prolog.org 6 Copyright (c) 2002-2023, University of Amsterdam, 7 VU University Amsterdam 8 SWI-Prolog Solutions b.v. 9 All rights reserved. 10 11 Redistribution and use in source and binary forms, with or without 12 modification, are permitted provided that the following conditions 13 are met: 14 15 1. Redistributions of source code must retain the above copyright 16 notice, this list of conditions and the following disclaimer. 17 18 2. Redistributions in binary form must reproduce the above copyright 19 notice, this list of conditions and the following disclaimer in 20 the documentation and/or other materials provided with the 21 distribution. 22 23 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 24 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 25 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 26 FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 27 COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 28 INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, 29 BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 30 LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 31 CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 32 LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN 33 ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 34 POSSIBILITY OF SUCH DAMAGE. 35*/ 36 37:- module(http_client, 38 [ http_get/3, % +URL, -Reply, +Options 39 http_delete/3, % +URL, -Reply, +Options 40 http_post/4, % +URL, +In, -Reply, +Options 41 http_put/4, % +URL, +In, -Reply, +Options 42 http_patch/4, % +URL, +In, -Reply, +Options 43 http_read_data/3, % +Header, -Data, :Options 44 http_disconnect/1 % +What 45 ]). 46:- autoload(http_header,[http_post_data/3]). 47:- autoload(http_stream,[http_chunked_open/3,stream_range_open/3]). 48:- autoload(library(error),[must_be/2]). 49:- autoload(library(lists),[delete/3,select/3]). 50:- autoload(library(memfile), 51 [ new_memory_file/1, open_memory_file/4, free_memory_file/1, 52 memory_file_to_atom/3, memory_file_to_string/3, 53 memory_file_to_codes/3, open_memory_file/3 54 ]). 55:- autoload(library(option), 56 [option/3,option/2,meta_options/3,select_option/3]). 57:- autoload(library(uri),[uri_query_components/2]). 58:- autoload(library(http/http_open), 59 [http_open/3,http_close_keep_alive/1]). 60 61:- meta_predicate 62 http_read_data(, , ). 63 64:- multifile 65 http_convert_data/4, % http_read_data plugin-hook 66 http:post_data_hook/3. 67 68:- predicate_options(http_get/3, 3, 69 [ pass_to(http_open:http_open/3, 3), 70 pass_to(http_open:http_read_data/3, 3), 71 pass_to(http_json:http_read_json/3, 3) 72 ]). 73:- predicate_options(http_delete/3, 3, [pass_to(http_get/3, 3)]). 74:- predicate_options(http_post/4, 4, [pass_to(http_get/3, 3)]). 75:- predicate_options(http_put/4, 4, [pass_to(http_post/4, 4)]). 76:- predicate_options(http_read_data/3, 3, 77 [ to(any), 78 content_type(any), 79 form_data(oneof([form,mime])), 80 input_encoding(encoding), % multipart 81 on_filename(callable), 82 module(atom), % x-prolog data 83 variable_names(-list), 84 json_object(oneof([term,dict])), % JSON data 85 value_string_as(oneof([string,atom])) 86 ]). 87 88 89/** <module> HTTP client library 90 91This library provides the four basic HTTP client actions: =GET=, 92=DELETE=, =POST= and =PUT=. In addition, it provides http_read_data/3, 93which is used by library(http/http_parameters) to decode =POST= data in 94server applications. 95 96This library is based on http_open/3, which opens a URL as a Prolog 97stream. The reply is processed by http_read_data/3. The following 98content-types are supported. Options passed to http_get/3 and friends 99are passed to http_read_data/3, which in turn passes them to the 100conversion predicates. Support for additional content types can be added 101by extending the multifile predicate http_client:http_convert_data/4. 102 103 - 'application/x-www-form-urlencoded' 104 Built in. Converts form-data into a list of `Name=Value` terms. 105 - 'application/x-prolog' 106 Built in. Reads a single Prolog term. 107 - 'multipart/form-data' 108 Processed if library(http/http_multipart_plugin) is loaded. This 109 format should be used to handle web forms that upload a file. 110 - 'text/html' | 'text/xml' 111 Processed if library(http/http_sgml_plugin) is loaded. See load_html/3 112 for details and load_xml/3 for details. The output is often processed 113 using xpath/3. 114 - 'application/json' | 'application/jsonrequest' 115 Processed if library(http/http_json) is loaded. The option 116 json_object(As) can be used to return a term json(Attributes) 117 (`As` is `term`) or a dict (`As` is `dict`). 118*/ 119 120 /******************************* 121 * GET * 122 *******************************/ 123 124%! http_get(+URL, -Data, +Options) is det. 125% 126% Get data from a URL server and convert it to a suitable Prolog 127% representation based on the =|Content-Type|= header and plugins. 128% This predicate is the common implementation of the HTTP client 129% operations. The predicates http_delete/3, http_post/4 and 130% http_put/4 call this predicate with an appropriate 131% method(+Method) option and ---for http_post/4 and http_put/4--- 132% a post(+Data) option. 133% 134% Options are passed to http_open/3 and http_read_data/3. Other 135% options: 136% 137% - reply_header(-Fields) 138% Synonym for headers(Fields) from http_open/3. Provided for 139% backward compatibility. Note that http_version(Major-Minor) 140% is missing in the new version. 141 142http_get(URL, Data, Options) :- 143 headers_option(Options, Options1, Headers), 144 option(reply_header(Headers), Options, _), 145 http_open(URL, In, Options1), 146 delete(Headers, transfer_encoding(_), Headers1), 147 call_cleanup( 148 http_read_data(In, Headers1, Data, Options), 149 close(In)). 150 151headers_option(Options, Options1, Headers) :- 152 option(headers(Headers), Options), 153 !, 154 Options1 = Options. 155headers_option(Options, [headers(Headers)|Options], Headers). 156 157 158%! http_delete(+URL, -Data, +Options) is det. 159% 160% Execute a =DELETE= method on the server. Arguments are the same 161% as for http_get/3. Typically one should pass the option 162% status_code(-Code) to assess and evaluate the returned status 163% code. Without, codes other than 200 are interpreted as an error. 164% 165% @tbd Properly map the 201, 202 and 204 replies. 166% @see Implemented on top of http_get/3. 167 168http_delete(URL, Data, Options) :- 169 http_get(URL, Data, [method(delete)|Options]). 170 171 172%! http_post(+URL, +Data, -Reply, +Options) is det. 173% 174% Issue an HTTP =POST= request. Data is posted using 175% http_post_data/3. The HTTP server reply is returned in Reply, 176% using the same rules as for http_get/3. 177% 178% @see Implemented on top of http_get/3. 179 180http_post(URL, Data, Reply, Options) :- 181 http_get(URL, Reply, 182 [ post(Data) 183 | Options 184 ]). 185 186%! http_put(+URL, +Data, -Reply, +Options) 187% 188% Issue an HTTP =PUT= request. Arguments are the same as for 189% http_post/4. 190% 191% @see Implemented on top of http_post/4. 192 193http_put(URL, In, Out, Options) :- 194 http_post(URL, In, Out, [method(put)|Options]). 195 196%! http_patch(+URL, +Data, -Reply, +Options) 197% 198% Issue an HTTP =PATCH= request. Arguments are the same as for 199% http_post/4. 200% 201% @see Implemented on top of http_post/4. 202 203http_patch(URL, In, Out, Options) :- 204 http_post(URL, In, Out, [method(patch)|Options]). 205 206%! http_read_data(+Request, -Data, +Options) is det. 207% 208% Read data from an HTTP connection and convert it according to 209% the supplied to(Format) option or based on the =|Content-type|= 210% in the Request. The following options are supported: 211% 212% * to(Format) 213% Convert data into Format. Values are: 214% - stream(+WriteStream)) 215% Append the content of the message to Stream 216% - atom 217% Return the reply as an atom 218% - string 219% Return the reply as a string 220% - codes 221% Return the reply as a list of codes 222% * form_data(AsForm) 223% * input_encoding(+Encoding) 224% * on_filename(:CallBack) 225% These options are implemented by the plugin 226% library(http/http_multipart_plugin) and apply to processing 227% =|multipart/form-data|= content. 228% * content_type(+Type) 229% Overrule the content-type that is part of Request as a 230% work-around for wrongly configured servers. 231% 232% Without plugins, this predicate handles 233% 234% * 'application/x-www-form-urlencoded' 235% Converts form-data into a list of `Name=Value` terms. 236% * 'application/x-prolog' 237% Converts data into a Prolog term. 238% 239% @param Request is a parsed HTTP request as returned by 240% http_read_request/2 or available from the HTTP server's request 241% dispatcher. Request must contain a term input(In) that provides 242% the input stream from the HTTP server. 243 244http_read_data(Fields, Data, QOptions) :- 245 meta_options(is_meta, QOptions, Options), 246 memberchk(input(In), Fields), 247 ( catch(http_read_data(In, Fields, Data, Options), 248 error(Formal,_), 249 throw(error(Formal, context(_, in_http_request)))) 250 -> true 251 ; throw(error(failed(http_read_data), _)) 252 ). 253 254is_meta(on_filename). 255 256http_read_data(_In, Fields, Data, _Options) :- 257 option(status_code(Code), Fields), 258 no_content_status(Code), 259 \+ ( option(content_length(Len), Fields), 260 Len > 0 261 ), 262 !, 263 Data = ''. 264http_read_data(In, Fields, Data, Options) :- % Transfer-encoding: chunked 265 select(transfer_encoding(chunked), Fields, RestFields), 266 !, 267 setup_call_cleanup( 268 http_chunked_open(In, DataStream, []), 269 http_read_data(DataStream, RestFields, Data, Options), 270 close(DataStream)). 271http_read_data(In, Fields, Data, Options) :- 272 option(to(X), Options), 273 !, 274 ( X = stream(Stream) 275 -> ( memberchk(content_length(Bytes), Fields) 276 -> copy_stream_data(In, Stream, Bytes) 277 ; copy_stream_data(In, Stream) 278 ) 279 ; must_be(oneof([atom,string,codes]), X), 280 setup_call_cleanup( 281 new_memory_file(MemFile), 282 ( setup_call_cleanup( 283 open_memory_file(MemFile, write, Stream, 284 [encoding(octet)]), 285 ( memberchk(content_length(Bytes), Fields) 286 -> copy_stream_data(In, Stream, Bytes) 287 ; copy_stream_data(In, Stream) 288 ), 289 close(Stream)), 290 encoding(Fields, Encoding, Options), 291 memory_file_to(X, MemFile, Encoding, Data0) 292 ), 293 free_memory_file(MemFile)), 294 Data = Data0 295 ). 296http_read_data(In, Fields, Data, _) :- 297 option(content_type(ContentType), Fields), 298 is_content_type(ContentType, 'application/x-www-form-urlencoded'), 299 !, 300 http_read_data(In, Fields, Codes, [to(string)]), 301 uri_query_components(Codes, Data). 302http_read_data(In, Fields, Data, Options) :- % call hook 303 ( select_option(content_type(Type), Options, Options1) 304 -> delete(Fields, content_type(_), Fields1), 305 http_convert_data(In, [content_type(Type)|Fields1], Data, Options1) 306 ; http_convert_data(In, Fields, Data, Options) 307 ), 308 !. 309http_read_data(In, Fields, Data, Options) :- 310 http_read_data(In, Fields, Data, [to(atom)|Options]). 311 312memory_file_to(atom, MemFile, Encoding, Data) :- 313 memory_file_to_atom(MemFile, Data, Encoding). 314memory_file_to(string, MemFile, Encoding, Data) :- 315 memory_file_to_string(MemFile, Data, Encoding). 316memory_file_to(codes, MemFile, Encoding, Data) :- 317 memory_file_to_codes(MemFile, Data, Encoding). 318 319 320encoding(_Fields, Encoding, Options) :- 321 option(input_encoding(Encoding), Options), 322 !. 323encoding(Fields, utf8, _) :- 324 memberchk(content_type(Type), Fields), 325 ( sub_atom(Type, _, _, _, 'UTF-8') 326 -> true 327 ; sub_atom(Type, _, _, _, 'utf-8') 328 ), 329 !. 330encoding(_, octet, _). 331 332is_content_type(ContentType, Check) :- 333 sub_atom(ContentType, 0, Len, After, Check), 334 ( After == 0 335 -> true 336 ; sub_atom(ContentType, Len, 1, _, ';') 337 ). 338 339%! no_content_status(+Code) is semidet. 340% 341% True when Code is an HTTP status code that carries no content. 342% 343% @see Issue#157 344 345no_content_status(Code) :- 346 between(100, 199, Code), 347 !. 348no_content_status(204). 349 350%! http_convert_data(+In, +Fields, -Data, +Options) is semidet. 351% 352% Multi-file hook to convert a HTTP payload according to the 353% _Content-Type_ header. The default implementation deals with 354% application/x-prolog. The HTTP framework provides 355% implementations for JSON (library(http/http_json)), HTML/XML 356% (library(http/http_sgml_plugin)) 357 358http_convert_data(In, Fields, Data, Options) :- 359 memberchk(content_type(Type), Fields), 360 is_content_type(Type, 'application/x-prolog'), 361 !, 362 ( memberchk(content_length(Bytes), Fields) 363 -> setup_call_cleanup( 364 ( stream_range_open(In, Range, [size(Bytes)]), 365 set_stream(Range, encoding(utf8)), 366 set_stream(Range, file_name('HTTP:DATA')) 367 ), 368 read_term(Range, Data, Options), 369 close(Range)) 370 ; set_stream(In, encoding(utf8)), 371 read_term(In, Data, Options) 372 ). 373 374%! http_disconnect(+Connections) is det. 375% 376% Close down some connections. Currently Connections must have the 377% value =all=, closing all connections. 378% 379% @deprecated New code should use http_close_keep_alive/1 from 380% library(http/http_open). 381 382http_disconnect(all) :- 383 http_close_keep_alive(_). 384 385%! http:post_data_hook(+Term, +Out, +Options) is semidet. 386% 387% Hook to extend the datatypes supported by the post(Data) option 388% of http_open/3. The default implementation supports 389% prolog(Term), sending a Prolog term as =|application/x-prolog|=. 390 391httppost_data_hook(prolog(Term), Out, HdrExtra) :- 392 setup_call_cleanup( 393 ( new_memory_file(MemFile), 394 open_memory_file(MemFile, write, Handle) 395 ), 396 ( format(Handle, 397 'Content-Type: application/x-prolog; charset=UTF-8~n~n', 398 []), 399 write_term(Handle, Term, 400 [ quoted(true), 401 ignore_ops(true), 402 fullstop(true), 403 nl(true) 404 ]) 405 ), 406 close(Handle)), 407 setup_call_cleanup( 408 open_memory_file(MemFile, read, RdHandle, 409 [ free_on_close(true) 410 ]), 411 http_post_data(cgi_stream(RdHandle), Out, HdrExtra), 412 close(RdHandle))