Did you know ... | Search Documentation: |
mqi -- Python and Other Programming Languge Integration for SWI Prolog |
This package provides the library mqi.pl
that enables
embedding SWI Prolog into just about any programming language (Python,
Go, C#, etc) in a straightforward way. It is designed for scenarios that
need to use SWI Prolog as a local implementation detail of another
language. Think of it as running SWI Prolog “like a library” .
It can support any programming language that can launch processes, read
their STDOUT pipe, and send and receive JSON over TCP/IP.
A Python 3.x library is provided.
The SWI Prolog Machine Query Interface ('MQI’) is designed to enable embedding SWI Prolog into just about any programming language (Python, Go, C#, etc) in a straightforward way. It is designed for scenarios that need to use SWI Prolog as a local implementation detail of another language. Think of it as running SWI Prolog "like a library". It can support any programming language that can launch processes, read their STDOUT pipe, and send and receive JSON over TCP/IP. A Python 3 library is included as a part of SWI Prolog, see Installation Steps for Python (section 1.1).
Key features of the MQI:
?-
").The server can be used in two different modes:
Note that the MQI is related to the Pengines library, but where the Pengines library is focused on a client/server, multi-tenet, sandboxed environment, the MQI is local, single tenet and unconstrained. Thus, when the requirement is to embed Prolog within another programming language "like a library", it can be a good solution for exposing the full power of Prolog with low integration overhead.
A Python 3.x library that integrates Python with SWI Prolog using the
Machine Query Interface is included within the libs
directory of the SWI Prolog installation. It is also available using pip install swiplserver
.
See the Python
swiplserver library documentation for more information on how to use
and install it from either location.
In general, to use the Machine Query Interface with any programming language:
swipl
executable from that installation.
The Machine Query Interface is designed to act like using the "top
level" prompt of SWI Prolog itself (i.e. the "?-
"
prompt). If you've built the Prolog part of your application by loading
code, running it and debugging it using the normal SWI Prolog top level,
integrating it with your native language should be straightforward:
simply run the commands you'd normally run on the top level, but now run
them using the query APIs provided by the library built for your target
language. Those APIs will allow you to send the exact same text to
Prolog and they should execute the same way. Here's an example using the
Python swiplserver
library:
% Prolog Top Level ?- member(X, [first, second, third]). X = first ; X = second ; X = third.
# Python using the swiplserver library from swiplserver import PrologMQI, PrologThread with PrologMQI() as mqi: with mqi.create_thread() as prolog_thread: result = prolog_thread.query("member(X, [first, second, third]).") print(result) first second third
While the query functionality of the MQI does run on a thread, it will always be the same thread, and, if you use a single connection, it will only allow queries to be run one at a time, just like the top level. Of course, the queries you send can launch threads, just like the top level, so you are not limited to a single threaded application. There are a few differences from the top level, however:
user_input
are initially bound
to the standard operating system I/O streams (like STDIN) and, since the
Prolog process is running invisibly, will obviously not work as
expected. Those streams can be changed, however, by issuing commands
using system predicates as defined in the SWI Prolog documentation.$residuals
that
is added to the results.s(CASP)
has a model and justification.A basic rule to remember is: any predicates designed to interact with or change the default behavior of the top level itself probably won't have any effect.
The most common way to use the Machine Query Interface is to find a
library that wraps and exposes it as a native part of another
programming language such as the Python swiplserver
library
(section 1.1). This section describes
how to build one if there isn't yet a library for your language. To do
this, you'll need to familiarize yourself with the MQI protocol as
described in the mqi_start/1
documentation. However, to give an idea of the scope of work required,
below is a typical interaction done (invisibly to the user) in the
implementation of any programming language library:
swipl mqi --write_connection_values=true
.
To work, the swipl
Prolog executable will need to be on the
path or the path needs to be specified in the command. This launches SWI
Prolog, starts the MQI, and writes the chosen port and password to
STDOUT. This way of launching invokes the mqi_start/0
predicate that turns off the int
(i.e. Interrupt/SIGINT)
signal to Prolog. This is because some languages (such as Python) use
that signal during debugging and it would be otherwise passed to the
client Prolog process and switch it into the debugger. See the mqi_start/0
predicate for more information on other command line options.\n
’.$ swipl mqi --write_connection_values=true 54501 185786669688147744015809740744888120144
Now the server is started. To create a connection:
<stringByteLength>.\n<stringBytes>.\n
where stringByteLength
includes the .\n
from
the string. For example: 7.\nhello.\n
More information on
the message format (section 1.6.1)
is below.true([[threads(Comm_Thread_ID, Goal_Thread_ID), version(Major, Minor)]])
(which will be in JSON form) indicating successful creation of the
connection. Comm_Thread_ID and Goal_Thread_ID are
the internal Prolog IDs of the two threads that are used for the
connection. They are sent solely for monitoring and debugging purposes. version
was introduced in version 1.0 of the protocol to allow for detecting the
protocol version and should be checked to ensure the protocol version is
supported by your library. See mqi_version/2
for more information and a version history.
We can try all of this using the Unix tool nc
(netcat)
(also available for Windows) to interactively connect to the MQI. In nc
hitting enter
sends \n
which is what the
message format requires. The server responses are show indented inline.
We'll use the port and password that were sent to STDOUT above:
$ nc 127.0.0.1 54501 41. 185786669688147744015809740744888120144. 173. { "args": [ [ [ { "args": ["mqi1_conn2_comm", "mqi1_conn2_goal" ], "functor":"threads" }, { "args": ["1", "0" ], "functor":"version" } ] ] ], "functor":"true" }
Now the connection is established. To run queries and shutdown:
run(atom(a), -1)
to run the synchronous
query atom(a)
with no timeout and wait for the response
message. It will be true([[]])
(in JSON form).close
,
waiting for the response message of true([[]])
(in JSON
form), and then closing the socket using the socket API of the language.
If the socket is closed (or fails) before the close
message
is sent, the default behavior of the MQI is to exit the SWI Prolog
process to avoid leaving the process around. This is to support
scenarios where the user is running and halting their language debugger
without cleanly exiting.quit
message and waiting for the response
message of true([[]])
(in JSON form). This will cause an
orderly shutdown and exit of the process.
Continuing with the nc
session (the quit
message isn't shown since the close
message closes the
connection):
18. run(atom(a), -1). 39. {"args": [ [ [] ] ], "functor":"true"} 7. close. 39. {"args": [ [ [] ] ], "functor":"true"}
Note that Unix Domain Sockets can be used instead of a TCP/IP port. How to do this is described with mqi_start/1.
Here's the same example running in the R language. Note that this is not an example of how to use the MQI from R, it just shows the first code a developer would write as they begin to build a nice library to connect R to Prolog using the MQI:
# Server run with: swipl mqi.pl --port=40001 --password=123 # R Source print("# Establish connection") sck = make.socket('localhost', 40001) print("# Send password") write.socket(sck, '5.\n') # message length write.socket(sck, '123.\n') # password print(read.socket(sck)) print("# Run query") query = 'run(member(X, [1, 2, 3]), -1).\n' write.socket(sck, paste(nchar(query), '.\n', sep='')) # message length write.socket(sck, query) # query print(read.socket(sck)) print("# Close session") close.socket(sck)
And here's the output:
[1] "# Establish connection" [1] "# Send password" [1] "172.\n{\n "args": [\n [\n [\n\t{\n\t "args": ["mqi1_conn1_comm", "mqi1_conn1_goal" ],\n\t "functor":"threads"\n\t}\n ]\n ]\n ],\n "functor":"true"\n}" [1] "# Run query" [1] "188.\n{\n "args": [\n [\n [ {"args": ["X", 1 ], "functor":"="} ],\n [ {"args": ["X", 2 ], "functor":"="} ],\n [ {"args": ["X", 3 ], "functor":"="} ]\n ]\n ],\n "functor":"true"\n}" [1] "# Close session"
Other notes about creating a new library to communicate with the MQI:
When using the Machine Query Interface from another language, debugging the Prolog code itself can often be done by viewing traces from the Prolog native writeln/1 or debug/3 predicates. Their output will be shown in the debugger of the native language used. Sometimes an issue surfaces deep in an application. When this happens, running the application in the native language while setting breakpoints and viewing traces in Prolog itself is often the best debugging approach. Standalone mode is designed for this scenario.
As the MQI is a multithreaded application, debugging the running code requires using the multithreaded debugging features of SWI Prolog as described in the section on "Debugging Threads" in the SWI Prolog documentation. A typical flow for Standalone Mode is:
tdebug, mqi_start([port(4242), password(debugnow)])
.In Python this would look like:
% From the SWI Prolog top level ?- tdebug, mqi_start([port(4242), password(debugnow)]). % The graphical front-end will be used for subsequent tracing true.
# Python using the swiplserver library {#mqi-library} from swiplserver import PrologMQI, PrologThread with PrologMQI(launch_mqi=False, port=4242, password="debugnow") as mqi: with mqi.create_thread() as prolog_thread: # Your code to be debugged here
At this point, all of the multi-threaded debugging tools in SWI Prolog are available for debugging the problem. If the issue is an unexpected exception, the exception debugging features of SWI Prolog can be used to break on the exception and examine the state of the application. If it is a logic error, breakpoints can be set to halt at the point where the problem appears, etc.
Note that, while using an MQI library to access Prolog will normally end and restart the process between runs of the code, running the server in standalone mode doesn't clear state between launches of the application. You'll either need to relaunch between runs or build your application so that it does the initialization at startup.
The messages the Machine Query Interface responds to are described below. A few things are true for all of them:
close
and waiting
for a response will halt the process if running in "Embedded Mode" (section
1.4). This is so that stopping a debugger doesn't leave the process
orphaned.-1
means no timeout.user
. module/1
has no effect.
Every Machine Query Interface message is a single valid Prolog term.
Those that run queries have an argument which represents the query as a
single term. To run several goals at once use (goal1, goal2, ...)
as the goal term.
The format of sent and received messages is identical (\n
stands for the ASCII newline character which is a single byte):
<stringByteLength>.\n<stringBytes>.\n.
For example, to send hello
as a message you would send
this:
7.\nhello.\n
<stringByteLength>
is the number of bytes of the
string to follow (including the .\n
), in human readable
numbers, such as 15
for a 15 byte string. It must be
followed by .\n
.<stringBytes>
is the actual message string being
sent, such as run(atom(a), -1).\n
. It must always end with .\n
.
The character encoding used to decode and encode the string is UTF-8.Important: The very first version of MQI (version 0.0) had a bug that required messages sent to (but not received from) MQI to use the count of Unicode code points (not bytes). This was fixed to properly require byte count in the next version, version 1.0.
To send a message to the MQI, send a message using the message format
above to the localhost port or Unix Domain Socket that the MQI is
listening on. For example, to run the synchronous goal atom(a)
,
send the following message:
18.\nrun(atom(a), -1).\n<end of stream>
You will receive the response below on the receive stream of the same connection you sent on. Note that the answer is in JSON format. If a message takes longer than 2 seconds, there will be "heartbeat" characters (".") at the beginning of the response message, approximately 1 every 2 seconds. So, if the query takes 6 seconds for some reason, there will be three "." characters first:
...12\ntrue([[]]).\n
The full list of Machine Query Interface messages is described below:
Timeout is in seconds and indicates a timeout for
generating all results for the query. Sending a variable (e.g. _)
will use the default timeout passed to the initial mqi_start/1
predicate and -1
means no timeout.
While it is waiting for the query to complete, sends a "." character not in message format, just as a single character, once every two seconds to proactively ensure that the client is alive. Those should be read and discarded by the client.
If a communication failure happens (during a heartbeat or otherwise), the connection is terminated, the query is aborted and (if running in "Embedded Mode" (section 1.4)) the SWI Prolog process shuts down.
When completed, sends a response message using the normal message format indicating the result.
Response:
true([Answer1, Answer2, ... ])
The goal succeeded at least once. The response always includes all answers as if run with findall()
(see run_async/3 below to get individual results back iteratively). Each Answer is a list of the assignments of free variables in the answer. A special variable called$residuals
will be added to each answer that has residual variable constraints on it. This will contain a list of all the constraints on all the variables for that answer. If there are no free variables, Answer is an empty list.false
The goal failed. exception(time_limit_exceeded)
The query timed out. exception(Exception)
An arbitrary exception was not caught while running the goal. exception(connection_failed)
The query thread unexpectedly exited. The MQI will no longer be listening after this exception.
async_result
message
(described below). The query can be cancelled by sending the cancel_async
message. If a previous query is still in progress, waits until that
query finishes (discarding that query's results) before responding.
Timeout is in seconds and indicates a timeout for
generating all results for the query. Sending a variable (e.g. _)
will use the default timeout passed to the initial mqi_start/1
predicate and -1
means no timeout.
If the socket closes before a response is sent, the connection is terminated, the query is aborted and (if running in "Embedded Mode" (section 1.4)) the SWI Prolog process shuts down.
If it needs to wait for the previous query to complete, it will send heartbeat messages (see "Machine Query Interface Message Format" (section 1.6.1)) while it waits. After it responds, however, it does not send more heartbeats. This is so that it can begin accepting new commands immediately after responding so the client.
Find_All == true
means generate one response to an async_result
message with all of the answers to the query (as in the run
message above). Find_All == false
generates a single
response to an async_result
message per answer.
Response:
true([[]])
The goal was successfully parsed. exception(Exception)
An error occurred parsing the goal. exception(connection_failed)
The goal thread unexpectedly shut down. The MQI will no longer be listening after this exception.
run_async
message
in a way that allows further queries to be run on this Prolog thread
afterwards.
If there is a goal running, injects a throw(cancel_goal)
into the executing goal to attempt to stop the goal's execution. Begins
accepting new commands immediately after responding. Does not inject abort/0
because this would kill the connection's designated thread and the
system is designed to maintain thread local data for the client. This
does mean it is a "best effort" cancel since the exception can be
caught.
cancel_async
is guaranteed to either respond with an
exception (if there is no query or pending results from the last query),
or safely attempt to stop the last executed query even if it has already
finished.
To guarantee that a query is cancelled, send close
and
close the socket.
It is not necessary to determine the outcome of cancel_async
after sending it and receiving a response. Further queries can be
immediately run. They will start after the current query stops.
However, if you do need to determine the outcome or determine when
the query stops, send async_result
. Using Timeout = 0
is recommended since the query might have caught the exception or still
be running. Sending async_result
will find out the
"natural" result of the goal's execution. The "natural" result depends
on the particulars of what the code actually did. The response could be:
exception(cancel_goal)
The query was running and did not catch the exception. I.e. the goal was successfully cancelled. exception(time_limit_exceeded)
The query timed out before getting cancelled. exception(Exception)
They query hits another exception before it has a chance to be cancelled. A valid answer The query finished before being cancelled.
Note that you will need to continue sending async_result
until you receive an exception(Exception)
message if you
want to be sure the query is finished (see documentation for async_result
).
Response:
true([[]])
There is a query running or there are pending results for the last query. exception(no_query)
There is no query or pending results from a query to cancel. exception(connection_failed)
The connection has been unexpectedly shut down. The MQI will no longer be listening after this exception.
run_async
message. Used to get results for all cases: if the query terminates
normally, is cancelled by sending a cancel_async
message,
or times out.
Each response to an async_result
message responds with
one result and, when there are no more results, responds with exception(no_more_results)
or whatever exception stopped the query. Receiving any exception
response except exception(result_not_available)
means there
are no more results. If run_async
was run with Find_All == false
,
multiple async_result
messages may be required before
receiving the final exception.
Waits Timeout seconds for a result. Timeout == -1
or sending a variable for Timeout indicates no timeout. If
the timeout is exceeded and no results are ready, sends exception(result_not_available)
.
Some examples:
If the query succeeds with N answers... async_result
messages 1 to N will receive each answer, in order, andasync_result
message N+1 will receiveexception(no_more_results)
If the query fails (i.e. has no answers)... async_result
message 1 will receivefalse
andasync_result
message 2 will receiveexception(no_more_results)
If the query times out after one answer... async_result
message 1 will receive the first answer andasync_result
message 2 will receiveexception(time_limit_exceeded)
If the query is cancelled after it had a chance to get 3 answers... async_result
messages 1 to 3 will receive each answer, in order, andasync_result
message 4 will receiveexception(cancel_goal)
If the query throws an exception before returning any results... async_result
message 1 will receiveexception(Exception)
Note that, after sending cancel_async
, calling async_result
will return the "natural" result of the goal's execution. The "natural"
result depends on the particulars of what the code actually did since
this is multi-threaded and there are race conditions. This is described
more below in the response section and above in cancel_async
.
Response:
true([Answer1, Answer2, ... ])
The next answer from the query is a successful answer. Whether there are more than one Answer in the response depends on the findall
setting. Each Answer is a list of the assignments of free variables in the answer. A special variable called$residuals
will be added to each answer that has residual variable constraints on it. This will contain a list of all the constraints on all the variables for that answer. If there are no free variables, Answer is an empty list.false
The query failed with no answers. exception(no_query)
There is no query in progress. exception(result_not_available)
There is a running query and no results were available in Timeout seconds. exception(no_more_results)
There are no more answers and no other exception occurred. exception(cancel_goal)
The next answer is an exception caused by cancel_async
. Indicates no more answers.exception(time_limit_exceeded)
The query timed out generating the next answer (possibly in a race condition before getting cancelled). Indicates no more answers. exception(Exception)
The next answer is an arbitrary exception. This can happen after cancel_async
if thecancel_async
exception is caught or the code hits another exception first. Indicates no more answers.exception(connection_failed)
The goal thread unexpectedly exited. The MQI will no longer be listening after this exception.
Any asynchronous query that is still running will be halted by using abort/0 in the connection's query thread.
Response:
true([[]])
Response:
true([[]])
swiplserver
Python library (section 1.1), but
starting manually can be useful when debugging Prolog code in some
scenarios. See the documentation on "Standalone Mode" (section
1.5) for more information.
Once started, the MQI listens for TCP/IP or Unix Domain Socket connections and authenticates them using the password provided (or created depending on options) before processing any messages. The messages processed by the MQI are described below (section 1.6).
For debugging, the server outputs traces using the debug/3 predicate so that the server operation can be observed by using the debug/1 predicate. Run the following commands to see them:
debug(mqi(protocol))
: Traces protocol messages to show
the flow of commands and connections. It is designed to avoid filling
the screen with large queries and results to make it easier to read.debug(mqi(query))
: Traces messages that involve each
query and its results. Therefore it can be quite verbose depending on
the query.
Options
Options is a list containing any combination of the following options. When used in the Prolog top level (i.e. in Standalone Mode (section 1.5)), these are specified as normal Prolog options like this:
mqi_start([unix_domain_socket(Socket), password('a password')])
When using "Embedded Mode" (section 1.4) they are passed using the same name but as normal command line arguments like this:
swipl mqi --write_connection_values=true --password="a password" --create_unix_domain_socket=true
Note the use of quotes around values that could confuse command line
processing like spaces (e.g. "a password") and that
unix_domain_socket(Variable)
is written as
--create_unix_domain_socket=true
on the command line. See
below for more information.
write_connection_values(true)
is set, the selected port is output to STDOUT followed by \n
on startup to allow the client language library to retrieve it in
"Embedded Mode" (section 1.4).
To have one generated instead (recommended), pass Unix_Domain_Socket_Path_And_File
as a variable when calling from the Prolog top level and the variable
will be unified with a created filename. If launching in "Embedded Mode"
(section 1.4), instead pass --create_unix_domain_socket=true
since there isn't a way to specify variables from the command line. When
generating the file, a temporary directory will be created using tmp_file/2
and a socket file will be created within that directory following the
below requirements. If the directory and file are unable to be created
for some reason, mqi_start/1
fails.
Regardless of whether the file is specified or generated, if the
option write_connection_values(true)
is set, the fully
qualified path to the generated file is output to STDOUT followed by \n
on startup to allow the client language library to retrieve it.
Specifying a file to use should follow the same guidelines as the generated file:
write_connection_values(true)
is set, the password is output to STDOUT followed by \n
on
startup to allow the client language library to retrieve it. This is the
recommended way to integrate the MQI with a language as it avoids
including the password as source code. This option is only included so
that a known password can be supplied for when the MQI is running in
Standalone Mode.query_timeout(+Seconds)
Sets the default time in
seconds that a query is allowed to run before it is cancelled. This can
be overridden on a query by query basis. If not set, the default is no
timeout (-1
).pending_connections(+Count)
Sets the number of pending
connections allowed for the MQI as in tcp_listen/2.
If not provided, the default is 5
.run_server_on_thread(+Run_Server_On_Thread)
Determines
whether mqi_start/1 runs in the
background on its own thread or blocks until the MQI shuts down. Must be
missing or set to true
when running in "Embedded Mode" (section
1.4) so that the SWI Prolog process can exit properly. If not set,
the default is true
.run_server_on_thread(true)
.
Passing in an atom for Server_Thread will only set the server thread
name if run_server_on_thread(true)
. If Server_Thread
is a variable, it is unified with a generated name.write_connection_values(+Write_Connection_Values)
Determines whether the server writes the port (or generated Unix Domain
Socket) and password to STDOUT as it initializes. Used by language
libraries to retrieve this information for connecting. If not set, the
default is false
.write_output_to_file(+File)
Redirects STDOUT and STDERR
to the file path specified. Useful for debugging the MQI when it is
being used in "Embedded Mode" (section
1.4). If using multiple MQI instances in one SWI Prolog instance,
only set this on the first one. Each time it is set the output will be
redirected.Note that the initial version of the MQI did not have a version predicate so The proper way for callers to check the version is:
use_module(library(mqi))
, ( current_predicate(mqi_version/2)
->
mqi_version(Major_Version, Minor_Version)
; Major_Version = 0, Minor_Version = 0 )
Major versions are increased when there is a change to the protocol that will likely break clients written to the previous version. Minor versions are increased when there is new functionality that will not break clients written to the old version
This allows a client written to MQI version’Client_Major_Version.Client_Minor_Version’to check for non-breaking compatibility like this:
Client_Major_Version = MQI_Major_Version and Client_Minor_Version <=
MQI_Minor_Version
Breaking changes (i.e. Major version increments) should be very rare as the goal is to have the broadest adoption possible.
Protocol Version History:
To launch embedded mode:
swipl mqi --write_connection_values=true
This will start SWI Prolog and invoke the mqi_start/0
predicate and exit the process when that predicate stops. Any command
line arguments after the standalone --
will be passed as
Options. These are the same Options that mqi_start/1
accepts and are passed to it directly. Some options are expressed
differently due to command line limitations, see mqi_start/1
Options for more information.
Any Option values that cause issues during command line parsing (such
as spaces) should be passed with ""
like this:
swipl mqi --write_connection_values=true --password="HGJ SOWLWW"
For help on commandline options run
swipl mqi --help
Always succeeds.