From 1e734938ae01f31acfbc8d2d235e54f432ef7fa1 Mon Sep 17 00:00:00 2001 From: lethosor Date: Sun, 19 Jul 2020 01:00:00 -0400 Subject: [PATCH 1/6] Add isoworldremote to Plugins.rst --- docs/Plugins.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/Plugins.rst b/docs/Plugins.rst index ee0aa074c..7c37ea78c 100644 --- a/docs/Plugins.rst +++ b/docs/Plugins.rst @@ -59,6 +59,11 @@ remotefortressreader An in-development plugin for realtime fortress visualisation. See :forums:`Armok Vision <146473>`. +.. _isoworldremote: + +isoworldremote +============== +A plugin that implements a `remote API ` used by Isoworld. .. _cursecheck: From d88137b1d653f2efcd7407af10e3bd4907d7b86a Mon Sep 17 00:00:00 2001 From: lethosor Date: Sun, 19 Jul 2020 01:13:43 -0400 Subject: [PATCH 2/6] Split remote API info into separate doc, add example tools and clients --- docs/Dev-intro.rst | 9 ++------- docs/Remote.rst | 47 ++++++++++++++++++++++++++++++++++++++++++++++ docs/index-dev.rst | 1 + 3 files changed, 50 insertions(+), 7 deletions(-) create mode 100644 docs/Remote.rst diff --git a/docs/Dev-intro.rst b/docs/Dev-intro.rst index 9a08533c0..2a17313fe 100644 --- a/docs/Dev-intro.rst +++ b/docs/Dev-intro.rst @@ -77,11 +77,6 @@ some functions (and some entire modules) are currently only available in C++. Remote access interface ----------------------- -DFHack supports remote access by exchanging Google protobuf messages via a TCP -socket. Both the core and plugins can define remotely accessible methods. The -``dfhack-run`` command uses this interface to invoke ordinary console commands. - -Currently the supported set of requests is limited, because the developers don't -know what exactly is most useful. `remotefortressreader` provides a fairly -comprehensive interface for visualisers such as :forums:`Armok Vision <146473>`. +DFHack provides a remote access interface that external tools can connect to and +use to interact with DF. See `remote` for more information. diff --git a/docs/Remote.rst b/docs/Remote.rst new file mode 100644 index 000000000..b7c65b032 --- /dev/null +++ b/docs/Remote.rst @@ -0,0 +1,47 @@ +.. _remote: + +======================= +DFHack Remote Interface +======================= + +DFHack provides a remote access interface that external tools can connect to and +use to interact with DF. This is implemented with `Google protobuf`_ messages +exchanged over a TCP socket (which only accepts connections from the local +machine by default). Both the core and plugins can define remotely-accessible +methods (often referred to as **RPC methods**). The RPC methods currently +available are not comprehensive, but can be extended with plugins. + +.. _Google protobuf: https://developers.google.com/protocol-buffers + +.. contents:: + :local: + +Examples +======== + +The `dfhack-run` command uses the RPC interface to invoke DFHack commands +(or Lua functions) externally. + +Plugins that implement RPC methods include: + +- `rename` +- `remotefortressreader` +- `isoworldremote` + +Plugins that use the RPC API include: + +- `stonesense` + +Third-party tools that use the RPC API include: + +- `Armok Vision `_ (:forums:`Bay12 forums thread <146473>`) + +Client libraries +================ + +Some external libraries are available for interacting with the remote interface +from other (non-C++) languages, including: + +- `RemoteClientDF-Net `_ for C# +- `dfhackrpc `_ for Go +- `dfhack-remote `_ for JavaScript diff --git a/docs/index-dev.rst b/docs/index-dev.rst index 617c0c0d7..560137609 100644 --- a/docs/index-dev.rst +++ b/docs/index-dev.rst @@ -15,5 +15,6 @@ These are pages relevant to people developing for DFHack. /docs/Documentation /docs/Structures-intro /docs/Memory-research + /docs/Remote /docs/Binpatches From b34aa86d3c3dd606c22877d9cf95d25204e5d567 Mon Sep 17 00:00:00 2001 From: lethosor Date: Tue, 21 Jul 2020 21:39:37 -0400 Subject: [PATCH 3/6] Document remote server config --- docs/Core.rst | 3 +++ docs/Remote.rst | 25 ++++++++++++++++++++++--- 2 files changed, 25 insertions(+), 3 deletions(-) diff --git a/docs/Core.rst b/docs/Core.rst index 6ef1d0575..196b12918 100644 --- a/docs/Core.rst +++ b/docs/Core.rst @@ -442,6 +442,8 @@ Other init files directory, will be run when any world or that save is loaded. +.. _env-vars: + Environment variables ===================== @@ -453,6 +455,7 @@ on UNIX-like systems:: - ``DFHACK_PORT``: the port to use for the RPC server (used by ``dfhack-run`` and `remotefortressreader` among others) instead of the default ``5000``. As with the default, if this port cannot be used, the server is not started. + See `remote` for more details. - ``DFHACK_DISABLE_CONSOLE``: if set, the DFHack console is not set up. This is the default behavior if ``PRINT_MODE:TEXT`` is set in ``data/init/init.txt``. diff --git a/docs/Remote.rst b/docs/Remote.rst index b7c65b032..5fa556ef2 100644 --- a/docs/Remote.rst +++ b/docs/Remote.rst @@ -6,9 +6,8 @@ DFHack Remote Interface DFHack provides a remote access interface that external tools can connect to and use to interact with DF. This is implemented with `Google protobuf`_ messages -exchanged over a TCP socket (which only accepts connections from the local -machine by default). Both the core and plugins can define remotely-accessible -methods (often referred to as **RPC methods**). The RPC methods currently +exchanged over a TCP socket. Both the core and plugins can define +remotely-accessible methods, or **RPC methods**. The RPC methods currently available are not comprehensive, but can be extended with plugins. .. _Google protobuf: https://developers.google.com/protocol-buffers @@ -16,6 +15,26 @@ available are not comprehensive, but can be extended with plugins. .. contents:: :local: + +Server configuration +==================== + +DFHack attempts to start a TCP server to listen for remote connections on +startup. If this fails (due to the port being in use, for example), an error +message will be logged to stderr.log. + +The server can be configured by setting options in ``dfhack-config/remote-server.json``: + +- ``allow_remote`` (default: ``false``): if true, allows connections from hosts + other than the local machine. This is insecure and may allow arbitrary code + execution on your machine, so it is disabled by default. +- ``port`` (default: ``5000``): the port that the remote server listens on. + Overriding this will allow the server to work if you have multiple instances + of DF running, or if you have something else running on port 5000. Note that + the ``DFHACK_PORT`` `environment variable ` takes precedence over + this setting and may be more useful for overriding the port temporarily. + + Examples ======== From dfc9f78fa4f2deeb576a7c334b8a75f69dfe01f4 Mon Sep 17 00:00:00 2001 From: lethosor Date: Tue, 21 Jul 2020 23:14:59 -0400 Subject: [PATCH 4/6] Refer to server config docs from dfhack-run --- docs/Core.rst | 10 +++++++++- docs/Remote.rst | 2 ++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/Core.rst b/docs/Core.rst index 196b12918..c5d5670af 100644 --- a/docs/Core.rst +++ b/docs/Core.rst @@ -82,7 +82,7 @@ dfhack-run If DF and DFHack are already running, calling ``dfhack-run my command`` in an external terminal is equivalent to calling ``my command`` in the -DFHack console. Direct use of the DFhack console is generally easier, +DFHack console. Direct use of the DFHack console is generally easier, but ``dfhack-run`` can be useful in a variety of circumstances: - if the console is unavailable @@ -101,6 +101,14 @@ Examples:: The first (\*nix) example `checks for vampires `; the second (Windows) example uses `kill-lua` to stop a Lua script. +.. note:: + + ``dfhack-run`` attempts to connect to a server on TCP port 5000. If DFHack + was unable to start this server, ``dfhack-run`` will not be able to connect. + This could happen if you have other software listening on port 5000, or if + you have multiple copies of DF running simultaneously. To assign a different + port, see `remote-server-config`. + Built-in Commands ================= diff --git a/docs/Remote.rst b/docs/Remote.rst index 5fa556ef2..24426c819 100644 --- a/docs/Remote.rst +++ b/docs/Remote.rst @@ -16,6 +16,8 @@ available are not comprehensive, but can be extended with plugins. :local: +.. _remote-server-config: + Server configuration ==================== From 4b6325eeabd6311ed30c38199f1e4111df519c19 Mon Sep 17 00:00:00 2001 From: lethosor Date: Wed, 22 Jul 2020 01:34:51 -0400 Subject: [PATCH 5/6] Add RPC protocol description to docs and reformat From https://gist.github.com/BenLubar/1f51cf570b4c8dca275db9687fa6b9e3 Ref dfhack/dfhack#1574 --- docs/Remote.rst | 166 +++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 164 insertions(+), 2 deletions(-) diff --git a/docs/Remote.rst b/docs/Remote.rst index 24426c819..689f9978b 100644 --- a/docs/Remote.rst +++ b/docs/Remote.rst @@ -37,8 +37,16 @@ The server can be configured by setting options in ``dfhack-config/remote-server this setting and may be more useful for overriding the port temporarily. +Developing with the remote API +============================== + +At a high level, the core and plugins define RPC methods, and external clients +can call these methods. Each method is assigned an ID internally, which clients +use to call it. These method IDs can be obtained using the special ``BindMethod`` +method, which has an ID of 0. + Examples -======== +-------- The `dfhack-run` command uses the RPC interface to invoke DFHack commands (or Lua functions) externally. @@ -58,7 +66,7 @@ Third-party tools that use the RPC API include: - `Armok Vision `_ (:forums:`Bay12 forums thread <146473>`) Client libraries -================ +---------------- Some external libraries are available for interacting with the remote interface from other (non-C++) languages, including: @@ -66,3 +74,157 @@ from other (non-C++) languages, including: - `RemoteClientDF-Net `_ for C# - `dfhackrpc `_ for Go - `dfhack-remote `_ for JavaScript + + +Protocol description +==================== + +This is a low-level description of the RPC protocol, which may be useful when +developing custom clients. + +Built-in messages +----------------- +These messages have hardcoded IDs; all others must be obtained through ``BindMethod``. + +=== ============ =============================== ======================= +ID Method Input Output +=== ============ =============================== ======================= + 0 BindMethod dfproto.CoreBindRequest dfproto.CoreBindReply + 1 RunCommand dfproto.CoreRunCommandRequest dfproto.EmptyMessage +=== ============ =============================== ======================= + + + +Conversation flow +----------------- + +* Client → Server: `handshake request`_ +* Server → Client: `handshake reply`_ +* Repeated 0 or more times: + * Client → Server: `request`_ + * Server → Client: `text`_ (0 or more times) + * Server → Client: `result`_ or `failure`_ +* Client → Server: `quit`_ + +Raw message types +----------------- + +* All numbers are little-endian +* All strings are ASCII +* A payload size of greater than 64MiB is an error +* See ``RemoteClient.h`` for definitions of constants starting with ``RPC`` + +handshake request +~~~~~~~~~~~~~~~~~ + +.. csv-table:: + :align: left + :header-rows: 1 + + Type, Name, Value + char[8], magic, ``DFHack?\n`` + int32_t, version, 1 + +handshake reply +~~~~~~~~~~~~~~~ + +.. csv-table:: + :align: left + :header-rows: 1 + + Type, Name, Value + char[8], magic, ``DFHack!\n`` + int32_t, version, 1 + +header +~~~~~~ + +**Note:** the two fields of this message are sometimes repurposed. Uses of this +message are represented as ``header(x, y)``, where ``x`` corresponds to the ``id`` +field and ``y`` corresponds to ``size``. + +.. csv-table:: + :align: left + :header-rows: 1 + + Type, Name + int16_t, id + int16_t, (padding - unused) + int32_t, size + +request +~~~~~~~ + +.. list-table:: + :align: left + :header-rows: 1 + :widths: 25 75 + + * - Type + - Description + * - `header`_ + - ``header(id, size)`` + * - buffer + - Protobuf-encoded payload of the input message type of the method specified by ``id``; length of ``size`` bytes + +text +~~~~ + +.. list-table:: + :align: left + :header-rows: 1 + :widths: 25 75 + + * - Type + - Description + * - `header`_ + - ``header(RPC_REPLY_TEXT, size)`` + * - buffer + - Protobuf-encoded payload of type ``dfproto.CoreTextNotification``; length of ``size`` bytes + +result +~~~~~~ + +.. list-table:: + :align: left + :header-rows: 1 + :widths: 25 75 + + * - Type + - Description + * - `header`_ + - ``header(RPC_REPLY_RESULT, size)`` + * - buffer + - Protobuf-encoded payload of the output message type of the oldest incomplete method call; when received, + that method call is considered completed. Length of ``size`` bytes. + +failure +~~~~~~~ + +.. list-table:: + :align: left + :header-rows: 1 + :widths: 25 75 + + * - Type + - Description + * - `header`_ + - ``header(RPC_REPLY_FAIL, command_result)`` + * - command_result + - return code of the command (a constant starting with ``CR_``; see ``RemoteClient.h``) + +quit +~~~~ + +**Note:** the server closes the connection after sending this message. + +.. list-table:: + :align: left + :header-rows: 1 + :widths: 25 75 + :width: 100% + + * - Type + - Description + * - `header`_ + - ``header(RPC_REQUEST_QUIT, 0)`` From eb074b8a493fb971a98df156b95926dca1146dec Mon Sep 17 00:00:00 2001 From: lethosor Date: Wed, 22 Jul 2020 02:04:02 -0400 Subject: [PATCH 6/6] Link to Wireshark script --- docs/Remote.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/Remote.rst b/docs/Remote.rst index 689f9978b..a40161fee 100644 --- a/docs/Remote.rst +++ b/docs/Remote.rst @@ -82,6 +82,10 @@ Protocol description This is a low-level description of the RPC protocol, which may be useful when developing custom clients. +A WireShark dissector for this protocol is available in the +`df_misc repo `_. + + Built-in messages ----------------- These messages have hardcoded IDs; all others must be obtained through ``BindMethod``. @@ -216,7 +220,7 @@ failure quit ~~~~ -**Note:** the server closes the connection after sending this message. +**Note:** the server closes the connection after receiving this message. .. list-table:: :align: left