From 128050c68435eb9633aac4af35070f57fd25e55c Mon Sep 17 00:00:00 2001 From: lethosor Date: Sat, 18 Jul 2020 03:16:52 -0400 Subject: [PATCH 01/13] Move installation docs to a separate document and expand --- docs/Installing.rst | 132 ++++++++++++++++++++++++++++++++++++++++++ docs/Introduction.rst | 46 +++------------ index.rst | 1 + 3 files changed, 142 insertions(+), 37 deletions(-) create mode 100644 docs/Installing.rst diff --git a/docs/Installing.rst b/docs/Installing.rst new file mode 100644 index 000000000..92e5cfa0a --- /dev/null +++ b/docs/Installing.rst @@ -0,0 +1,132 @@ +.. _installing: + +================= +Installing DFHack +================= + +.. contents:: + :local: + + +Requirements +============ + +DFHack supports Windows, Linux, and macOS, and both 64-bit and 32-bit builds +of Dwarf Fortress. + +.. _installing-df-version: + +DFHack releases generally only support the version of Dwarf Fortress that they +are named after. For example, DFHack 0.40.24-r5 only supported DF 0.40.24. +DFHack releases *never* support newer versions of DF, because DFHack requires +data about DF that is only possible to obtain after DF has been released. +Occasionally, DFHack releases will be able to maintain support for older +versions of DF - for example, DFHack 0.34.11-r5 supported both DF 0.34.11 and +0.34.10. For maximum stability, you should usually use the latest versions of +both DF and DFHack. + +Windows +------- + +* DFHack only supports the SDL version of Dwarf Fortress. The "legacy" version + will *not* work with DFHack (the "small" SDL version is acceptable, however). +* Windows XP and older are *not* supported, due in part to a + `Visual C++ 2015 bug `_ + +The Windows build of DFHack should work under Wine on other operating systems, +although this is not tested very often. It is recommended to use the native +build for your operating system instead. + +.. _installing-reqs-linux: + +Linux +----- + +Generally, DFHack should work on any modern Linux distribution. There are +multiple release binaries provided - as of DFHack 0.47.04-r1, there are built +with GCC 7 and GCC 4.8 (as indicated by the ``gcc`` component of their +filenames). Using the newest build that works on your system is recommended. +The GCC 4.8 build is built on Ubuntu 14.04 and targets an older glibc, so it +should work on older distributions. + +In the event that none of the provided binaries work on your distribution, +you may need to `compile DFHack from source `. + +macOS +----- + +OS X 10.6.8 or later is required. + + +.. _downloading: + +Downloading DFHack +================== + +Stable builds of DFHack are available on `GitHub `_. +GitHub has been known to change their layout periodically, but as of July 2020, +downloads are available at the bottom of the release notes for each release, under a section +named "Assets" (which you may have to expand). The name of the file indicates +which DF version, platform, and architecture the build supports - the platform +and architecture (64-bit or 32-bit) **must** match your build of DF. The DF +version should also match your DF version - see `above ` +for details. For example: + +* ``dfhack-0.47.04-r1-Windows-64bit.zip`` supports 64-bit DF on Windows +* ``dfhack-0.47.04-r1-Linux-32bit-gcc-7.tar.bz2`` supports 32-bit DF on Linux + (see `installing-reqs-linux` for details on the GCC version indicator) + +The `DFHack website `_ also provides links to +unstable builds. These files have a different naming scheme, but the same +restrictions apply (e.g. a file named ``Windows64`` is for 64-bit Windows DF). + +.. warning:: + + Do *not* download the source code from GitHub, either from the releases page + or by clicking "Download ZIP" on the repo homepage. This will give you an + incomplete copy of the DFHack source code, which will not work as-is. (If + you want to compile DFHack instead of using a pre-built release, see + `compile` for instructions.) + +Installing DFHack +================= + +When you `download DFHack `, you will end up with a release archive +(a ``.zip`` file on Windows, or a ``.tar.bz2`` file on other platforms). Your +operating system should have built-in utilities capable of extracting files from +these archives. + +The release archives contain several files and folders, including a ``hack`` +folder, a ``dfhack-config`` folder, and a ``dfhack.init-example`` file. All +files from the DFHack archive need to be copied into the root DF folder, which +should already include a ``data`` folder and a ``raw`` folder, among other +things. Some packs and other redistributions of Dwarf Fortress may place DF in +another folder, so ensure that the ``hack`` folder ends up next to the ``data`` +folder. + +.. note:: + + On Windows, installing DFHack will overwrite ``SDL.dll``. This is + intentional and necessary for DFHack to work, so be sure to choose to + overwrite ``SDL.dll`` if prompted. (If you are not prompted, you may be + installing DFHack in the wrong place.) + + +Uninstalling DFHack +=================== + +Uninstalling DFHack essentially involves reversing what you did to install +DFHack. On Windows, replace ``SDL.dll`` with ``SDLreal.dll`` first. Then, you +can remove any files that were part of the DFHack archive. DFHack does not +currently maintain a list of these files, so if you want to completely remove +them, you should consult the DFHack archive that you installed for a full list. +Generally, any files left behind should not negatively affect DF. + +Pre-packaged DFHack installations +================================= + +There are :wiki:`several packs available ` that include +DF, DFHack, and other utilities. If you are new to Dwarf Fortress and DFHack, +these may be easier to set up. Note that these packs are not maintained by the +DFHack team and vary in their release schedules and contents. Some may make +significant configuration changes, and some may not include DFHack at all. diff --git a/docs/Introduction.rst b/docs/Introduction.rst index 4490086f3..7d510422f 100644 --- a/docs/Introduction.rst +++ b/docs/Introduction.rst @@ -35,47 +35,19 @@ allows easier development of new tools. As an open-source project under :local: -.. _installing: - -Installing DFHack -================= -DFHack is available for the SDL version of Dwarf Fortress on Windows, -any modern Linux distribution, and Mac OS X (10.6.8 and later). -It is possible to use Windows DF+DFHack under Wine on Linux or OS X. - -Most releases only support the version of DF mentioned in their title - for -example, DFHack 0.40.24-r2 only supports DF 0.40.24 - but some releases -support earlier DF versions as well. Wherever possible, use the latest version -of DFHack built for the target version of DF. - -Installing DFhack involves copying files from a release archive -into your DF folder, so that: - -* On Windows, ``SDL.dll`` is replaced -* On Linux or OS X, the ``dfhack`` script is placed in the same folder as the ``df`` script - -Uninstalling is basically the same, in reverse: - -* On Windows, replace ``SDL.dll`` with ``SDLreal.dll``, then remove the DFHack files. -* On Linux or OS X, remove the DFHack files. - -New players may wish to :wiki:`get a pack ` -with DFHack preinstalled. - - Getting started =============== -DFHack basically extends DF with something similar to the -console found in many PC games. +See `installing` for details on installing DFHack. -If DFHack is installed correctly, it will automatically pop up a console -window once DF is started as usual on Windows. Linux and Mac OS X require -running the dfhack script from the terminal, and will use that terminal for -the console. +Once DFHack is installed, it extends DF with a console that can be used to run +commands. On Windows, this console will open automatically when DF is started. +On Linux and macOS, you will need to run the ``dfhack`` script from a terminal +(instead of the ``df`` script included with DF), and that terminal will be +used by the DFHack console. -* Basic interaction with dfhack involves entering commands into the console. - To learn what commands are available, you can keep reading this documentation - or skip ahead and use the `ls` and `help` commands. +* Basic interaction with DFHack involves entering commands into the console. To + learn what commands are available, you can keep reading this documentation or + skip ahead and use the `ls` and `help` commands. * Another way to interact with DFHack is to set in-game `keybindings ` for certain commands. Many of the newer and user-friendly tools are designed diff --git a/index.rst b/index.rst index 91c883b51..d0e90b741 100644 --- a/index.rst +++ b/index.rst @@ -27,6 +27,7 @@ User Manual :maxdepth: 2 /docs/Introduction + /docs/Installing /docs/Core /docs/Plugins /docs/Scripts From 6921277e8104187d74d7a68be0f4df2b94ecc6ce Mon Sep 17 00:00:00 2001 From: lethosor Date: Sat, 18 Jul 2020 12:35:04 -0400 Subject: [PATCH 02/13] Add upgrading instructions --- docs/Installing.rst | 32 ++++++++++++++++++++++++++------ 1 file changed, 26 insertions(+), 6 deletions(-) diff --git a/docs/Installing.rst b/docs/Installing.rst index 92e5cfa0a..c0175b820 100644 --- a/docs/Installing.rst +++ b/docs/Installing.rst @@ -97,12 +97,12 @@ operating system should have built-in utilities capable of extracting files from these archives. The release archives contain several files and folders, including a ``hack`` -folder, a ``dfhack-config`` folder, and a ``dfhack.init-example`` file. All -files from the DFHack archive need to be copied into the root DF folder, which -should already include a ``data`` folder and a ``raw`` folder, among other -things. Some packs and other redistributions of Dwarf Fortress may place DF in -another folder, so ensure that the ``hack`` folder ends up next to the ``data`` -folder. +folder, a ``dfhack-config`` folder, and a ``dfhack.init-example`` file. To +install DFHack, copy all of the files from the DFHack archive into the root DF +folder, which should already include a ``data`` folder and a ``raw`` folder, +among other things. Some packs and other redistributions of Dwarf Fortress may +place DF in another folder, so ensure that the ``hack`` folder ends up next to +the ``data`` folder. .. note:: @@ -122,6 +122,26 @@ currently maintain a list of these files, so if you want to completely remove them, you should consult the DFHack archive that you installed for a full list. Generally, any files left behind should not negatively affect DF. + +Upgrading DFHack +================ + +The recommended approach to upgrade DFHack is to uninstall DFHack first, then +install the new version. This will ensure that any files that are only part +of the older DFHack installation do not affect the new DFHack installation +(although this is unlikely to occur). + +It is also possible to overwrite an existing DFHack installation in-place. +To do this, follow the installation instructions above, but overwrite all files +that exist in the new DFHack archive (on Windows, this includes ``SDL.dll`` again). + +.. note:: + + You may wish to make a backup of your ``dfhack-config`` folder first if you + have made changes to it. Some archive managers (e.g. Archive Utility on macOS) + will overwrite the entire folder, removing any files that you have added. + + Pre-packaged DFHack installations ================================= From 044add214710434a6d0fddb30b48209d3ed0f017 Mon Sep 17 00:00:00 2001 From: lethosor Date: Sat, 18 Jul 2020 12:48:20 -0400 Subject: [PATCH 03/13] Reorder .proto comments --- plugins/proto/AdventureControl.proto | 2 +- plugins/proto/DwarfControl.proto | 2 +- plugins/proto/ItemdefInstrument.proto | 2 +- plugins/proto/RemoteFortressReader.proto | 2 +- plugins/proto/isoworldremote.proto | 2 +- plugins/proto/ui_sidebar_mode.proto | 4 ++-- 6 files changed, 7 insertions(+), 7 deletions(-) diff --git a/plugins/proto/AdventureControl.proto b/plugins/proto/AdventureControl.proto index 0bbcc8da8..840576c2b 100644 --- a/plugins/proto/AdventureControl.proto +++ b/plugins/proto/AdventureControl.proto @@ -1,6 +1,6 @@ +//Attempts to provide a complete framework for reading everything from a fortress needed for vizualization package AdventureControl; -//Attempts to provide a complete framework for reading everything from a fortress needed for vizualization option optimize_for = LITE_RUNTIME; import "RemoteFortressReader.proto"; diff --git a/plugins/proto/DwarfControl.proto b/plugins/proto/DwarfControl.proto index 9bd63b8d3..ee750f8bc 100644 --- a/plugins/proto/DwarfControl.proto +++ b/plugins/proto/DwarfControl.proto @@ -1,7 +1,7 @@ +//Attempts to provide a complete framework for reading everything from a fortress needed for vizualization syntax = "proto2"; package DwarfControl; -//Attempts to provide a complete framework for reading everything from a fortress needed for vizualization option optimize_for = LITE_RUNTIME; // Plugin: RemoteFortressReader diff --git a/plugins/proto/ItemdefInstrument.proto b/plugins/proto/ItemdefInstrument.proto index 02d4c7a90..0b64eb6d1 100644 --- a/plugins/proto/ItemdefInstrument.proto +++ b/plugins/proto/ItemdefInstrument.proto @@ -1,6 +1,6 @@ +//Attempts to provide a complete framework for reading everything from a fortress needed for vizualization package ItemdefInstrument; -//Attempts to provide a complete framework for reading everything from a fortress needed for vizualization option optimize_for = LITE_RUNTIME; // Plugin: RemoteFortressReader diff --git a/plugins/proto/RemoteFortressReader.proto b/plugins/proto/RemoteFortressReader.proto index 8255f91cd..a128a162f 100644 --- a/plugins/proto/RemoteFortressReader.proto +++ b/plugins/proto/RemoteFortressReader.proto @@ -1,6 +1,6 @@ +//Attempts to provide a complete framework for reading everything from a fortress needed for vizualization package RemoteFortressReader; -//Attempts to provide a complete framework for reading everything from a fortress needed for vizualization option optimize_for = LITE_RUNTIME; // Plugin: RemoteFortressReader diff --git a/plugins/proto/isoworldremote.proto b/plugins/proto/isoworldremote.proto index 23abdab73..c972e98a8 100644 --- a/plugins/proto/isoworldremote.proto +++ b/plugins/proto/isoworldremote.proto @@ -1,6 +1,6 @@ +//Describes a very basic material structure for the map embark package isoworldremote; -//Describes a very basic material structure for the map embark option optimize_for = LITE_RUNTIME; // Plugin: isoworldremote diff --git a/plugins/proto/ui_sidebar_mode.proto b/plugins/proto/ui_sidebar_mode.proto index b07bb1baa..af9dbabd7 100644 --- a/plugins/proto/ui_sidebar_mode.proto +++ b/plugins/proto/ui_sidebar_mode.proto @@ -1,6 +1,6 @@ +//Attempts to provide a complete framework for reading everything from a fortress needed for vizualization package proto.enums.ui_sidebar_mode; -//Attempts to provide a complete framework for reading everything from a fortress needed for vizualization option optimize_for = LITE_RUNTIME; enum ui_sidebar_mode @@ -60,4 +60,4 @@ enum ui_sidebar_mode Hauling = 52; ArenaWeather = 53; ArenaTrees = 54; -} \ No newline at end of file +} From 1e734938ae01f31acfbc8d2d235e54f432ef7fa1 Mon Sep 17 00:00:00 2001 From: lethosor Date: Sun, 19 Jul 2020 01:00:00 -0400 Subject: [PATCH 04/13] Add isoworldremote to Plugins.rst --- docs/Plugins.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/Plugins.rst b/docs/Plugins.rst index ee0aa074c..7c37ea78c 100644 --- a/docs/Plugins.rst +++ b/docs/Plugins.rst @@ -59,6 +59,11 @@ remotefortressreader An in-development plugin for realtime fortress visualisation. See :forums:`Armok Vision <146473>`. +.. _isoworldremote: + +isoworldremote +============== +A plugin that implements a `remote API ` used by Isoworld. .. _cursecheck: From d88137b1d653f2efcd7407af10e3bd4907d7b86a Mon Sep 17 00:00:00 2001 From: lethosor Date: Sun, 19 Jul 2020 01:13:43 -0400 Subject: [PATCH 05/13] Split remote API info into separate doc, add example tools and clients --- docs/Dev-intro.rst | 9 ++------- docs/Remote.rst | 47 ++++++++++++++++++++++++++++++++++++++++++++++ docs/index-dev.rst | 1 + 3 files changed, 50 insertions(+), 7 deletions(-) create mode 100644 docs/Remote.rst diff --git a/docs/Dev-intro.rst b/docs/Dev-intro.rst index 9a08533c0..2a17313fe 100644 --- a/docs/Dev-intro.rst +++ b/docs/Dev-intro.rst @@ -77,11 +77,6 @@ some functions (and some entire modules) are currently only available in C++. Remote access interface ----------------------- -DFHack supports remote access by exchanging Google protobuf messages via a TCP -socket. Both the core and plugins can define remotely accessible methods. The -``dfhack-run`` command uses this interface to invoke ordinary console commands. - -Currently the supported set of requests is limited, because the developers don't -know what exactly is most useful. `remotefortressreader` provides a fairly -comprehensive interface for visualisers such as :forums:`Armok Vision <146473>`. +DFHack provides a remote access interface that external tools can connect to and +use to interact with DF. See `remote` for more information. diff --git a/docs/Remote.rst b/docs/Remote.rst new file mode 100644 index 000000000..b7c65b032 --- /dev/null +++ b/docs/Remote.rst @@ -0,0 +1,47 @@ +.. _remote: + +======================= +DFHack Remote Interface +======================= + +DFHack provides a remote access interface that external tools can connect to and +use to interact with DF. This is implemented with `Google protobuf`_ messages +exchanged over a TCP socket (which only accepts connections from the local +machine by default). Both the core and plugins can define remotely-accessible +methods (often referred to as **RPC methods**). The RPC methods currently +available are not comprehensive, but can be extended with plugins. + +.. _Google protobuf: https://developers.google.com/protocol-buffers + +.. contents:: + :local: + +Examples +======== + +The `dfhack-run` command uses the RPC interface to invoke DFHack commands +(or Lua functions) externally. + +Plugins that implement RPC methods include: + +- `rename` +- `remotefortressreader` +- `isoworldremote` + +Plugins that use the RPC API include: + +- `stonesense` + +Third-party tools that use the RPC API include: + +- `Armok Vision `_ (:forums:`Bay12 forums thread <146473>`) + +Client libraries +================ + +Some external libraries are available for interacting with the remote interface +from other (non-C++) languages, including: + +- `RemoteClientDF-Net `_ for C# +- `dfhackrpc `_ for Go +- `dfhack-remote `_ for JavaScript diff --git a/docs/index-dev.rst b/docs/index-dev.rst index 617c0c0d7..560137609 100644 --- a/docs/index-dev.rst +++ b/docs/index-dev.rst @@ -15,5 +15,6 @@ These are pages relevant to people developing for DFHack. /docs/Documentation /docs/Structures-intro /docs/Memory-research + /docs/Remote /docs/Binpatches From b34aa86d3c3dd606c22877d9cf95d25204e5d567 Mon Sep 17 00:00:00 2001 From: lethosor Date: Tue, 21 Jul 2020 21:39:37 -0400 Subject: [PATCH 06/13] Document remote server config --- docs/Core.rst | 3 +++ docs/Remote.rst | 25 ++++++++++++++++++++++--- 2 files changed, 25 insertions(+), 3 deletions(-) diff --git a/docs/Core.rst b/docs/Core.rst index 6ef1d0575..196b12918 100644 --- a/docs/Core.rst +++ b/docs/Core.rst @@ -442,6 +442,8 @@ Other init files directory, will be run when any world or that save is loaded. +.. _env-vars: + Environment variables ===================== @@ -453,6 +455,7 @@ on UNIX-like systems:: - ``DFHACK_PORT``: the port to use for the RPC server (used by ``dfhack-run`` and `remotefortressreader` among others) instead of the default ``5000``. As with the default, if this port cannot be used, the server is not started. + See `remote` for more details. - ``DFHACK_DISABLE_CONSOLE``: if set, the DFHack console is not set up. This is the default behavior if ``PRINT_MODE:TEXT`` is set in ``data/init/init.txt``. diff --git a/docs/Remote.rst b/docs/Remote.rst index b7c65b032..5fa556ef2 100644 --- a/docs/Remote.rst +++ b/docs/Remote.rst @@ -6,9 +6,8 @@ DFHack Remote Interface DFHack provides a remote access interface that external tools can connect to and use to interact with DF. This is implemented with `Google protobuf`_ messages -exchanged over a TCP socket (which only accepts connections from the local -machine by default). Both the core and plugins can define remotely-accessible -methods (often referred to as **RPC methods**). The RPC methods currently +exchanged over a TCP socket. Both the core and plugins can define +remotely-accessible methods, or **RPC methods**. The RPC methods currently available are not comprehensive, but can be extended with plugins. .. _Google protobuf: https://developers.google.com/protocol-buffers @@ -16,6 +15,26 @@ available are not comprehensive, but can be extended with plugins. .. contents:: :local: + +Server configuration +==================== + +DFHack attempts to start a TCP server to listen for remote connections on +startup. If this fails (due to the port being in use, for example), an error +message will be logged to stderr.log. + +The server can be configured by setting options in ``dfhack-config/remote-server.json``: + +- ``allow_remote`` (default: ``false``): if true, allows connections from hosts + other than the local machine. This is insecure and may allow arbitrary code + execution on your machine, so it is disabled by default. +- ``port`` (default: ``5000``): the port that the remote server listens on. + Overriding this will allow the server to work if you have multiple instances + of DF running, or if you have something else running on port 5000. Note that + the ``DFHACK_PORT`` `environment variable ` takes precedence over + this setting and may be more useful for overriding the port temporarily. + + Examples ======== From dfc9f78fa4f2deeb576a7c334b8a75f69dfe01f4 Mon Sep 17 00:00:00 2001 From: lethosor Date: Tue, 21 Jul 2020 23:14:59 -0400 Subject: [PATCH 07/13] Refer to server config docs from dfhack-run --- docs/Core.rst | 10 +++++++++- docs/Remote.rst | 2 ++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/Core.rst b/docs/Core.rst index 196b12918..c5d5670af 100644 --- a/docs/Core.rst +++ b/docs/Core.rst @@ -82,7 +82,7 @@ dfhack-run If DF and DFHack are already running, calling ``dfhack-run my command`` in an external terminal is equivalent to calling ``my command`` in the -DFHack console. Direct use of the DFhack console is generally easier, +DFHack console. Direct use of the DFHack console is generally easier, but ``dfhack-run`` can be useful in a variety of circumstances: - if the console is unavailable @@ -101,6 +101,14 @@ Examples:: The first (\*nix) example `checks for vampires `; the second (Windows) example uses `kill-lua` to stop a Lua script. +.. note:: + + ``dfhack-run`` attempts to connect to a server on TCP port 5000. If DFHack + was unable to start this server, ``dfhack-run`` will not be able to connect. + This could happen if you have other software listening on port 5000, or if + you have multiple copies of DF running simultaneously. To assign a different + port, see `remote-server-config`. + Built-in Commands ================= diff --git a/docs/Remote.rst b/docs/Remote.rst index 5fa556ef2..24426c819 100644 --- a/docs/Remote.rst +++ b/docs/Remote.rst @@ -16,6 +16,8 @@ available are not comprehensive, but can be extended with plugins. :local: +.. _remote-server-config: + Server configuration ==================== From 4b6325eeabd6311ed30c38199f1e4111df519c19 Mon Sep 17 00:00:00 2001 From: lethosor Date: Wed, 22 Jul 2020 01:34:51 -0400 Subject: [PATCH 08/13] Add RPC protocol description to docs and reformat From https://gist.github.com/BenLubar/1f51cf570b4c8dca275db9687fa6b9e3 Ref dfhack/dfhack#1574 --- docs/Remote.rst | 166 +++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 164 insertions(+), 2 deletions(-) diff --git a/docs/Remote.rst b/docs/Remote.rst index 24426c819..689f9978b 100644 --- a/docs/Remote.rst +++ b/docs/Remote.rst @@ -37,8 +37,16 @@ The server can be configured by setting options in ``dfhack-config/remote-server this setting and may be more useful for overriding the port temporarily. +Developing with the remote API +============================== + +At a high level, the core and plugins define RPC methods, and external clients +can call these methods. Each method is assigned an ID internally, which clients +use to call it. These method IDs can be obtained using the special ``BindMethod`` +method, which has an ID of 0. + Examples -======== +-------- The `dfhack-run` command uses the RPC interface to invoke DFHack commands (or Lua functions) externally. @@ -58,7 +66,7 @@ Third-party tools that use the RPC API include: - `Armok Vision `_ (:forums:`Bay12 forums thread <146473>`) Client libraries -================ +---------------- Some external libraries are available for interacting with the remote interface from other (non-C++) languages, including: @@ -66,3 +74,157 @@ from other (non-C++) languages, including: - `RemoteClientDF-Net `_ for C# - `dfhackrpc `_ for Go - `dfhack-remote `_ for JavaScript + + +Protocol description +==================== + +This is a low-level description of the RPC protocol, which may be useful when +developing custom clients. + +Built-in messages +----------------- +These messages have hardcoded IDs; all others must be obtained through ``BindMethod``. + +=== ============ =============================== ======================= +ID Method Input Output +=== ============ =============================== ======================= + 0 BindMethod dfproto.CoreBindRequest dfproto.CoreBindReply + 1 RunCommand dfproto.CoreRunCommandRequest dfproto.EmptyMessage +=== ============ =============================== ======================= + + + +Conversation flow +----------------- + +* Client → Server: `handshake request`_ +* Server → Client: `handshake reply`_ +* Repeated 0 or more times: + * Client → Server: `request`_ + * Server → Client: `text`_ (0 or more times) + * Server → Client: `result`_ or `failure`_ +* Client → Server: `quit`_ + +Raw message types +----------------- + +* All numbers are little-endian +* All strings are ASCII +* A payload size of greater than 64MiB is an error +* See ``RemoteClient.h`` for definitions of constants starting with ``RPC`` + +handshake request +~~~~~~~~~~~~~~~~~ + +.. csv-table:: + :align: left + :header-rows: 1 + + Type, Name, Value + char[8], magic, ``DFHack?\n`` + int32_t, version, 1 + +handshake reply +~~~~~~~~~~~~~~~ + +.. csv-table:: + :align: left + :header-rows: 1 + + Type, Name, Value + char[8], magic, ``DFHack!\n`` + int32_t, version, 1 + +header +~~~~~~ + +**Note:** the two fields of this message are sometimes repurposed. Uses of this +message are represented as ``header(x, y)``, where ``x`` corresponds to the ``id`` +field and ``y`` corresponds to ``size``. + +.. csv-table:: + :align: left + :header-rows: 1 + + Type, Name + int16_t, id + int16_t, (padding - unused) + int32_t, size + +request +~~~~~~~ + +.. list-table:: + :align: left + :header-rows: 1 + :widths: 25 75 + + * - Type + - Description + * - `header`_ + - ``header(id, size)`` + * - buffer + - Protobuf-encoded payload of the input message type of the method specified by ``id``; length of ``size`` bytes + +text +~~~~ + +.. list-table:: + :align: left + :header-rows: 1 + :widths: 25 75 + + * - Type + - Description + * - `header`_ + - ``header(RPC_REPLY_TEXT, size)`` + * - buffer + - Protobuf-encoded payload of type ``dfproto.CoreTextNotification``; length of ``size`` bytes + +result +~~~~~~ + +.. list-table:: + :align: left + :header-rows: 1 + :widths: 25 75 + + * - Type + - Description + * - `header`_ + - ``header(RPC_REPLY_RESULT, size)`` + * - buffer + - Protobuf-encoded payload of the output message type of the oldest incomplete method call; when received, + that method call is considered completed. Length of ``size`` bytes. + +failure +~~~~~~~ + +.. list-table:: + :align: left + :header-rows: 1 + :widths: 25 75 + + * - Type + - Description + * - `header`_ + - ``header(RPC_REPLY_FAIL, command_result)`` + * - command_result + - return code of the command (a constant starting with ``CR_``; see ``RemoteClient.h``) + +quit +~~~~ + +**Note:** the server closes the connection after sending this message. + +.. list-table:: + :align: left + :header-rows: 1 + :widths: 25 75 + :width: 100% + + * - Type + - Description + * - `header`_ + - ``header(RPC_REQUEST_QUIT, 0)`` From eb074b8a493fb971a98df156b95926dca1146dec Mon Sep 17 00:00:00 2001 From: lethosor Date: Wed, 22 Jul 2020 02:04:02 -0400 Subject: [PATCH 09/13] Link to Wireshark script --- docs/Remote.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/Remote.rst b/docs/Remote.rst index 689f9978b..a40161fee 100644 --- a/docs/Remote.rst +++ b/docs/Remote.rst @@ -82,6 +82,10 @@ Protocol description This is a low-level description of the RPC protocol, which may be useful when developing custom clients. +A WireShark dissector for this protocol is available in the +`df_misc repo `_. + + Built-in messages ----------------- These messages have hardcoded IDs; all others must be obtained through ``BindMethod``. @@ -216,7 +220,7 @@ failure quit ~~~~ -**Note:** the server closes the connection after sending this message. +**Note:** the server closes the connection after receiving this message. .. list-table:: :align: left From 655b3632386f17f34ccaf5ddb223114e9184e623 Mon Sep 17 00:00:00 2001 From: lethosor Date: Wed, 22 Jul 2020 02:23:29 -0400 Subject: [PATCH 10/13] Fix some spelling errors --- docs/Compile.rst | 2 +- docs/Core.rst | 12 ++++++------ docs/Documentation.rst | 6 +++--- docs/Lua API.rst | 8 ++++---- docs/Plugins.rst | 8 ++++---- plugins/stonesense | 2 +- scripts | 2 +- 7 files changed, 20 insertions(+), 20 deletions(-) diff --git a/docs/Compile.rst b/docs/Compile.rst index 68187e7fa..c98cb939a 100644 --- a/docs/Compile.rst +++ b/docs/Compile.rst @@ -112,7 +112,7 @@ in the platform-specific sections below first, then come back here. Generator --------- -The ``Ninja`` CMake build generator is the prefered build method on Linux and +The ``Ninja`` CMake build generator is the preferred build method on Linux and macOS, instead of ``Unix Makefiles``, which is the default. You can select Ninja by passing ``-G Ninja`` to CMake. Incremental builds using Unix Makefiles can be much slower than Ninja builds. Note that you will probably need to install diff --git a/docs/Core.rst b/docs/Core.rst index c5d5670af..5ef3e6a4d 100644 --- a/docs/Core.rst +++ b/docs/Core.rst @@ -15,12 +15,12 @@ DFHack commands can be implemented in three ways, all of which are used in the same way: :builtin: commands are implemented by the core of DFHack. They manage - other DFhack tools, interpret commands, and control basic + other DFHack tools, interpret commands, and control basic aspects of DF (force pause or quit). :plugins: are stored in ``hack/plugins/`` and must be compiled with the same version of DFHack. They are less flexible than scripts, - but used for complex or ongoing tasks becasue they run faster. + but used for complex or ongoing tasks because they run faster. :scripts: are Ruby or Lua scripts stored in ``hack/scripts/``. Because they don't need to be compiled, scripts are @@ -88,7 +88,7 @@ but ``dfhack-run`` can be useful in a variety of circumstances: - if the console is unavailable - with the init setting ``PRINT_MODE:TEXT`` - - while running an interactive command (eg. `liquids` or `tiletypes`) + - while running an interactive command (e.g. `liquids` or `tiletypes`) - from external programs or scripts - if DF or DFHack are not responding @@ -113,7 +113,7 @@ second (Windows) example uses `kill-lua` to stop a Lua script. Built-in Commands ================= The following commands are provided by the 'core' components -of DFhack, rather than plugins or scripts. +of DFHack, rather than plugins or scripts. .. contents:: :local: @@ -419,7 +419,7 @@ onLoad*.init When a world is loaded, DFHack looks for files of the form ``onLoad*.init``, where ``*`` can be any string, including the empty string. -All matching init files will be executed in alphebetical order. +All matching init files will be executed in alphabetical order. A world being loaded can mean a fortress, an adventurer, or legends mode. These files are best used for non-persistent commands, such as setting @@ -444,7 +444,7 @@ Other init files * ``onMapLoad*.init`` and ``onMapUnload*.init`` are run when a map, distinct from a world, is loaded. This is good for map-affecting - commands (eg `clean`), or avoiding issues in Legends mode. + commands (e.g. `clean`), or avoiding issues in Legends mode. * Any lua script named ``raw/init.d/*.lua``, in the save or main DF directory, will be run when any world or that save is loaded. diff --git a/docs/Documentation.rst b/docs/Documentation.rst index 37fcc6ee6..067f6bf90 100644 --- a/docs/Documentation.rst +++ b/docs/Documentation.rst @@ -53,7 +53,7 @@ directive - anything between the tokens is copied into the appropriate scripts documentation page. For Ruby, we follow the built-in docstring convention (``=begin`` and ``=end``). For Lua, the tokens are ``[====[`` and ``]====]`` - ordinary multi-line strings. It is highly encouraged to reuse this string -as the in-console documentation by (eg.) printing it when a ``-help`` argument +as the in-console documentation by (e.g.) printing it when a ``-help`` argument is given. The docs **must** have a heading which exactly matches the command, underlined @@ -101,7 +101,7 @@ describe the effect:: If it would be helpful to mention another DFHack command, don't just type the name - add a hyperlink! Specify the link target in backticks, and it will be -replaced with the corresponding title and linked: eg ```autolabor``` +replaced with the corresponding title and linked: e.g. ```autolabor``` => `autolabor`. Link targets should be equivalent to the command described (without file extension), and placed above the heading of that section like this:: @@ -159,7 +159,7 @@ Once pip is available, you can then install Sphinx with:: pip3 install sphinx -If you run this as an unpriviliged user, it may install a local copy of Sphinx +If you run this as an unprivileged user, it may install a local copy of Sphinx for your user only. The ``sphinx-build`` executable will typically end up in ``~/.local/bin/`` in this case. Alternatively, you can install Sphinx system-wide by running pip with ``sudo``. In any case, you will need the folder diff --git a/docs/Lua API.rst b/docs/Lua API.rst index c62d0f379..e9392f9c4 100644 --- a/docs/Lua API.rst +++ b/docs/Lua API.rst @@ -42,11 +42,11 @@ to lua code as a tree of objects and functions under the ``df`` global, which also broadly maps to the ``df`` namespace in the headers generated for C++. .. warning:: - The wrapper provides almost raw access to the memory - of the game, so mistakes in manipulating objects are as likely to - crash the game as equivalent plain C++ code would be. - eg. NULL pointer access is safely detected, but dangling pointers aren't. + The wrapper provides almost raw access to the memory of the game, so + mistakes in manipulating objects are as likely to crash the game as + equivalent plain C++ code would be - e.g. null pointer access is safely + detected, but dangling pointers aren't. Objects managed by the wrapper can be broadly classified into the following groups: diff --git a/docs/Plugins.rst b/docs/Plugins.rst index 7c37ea78c..3a4133873 100644 --- a/docs/Plugins.rst +++ b/docs/Plugins.rst @@ -191,7 +191,7 @@ Usage and related commands: :revflood: Hide everything, then reveal tiles with a path to the cursor (useful to make walled-off rooms vanish) :revforget: Discard info about what was visible before revealing the map. - Only useful where (eg) you abandoned with the fort revealed + Only useful where (e.g.) you abandoned with the fort revealed and no longer want the data. .. _showmood: @@ -737,7 +737,7 @@ by moving the view manually. mousequery ========== -Adds mouse controls to the DF interface, eg click-and-drag designations. +Adds mouse controls to the DF interface, e.g. click-and-drag designations. Options: @@ -840,10 +840,10 @@ See `gui/stockpiles` for an in-game interface. :savestock: Saves the currently highlighted stockpile's settings to a file in your Dwarf Fortress folder. This file can be used to copy settings between game saves or - players. eg: ``savestock food_settings.dfstock`` + players. e.g.: ``savestock food_settings.dfstock`` :loadstock: Loads a saved stockpile settings file and applies it to the currently selected - stockpile. eg: ``loadstock food_settings.dfstock`` + stockpile. e.g.: ``loadstock food_settings.dfstock`` To use savestock and loadstock, use the :kbd:`q` command to highlight a stockpile. Then run savestock giving it a descriptive filename. Then, in a different (or diff --git a/plugins/stonesense b/plugins/stonesense index bdab71c99..87e6cf02b 160000 --- a/plugins/stonesense +++ b/plugins/stonesense @@ -1 +1 @@ -Subproject commit bdab71c99a0a7cc268ca517a0cd3f0a5fb41042a +Subproject commit 87e6cf02b914daf6f7811d967db6fadc7ca115fe diff --git a/scripts b/scripts index af749e708..a5e385bb2 160000 --- a/scripts +++ b/scripts @@ -1 +1 @@ -Subproject commit af749e7086739a058cd5095a6ee1a60d7e795a7c +Subproject commit a5e385bb2a3c72f878be614cb0cc004aafa17f79 From 8541db17cd660bb3a2e6745151be33d5338ac32d Mon Sep 17 00:00:00 2001 From: lethosor Date: Wed, 22 Jul 2020 02:30:41 -0400 Subject: [PATCH 11/13] Remove :width: table directive (requires docutils 0.15) --- docs/Remote.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/Remote.rst b/docs/Remote.rst index a40161fee..a6f07ebc0 100644 --- a/docs/Remote.rst +++ b/docs/Remote.rst @@ -226,7 +226,6 @@ quit :align: left :header-rows: 1 :widths: 25 75 - :width: 100% * - Type - Description From e17d49220359ca0bd1dfe757532cbb0fb161afb6 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Thu, 23 Jul 2020 21:55:05 -0700 Subject: [PATCH 12/13] address PR review comments on api docs --- docs/changelog.txt | 2 +- library/include/modules/Filesystem.h | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/changelog.txt b/docs/changelog.txt index 5f3510ed9..58669e321 100644 --- a/docs/changelog.txt +++ b/docs/changelog.txt @@ -51,7 +51,7 @@ changelog.txt uses a syntax similar to RST, with a few special sequences: - `confirm`: added a confirmation dialog for convicting dwarves of crimes ## API -- Added ``Filesystem::mkdir_recursive`` (returns true if directories to create already exist) +- Added ``Filesystem::mkdir_recursive`` - Extended ``Filesystem::listdir_recursive`` to optionally make returned filenames relative to the start directory ## Lua diff --git a/library/include/modules/Filesystem.h b/library/include/modules/Filesystem.h index 3252be007..c9218ba33 100644 --- a/library/include/modules/Filesystem.h +++ b/library/include/modules/Filesystem.h @@ -149,6 +149,7 @@ namespace DFHack { DFHACK_EXPORT bool chdir (std::string path); DFHACK_EXPORT std::string getcwd (); DFHACK_EXPORT bool mkdir (std::string path); + // returns true on success or if directory already exists DFHACK_EXPORT bool mkdir_recursive (std::string path); DFHACK_EXPORT bool rmdir (std::string path); DFHACK_EXPORT bool stat (std::string path, STAT_STRUCT &info); @@ -160,6 +161,8 @@ namespace DFHack { DFHACK_EXPORT int64_t ctime (std::string path); DFHACK_EXPORT int64_t mtime (std::string path); DFHACK_EXPORT int listdir (std::string dir, std::vector &files); + // set include_prefix to false to prevent dir from being prepended to + // paths returned in files DFHACK_EXPORT int listdir_recursive (std::string dir, std::map &files, int depth = 10, bool include_prefix = true); } From e61f2b74516c208fed05ee3e420e231f15f282a3 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Thu, 23 Jul 2020 22:44:34 -0700 Subject: [PATCH 13/13] update Lua API.rst --- docs/Lua API.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/Lua API.rst b/docs/Lua API.rst index c62d0f379..569dc4c49 100644 --- a/docs/Lua API.rst +++ b/docs/Lua API.rst @@ -2116,8 +2116,10 @@ unless otherwise noted. * ``dfhack.filesystem.listdir(path)`` Lists files/directories in a directory. Returns ``{}`` if ``path`` does not exist. + Set include_prefix to false if you don't want the ``path`` string prepended to the + returned filenames. -* ``dfhack.filesystem.listdir_recursive(path [, depth = 10])`` +* ``dfhack.filesystem.listdir_recursive(path [, depth = 10[, include_prefix = true]])`` Lists all files/directories in a directory and its subdirectories. All directories are listed before their contents. Returns a table with subtables of the format::