diff --git a/COMPILE.rst b/COMPILE.rst index beb738363..8fca0e1f5 100644 --- a/COMPILE.rst +++ b/COMPILE.rst @@ -152,10 +152,13 @@ Valid and useful build types include 'Release', 'Debug' and ================================ Using the library as a developer ================================ -Currently, the only way to use the library is to write a plugin that can be loaded by it. + +Currently, the most direct way to use the library is to write a plugin that can be loaded by it. All the plugins can be found in the 'plugins' folder. There's no in-depth documentation on how to write one yet, but it should be easy enough to copy one and just follow the pattern. +Other than through plugins, it is possible to use DFHack via remote access interface, or by writing Lua scripts. + The most important parts of DFHack are the Core, Console, Modules and Plugins. * Core acts as the centerpiece of DFHack - it acts as a filter between DF and SDL and synchronizes the various plugins with DF. @@ -171,6 +174,24 @@ The main license is zlib/libpng, some bits are MIT licensed, and some are BSD li Feel free to add your own extensions and plugins. Contributing back to the dfhack repository is welcome and the right thing to do :) +DF data structure definitions +============================= + +DFHack uses information about the game data structures, represented via xml files in the library/xml/ submodule. + +Data structure layouts are described in files following the df.*.xml name pattern. This information is transformed by a perl script into C++ headers describing the structures, and associated metadata for the Lua wrapper. These headers and data are then compiled into the DFHack libraries, thus necessitating a compatibility break every time layouts change; in return it significantly boosts the efficiency and capabilities of DFHack code. + +Global object addresses are stored in symbols.xml, which is copied to the dfhack release package and loaded as data at runtime. + +Remote access interface +======================= + +DFHack supports remote access by exchanging Google protobuf messages via a TCP socket. Both the core and plugins can define remotely accessible methods. The ``dfhack-run`` command uses this interface to invoke ordinary console commands. + +Currently the supported set of requests is limited, because the developers don't know what exactly is most useful. + +Protocol client implementations exist for Java and C#. + Contributing to DFHack ====================== diff --git a/Compile.html b/Compile.html index b0f9e9c6e..e17e57e22 100644 --- a/Compile.html +++ b/Compile.html @@ -334,10 +334,12 @@ ul.auto-toc {
Currently, the only way to use the library is to write a plugin that can be loaded by it. +
Currently, the most direct way to use the library is to write a plugin that can be loaded by it. All the plugins can be found in the 'plugins' folder. There's no in-depth documentation on how to write one yet, but it should be easy enough to copy one and just follow the pattern.
+Other than through plugins, it is possible to use DFHack via remote access interface, or by writing Lua scripts.
The most important parts of DFHack are the Core, Console, Modules and Plugins.
Feel free to add your own extensions and plugins. Contributing back to the dfhack repository is welcome and the right thing to do :)
+DFHack uses information about the game data structures, represented via xml files in the library/xml/ submodule.
+Data structure layouts are described in files following the df.*.xml name pattern. This information is transformed by a perl script into C++ headers describing the structures, and associated metadata for the Lua wrapper. These headers and data are then compiled into the DFHack libraries, thus necessitating a compatibility break every time layouts change; in return it significantly boosts the efficiency and capabilities of DFHack code.
+Global object addresses are stored in symbols.xml, which is copied to the dfhack release package and loaded as data at runtime.
+DFHack supports remote access by exchanging Google protobuf messages via a TCP socket. Both the core and plugins can define remotely accessible methods. The dfhack-run command uses this interface to invoke ordinary console commands.
+Currently the supported set of requests is limited, because the developers don't know what exactly is most useful.
+Protocol client implementations exist for Java and C#.
+Several things should be kept in mind when contributing to DFHack.
DFhack uses ANSI formatting and four spaces as indentation. Line endings are UNIX. The files use UTF-8 encoding. Code not following this won't make me happy, because I'll have to fix it. There's a good chance I'll make you fix it ;)
You can send patches or make a clone of the github repo and ask me on the IRC channel to pull your code in. I'll review it and see if there are any problems. I'll fix them if they are minor.
@@ -506,7 +521,7 @@ this is also a good place to dump new ideas and/or bugs that need fixing.If you want to do memory research, you'll need some tools and some knowledge. In general, you'll need a good memory viewer and optionally something to look at machine code without getting crazy :)
diff --git a/LUA_API.rst b/LUA_API.rst index f40b786d2..5fc653bb3 100644 --- a/LUA_API.rst +++ b/LUA_API.rst @@ -4,9 +4,26 @@ DFHack Lua API .. contents:: -==================== -DF structure wrapper -==================== +The current version of DFHack has extensive support for +the Lua scripting language, providing access to: + +1. Raw data structures used by the game. +2. Many C++ functions for high-level access to these + structures, and interaction with dfhack itself. +3. Some functions exported by C++ plugins. + +Lua code can be used both for writing scripts, which +are treated by DFHack command line prompt almost as +native C++ commands, and invoked by plugins written in c++. + +This document describes native API available to Lua in detail. +For the most part it does not describe utility functions +implemented by Lua files located in hack/lua/... + + +========================= +DF data structure wrapper +========================= DF structures described by the xml files in library/xml are exported to lua code as a tree of objects and functions under the ``df`` global, @@ -426,13 +443,17 @@ not destroy any objects allocated in this way, so the user should be prepared to catch the error and do the necessary cleanup. -================ -DFHack utilities -================ +========== +DFHack API +========== DFHack utility functions are placed in the ``dfhack`` global tree. -Currently it defines the following features: +Native utilities +================ + +Input & Output +-------------- * ``dfhack.print(args...)`` @@ -451,6 +472,7 @@ Currently it defines the following features: * ``dfhack.color([color])`` Sets the current output color. If color is *nil* or *-1*, resets to default. + Returns the previous color value. * ``dfhack.is_interactive()`` @@ -473,15 +495,27 @@ Currently it defines the following features: If the interactive console is not accessible, returns *nil, error*. + +Exception handling +------------------ + +* ``dfhack.error(msg[,level[,verbose]])`` + + Throws a dfhack exception object with location and stack trace. + The verbose parameter controls whether the trace is printed by default. + +* ``qerror(msg[,level])`` + + Calls ``dfhack.error()`` with ``verbose`` being *false*. Intended to + be used for user-caused errors in scripts, where stack traces are not + desirable. + * ``dfhack.pcall(f[,args...])`` Invokes f via xpcall, using an error function that attaches a stack trace to the error. The same function is used by SafeCall in C++, and dfhack.safecall. - The returned error is a table with separate ``message`` and - ``stacktrace`` string fields; it implements ``__tostring``. - * ``safecall(f[,args...])``, ``dfhack.safecall(f[,args...])`` Just like pcall, but also prints the error using printerr before @@ -491,6 +525,34 @@ Currently it defines the following features: Compares to coroutine.resume like dfhack.safecall vs pcall. +* ``dfhack.exception`` + + Metatable of error objects used by dfhack. The objects have the + following properties: + + ``err.where`` + The location prefix string, or *nil*. + ``err.message`` + The base message string. + ``err.stacktrace`` + The stack trace string, or *nil*. + ``err.cause`` + A different exception object, or *nil*. + ``err.thread`` + The coroutine that has thrown the exception. + ``err.verbose`` + Boolean, or *nil*; specifies if where and stacktrace should be printed. + ``tostring(err)``, or ``err:tostring([verbose])`` + Converts the exception to string. + +* ``dfhack.exception.verbose`` + + The default value of the ``verbose`` argument of ``err:tostring()``. + + +Locking and finalization +------------------------ + * ``dfhack.with_suspend(f[,args...])`` Calls ``f`` with arguments after grabbing the DF core suspend lock. @@ -529,7 +591,7 @@ Currently it defines the following features: Persistent configuration storage -================================ +-------------------------------- This api is intended for storing configuration options in the world itself. It probably should be restricted to data that is world-dependent. @@ -571,7 +633,7 @@ functions can just copy values in memory without doing any actual I/O. However, currently every entry has a 180+-byte dead-weight overhead. Material info lookup -==================== +-------------------- A material info record has fields: @@ -1148,6 +1210,11 @@ Internal API These functions are intended for the use by dfhack developers, and are only documented here for completeness: +* ``dfhack.internal.scripts`` + + The table used by ``dfhack.run_script()`` to give every script its own + global environment, persistent between calls to the script. + * ``dfhack.internal.getAddress(name)`` Returns the global address ``name``, or *nil*. @@ -1156,14 +1223,38 @@ and are only documented here for completeness: Sets the global address ``name``. Returns the value of ``getAddress`` before the change. -* ``dfhack.internal.getBase()`` +* ``dfhack.internal.getVTable(name)`` + + Returns the pre-extracted vtable address ``name``, or *nil*. - Returns the base address of the process. +* ``dfhack.internal.getRebaseDelta()`` + + Returns the ASLR rebase offset of the DF executable. * ``dfhack.internal.getMemRanges()`` Returns a sequence of tables describing virtual memory ranges of the process. +* ``dfhack.internal.memmove(dest,src,count)`` + + Wraps the standard memmove function. Accepts both numbers and refs as pointers. + +* ``dfhack.internal.memcmp(ptr1,ptr2,count)`` + + Wraps the standard memcmp function. + +* ``dfhack.internal.memscan(haystack,count,step,needle,nsize)`` + + Searches for ``needle`` of ``nsize`` bytes in ``haystack``, + using ``count`` steps of ``step`` bytes. + Returns: *step_idx, sum_idx, found_ptr*, or *nil* if not found. + +* ``dfhack.internal.diffscan(old_data, new_data, start_idx, end_idx, eltsize[, oldval, newval, delta])`` + + Searches for differences between buffers at ptr1 and ptr2, as integers of size eltsize. + The oldval, newval or delta arguments may be used to specify additional constraints. + Returns: *found_index*, or *nil* if end reached. + Core interpreter context ======================== @@ -1231,6 +1322,42 @@ Features: Invokes all listeners contained in the event in an arbitrary order using ``dfhack.safecall``. + +======= +Modules +======= + +DFHack sets up the lua interpreter so that the built-in ``require`` +function can be used to load shared lua code from hack/lua/. +The ``dfhack`` namespace reference itself may be obtained via +``require('dfhack')``, although it is initially created as a +global by C++ bootstrap code. + +The following functions are provided: + +* ``mkmodule(name)`` + + Creates an environment table for the module. Intended to be used as:: + + local _ENV = mkmodule('foo') + ... + return _ENV + + If called the second time, returns the same table; thus providing reload support. + +* ``reload(name)`` + + Reloads a previously ``require``-d module *"name"* from the file. + Intended as a help for module development. + +* ``dfhack.BASE_G`` + + This variable contains the root global environment table, which is + used as a base for all module and script environments. Its contents + should be kept limited to the standard Lua library and API described + in this document. + + ======= Plugins ======= @@ -1292,3 +1419,40 @@ sort Does not export any native functions as of now. Instead, it calls lua code to perform the actual ordering of list items. + + +======= +Scripts +======= + +Any files with the .lua extension placed into hack/scripts/* +are automatically used by the DFHack core as commands. The +matching command name consists of the name of the file sans +the extension. + +**NOTE:** Scripts placed in subdirectories still can be accessed, but +do not clutter the ``ls`` command list; thus it is preferred +for obscure developer-oriented scripts and scripts used by tools. +When calling such scripts, always use '/' as the separator for +directories, e.g. ``devel/lua-example``. + +Scripts are re-read from disk every time they are used +(this may be changed later to check the file change time); however +the global variable values persist in memory between calls. +Every script gets its own separate environment for global +variables. + +Arguments are passed in to the scripts via the **...** built-in +quasi-variable; when the script is called by the DFHack core, +they are all guaranteed to be non-nil strings. + +DFHack core invokes the scripts in the *core context* (see above); +however it is possible to call them from any lua code (including +from other scripts) in any context, via the same function the core uses: + +* ``dfhack.run_script(name[,args...])`` + + Run a lua script in hack/scripts/, as if it was started from dfhack command-line. + The ``name`` argument should be the name stem, as would be used on the command line. + +Note that this function lets errors propagate to the caller. diff --git a/Lua API.html b/Lua API.html index 5574c55b8..04e899366 100644 --- a/Lua API.html +++ b/Lua API.html @@ -320,7 +320,7 @@ ul.auto-toc {Contents
The current version of DFHack has extensive support for +the Lua scripting language, providing access to:
+Lua code can be used both for writing scripts, which +are treated by DFHack command line prompt almost as +native C++ commands, and invoked by plugins written in c++.
+This document describes native API available to Lua in detail. +For the most part it does not describe utility functions +implemented by Lua files located in hack/lua/...
+DF structures described by the xml files in library/xml are exported to lua code as a tree of objects and functions under the df global, which broadly maps to the df namespace in C++.
@@ -717,10 +739,13 @@ should be prepared to catch the error and do the necessary cleanup.DFHack utility functions are placed in the dfhack global tree.
-Currently it defines the following features:
+dfhack.print(args...)
Output tab-separated args as standard lua print would do, @@ -734,7 +759,8 @@ works with DFHack output infrastructure.
Same as println; intended for errors. Uses red color and logs to stderr.log.
dfhack.color([color])
-Sets the current output color. If color is nil or -1, resets to default.
+Sets the current output color. If color is nil or -1, resets to default. +Returns the previous color value.
dfhack.is_interactive()
Checks if the thread can access the interactive console and returns true or false.
@@ -752,12 +778,24 @@ this, forcing the function to block on input with lock held. string, global environment and command-line history file.If the interactive console is not accessible, returns nil, error.
dfhack.error(msg[,level[,verbose]])
+Throws a dfhack exception object with location and stack trace. +The verbose parameter controls whether the trace is printed by default.
+qerror(msg[,level])
+Calls dfhack.error() with verbose being false. Intended to +be used for user-caused errors in scripts, where stack traces are not +desirable.
+dfhack.pcall(f[,args...])
Invokes f via xpcall, using an error function that attaches a stack trace to the error. The same function is used by SafeCall in C++, and dfhack.safecall.
-The returned error is a table with separate message and -stacktrace string fields; it implements __tostring.
safecall(f[,args...]), dfhack.safecall(f[,args...])
Just like pcall, but also prints the error using printerr before @@ -766,6 +804,41 @@ returning. Intended as a convenience function.
dfhack.saferesume(coroutine[,args...])
Compares to coroutine.resume like dfhack.safecall vs pcall.
dfhack.exception
+Metatable of error objects used by dfhack. The objects have the +following properties:
+The location prefix string, or nil.
+The base message string.
+The stack trace string, or nil.
+A different exception object, or nil.
+The coroutine that has thrown the exception.
+Boolean, or nil; specifies if where and stacktrace should be printed.
+Converts the exception to string.
+dfhack.exception.verbose
+The default value of the verbose argument of err:tostring().
+dfhack.with_suspend(f[,args...])
Calls f with arguments after grabbing the DF core suspend lock. Suspending is necessary for accessing a consistent state of DF memory.
@@ -795,8 +868,9 @@ Implemented using call_with_finalCalls fn(obj,args...), then finalizes with obj:delete().
This api is intended for storing configuration options in the world itself. It probably should be restricted to data that is world-dependent.
Entries are identified by a string key, but it is also possible to manage @@ -831,7 +905,7 @@ functions can just copy values in memory without doing any actual I/O. However, currently every entry has a 180+-byte dead-weight overhead.
A material info record has fields:
type, index, material
@@ -874,8 +948,9 @@ Accept dfhack_material_category auto-assign table.Thin wrappers around C++ functions, similar to the ones for virtual methods. One notable difference is that these explicit wrappers allow argument count adjustment according to the usual lua rules, so trailing false/nil arguments @@ -904,7 +979,7 @@ can be omitted.
dfhack.gui.getCurViewscreen()
Returns the viewscreen that is current in the core.
@@ -940,7 +1015,7 @@ The is_bright boolean actually seems to invert the brightness.dfhack.job.cloneJobStruct(job)
Creates a deep copy of the given job.
@@ -977,7 +1052,7 @@ a lua list containing them.dfhack.units.getPosition(unit)
Returns true x,y,z of the unit, or nil if invalid; may be not equal to unit.pos if caged.
@@ -1031,7 +1106,7 @@ or raws. The ignore_noble boolean disables thedfhack.items.getPosition(item)
Returns true x,y,z of the item, or nil if invalid; may be not equal to item.pos if in inventory.
@@ -1074,7 +1149,7 @@ Returns false in case of error.dfhack.maps.getSize()
Returns map size in blocks: x, y, z
@@ -1115,7 +1190,7 @@ burrows, or the presence of invaders.dfhack.burrows.findByName(name)
Returns the burrow pointer or nil.
@@ -1150,7 +1225,7 @@ burrows, or the presence of invaders.dfhack.buildings.getSize(building)
Returns width, height, centerx, centery.
@@ -1290,7 +1365,7 @@ can be determined this way, constructBuildingdfhack.constructions.designateNew(pos,type,item_type,mat_index)
Designates a new construction at given position. If there already is @@ -1306,27 +1381,50 @@ Returns true, was_only_planned if removed; or false if none fo
These functions are intended for the use by dfhack developers, and are only documented here for completeness:
dfhack.internal.scripts
+The table used by dfhack.run_script() to give every script its own +global environment, persistent between calls to the script.
+dfhack.internal.getAddress(name)
Returns the global address name, or nil.
dfhack.internal.setAddress(name, value)
Sets the global address name. Returns the value of getAddress before the change.
dfhack.internal.getBase()
-Returns the base address of the process.
+dfhack.internal.getVTable(name)
+Returns the pre-extracted vtable address name, or nil.
+dfhack.internal.getRebaseDelta()
+Returns the ASLR rebase offset of the DF executable.
dfhack.internal.getMemRanges()
Returns a sequence of tables describing virtual memory ranges of the process.
dfhack.internal.memmove(dest,src,count)
+Wraps the standard memmove function. Accepts both numbers and refs as pointers.
+dfhack.internal.memcmp(ptr1,ptr2,count)
+Wraps the standard memcmp function.
+dfhack.internal.memscan(haystack,count,step,needle,nsize)
+Searches for needle of nsize bytes in haystack, +using count steps of step bytes. +Returns: step_idx, sum_idx, found_ptr, or nil if not found.
+dfhack.internal.diffscan(old_data, new_data, start_idx, end_idx, eltsize[, oldval, newval, delta])
+Searches for differences between buffers at ptr1 and ptr2, as integers of size eltsize. +The oldval, newval or delta arguments may be used to specify additional constraints. +Returns: found_index, or nil if end reached.
+While plugins can create any number of interpreter instances, there is one special context managed by dfhack core. It is the only context that can receive events from DF and plugins.
@@ -1357,7 +1455,7 @@ Using timeout_active(id,nil) cancels the timerAn event is just a lua table with a predefined metatable that contains a __call metamethod. When it is invoked, it loops through the table with next and calls all contained values. @@ -1382,15 +1480,45 @@ order using dfhack.safecall.
DFHack sets up the lua interpreter so that the built-in require +function can be used to load shared lua code from hack/lua/. +The dfhack namespace reference itself may be obtained via +require('dfhack'), although it is initially created as a +global by C++ bootstrap code.
+The following functions are provided:
+mkmodule(name)
+Creates an environment table for the module. Intended to be used as:
++local _ENV = mkmodule('foo') +... +return _ENV ++
If called the second time, returns the same table; thus providing reload support.
+reload(name)
+Reloads a previously require-d module "name" from the file. +Intended as a help for module development.
+dfhack.BASE_G
+This variable contains the root global environment table, which is +used as a base for all module and script environments. Its contents +should be kept limited to the standard Lua library and API described +in this document.
+DFHack plugins may export native functions and events to lua contexts. They are automatically imported by mkmodule('plugins.<name>'); this means that a lua module file is still necessary for require to read.
The following plugins have lua support.
Any files with the .lua extension placed into hack/scripts/* +are automatically used by the DFHack core as commands. The +matching command name consists of the name of the file sans +the extension.
+NOTE: Scripts placed in subdirectories still can be accessed, but +do not clutter the ls command list; thus it is preferred +for obscure developer-oriented scripts and scripts used by tools. +When calling such scripts, always use '/' as the separator for +directories, e.g. devel/lua-example.
+Scripts are re-read from disk every time they are used +(this may be changed later to check the file change time); however +the global variable values persist in memory between calls. +Every script gets its own separate environment for global +variables.
+Arguments are passed in to the scripts via the ... built-in +quasi-variable; when the script is called by the DFHack core, +they are all guaranteed to be non-nil strings.
+DFHack core invokes the scripts in the core context (see above); +however it is possible to call them from any lua code (including +from other scripts) in any context, via the same function the core uses:
+dfhack.run_script(name[,args...])
+Run a lua script in hack/scripts/, as if it was started from dfhack command-line. +The name argument should be the name stem, as would be used on the command line.
+Note that this function lets errors propagate to the caller.
+