From 2302dfe3d62cc37d6ce7c815ad818ba5ca162ff8 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Thu, 6 Aug 2020 23:32:31 -0700 Subject: [PATCH 01/61] add quickfort user guide (first draft) Much of the text was reused (with permission) from Joel Thornton's Python Quickfort 2.0 user guide. --- data/blueprints/README.md | 458 +++++++++++++++++++++++++++++++++++++- 1 file changed, 446 insertions(+), 12 deletions(-) diff --git a/data/blueprints/README.md b/data/blueprints/README.md index d81f0e3b4..6f11df32a 100644 --- a/data/blueprints/README.md +++ b/data/blueprints/README.md @@ -1,15 +1,449 @@ -This directory is for quickfort blueprints. You can apply them to your fortress -map with the DFHack -[quickfort script](https://docs.dfhack.org/en/stable/docs/_auto/base.html#quickfort). -You can create blueprints by hand or by using any spreadsheet application, -saving them as .xlsx or .csv files. You can also build your plan "for real" in -Dwarf Fortress, and then export your map using the DFHack -[blueprint plugin](https://docs.dfhack.org/en/stable/docs/Plugins.html#blueprint) -for later replay in a different fort. +[//]: # (The online version of this manual, which may be easier to read, is) +[//]: # (at https://github.com/DFHack/dfhack/tree/develop/data/blueprints) -DFHack blueprints are compatible with the original Quickfort 2.0 syntax. If you -have existing blueprints that worked with https://github.com/joelpt/quickfort, -then they should work just fine in DFHack quickfort. +[//]: # (Note to editors: don't word wrap -- it breaks the formatting on GitHub) -There are many ready-to-use examples of blueprints in the library subfolder. +DFHack Quickfort User Manual +============================ + +DFHack Quickfort is a DFHack script that helps you build fortresses from "blueprint" .csv and .xlsx files. Many applications exist to edit these files, such as MS Excel and [Google Sheets](https://sheets.new). You can also build your plan "for real" in Dwarf Fortress, and then export your map using DFHack's [blueprint](https://docs.dfhack.org/en/stable/docs/Plugins.html#blueprint) plugin. Most layout and building-oriented DF commands are supported through the use of multiple files or spreadsheets, each describing a different phase of DF construction: designation, building, placing stockpiles, and setting configuration. + +The original idea and 1.0 codebase came from [Valdemar's](https://dwarffortresswiki.org/index.php/User:Valdemar) auto-designation macro. Joel Thornton (joelpt) reimplemented the core logic in Python and extended its functionality with [Quickfort 2.0](https://github.com/joelpt/quickfort). This DFHack-native implementation, called "DFHack Quickfort" or just "quickfort", builds upon Quickfort 2.0's formats and features. DFHack Quickfort is written in Lua and interacts with Dwarf Fortress memory structures directly, allowing for instantaneous blueprint application, error checking and recovery, and many other advanced features. + +This document focuses on DFHack Quickfort's capabilities and teaches players how to understand and build blueprint files. Much of the text was originally written by Joel Thornton, reused here with his permission. + +For those just looking to apply blueprints, check out the [quickfort command syntax](https://docs.dfhack.org/en/stable/docs/_auto/base.html#quickfort) in the DFHack Scripts documentation. There are many ready-to-use blueprints available in the `blueprints/library` subfolder in your DFHack installation. Browse them on your computer or [online](https://github.com/DFHack/dfhack/tree/develop/data/blueprints/library), or run `quickfort list -l` at the `DFHack#` prompt to list them, and then `quickfort run` to apply them to your fort! + +See the [Links section](#links) for more information and online resources. + + +Table of Contents +----------------- + +* [Features](#features) +* [Editing Blueprints](#editing-blueprints) + * [Area expansion syntax](#area-expansion-syntax) + * [Automatic area expansion](#automatic-area-expansion) + * [Specifying a starting position](#specifying-a-starting-position) + * [Multilevel blueprints](#multilevel-blueprints) + * [Minecart tracks](#minecart-tracks) +* [Packaging a set of blueprints](#packaging-a-set-of-blueprints) +* [Troubleshooting](#troubleshooting) +* [Tips and tricks](#tips-and-tricks) +* [Links](#links) + + +Features +-------- + +* General + * Manages complete blueprints to handle the four main phases of DF construction + * Supports .csv and multi-worksheet .xlsx blueprint files + * Near-instant application, even for very large and complex blueprints + * Blueprints can span multiple z-levels + * Undo functionality for dig, build, and place blueprints + * Automatic cropping of blueprints so you don't get errors if the blueprint extends off the map + * Can generate manager orders for everything required to apply build blueprints + * Library of ready-to-use blueprints included + * Verbose output mode for debugging blueprints +* Dig mode + * Supports all types of designations, including dumping/forbidding items and setting traffic areas + * Supports applying dig blueprints in marker mode + * Handles carving arbitrarily complex minecart tracks, including tracks that cross other tracks +* Build and place modes + * Supports stockpiles of all shapes, not just rectangular blocks + * Configurable maximums for bins, barrels and wheelbarrows assigned to created stockpiles + * Automatic expansion of building footprints to their minimum dimensions, so only the center tile of a multi-tile building needs to be recorded in the blueprint + * Designates complete constructions at once, without having to wait for each tile to become supported before you can build it + * Automatic splitting of stockpiles and buildings that exceed maximum dimension limits + * Tile occupancy and validity checking so, for example, buildings that cannot be placed on a certain tile will simply be skipped instead of the blueprint failing to apply. Blueprints that are only partially applied for any reason (for example, you need to dig out some more tiles) can be safely reapplied to build the remaining buildings. + * Relaxed rules for farm plot and road placement, allowing tiles that are separated by invalid tiles (e.g. stone tiles for farm plots) to be part of the same structure +* Query mode + * Support sending arbitrary keystroke sequences to the UI -- configure *anything* + * Supports aliases to automate frequent keystroke combos + * Supports including aliases in other aliases, and repeating key sequences a specified number of times + * Includes a library of pre-made and tested aliases to automate most common tasks, such as configuring stockpiles for important item types or creating hauling routes for quantum stockpiles. + * Skips sending key sequences when the cursor is over a tile that does not have a stockpile or building + * Instant halting of query blueprint application when keystroke errors are detected, such as when a key sequence leaves us stuck in a submenu + + +Editing Blueprints +------------------ + +The format of Quickfort-compatible blueprint files is straightforward. + +It is recommended to use a spreadsheet editor such as Excel, [Google Sheets](https://sheets.new), or [LibreOffice](https://www.libreoffice.org) to edit these files, but any text editor will do. + +The first line (or upper-left cell) of the spreadsheet should look like this: + + #dig This is a decription. + +The keyword "dig" tells Quickfort we are going to be using the Designations menu in DF. The following "mode" keywords are understood: + + dig Designations menu (d) + build Build menu (b) + place Place stockpiles menu (p) + query Set building tasks/prefs menu (q) + +Optionally following this keyword and a space, you may enter a comment. This comment will appear in the output of `quickfort list` when run from the `DFHack#` prompt. You can use this space for explanations, attribution, etc. + +Below this line begin entering the keys you want sent in each cell. For example, we could dig out a 4x4 room like so (spaces are used as column separators here for clarity, but a real .csv file would have commas): + + #dig + d d d d # + d d d d # + d d d d # + d d d d # + # # # # # + +Note the # symbols at the right end of each row and below the last row. These are completely optional, but can be helpful when visualizing the layout. + +Once the dwarves have that dug out, let's build a walled-in bedroom within our dug-out area: + + #build + Cw Cw Cw Cw # + Cw b h Cw # + Cw Cw # + Cw Cw Cw # + # # # # # + +Note my generosity - in addition to the bed (b) I've built a chest (h) here for the dwarf as well. Note that you must use the full series of keys needed to build something in each cell, e.g. 'Cw' enters DF's constructions submenu (C) and selects walls (w). + +I'd also like to place a booze stockpile in the 2 unoccupied tiles in the room. + + #place Place a food stockpile + ` ` ` ` # + ` ` ` ` # + ` f(2x1)# + ` ` ` ` # + # # # # # + +This illustration may be a little hard to understand. The f(2x1) is in column 2, row 3. All the other cells are empty. QF considers both "`" (backtick -- the character under the tilde) and "~" (tilde) characters within cells to be empty cells; this can help with multilayer or fortress-wide blueprint layouts as 'chalk lines'. + +With f(2x1), we've asked QF to place a food stockpile 2 units wide by 1 high unit. Note that the f(2x1) syntax isn't actually necessary here; we could have just used: + + #place Place a food stockpile + ` ` ` ` # + ` ` ` ` # + ` f f ` # + ` ` ` ` # + # # # # # + +QF is smart enough to recognize this as a 2x1 food stockpile, and creates it as such rather than as two 1x1 food stockpiles. Quickfort recognizes any connected region of identical designations as a single stockpile. The tiles can be connected orthogonally or diagonally, just as long as they are touching somehow. + +Lastly, let's turn the bed into a bedroom and set the food stockpile to hold only booze. + + #query + ` ` ` ` # + ` r& ` # + ` booze # + ` ` ` ` # + # # # # # + +In column 2, row 2 we have "r&". This sends the "r" key to DF when the cursor is over the bed, causing us to 'make room' and "&", which is a special symbol that expands to "{Enter}", to indicate that we're done. + +In column 2, row 3 we have "booze". This is one of many alias keywords defined in the included [baseline aliases file](https://github.com/DFHack/dfhack/tree/develop/data/quickfort/aliases-common.txt). This particular alias sets a food stockpile to carry booze only. It sends the keys needed to navigate DF's stockpile settings menu, and then sends an Escape character ("^" or "{ESC}") to exit back to the map. It is important to exit out of any menus that you enter while in query mode so that the cursor can move to the next tile when it is done configuring the current tile. + +Check out the included [blueprint library](https://github.com/DFHack/dfhack/tree/develop/data/blueprints/library) to see many more examples. Read the baseline aliases file for helpful pre-packaged aliases, or create your own in [dfhack-config/quickfort/aliases.txt](https://github.com/DFHack/dfhack/tree/develop/dfhack-config/quickfort/aliases.txt) in your DFHack installation. + + +Area expansion syntax +--------------------- + +In Quickfort, the following blueprints are equivalent: + + #dig a 3x3 area + d d d # + d d d # + d d d # + # # # # + + #dig the same area with d(3x3) specified in row 1, col 1 + d(3x3)# + ` ` ` # + ` ` ` # + # # # # + +The second example uses Quickfort's "area expansion syntax", which takes the form: + + cmds(WxH) + +In Quickfort the above two examples of specifying a contiguous 3x3 area produce identical output: a single 3x3 designation will be performed, rather than nine 1x1 designations as the first example might suggest. + +Area expansion syntax can only specify rectangular areas. If you want to create extent-based structures (e.g. farm plots or stockpiles) in different shapes, use the first format above. For example: + + #place L shaped food stockpile + f f ` ` # + f f ` ` # + f f f f # + f f f f # + # # # # # + +Area expansion syntax also sets boundaries, which can be useful if you want adjacent, but separate, stockpiles of the same type: + + #place Two touching but separate food stockpiles + f(4x2) # + ~ ~ ~ ~ # + f(4x2) # + ~ ~ ~ ~ # + # # # # # + +As mentioned previously, "~" characters are ignored as comment characters and can be used for visualizing the blueprint layout. The blueprint can be equivalently written as: + + #place Two touching but separate food stockpiles + f(4x2) # + ~ ~ ~ ~ # + f f f f # + f f f f # + # # # # # + +since the area expansion syntax of the upper stockpile prevents it from combining with the lower, freeform syntax stockpile. + +Area expansion syntax can also be used for buildings which have an adjustable size, like bridges. The following blueprints are equivalent: + + #build a 4x2 bridge from row 1, col 1 + ga(4x2) ` # + ` ` ` ` # + # # # # # + + #build a 4x2 bridge from row 1, col 1 + ga ga ga ga # + ga ga ga ga # + # # # # # + + +Automatic area expansion +------------------------ + +Buildings larger than 1x1, like workshops, can be represented in any of three ways. You can designate just their center tile with empty cells around it to leave room for the footprint, like this: + + #build a mason workshop in row 2, col 2 that will occupy the 3x3 area + ` ` ` # + ` wm ` # + ` ` ` # + # # # # + +Or you can fill out the entire footprint like this: + + #build a mason workshop + wm wm wm # + wm wm wm # + wm wm wm # + # # # # + +This format may be verbose for regular workshops, but it can be very helpful for laying out structures like screw pump towers and waterwheels, whose "center point" can be non-obvious. + +Finally, you can use area expansion syntax to represent the workshop: + + #build a mason workshop + wm(3x3) # + ` ` ` # + ` ` ` # + # # # # + +This style can be convenient for laying out multiple buildings of the same type. If you are building a large-scale block factory, for example, this will create 20 mason workshops all in a row: + + #build line of 20 mason workshops + wm(60x3) # + +Quickfort will intelligently break large areas of the same designation into appropriately-sized chunks. + + +Specifying a starting position +------------------------------ + +You can optionally specify a cursor starting position for a particular blueprint, simplifying the task of blueprint alignment. This can be helpful for blueprints that are based on a central staircase, for example. + +To specify a cursor starting position, use the following modified format for the header line of your blueprint: + + #mode start(X;Y;STARTCOMMENT) comment + +where X and Y specify the starting cursor position (1;1 is the top left cell) and STARTCOMMENT (optional) is information about where to position the cursor. This information appears in the `quickfort list` output. + +A couple examples: + + #dig start(3; 3; Center tile of a 5-tile square) Regular blueprint comment + #build start(10;15) + + +Multilevel blueprints +--------------------- + +Multilevel blueprints are accommodated by separating Z-levels of the blueprint with #> (go down one z-level) or #< (go up one z-level) at the end of each floor. + + #dig Stairs leading down to a small room below + j ` ` # + ` ` ` # + ` ` ` # + #> # # # + u d d # + d d d # + d d d # + # # # # + + +Minecart tracks +--------------- + +There are two ways to produce minecart tracks, and they are handled very differently by the game. You can carve them into hard natural floors or you can construct them out of building materials. Constructed tracks are conceptually simpler, so we'll start with them. + +### Constructed Tracks ### + +Quickfort supports the designation of track stops and rollers through the normal mechanisms: a #build blueprint with `CS` and some number of 'd' and 'a' characters (for selecting dump direction and friction) in a cell designates a track stop and a #build blueprint with `Mr` and some number of 's' and 'q' characters (for direction and speed) designates a roller. This can get confusing very quickly and is very difficult to read in a blueprint. Constructed track segments don't even have keys associated with them at all! + +To solve this problem, Quickfort provides the following keywords for use in build blueprints: + + -- Track segments -- + trackN + trackS + trackE + trackW + trackNS + trackNE + trackNW + trackSE + trackSW + trackEW + trackNSE + trackNSW + trackNEW + trackSEW + trackNSEW + + -- Track/ramp segments -- + trackrampN + trackrampS + trackrampE + trackrampW + trackrampNS + trackrampNE + trackrampNW + trackrampSE + trackrampSW + trackrampEW + trackrampNSE + trackrampNSW + trackrampNEW + trackrampSEW + trackrampNSEW + + -- Horizontal and vertical roller segments -- + rollerH + rollerV + rollerNS + rollerSN + rollerEW + rollerWE + + Note: append up to four 'q' characters to roller keywords to set roller + speed. E.g. a roller that propels from East to West at the slowest speed can + be specified with 'rollerEWqqqq'. + + -- Track stops that (optionally) dump to the N/S/E/W -- + trackstop + trackstopN + trackstopS + trackstopE + trackstopW + + Note: append up to four 'a' characters to trackstop keywords to set friction + amount. E.g. a stop that applies the smallest amount of friction can be + specified with 'trackstopaaaa'. + +As an example, you can create an E-W track with stops at each end that dump to their outside directions with the following blueprint: + + #build Example track + trackstopW trackEW trackEW trackEW trackstopE + +Note that the **only** way to build track and track/ramp segments is with the keywords. The UI method of using "+" and "-" keys to select the track type from a list does not work since DFHack Quickfort doesn't actually send keys to the UI in order to build buildings. The text in your spreadsheet cells is mapped directly into DFHack API calls. Only query-mode blueprints still send actual keycodes to the UI. + +### Carved Tracks ### + +In the game, you carve a minecart track by specifying a beginning and ending tile and the game "adds" the designation to the tiles. You cannot designate single tiles. For example to carve two track segments that cross each other, you might use the cursor to designate a line of three vertical tiles like this: + + ` start here ` # + ` ` ` # + ` end here ` # + +Then to carve the cross, you'd do a horizonal segment: + + ` ` ` # + start here ` end here # + ` ` ` # + +This will result in a carved track that would be equivalent to a constructed track of the form: + + #build + ` trackS ` # + trackE trackNSEW trackW # + ` trackN ` # + # # # # + +To carve this same track with a dig blueprint, you'd use area expansion syntax with a height or width of 1 to indicate the segments to designate: + + #dig + ` T(1x3) ` # + T(3x1) ` ` # + ` ` ` # + # # # # + +"But wait!", I can hear you say, "How do you designate a track corner that opens to the South and East? You can't put both T(1xH) and T(Wx1) in the same cell!" This is true, but you can specify both width and height, and for tracks, QF interprets it as an upper-left corner extending to the right W tiles and down H tiles. For example, to carve a track in a closed ring, you'd write: + + #dig + T(3x3) ` T(1x3) # + ` ` ` # + T(3x1) ` ` # + # # # # + +Which would result in a carved track simliar to a constructed track of the form: + + #build + trackSE trackEW trackSW # + trackNS ` trackNS # + trackNE trackEW trackNW # + # # # # + + +Packaging a set of blueprints +----------------------------- + +A complete QF specification for a section of your fortress may contain 4 or more separate blueprints, one for each "phase" of construction (dig, build, place stockpiles, query building adjustments). + +To manage all the separate blueprints, it is often convenient to keep related blueprints in a single .xlsx file, with each blueprint in a separate sheet within the file. Online spreadsheet applications like [Google Sheets](https://sheets.new) make it easy to work with multiple related blueprints, and, as a bonus, they retain any formatting you've set, like column sizes and coloring. + +Of course, you could still choose to keep your blueprints in single-sheet .csv files and just give related blueprints similar names. The blueprint modes suggest a convenient naming scheme for .csv-based blueprint "stacks": + + bedroom.1.dig.csv + bedroom.2.build.csv + bedroom.3.place.csv + bedroom.4.query.csv + +But the naming is completely up to you, of course. + + +Tips and Tricks +--------------- + +* During blueprint application, especially query blueprints, don't click the mouse on the DF window or type any keys. They can change the state of the game while the blueprint is being applied, resulting in strange errors. + +* After digging out an area, you may wish to smooth and/or engrave the area before starting the build phase, as dwarves may be unable to access walls or floors that are behind/under built objects. + +* As of DF 0.34.x, it is no longer possible to build doors (d) at the same time that you build adjacent walls (Cw). Doors must now be built *after* walls are constructed for them to be next to. This does not affect the more common case where walls exist as a side-effect of having dug-out a room in a #dig blueprint. + + +Links +----- + +### Quickfort links ### + +* [Quickfort command syntax](https://docs.dfhack.org/en/stable/docs/_auto/base.html#quickfort) +* [Quickfort forum thread](http://www.bay12forums.com/smf/index.php?topic=176889.0) +* [Quickfort blueprints library](https://github.com/DFHack/dfhack/tree/develop/data/blueprints/library) +* [DFHack issue tracker](https://github.com/DFHack/dfhack/issues) +* [Quickfort source code](https://github.com/DFHack/scripts/tree/master/internal/quickfort) + +### Related tools ### + +* DFHack's [blueprint plugin](https://docs.dfhack.org/en/stable/docs/Plugins.html#blueprint) can generate blueprints from actual DF maps. +* [Python Quickfort](http://joelpt.net/quickfort) is the previous, Python-based implementation that DFHack's quickfort script was inspired by. From b033e4644f8ea414aa12975b2261c797ccf5619c Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Fri, 7 Aug 2020 12:44:40 -0700 Subject: [PATCH 02/61] add xlsxio as a submodule --- .gitmodules | 4 ++++ depends/xlsxio | 1 + 2 files changed, 5 insertions(+) create mode 160000 depends/xlsxio diff --git a/.gitmodules b/.gitmodules index 98e06473d..0ccd3810a 100644 --- a/.gitmodules +++ b/.gitmodules @@ -16,3 +16,7 @@ [submodule "depends/jsoncpp"] path = depends/jsoncpp-sub url = ../../DFHack/jsoncpp.git +[submodule "depends/xlsxio"] + path = depends/xlsxio + url = ../../brechtsanders/xlsxio.git + shallow = true diff --git a/depends/xlsxio b/depends/xlsxio new file mode 160000 index 000000000..2d12a8844 --- /dev/null +++ b/depends/xlsxio @@ -0,0 +1 @@ +Subproject commit 2d12a88448b6f40594b2b2e0d0ce7207d0efa021 From 1fc800e7652d344a53f988b3977cca37f08fa568 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Fri, 7 Aug 2020 13:08:06 -0700 Subject: [PATCH 03/61] fix xlsxio lib to current commit so it doesn't break us with incompatibile changes. we only need to update this lib if/when we run into a bug with it. no need to introduce instability by tracking HEAD --- depends/xlsxio | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/depends/xlsxio b/depends/xlsxio index 2d12a8844..261d56815 160000 --- a/depends/xlsxio +++ b/depends/xlsxio @@ -1 +1 @@ -Subproject commit 2d12a88448b6f40594b2b2e0d0ce7207d0efa021 +Subproject commit 261d56815b29908fc960fecb9cb3143db4b485ad From 89a6309788a6ff2d1a64ba951c5427f9de8f0f79 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sat, 8 Aug 2020 15:24:12 -0700 Subject: [PATCH 04/61] build xlsxio and link it into dfhack, add lua api --- CMakeLists.txt | 19 +++++ depends/CMakeLists.txt | 11 +++ library/CMakeLists.txt | 4 +- library/LuaApi.cpp | 43 ++++++++++ library/include/modules/XlsxReader.h | 45 +++++++++++ library/modules/XlsxReader.cpp | 115 +++++++++++++++++++++++++++ 6 files changed, 236 insertions(+), 1 deletion(-) create mode 100644 library/include/modules/XlsxReader.h create mode 100644 library/modules/XlsxReader.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index 44d6a02ce..6a2230721 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -364,6 +364,8 @@ endif() #### expose depends #### +include(ExternalProject) + if(UNIX) # Rescan for pthread and zlib if the build arch changed if(NOT "${DFHACK_BUILD_ARCH}" STREQUAL "${DFHACK_BUILD_ARCH_PREV}") @@ -415,6 +417,23 @@ endif() # build the lib itself if(BUILD_LIBRARY) + ExternalProject_Get_Property(xlsxio_project INSTALL_DIR) + set(XLSXIO_INSTALL_DIR ${INSTALL_DIR}) + include_directories(${XLSXIO_INSTALL_DIR}/include) + add_library(xlsxio_read STATIC IMPORTED) + set_target_properties(xlsxio_read PROPERTIES + IMPORTED_LOCATION ${XLSXIO_INSTALL_DIR}/lib/libxlsxio_read.a) + add_dependencies(xlsxio_read xlsxio_project) + + # just pull from the system until I figure out how to integrate these deps + # into the build + add_library(zip SHARED IMPORTED) + set_target_properties(zip PROPERTIES + IMPORTED_LOCATION /usr/lib64/libzip.so) + add_library(expat SHARED IMPORTED) + set_target_properties(expat PROPERTIES + IMPORTED_LOCATION /usr/lib64/libexpat.so) + add_subdirectory(library) install(FILES LICENSE.rst docs/changelog.txt DESTINATION ${DFHACK_USERDOC_DESTINATION}) endif() diff --git a/depends/CMakeLists.txt b/depends/CMakeLists.txt index aac250c39..e43c80266 100644 --- a/depends/CMakeLists.txt +++ b/depends/CMakeLists.txt @@ -17,3 +17,14 @@ option(CLSOCKET_SHARED "Build clsocket lib as shared." OFF) option(CLSOCKET_DEP_ONLY "Build for use inside other CMake projects as dependency." ON) add_subdirectory(clsocket) ide_folder(clsocket "Depends") + +set(XLSXIO_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/xlsxio) +ExternalProject_Add(xlsxio_project + PREFIX xlsxio + SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/xlsxio + INSTALL_DIR ${XLSXIO_INSTALL_DIR} + BUILD_BYPRODUCTS ${XLSXIO_INSTALL_DIR}/lib/libxlsxio_read.a + CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${XLSXIO_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_STATIC=ON -DBUILD_SHARED=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_TOOLS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_DOCUMENTATION=OFF -DWITH_LIBZIP=ON -DZLIB_DIR=${ZLIB_DIR} +) + #-DLIBZIP_DIR= + #-DEXPAT_DIR= diff --git a/library/CMakeLists.txt b/library/CMakeLists.txt index 2de293e15..583c8768f 100644 --- a/library/CMakeLists.txt +++ b/library/CMakeLists.txt @@ -144,6 +144,7 @@ set(MODULE_HEADERS include/modules/Translation.h include/modules/Units.h include/modules/World.h + include/modules/XlsxReader.h ) set(MODULE_SOURCES @@ -172,6 +173,7 @@ set(MODULE_SOURCES modules/Units.cpp modules/Windows.cpp modules/World.cpp + modules/XlsxReader.cpp ) set(STATIC_FIELDS_FILES) @@ -409,7 +411,7 @@ if(APPLE) set_target_properties(dfhack PROPERTIES SOVERSION 1.0.0) endif() -target_link_libraries(dfhack protobuf-lite clsocket lua jsoncpp_lib_static dfhack-version ${PROJECT_LIBS}) +target_link_libraries(dfhack protobuf-lite clsocket lua jsoncpp_lib_static xlsxio_read zip expat dfhack-version ${PROJECT_LIBS}) set_target_properties(dfhack PROPERTIES INTERFACE_LINK_LIBRARIES "") target_link_libraries(dfhack-client protobuf-lite clsocket jsoncpp_lib_static) diff --git a/library/LuaApi.cpp b/library/LuaApi.cpp index 25843875d..49fdf4c73 100644 --- a/library/LuaApi.cpp +++ b/library/LuaApi.cpp @@ -60,6 +60,7 @@ distribution. #include "modules/Translation.h" #include "modules/Units.h" #include "modules/World.h" +#include "modules/XlsxReader.h" #include "LuaWrapper.h" #include "LuaTools.h" @@ -2474,6 +2475,47 @@ static const LuaWrapper::FunctionReg dfhack_kitchen_module[] = { {NULL, NULL} }; +/***** XlsxReader module *****/ + +static const LuaWrapper::FunctionReg dfhack_xlsxreader_module[] = { + WRAPM(XlsxReader, open_xlsx_file), + WRAPM(XlsxReader, close_xlsx_file), + WRAPM(XlsxReader, list_sheets), + WRAPM(XlsxReader, open_sheet), + WRAPM(XlsxReader, close_sheet), + WRAPM(XlsxReader, get_next_row), + {NULL, NULL} +}; + +// takes the sheet handle and returns a string, or nil if there is no next cell +// in the current row. +static int xlsxreader_get_next_cell(lua_State *L) +{ + if (lua_gettop(L) != 1 || lua_isnil(L, 1)) + { + luaL_error(L, "invalid sheet handle"); + } + luaL_checktype(L, 1, LUA_TUSERDATA); + XlsxReader::xlsx_sheet_handle sheet_handle = lua_touserdata(L, 1); + + std::string value; + bool ok = XlsxReader::get_next_cell(sheet_handle, value); + if (!ok) + { + lua_pushnil(L); + } + else + { + lua_pushstring(L, value.c_str()); + } + return 1; +} + +static const luaL_Reg dfhack_xlsxreader_funcs[] = { + {"get_next_cell", xlsxreader_get_next_cell}, + {NULL, NULL} +}; + /***** Console module *****/ namespace console { @@ -3027,6 +3069,7 @@ void OpenDFHackApi(lua_State *state) OpenModule(state, "filesystem", dfhack_filesystem_module, dfhack_filesystem_funcs); OpenModule(state, "designations", dfhack_designations_module, dfhack_designations_funcs); OpenModule(state, "kitchen", dfhack_kitchen_module); + OpenModule(state, "xlsxreader", dfhack_xlsxreader_module, dfhack_xlsxreader_funcs); OpenModule(state, "console", dfhack_console_module); OpenModule(state, "internal", dfhack_internal_module, dfhack_internal_funcs); } diff --git a/library/include/modules/XlsxReader.h b/library/include/modules/XlsxReader.h new file mode 100644 index 000000000..93ea1cf20 --- /dev/null +++ b/library/include/modules/XlsxReader.h @@ -0,0 +1,45 @@ +/* + * Wrapper for xlsxio_read library functions. + */ + +#pragma once + +#include +#include + +#include "Export.h" + +/** + * \defgroup grp_xlsx_reader Xlsx file reader + * @ingroup grp_modules + */ +namespace DFHack +{ +namespace XlsxReader +{ + +typedef void* xlsx_file_handle; +typedef void* xlsx_sheet_handle; + +// returns NULL on error +DFHACK_EXPORT xlsx_file_handle open_xlsx_file(std::string filename); +DFHACK_EXPORT void close_xlsx_file(xlsx_file_handle file_handle); +DFHACK_EXPORT std::vector list_sheets(xlsx_file_handle file_handle); + +// returns XLSXIOReaderSheet object or NULL on error +DFHACK_EXPORT xlsx_sheet_handle open_sheet( + xlsx_file_handle file_handle, std::string sheet_name); +DFHACK_EXPORT void close_sheet(xlsx_sheet_handle sheet_handle); + +// start reading the next row of data; must be called before GetNextCell . +// returns false if there is no next row to get. +DFHACK_EXPORT bool get_next_row(xlsx_sheet_handle sheet_handle); + +// fills the value param with the contents of the cell in the next column cell +// in the current row. +// returns false if there are no more cells in this row. +DFHACK_EXPORT bool get_next_cell( + xlsx_sheet_handle sheet_handle, std::string& value); + +} +} diff --git a/library/modules/XlsxReader.cpp b/library/modules/XlsxReader.cpp new file mode 100644 index 000000000..d13582450 --- /dev/null +++ b/library/modules/XlsxReader.cpp @@ -0,0 +1,115 @@ +/* + * Wrapper for xlsxio_read library functions. + * + * Sample usage: + * + * std::string filename = "sample_file.xlsx"; + * xlsxioreader xlsxfile = XlsxReader::open_xlsx_file(filename); + * if (xlsxfile == NULL) { + * printf("cannot open file: '%s'", filename.c_str()); + * return false; + * } + * auto sheetNames = XlsxReader::list_sheets(xlsxfile); + * for (auto sheetName = sheetNames.begin(); + * sheetName != sheetNames.end(); + * ++sheetName) { + * printf("reading sheet: %s\n", sheetName->c_str()); + * xlsxioreadersheet xlsxsheet = + * XlsxReader::open_sheet(xlsxfile, *sheetName); + * if (xlsxsheet == NULL) { + * printf("cannot open sheet: '%s'", sheetName->c_str()); + * continue; + * } + * std::string value; + * int row_num = 1; + * while (XlsxReader::GetNextRow(xlsxsheet)) { + * std::string s; + * printf("%d:\t", row_num); + * while (XlsxReader::GetNextCell(xlsxsheet, s)) { + * printf("%s\t", s.c_str()); + * } + * printf("\n"); + * ++row_num; + * } + * XlsxReader::close_sheet(xlsxsheet); + * } + * XlsxReader::close_xlsx_file(xlsxfile); + * return true; + */ + +#include + +#include "modules/XlsxReader.h" + +using namespace DFHack; + + +// returns NULL on error +DFHACK_EXPORT XlsxReader::xlsx_file_handle XlsxReader::open_xlsx_file( + std::string filename) +{ + return xlsxioread_open(filename.c_str()); +} + +DFHACK_EXPORT void XlsxReader::close_xlsx_file( + XlsxReader::xlsx_file_handle file_handle) +{ + xlsxioread_close((xlsxioreader)file_handle); +} + +static int list_callback(const XLSXIOCHAR* name, void* cbdata) +{ + auto sheetNames = (std::vector *)cbdata; + sheetNames->push_back(name); + return 0; +} + +DFHACK_EXPORT std::vector XlsxReader::list_sheets( + XlsxReader::xlsx_file_handle file_handle) +{ + auto sheetNames = std::vector(); + xlsxioread_list_sheets( + (xlsxioreader)file_handle, list_callback, &sheetNames); + return sheetNames; +} + +// returns XLSXIOReaderSheet object or NULL on error +DFHACK_EXPORT XlsxReader::xlsx_sheet_handle XlsxReader::open_sheet( + XlsxReader::xlsx_file_handle file_handle, std::string sheet_name) +{ + if (file_handle == NULL) + return NULL; + return xlsxioread_sheet_open( + (xlsxioreader)file_handle, sheet_name.c_str(), XLSXIOREAD_SKIP_NONE); +} + +DFHACK_EXPORT void XlsxReader::close_sheet( + XlsxReader::xlsx_sheet_handle sheet_handle) +{ + xlsxioread_sheet_close((xlsxioreadersheet)sheet_handle); +} + +// start reading the next row of data; must be called before GetNextCell . +// returns false if there is no next row to get. +DFHACK_EXPORT bool XlsxReader::get_next_row( + XlsxReader::xlsx_sheet_handle sheet_handle) +{ + return xlsxioread_sheet_next_row((xlsxioreadersheet)sheet_handle) != 0; +} + +// fills the value param with the contents of the cell in the next column cell +// in the current row. +// returns false if there are no more cells in this row. +DFHACK_EXPORT bool XlsxReader::get_next_cell( + XlsxReader::xlsx_sheet_handle sheet_handle, std::string& value) +{ + char* result; + if (!xlsxioread_sheet_next_cell_string((xlsxioreadersheet)sheet_handle, + &result)) { + value.clear(); + return false; + } + value.assign(result); + free(result); + return true; +} From 8c7fb808e14f7cbfd0bbb5307f50202947061274 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sat, 8 Aug 2020 17:55:49 -0700 Subject: [PATCH 05/61] get xlsxio Lua API working --- library/LuaApi.cpp | 44 +++++++++++++++++++++++++++++++------------- 1 file changed, 31 insertions(+), 13 deletions(-) diff --git a/library/LuaApi.cpp b/library/LuaApi.cpp index 30a084635..0265eee20 100644 --- a/library/LuaApi.cpp +++ b/library/LuaApi.cpp @@ -2486,39 +2486,57 @@ static const LuaWrapper::FunctionReg dfhack_kitchen_module[] = { static const LuaWrapper::FunctionReg dfhack_xlsxreader_module[] = { WRAPM(XlsxReader, open_xlsx_file), WRAPM(XlsxReader, close_xlsx_file), - WRAPM(XlsxReader, list_sheets), WRAPM(XlsxReader, open_sheet), WRAPM(XlsxReader, close_sheet), - WRAPM(XlsxReader, get_next_row), {NULL, NULL} }; -// takes the sheet handle and returns a string, or nil if there is no next cell -// in the current row. -static int xlsxreader_get_next_cell(lua_State *L) +// internal function to factor out handle extraction +static void * get_xlsxreader_handle(lua_State *L) { - if (lua_gettop(L) != 1 || lua_isnil(L, 1)) + if (lua_gettop(L) < 1 || lua_isnil(L, 1)) { - luaL_error(L, "invalid sheet handle"); + luaL_error(L, "invalid xlsxreader handle"); } - luaL_checktype(L, 1, LUA_TUSERDATA); - XlsxReader::xlsx_sheet_handle sheet_handle = lua_touserdata(L, 1); + luaL_checktype(L, 1, LUA_TLIGHTUSERDATA); + return lua_touserdata(L, 1); +} + +// takes a file handle and returns a table-list of sheet names +static int xlsxreader_list_sheets(lua_State *L) +{ + XlsxReader::xlsx_file_handle file_handle = get_xlsxreader_handle(L); + Lua::PushVector(L, XlsxReader::list_sheets(file_handle), true); + return 1; +} - std::string value; - bool ok = XlsxReader::get_next_cell(sheet_handle, value); +// takes the sheet handle and returns a table-list of strings, or nil if we +// already processed the last row in the file. +static int xlsxreader_get_row(lua_State *L) +{ + XlsxReader::xlsx_sheet_handle sheet_handle = get_xlsxreader_handle(L); + bool ok = XlsxReader::get_next_row(sheet_handle); if (!ok) { lua_pushnil(L); } else { - lua_pushstring(L, value.c_str()); + std::string value; + auto cells = std::vector(); + while (XlsxReader::get_next_cell(sheet_handle, value)) + { + cells.push_back(value); + } + Lua::PushVector(L, cells, true); } + return 1; } static const luaL_Reg dfhack_xlsxreader_funcs[] = { - {"get_next_cell", xlsxreader_get_next_cell}, + {"list_sheets", xlsxreader_list_sheets}, + {"get_row", xlsxreader_get_row}, {NULL, NULL} }; From 4f0eb4bca6d80fb1d7f1822f509c4082a3e79158 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sat, 8 Aug 2020 19:47:02 -0700 Subject: [PATCH 06/61] add libzip dep --- .gitmodules | 3 +++ depends/libzip | 1 + 2 files changed, 4 insertions(+) create mode 160000 depends/libzip diff --git a/.gitmodules b/.gitmodules index 0ccd3810a..b41fa0437 100644 --- a/.gitmodules +++ b/.gitmodules @@ -20,3 +20,6 @@ path = depends/xlsxio url = ../../brechtsanders/xlsxio.git shallow = true +[submodule "depends/libzip"] + path = depends/libzip + url = ../../nih-at/libzip.git diff --git a/depends/libzip b/depends/libzip new file mode 160000 index 000000000..2134d8c34 --- /dev/null +++ b/depends/libzip @@ -0,0 +1 @@ +Subproject commit 2134d8c34dbb8988076e8f2fc3b8be75fdb780bf From eeeb638c1c5641c94771601558321ccbb0e5f4c3 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sat, 8 Aug 2020 19:48:11 -0700 Subject: [PATCH 07/61] fix libzip at v1.7.3 --- depends/libzip | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/depends/libzip b/depends/libzip index 2134d8c34..66e496489 160000 --- a/depends/libzip +++ b/depends/libzip @@ -1 +1 @@ -Subproject commit 2134d8c34dbb8988076e8f2fc3b8be75fdb780bf +Subproject commit 66e496489bdae81bfda8b0088172871d8fda0032 From 28eaf0ae8b6eba53d164d5a642202e16cbd42ff8 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sat, 8 Aug 2020 20:04:46 -0700 Subject: [PATCH 08/61] add libzip to the build --- CMakeLists.txt | 11 ++++++++--- depends/CMakeLists.txt | 9 +++++++++ 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 2c9ada74a..490e1bb55 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -425,11 +425,16 @@ if(BUILD_LIBRARY) IMPORTED_LOCATION ${XLSXIO_INSTALL_DIR}/lib/libxlsxio_read.a) add_dependencies(xlsxio_read xlsxio_project) + ExternalProject_Get_Property(libzip_project INSTALL_DIR) + set(LIBZIP_INSTALL_DIR ${INSTALL_DIR}) + include_directories(${LIBZIP_INSTALL_DIR}/include) + add_library(zip STATIC IMPORTED) + set_target_properties(zip PROPERTIES + IMPORTED_LOCATION ${LIBZIP_INSTALL_DIR}/lib64/libzip.a) + add_dependencies(zip libzip_project) + # just pull from the system until I figure out how to integrate these deps # into the build - add_library(zip SHARED IMPORTED) - set_target_properties(zip PROPERTIES - IMPORTED_LOCATION /usr/lib64/libzip.so) add_library(expat SHARED IMPORTED) set_target_properties(expat PROPERTIES IMPORTED_LOCATION /usr/lib64/libexpat.so) diff --git a/depends/CMakeLists.txt b/depends/CMakeLists.txt index e43c80266..d5669080e 100644 --- a/depends/CMakeLists.txt +++ b/depends/CMakeLists.txt @@ -18,6 +18,15 @@ option(CLSOCKET_DEP_ONLY "Build for use inside other CMake projects as dependenc add_subdirectory(clsocket) ide_folder(clsocket "Depends") +set(LIBZIP_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libzip) +ExternalProject_Add(libzip_project + PREFIX libzip + SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/libzip + INSTALL_DIR ${LIBZIP_INSTALL_DIR} + BUILD_BYPRODUCTS ${LIBZIP_INSTALL_DIR}/lib64/libzip.a + CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${LIBZIP_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_DOC=OFF -DBUILD_EXAMPLES=OFF -DBUILD_REGRESS=OFF -DBUILD_SHARED_LIBS=OFF -DBUILD_TOOLS=OFF -DENABLE_BZIP2=OFF -DENABLE_COMMONCRYPTO=OFF -DENABLE_GNUTLS=OFF -DENABLE_LZMA=OFF -DENABLE_MBEDTLS=OFF -DENABLE_OPENSSL=OFF -DENABLE_WINDOWS_CRYPTO=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON +) + set(XLSXIO_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/xlsxio) ExternalProject_Add(xlsxio_project PREFIX xlsxio From cffac8ed93078d1d3e22087d3e5f415466b4004a Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sat, 8 Aug 2020 20:07:54 -0700 Subject: [PATCH 09/61] fix line endings in XlsxReader.cpp --- library/modules/XlsxReader.cpp | 230 ++++++++++++++++----------------- 1 file changed, 115 insertions(+), 115 deletions(-) diff --git a/library/modules/XlsxReader.cpp b/library/modules/XlsxReader.cpp index d13582450..405369c85 100644 --- a/library/modules/XlsxReader.cpp +++ b/library/modules/XlsxReader.cpp @@ -1,115 +1,115 @@ -/* - * Wrapper for xlsxio_read library functions. - * - * Sample usage: - * - * std::string filename = "sample_file.xlsx"; - * xlsxioreader xlsxfile = XlsxReader::open_xlsx_file(filename); - * if (xlsxfile == NULL) { - * printf("cannot open file: '%s'", filename.c_str()); - * return false; - * } - * auto sheetNames = XlsxReader::list_sheets(xlsxfile); - * for (auto sheetName = sheetNames.begin(); - * sheetName != sheetNames.end(); - * ++sheetName) { - * printf("reading sheet: %s\n", sheetName->c_str()); - * xlsxioreadersheet xlsxsheet = - * XlsxReader::open_sheet(xlsxfile, *sheetName); - * if (xlsxsheet == NULL) { - * printf("cannot open sheet: '%s'", sheetName->c_str()); - * continue; - * } - * std::string value; - * int row_num = 1; - * while (XlsxReader::GetNextRow(xlsxsheet)) { - * std::string s; - * printf("%d:\t", row_num); - * while (XlsxReader::GetNextCell(xlsxsheet, s)) { - * printf("%s\t", s.c_str()); - * } - * printf("\n"); - * ++row_num; - * } - * XlsxReader::close_sheet(xlsxsheet); - * } - * XlsxReader::close_xlsx_file(xlsxfile); - * return true; - */ - -#include - -#include "modules/XlsxReader.h" - -using namespace DFHack; - - -// returns NULL on error -DFHACK_EXPORT XlsxReader::xlsx_file_handle XlsxReader::open_xlsx_file( - std::string filename) -{ - return xlsxioread_open(filename.c_str()); -} - -DFHACK_EXPORT void XlsxReader::close_xlsx_file( - XlsxReader::xlsx_file_handle file_handle) -{ - xlsxioread_close((xlsxioreader)file_handle); -} - -static int list_callback(const XLSXIOCHAR* name, void* cbdata) -{ - auto sheetNames = (std::vector *)cbdata; - sheetNames->push_back(name); - return 0; -} - -DFHACK_EXPORT std::vector XlsxReader::list_sheets( - XlsxReader::xlsx_file_handle file_handle) -{ - auto sheetNames = std::vector(); - xlsxioread_list_sheets( - (xlsxioreader)file_handle, list_callback, &sheetNames); - return sheetNames; -} - -// returns XLSXIOReaderSheet object or NULL on error -DFHACK_EXPORT XlsxReader::xlsx_sheet_handle XlsxReader::open_sheet( - XlsxReader::xlsx_file_handle file_handle, std::string sheet_name) -{ - if (file_handle == NULL) - return NULL; - return xlsxioread_sheet_open( - (xlsxioreader)file_handle, sheet_name.c_str(), XLSXIOREAD_SKIP_NONE); -} - -DFHACK_EXPORT void XlsxReader::close_sheet( - XlsxReader::xlsx_sheet_handle sheet_handle) -{ - xlsxioread_sheet_close((xlsxioreadersheet)sheet_handle); -} - -// start reading the next row of data; must be called before GetNextCell . -// returns false if there is no next row to get. -DFHACK_EXPORT bool XlsxReader::get_next_row( - XlsxReader::xlsx_sheet_handle sheet_handle) -{ - return xlsxioread_sheet_next_row((xlsxioreadersheet)sheet_handle) != 0; -} - -// fills the value param with the contents of the cell in the next column cell -// in the current row. -// returns false if there are no more cells in this row. -DFHACK_EXPORT bool XlsxReader::get_next_cell( - XlsxReader::xlsx_sheet_handle sheet_handle, std::string& value) -{ - char* result; - if (!xlsxioread_sheet_next_cell_string((xlsxioreadersheet)sheet_handle, - &result)) { - value.clear(); - return false; - } - value.assign(result); - free(result); - return true; -} +/* + * Wrapper for xlsxio_read library functions. + * + * Sample usage: + * + * std::string filename = "sample_file.xlsx"; + * xlsxioreader xlsxfile = XlsxReader::open_xlsx_file(filename); + * if (xlsxfile == NULL) { + * printf("cannot open file: '%s'", filename.c_str()); + * return false; + * } + * auto sheetNames = XlsxReader::list_sheets(xlsxfile); + * for (auto sheetName = sheetNames.begin(); + * sheetName != sheetNames.end(); + * ++sheetName) { + * printf("reading sheet: %s\n", sheetName->c_str()); + * xlsxioreadersheet xlsxsheet = + * XlsxReader::open_sheet(xlsxfile, *sheetName); + * if (xlsxsheet == NULL) { + * printf("cannot open sheet: '%s'", sheetName->c_str()); + * continue; + * } + * std::string value; + * int row_num = 1; + * while (XlsxReader::GetNextRow(xlsxsheet)) { + * std::string s; + * printf("%d:\t", row_num); + * while (XlsxReader::GetNextCell(xlsxsheet, s)) { + * printf("%s\t", s.c_str()); + * } + * printf("\n"); + * ++row_num; + * } + * XlsxReader::close_sheet(xlsxsheet); + * } + * XlsxReader::close_xlsx_file(xlsxfile); + * return true; + */ + +#include + +#include "modules/XlsxReader.h" + +using namespace DFHack; + + +// returns NULL on error +DFHACK_EXPORT XlsxReader::xlsx_file_handle XlsxReader::open_xlsx_file( + std::string filename) +{ + return xlsxioread_open(filename.c_str()); +} + +DFHACK_EXPORT void XlsxReader::close_xlsx_file( + XlsxReader::xlsx_file_handle file_handle) +{ + xlsxioread_close((xlsxioreader)file_handle); +} + +static int list_callback(const XLSXIOCHAR* name, void* cbdata) +{ + auto sheetNames = (std::vector *)cbdata; + sheetNames->push_back(name); + return 0; +} + +DFHACK_EXPORT std::vector XlsxReader::list_sheets( + XlsxReader::xlsx_file_handle file_handle) +{ + auto sheetNames = std::vector(); + xlsxioread_list_sheets( + (xlsxioreader)file_handle, list_callback, &sheetNames); + return sheetNames; +} + +// returns XLSXIOReaderSheet object or NULL on error +DFHACK_EXPORT XlsxReader::xlsx_sheet_handle XlsxReader::open_sheet( + XlsxReader::xlsx_file_handle file_handle, std::string sheet_name) +{ + if (file_handle == NULL) + return NULL; + return xlsxioread_sheet_open( + (xlsxioreader)file_handle, sheet_name.c_str(), XLSXIOREAD_SKIP_NONE); +} + +DFHACK_EXPORT void XlsxReader::close_sheet( + XlsxReader::xlsx_sheet_handle sheet_handle) +{ + xlsxioread_sheet_close((xlsxioreadersheet)sheet_handle); +} + +// start reading the next row of data; must be called before GetNextCell . +// returns false if there is no next row to get. +DFHACK_EXPORT bool XlsxReader::get_next_row( + XlsxReader::xlsx_sheet_handle sheet_handle) +{ + return xlsxioread_sheet_next_row((xlsxioreadersheet)sheet_handle) != 0; +} + +// fills the value param with the contents of the cell in the next column cell +// in the current row. +// returns false if there are no more cells in this row. +DFHACK_EXPORT bool XlsxReader::get_next_cell( + XlsxReader::xlsx_sheet_handle sheet_handle, std::string& value) +{ + char* result; + if (!xlsxioread_sheet_next_cell_string((xlsxioreadersheet)sheet_handle, + &result)) { + value.clear(); + return false; + } + value.assign(result); + free(result); + return true; +} From 8de39c2d7d1c64631043eab751d0ce4787b65612 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sat, 8 Aug 2020 20:09:29 -0700 Subject: [PATCH 10/61] add libexpat as a submodule --- .gitmodules | 5 +++++ depends/libexpat | 1 + 2 files changed, 6 insertions(+) create mode 160000 depends/libexpat diff --git a/.gitmodules b/.gitmodules index b41fa0437..e24bfb707 100644 --- a/.gitmodules +++ b/.gitmodules @@ -23,3 +23,8 @@ [submodule "depends/libzip"] path = depends/libzip url = ../../nih-at/libzip.git + shallow = true +[submodule "depends/libexpat"] + path = depends/libexpat + url = ../../libexpat/libexpat.git + shallow = true diff --git a/depends/libexpat b/depends/libexpat new file mode 160000 index 000000000..990e3d07e --- /dev/null +++ b/depends/libexpat @@ -0,0 +1 @@ +Subproject commit 990e3d07eaa127007f9d304a9b4c6ffadc61b1fe From e7afc4cfeb6d8c0e4670e7d517b926f2164dbbac Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sat, 8 Aug 2020 20:10:13 -0700 Subject: [PATCH 11/61] fix libexpat at v2.2.9 --- depends/libexpat | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/depends/libexpat b/depends/libexpat index 990e3d07e..a7bc26b69 160000 --- a/depends/libexpat +++ b/depends/libexpat @@ -1 +1 @@ -Subproject commit 990e3d07eaa127007f9d304a9b4c6ffadc61b1fe +Subproject commit a7bc26b69768f7fb24f0c7976fae24b157b85b13 From 8a05f09cd612249897c8767d4acc0f6ac956d92b Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sat, 8 Aug 2020 20:18:43 -0700 Subject: [PATCH 12/61] add libexpat to the build --- CMakeLists.txt | 10 ++++++---- depends/CMakeLists.txt | 15 ++++++++++++--- 2 files changed, 18 insertions(+), 7 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 490e1bb55..5dcd87b56 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -433,11 +433,13 @@ if(BUILD_LIBRARY) IMPORTED_LOCATION ${LIBZIP_INSTALL_DIR}/lib64/libzip.a) add_dependencies(zip libzip_project) - # just pull from the system until I figure out how to integrate these deps - # into the build - add_library(expat SHARED IMPORTED) + ExternalProject_Get_Property(libexpat_project INSTALL_DIR) + set(LIBEXPAT_INSTALL_DIR ${INSTALL_DIR}) + include_directories(${LIBEXPAT_INSTALL_DIR}/include) + add_library(expat STATIC IMPORTED) set_target_properties(expat PROPERTIES - IMPORTED_LOCATION /usr/lib64/libexpat.so) + IMPORTED_LOCATION ${LIBEXPAT_INSTALL_DIR}/lib64/libexpat.a) + add_dependencies(expat libexpat_project) add_subdirectory(library) install(FILES LICENSE.rst docs/changelog.txt DESTINATION ${DFHACK_USERDOC_DESTINATION}) diff --git a/depends/CMakeLists.txt b/depends/CMakeLists.txt index d5669080e..9735e7fc7 100644 --- a/depends/CMakeLists.txt +++ b/depends/CMakeLists.txt @@ -18,6 +18,15 @@ option(CLSOCKET_DEP_ONLY "Build for use inside other CMake projects as dependenc add_subdirectory(clsocket) ide_folder(clsocket "Depends") +set(LIBEXPAT_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libexpat) +ExternalProject_Add(libexpat_project + PREFIX libexpat + SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/libexpat/expat + INSTALL_DIR ${LIBEXPAT_INSTALL_DIR} + BUILD_BYPRODUCTS ${LIBEXPAT_INSTALL_DIR}/lib64/libexpat.a + CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${LIBEXPAT_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DEXPAT_BUILD_EXAMPLES=OFF -DEXPAT_BUILD_TESTS=OFF -DEXPAT_BUILD_TOOLS=OFF -DEXPAT_SHARED_LIBS=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON +) + set(LIBZIP_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libzip) ExternalProject_Add(libzip_project PREFIX libzip @@ -33,7 +42,7 @@ ExternalProject_Add(xlsxio_project SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/xlsxio INSTALL_DIR ${XLSXIO_INSTALL_DIR} BUILD_BYPRODUCTS ${XLSXIO_INSTALL_DIR}/lib/libxlsxio_read.a - CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${XLSXIO_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_STATIC=ON -DBUILD_SHARED=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_TOOLS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_DOCUMENTATION=OFF -DWITH_LIBZIP=ON -DZLIB_DIR=${ZLIB_DIR} + CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${XLSXIO_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_STATIC=ON -DBUILD_SHARED=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_TOOLS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_DOCUMENTATION=OFF -DWITH_LIBZIP=ON -DZLIB_DIR=${ZLIB_DIR} -DLIBZIP_DIR=${LIBZIP_INSTALL_DIR} -DEXPAT_DIR=${LIBEXPAT_INSTALL_DIR} ) - #-DLIBZIP_DIR= - #-DEXPAT_DIR= +add_dependencies(xlsxio_project libexpat_project) +add_dependencies(xlsxio_project libzip_project) From 8753f9feaff9df8367e6031e2cc299014dac1d55 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sat, 8 Aug 2020 20:42:03 -0700 Subject: [PATCH 13/61] use correct lib dir on 32-bit --- CMakeLists.txt | 15 +++++++++------ depends/CMakeLists.txt | 10 ++++++++-- 2 files changed, 17 insertions(+), 8 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 5dcd87b56..dbd028a36 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -419,26 +419,29 @@ endif() if(BUILD_LIBRARY) ExternalProject_Get_Property(xlsxio_project INSTALL_DIR) set(XLSXIO_INSTALL_DIR ${INSTALL_DIR}) + ExternalProject_Get_Property(xlsxio_project BUILD_BYPRODUCTS) + set(XLSXIO_LIB ${BUILD_BYPRODUCTS}) include_directories(${XLSXIO_INSTALL_DIR}/include) add_library(xlsxio_read STATIC IMPORTED) - set_target_properties(xlsxio_read PROPERTIES - IMPORTED_LOCATION ${XLSXIO_INSTALL_DIR}/lib/libxlsxio_read.a) + set_target_properties(xlsxio_read PROPERTIES IMPORTED_LOCATION ${XLSXIO_LIB}) add_dependencies(xlsxio_read xlsxio_project) ExternalProject_Get_Property(libzip_project INSTALL_DIR) set(LIBZIP_INSTALL_DIR ${INSTALL_DIR}) + ExternalProject_Get_Property(libzip_project BUILD_BYPRODUCTS) + set(LIBZIP_LIB ${BUILD_BYPRODUCTS}) include_directories(${LIBZIP_INSTALL_DIR}/include) add_library(zip STATIC IMPORTED) - set_target_properties(zip PROPERTIES - IMPORTED_LOCATION ${LIBZIP_INSTALL_DIR}/lib64/libzip.a) + set_target_properties(zip PROPERTIES IMPORTED_LOCATION ${LIBZIP_LIB}) add_dependencies(zip libzip_project) ExternalProject_Get_Property(libexpat_project INSTALL_DIR) set(LIBEXPAT_INSTALL_DIR ${INSTALL_DIR}) + ExternalProject_Get_Property(libexpat_project BUILD_BYPRODUCTS) + set(LIBEXPAT_LIB ${BUILD_BYPRODUCTS}) include_directories(${LIBEXPAT_INSTALL_DIR}/include) add_library(expat STATIC IMPORTED) - set_target_properties(expat PROPERTIES - IMPORTED_LOCATION ${LIBEXPAT_INSTALL_DIR}/lib64/libexpat.a) + set_target_properties(expat PROPERTIES IMPORTED_LOCATION ${LIBEXPAT_LIB}) add_dependencies(expat libexpat_project) add_subdirectory(library) diff --git a/depends/CMakeLists.txt b/depends/CMakeLists.txt index 9735e7fc7..4b11f4110 100644 --- a/depends/CMakeLists.txt +++ b/depends/CMakeLists.txt @@ -18,12 +18,18 @@ option(CLSOCKET_DEP_ONLY "Build for use inside other CMake projects as dependenc add_subdirectory(clsocket) ide_folder(clsocket "Depends") +if("${DFHACK_BUILD_ARCH}" STREQUAL "64") + set(OUTPUT_LIB_DIR "lib64") +else() + set(OUTPUT_LIB_DIR "lib") +endif() + set(LIBEXPAT_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libexpat) ExternalProject_Add(libexpat_project PREFIX libexpat SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/libexpat/expat INSTALL_DIR ${LIBEXPAT_INSTALL_DIR} - BUILD_BYPRODUCTS ${LIBEXPAT_INSTALL_DIR}/lib64/libexpat.a + BUILD_BYPRODUCTS ${LIBEXPAT_INSTALL_DIR}/${OUTPUT_LIB_DIR}/libexpat.a CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${LIBEXPAT_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DEXPAT_BUILD_EXAMPLES=OFF -DEXPAT_BUILD_TESTS=OFF -DEXPAT_BUILD_TOOLS=OFF -DEXPAT_SHARED_LIBS=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON ) @@ -32,7 +38,7 @@ ExternalProject_Add(libzip_project PREFIX libzip SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/libzip INSTALL_DIR ${LIBZIP_INSTALL_DIR} - BUILD_BYPRODUCTS ${LIBZIP_INSTALL_DIR}/lib64/libzip.a + BUILD_BYPRODUCTS ${LIBZIP_INSTALL_DIR}/${OUTPUT_LIB_DIR}/libzip.a CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${LIBZIP_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_DOC=OFF -DBUILD_EXAMPLES=OFF -DBUILD_REGRESS=OFF -DBUILD_SHARED_LIBS=OFF -DBUILD_TOOLS=OFF -DENABLE_BZIP2=OFF -DENABLE_COMMONCRYPTO=OFF -DENABLE_GNUTLS=OFF -DENABLE_LZMA=OFF -DENABLE_MBEDTLS=OFF -DENABLE_OPENSSL=OFF -DENABLE_WINDOWS_CRYPTO=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON ) From d65504eb828e76bdff7348c15e3d6a1629942524 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sat, 8 Aug 2020 21:19:38 -0700 Subject: [PATCH 14/61] ensure CFLAGS and CXXFLAGS get propagated to deps --- depends/CMakeLists.txt | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/depends/CMakeLists.txt b/depends/CMakeLists.txt index 4b11f4110..2e2872d20 100644 --- a/depends/CMakeLists.txt +++ b/depends/CMakeLists.txt @@ -18,19 +18,13 @@ option(CLSOCKET_DEP_ONLY "Build for use inside other CMake projects as dependenc add_subdirectory(clsocket) ide_folder(clsocket "Depends") -if("${DFHACK_BUILD_ARCH}" STREQUAL "64") - set(OUTPUT_LIB_DIR "lib64") -else() - set(OUTPUT_LIB_DIR "lib") -endif() - set(LIBEXPAT_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libexpat) ExternalProject_Add(libexpat_project PREFIX libexpat SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/libexpat/expat INSTALL_DIR ${LIBEXPAT_INSTALL_DIR} - BUILD_BYPRODUCTS ${LIBEXPAT_INSTALL_DIR}/${OUTPUT_LIB_DIR}/libexpat.a - CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${LIBEXPAT_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DEXPAT_BUILD_EXAMPLES=OFF -DEXPAT_BUILD_TESTS=OFF -DEXPAT_BUILD_TOOLS=OFF -DEXPAT_SHARED_LIBS=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON + BUILD_BYPRODUCTS ${LIBEXPAT_INSTALL_DIR}/lib/libexpat.a + CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${LIBEXPAT_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DEXPAT_BUILD_EXAMPLES=OFF -DEXPAT_BUILD_TESTS=OFF -DEXPAT_BUILD_TOOLS=OFF -DEXPAT_SHARED_LIBS=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS} -DCMAKE_C_FLAGS=${CMAKE_C_FLAGS} ) set(LIBZIP_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libzip) @@ -38,8 +32,8 @@ ExternalProject_Add(libzip_project PREFIX libzip SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/libzip INSTALL_DIR ${LIBZIP_INSTALL_DIR} - BUILD_BYPRODUCTS ${LIBZIP_INSTALL_DIR}/${OUTPUT_LIB_DIR}/libzip.a - CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${LIBZIP_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_DOC=OFF -DBUILD_EXAMPLES=OFF -DBUILD_REGRESS=OFF -DBUILD_SHARED_LIBS=OFF -DBUILD_TOOLS=OFF -DENABLE_BZIP2=OFF -DENABLE_COMMONCRYPTO=OFF -DENABLE_GNUTLS=OFF -DENABLE_LZMA=OFF -DENABLE_MBEDTLS=OFF -DENABLE_OPENSSL=OFF -DENABLE_WINDOWS_CRYPTO=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON + BUILD_BYPRODUCTS ${LIBZIP_INSTALL_DIR}/lib/libzip.a + CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${LIBZIP_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_DOC=OFF -DBUILD_EXAMPLES=OFF -DBUILD_REGRESS=OFF -DBUILD_SHARED_LIBS=OFF -DBUILD_TOOLS=OFF -DENABLE_BZIP2=OFF -DENABLE_COMMONCRYPTO=OFF -DENABLE_GNUTLS=OFF -DENABLE_LZMA=OFF -DENABLE_MBEDTLS=OFF -DENABLE_OPENSSL=OFF -DENABLE_WINDOWS_CRYPTO=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS} -DCMAKE_C_FLAGS=${CMAKE_C_FLAGS} ) set(XLSXIO_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/xlsxio) @@ -48,7 +42,7 @@ ExternalProject_Add(xlsxio_project SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/xlsxio INSTALL_DIR ${XLSXIO_INSTALL_DIR} BUILD_BYPRODUCTS ${XLSXIO_INSTALL_DIR}/lib/libxlsxio_read.a - CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${XLSXIO_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_STATIC=ON -DBUILD_SHARED=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_TOOLS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_DOCUMENTATION=OFF -DWITH_LIBZIP=ON -DZLIB_DIR=${ZLIB_DIR} -DLIBZIP_DIR=${LIBZIP_INSTALL_DIR} -DEXPAT_DIR=${LIBEXPAT_INSTALL_DIR} + CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${XLSXIO_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_STATIC=ON -DBUILD_SHARED=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_TOOLS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_DOCUMENTATION=OFF -DWITH_LIBZIP=ON -DZLIB_DIR=${ZLIB_DIR} -DLIBZIP_DIR=${LIBZIP_INSTALL_DIR} -DEXPAT_DIR=${LIBEXPAT_INSTALL_DIR} -DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS} -DCMAKE_C_FLAGS=${CMAKE_C_FLAGS} ) add_dependencies(xlsxio_project libexpat_project) add_dependencies(xlsxio_project libzip_project) From 1cfe2735d3eb59a006efb110641954f0fd1e1b9a Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sat, 8 Aug 2020 22:03:42 -0700 Subject: [PATCH 15/61] build deps in verbose mode for debugging --- depends/CMakeLists.txt | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/depends/CMakeLists.txt b/depends/CMakeLists.txt index 2e2872d20..2e5dde062 100644 --- a/depends/CMakeLists.txt +++ b/depends/CMakeLists.txt @@ -24,7 +24,7 @@ ExternalProject_Add(libexpat_project SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/libexpat/expat INSTALL_DIR ${LIBEXPAT_INSTALL_DIR} BUILD_BYPRODUCTS ${LIBEXPAT_INSTALL_DIR}/lib/libexpat.a - CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${LIBEXPAT_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DEXPAT_BUILD_EXAMPLES=OFF -DEXPAT_BUILD_TESTS=OFF -DEXPAT_BUILD_TOOLS=OFF -DEXPAT_SHARED_LIBS=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS} -DCMAKE_C_FLAGS=${CMAKE_C_FLAGS} + CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${LIBEXPAT_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DEXPAT_BUILD_EXAMPLES=OFF -DEXPAT_BUILD_TESTS=OFF -DEXPAT_BUILD_TOOLS=OFF -DEXPAT_SHARED_LIBS=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS} -DCMAKE_C_FLAGS=${CMAKE_C_FLAGS} -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON ) set(LIBZIP_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libzip) @@ -33,7 +33,7 @@ ExternalProject_Add(libzip_project SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/libzip INSTALL_DIR ${LIBZIP_INSTALL_DIR} BUILD_BYPRODUCTS ${LIBZIP_INSTALL_DIR}/lib/libzip.a - CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${LIBZIP_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_DOC=OFF -DBUILD_EXAMPLES=OFF -DBUILD_REGRESS=OFF -DBUILD_SHARED_LIBS=OFF -DBUILD_TOOLS=OFF -DENABLE_BZIP2=OFF -DENABLE_COMMONCRYPTO=OFF -DENABLE_GNUTLS=OFF -DENABLE_LZMA=OFF -DENABLE_MBEDTLS=OFF -DENABLE_OPENSSL=OFF -DENABLE_WINDOWS_CRYPTO=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS} -DCMAKE_C_FLAGS=${CMAKE_C_FLAGS} + CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${LIBZIP_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_DOC=OFF -DBUILD_EXAMPLES=OFF -DBUILD_REGRESS=OFF -DBUILD_SHARED_LIBS=OFF -DBUILD_TOOLS=OFF -DENABLE_BZIP2=OFF -DENABLE_COMMONCRYPTO=OFF -DENABLE_GNUTLS=OFF -DENABLE_LZMA=OFF -DENABLE_MBEDTLS=OFF -DENABLE_OPENSSL=OFF -DENABLE_WINDOWS_CRYPTO=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_C_FLAGS=${CMAKE_C_FLAGS} -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON ) set(XLSXIO_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/xlsxio) @@ -42,7 +42,7 @@ ExternalProject_Add(xlsxio_project SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/xlsxio INSTALL_DIR ${XLSXIO_INSTALL_DIR} BUILD_BYPRODUCTS ${XLSXIO_INSTALL_DIR}/lib/libxlsxio_read.a - CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${XLSXIO_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_STATIC=ON -DBUILD_SHARED=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_TOOLS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_DOCUMENTATION=OFF -DWITH_LIBZIP=ON -DZLIB_DIR=${ZLIB_DIR} -DLIBZIP_DIR=${LIBZIP_INSTALL_DIR} -DEXPAT_DIR=${LIBEXPAT_INSTALL_DIR} -DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS} -DCMAKE_C_FLAGS=${CMAKE_C_FLAGS} + CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${XLSXIO_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_STATIC=ON -DBUILD_SHARED=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_TOOLS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_DOCUMENTATION=OFF -DWITH_LIBZIP=ON -DZLIB_DIR=${ZLIB_DIR} -DLIBZIP_DIR=${LIBZIP_INSTALL_DIR} -DEXPAT_DIR=${LIBEXPAT_INSTALL_DIR} -DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS} -DCMAKE_C_FLAGS=${CMAKE_C_FLAGS} -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON ) add_dependencies(xlsxio_project libexpat_project) add_dependencies(xlsxio_project libzip_project) From 8d157012d0a43c1a2fe0518a274cd28598c50fbf Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sat, 8 Aug 2020 22:04:49 -0700 Subject: [PATCH 16/61] build dfhack with verbosity on too --- library/CMakeLists.txt | 2 ++ 1 file changed, 2 insertions(+) diff --git a/library/CMakeLists.txt b/library/CMakeLists.txt index 583c8768f..c78f7f58e 100644 --- a/library/CMakeLists.txt +++ b/library/CMakeLists.txt @@ -4,6 +4,8 @@ cmake_minimum_required(VERSION 2.8.12) # prevent CMake warnings about INTERFACE_LINK_LIBRARIES vs LINK_INTERFACE_LIBRARIES cmake_policy(SET CMP0022 NEW) +set(CMAKE_VERBOSE_MAKEFILE ON) + # build options option(BUILD_DEVEL "Install/package files required for development(For SDK)." OFF) if(UNIX) From b0cb40c5ea2941287450c1fd9352a40c2e50a49d Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sat, 8 Aug 2020 22:42:50 -0700 Subject: [PATCH 17/61] pass the cmake environment to deps --- depends/CMakeLists.txt | 16 +++++++++++++--- library/CMakeLists.txt | 2 -- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/depends/CMakeLists.txt b/depends/CMakeLists.txt index 2e5dde062..87e9d74bf 100644 --- a/depends/CMakeLists.txt +++ b/depends/CMakeLists.txt @@ -18,13 +18,23 @@ option(CLSOCKET_DEP_ONLY "Build for use inside other CMake projects as dependenc add_subdirectory(clsocket) ide_folder(clsocket "Depends") +# assemble environment args to pass on to dependency projects +get_cmake_property(vars CACHE_VARIABLES) +foreach(var ${vars}) + get_property(currentHelpString CACHE "${var}" PROPERTY HELPSTRING) + if("${currentHelpString}" MATCHES "No help, variable specified on the command line." OR "${currentHelpString}" STREQUAL "") + message("${var} = [${${var}}] -- ${currentHelpString}") # uncomment to see the variables being processed + list(APPEND CL_ARGS "-D${var}=${${var}}") + endif() +endforeach() + set(LIBEXPAT_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libexpat) ExternalProject_Add(libexpat_project PREFIX libexpat SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/libexpat/expat INSTALL_DIR ${LIBEXPAT_INSTALL_DIR} BUILD_BYPRODUCTS ${LIBEXPAT_INSTALL_DIR}/lib/libexpat.a - CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${LIBEXPAT_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DEXPAT_BUILD_EXAMPLES=OFF -DEXPAT_BUILD_TESTS=OFF -DEXPAT_BUILD_TOOLS=OFF -DEXPAT_SHARED_LIBS=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS} -DCMAKE_C_FLAGS=${CMAKE_C_FLAGS} -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON + CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${LIBEXPAT_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DEXPAT_BUILD_EXAMPLES=OFF -DEXPAT_BUILD_TESTS=OFF -DEXPAT_BUILD_TOOLS=OFF -DEXPAT_SHARED_LIBS=OFF -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON ) set(LIBZIP_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libzip) @@ -33,7 +43,7 @@ ExternalProject_Add(libzip_project SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/libzip INSTALL_DIR ${LIBZIP_INSTALL_DIR} BUILD_BYPRODUCTS ${LIBZIP_INSTALL_DIR}/lib/libzip.a - CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${LIBZIP_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_DOC=OFF -DBUILD_EXAMPLES=OFF -DBUILD_REGRESS=OFF -DBUILD_SHARED_LIBS=OFF -DBUILD_TOOLS=OFF -DENABLE_BZIP2=OFF -DENABLE_COMMONCRYPTO=OFF -DENABLE_GNUTLS=OFF -DENABLE_LZMA=OFF -DENABLE_MBEDTLS=OFF -DENABLE_OPENSSL=OFF -DENABLE_WINDOWS_CRYPTO=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_C_FLAGS=${CMAKE_C_FLAGS} -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON + CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${LIBZIP_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_DOC=OFF -DBUILD_EXAMPLES=OFF -DBUILD_REGRESS=OFF -DBUILD_SHARED_LIBS=OFF -DBUILD_TOOLS=OFF -DENABLE_BZIP2=OFF -DENABLE_COMMONCRYPTO=OFF -DENABLE_GNUTLS=OFF -DENABLE_LZMA=OFF -DENABLE_MBEDTLS=OFF -DENABLE_OPENSSL=OFF -DENABLE_WINDOWS_CRYPTO=OFF -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON ) set(XLSXIO_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/xlsxio) @@ -42,7 +52,7 @@ ExternalProject_Add(xlsxio_project SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/xlsxio INSTALL_DIR ${XLSXIO_INSTALL_DIR} BUILD_BYPRODUCTS ${XLSXIO_INSTALL_DIR}/lib/libxlsxio_read.a - CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${XLSXIO_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_STATIC=ON -DBUILD_SHARED=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_TOOLS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_DOCUMENTATION=OFF -DWITH_LIBZIP=ON -DZLIB_DIR=${ZLIB_DIR} -DLIBZIP_DIR=${LIBZIP_INSTALL_DIR} -DEXPAT_DIR=${LIBEXPAT_INSTALL_DIR} -DCMAKE_CXX_FLAGS=${CMAKE_CXX_FLAGS} -DCMAKE_C_FLAGS=${CMAKE_C_FLAGS} -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON + CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${XLSXIO_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_STATIC=ON -DBUILD_SHARED=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_TOOLS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_DOCUMENTATION=OFF -DWITH_LIBZIP=ON -DZLIB_DIR=${ZLIB_DIR} -DLIBZIP_DIR=${LIBZIP_INSTALL_DIR} -DEXPAT_DIR=${LIBEXPAT_INSTALL_DIR} -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON ) add_dependencies(xlsxio_project libexpat_project) add_dependencies(xlsxio_project libzip_project) diff --git a/library/CMakeLists.txt b/library/CMakeLists.txt index c78f7f58e..583c8768f 100644 --- a/library/CMakeLists.txt +++ b/library/CMakeLists.txt @@ -4,8 +4,6 @@ cmake_minimum_required(VERSION 2.8.12) # prevent CMake warnings about INTERFACE_LINK_LIBRARIES vs LINK_INTERFACE_LIBRARIES cmake_policy(SET CMP0022 NEW) -set(CMAKE_VERBOSE_MAKEFILE ON) - # build options option(BUILD_DEVEL "Install/package files required for development(For SDK)." OFF) if(UNIX) From 51794d91eda59faeadd1190b96a55f922f881b07 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sat, 8 Aug 2020 23:55:24 -0700 Subject: [PATCH 18/61] really get the env vars --- depends/CMakeLists.txt | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/depends/CMakeLists.txt b/depends/CMakeLists.txt index 87e9d74bf..f2ccff398 100644 --- a/depends/CMakeLists.txt +++ b/depends/CMakeLists.txt @@ -21,12 +21,12 @@ ide_folder(clsocket "Depends") # assemble environment args to pass on to dependency projects get_cmake_property(vars CACHE_VARIABLES) foreach(var ${vars}) - get_property(currentHelpString CACHE "${var}" PROPERTY HELPSTRING) - if("${currentHelpString}" MATCHES "No help, variable specified on the command line." OR "${currentHelpString}" STREQUAL "") - message("${var} = [${${var}}] -- ${currentHelpString}") # uncomment to see the variables being processed - list(APPEND CL_ARGS "-D${var}=${${var}}") - endif() + if(var MATCHES "^CMAKE_" AND NOT var MATCHES "^CMAKE_CACHE" AND NOT var MATCHES "^CMAKE_HOME" AND NOT var MATCHES "^CMAKE_PROJECT") + message("${var} = [${${var}}]") + list(APPEND CL_ARGS "-D${var}=${${var}}") + endif() endforeach() +message("final CL_ARGS = ${CL_ARGS}") set(LIBEXPAT_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libexpat) ExternalProject_Add(libexpat_project From bb988d25e674ede1d2fe74af18a0885708d7f156 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sun, 9 Aug 2020 00:24:25 -0700 Subject: [PATCH 19/61] try to get the library suffix right for windows --- depends/CMakeLists.txt | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/depends/CMakeLists.txt b/depends/CMakeLists.txt index f2ccff398..7e0a4027b 100644 --- a/depends/CMakeLists.txt +++ b/depends/CMakeLists.txt @@ -21,19 +21,31 @@ ide_folder(clsocket "Depends") # assemble environment args to pass on to dependency projects get_cmake_property(vars CACHE_VARIABLES) foreach(var ${vars}) - if(var MATCHES "^CMAKE_" AND NOT var MATCHES "^CMAKE_CACHE" AND NOT var MATCHES "^CMAKE_HOME" AND NOT var MATCHES "^CMAKE_PROJECT") + if(var MATCHES "^CMAKE_" + AND NOT var MATCHES "^CMAKE_CACHE" + AND NOT var MATCHES "^CMAKE_HOME" + AND NOT var MATCHES "^CMAKE_PROJECT") + message("${var} = [${${var}}]") + list(APPEND CL_ARGS "-D${var}=${${var}}") + endif() + if(var MATCHES "^ZLIB") message("${var} = [${${var}}]") list(APPEND CL_ARGS "-D${var}=${${var}}") endif() endforeach() -message("final CL_ARGS = ${CL_ARGS}") + +if(WIN32) + set(LIB_SUFFIX "MD.lib") +else() + set(LIB_SUFFIX ".a") +endif() set(LIBEXPAT_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libexpat) ExternalProject_Add(libexpat_project PREFIX libexpat SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/libexpat/expat INSTALL_DIR ${LIBEXPAT_INSTALL_DIR} - BUILD_BYPRODUCTS ${LIBEXPAT_INSTALL_DIR}/lib/libexpat.a + BUILD_BYPRODUCTS ${LIBEXPAT_INSTALL_DIR}/lib/libexpat${LIB_SUFFIX} CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${LIBEXPAT_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DEXPAT_BUILD_EXAMPLES=OFF -DEXPAT_BUILD_TESTS=OFF -DEXPAT_BUILD_TOOLS=OFF -DEXPAT_SHARED_LIBS=OFF -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON ) @@ -42,7 +54,7 @@ ExternalProject_Add(libzip_project PREFIX libzip SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/libzip INSTALL_DIR ${LIBZIP_INSTALL_DIR} - BUILD_BYPRODUCTS ${LIBZIP_INSTALL_DIR}/lib/libzip.a + BUILD_BYPRODUCTS ${LIBZIP_INSTALL_DIR}/lib/libzip${LIB_SUFFIX} CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${LIBZIP_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_DOC=OFF -DBUILD_EXAMPLES=OFF -DBUILD_REGRESS=OFF -DBUILD_SHARED_LIBS=OFF -DBUILD_TOOLS=OFF -DENABLE_BZIP2=OFF -DENABLE_COMMONCRYPTO=OFF -DENABLE_GNUTLS=OFF -DENABLE_LZMA=OFF -DENABLE_MBEDTLS=OFF -DENABLE_OPENSSL=OFF -DENABLE_WINDOWS_CRYPTO=OFF -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON ) @@ -51,7 +63,7 @@ ExternalProject_Add(xlsxio_project PREFIX xlsxio SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/xlsxio INSTALL_DIR ${XLSXIO_INSTALL_DIR} - BUILD_BYPRODUCTS ${XLSXIO_INSTALL_DIR}/lib/libxlsxio_read.a + BUILD_BYPRODUCTS ${XLSXIO_INSTALL_DIR}/lib/libxlsxio_read${LIB_SUFFIX} CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${XLSXIO_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_STATIC=ON -DBUILD_SHARED=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_TOOLS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_DOCUMENTATION=OFF -DWITH_LIBZIP=ON -DZLIB_DIR=${ZLIB_DIR} -DLIBZIP_DIR=${LIBZIP_INSTALL_DIR} -DEXPAT_DIR=${LIBEXPAT_INSTALL_DIR} -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON ) add_dependencies(xlsxio_project libexpat_project) From 43bcec4a57c55ed2cce404013820d378aaf973f1 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sun, 9 Aug 2020 00:46:50 -0700 Subject: [PATCH 20/61] move libzip to older version although they declare that they only need cmake 3.0.2, they started using add_compile_definitions(), which was added in cmake 3.12 git checkout 29b881d fixes this --- depends/libzip | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/depends/libzip b/depends/libzip index 66e496489..29b881d28 160000 --- a/depends/libzip +++ b/depends/libzip @@ -1 +1 @@ -Subproject commit 66e496489bdae81bfda8b0088172871d8fda0032 +Subproject commit 29b881d286f43189ff7392f609da78daa80c0606 From c7cf7fdd30fc2bef7193464cecc503008a6698c4 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sun, 9 Aug 2020 02:00:06 -0700 Subject: [PATCH 21/61] define EXPAT_LIBRARIES for xlsxio --- depends/CMakeLists.txt | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/depends/CMakeLists.txt b/depends/CMakeLists.txt index 7e0a4027b..21b244272 100644 --- a/depends/CMakeLists.txt +++ b/depends/CMakeLists.txt @@ -41,11 +41,12 @@ else() endif() set(LIBEXPAT_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libexpat) +set(LIBEXPAT_OUTPUT_LIB ${LIBEXPAT_INSTALL_DIR}/lib/libexpat${LIB_SUFFIX}) ExternalProject_Add(libexpat_project PREFIX libexpat SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/libexpat/expat INSTALL_DIR ${LIBEXPAT_INSTALL_DIR} - BUILD_BYPRODUCTS ${LIBEXPAT_INSTALL_DIR}/lib/libexpat${LIB_SUFFIX} + BUILD_BYPRODUCTS ${LIBEXPAT_OUTPUT_LIB} CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${LIBEXPAT_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DEXPAT_BUILD_EXAMPLES=OFF -DEXPAT_BUILD_TESTS=OFF -DEXPAT_BUILD_TOOLS=OFF -DEXPAT_SHARED_LIBS=OFF -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON ) @@ -64,7 +65,7 @@ ExternalProject_Add(xlsxio_project SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/xlsxio INSTALL_DIR ${XLSXIO_INSTALL_DIR} BUILD_BYPRODUCTS ${XLSXIO_INSTALL_DIR}/lib/libxlsxio_read${LIB_SUFFIX} - CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${XLSXIO_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_STATIC=ON -DBUILD_SHARED=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_TOOLS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_DOCUMENTATION=OFF -DWITH_LIBZIP=ON -DZLIB_DIR=${ZLIB_DIR} -DLIBZIP_DIR=${LIBZIP_INSTALL_DIR} -DEXPAT_DIR=${LIBEXPAT_INSTALL_DIR} -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON + CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${XLSXIO_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_STATIC=ON -DBUILD_SHARED=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_TOOLS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_DOCUMENTATION=OFF -DWITH_LIBZIP=ON -DZLIB_DIR=${ZLIB_DIR} -DLIBZIP_DIR=${LIBZIP_INSTALL_DIR} -DEXPAT_DIR=${LIBEXPAT_INSTALL_DIR} -DEXPAT_LIBRARIES=${LIBEXPAT_OUTPUT_LIB} -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON ) add_dependencies(xlsxio_project libexpat_project) add_dependencies(xlsxio_project libzip_project) From 204240f3e34d0d1522157cf9306117a2c80063e2 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sun, 9 Aug 2020 02:14:37 -0700 Subject: [PATCH 22/61] specify expat include dir for xlsxio --- depends/CMakeLists.txt | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/depends/CMakeLists.txt b/depends/CMakeLists.txt index 21b244272..9f2e98763 100644 --- a/depends/CMakeLists.txt +++ b/depends/CMakeLists.txt @@ -35,13 +35,15 @@ foreach(var ${vars}) endforeach() if(WIN32) - set(LIB_SUFFIX "MD.lib") + set(EXPAT_LIB_SUFFIX "MD.lib") + set(LIB_SUFFIX ".lib") else() + set(EXPAT_LIB_SUFFIX ".a") set(LIB_SUFFIX ".a") endif() set(LIBEXPAT_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libexpat) -set(LIBEXPAT_OUTPUT_LIB ${LIBEXPAT_INSTALL_DIR}/lib/libexpat${LIB_SUFFIX}) +set(LIBEXPAT_OUTPUT_LIB ${LIBEXPAT_INSTALL_DIR}/lib/libexpat${EXPAT_LIB_SUFFIX}) ExternalProject_Add(libexpat_project PREFIX libexpat SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/libexpat/expat @@ -65,7 +67,7 @@ ExternalProject_Add(xlsxio_project SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/xlsxio INSTALL_DIR ${XLSXIO_INSTALL_DIR} BUILD_BYPRODUCTS ${XLSXIO_INSTALL_DIR}/lib/libxlsxio_read${LIB_SUFFIX} - CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${XLSXIO_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_STATIC=ON -DBUILD_SHARED=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_TOOLS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_DOCUMENTATION=OFF -DWITH_LIBZIP=ON -DZLIB_DIR=${ZLIB_DIR} -DLIBZIP_DIR=${LIBZIP_INSTALL_DIR} -DEXPAT_DIR=${LIBEXPAT_INSTALL_DIR} -DEXPAT_LIBRARIES=${LIBEXPAT_OUTPUT_LIB} -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON + CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${XLSXIO_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_STATIC=ON -DBUILD_SHARED=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_TOOLS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_DOCUMENTATION=OFF -DWITH_LIBZIP=ON -DZLIB_DIR=${ZLIB_DIR} -DLIBZIP_DIR=${LIBZIP_INSTALL_DIR} -DEXPAT_DIR=${LIBEXPAT_INSTALL_DIR} -DEXPAT_LIBRARIES=${LIBEXPAT_OUTPUT_LIB} -DEXPAT_INCLUDE_DIR=${LIBEXPAT_INSTALL_DIR}/include -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON ) add_dependencies(xlsxio_project libexpat_project) add_dependencies(xlsxio_project libzip_project) From 79415cf097b1de8e9b33734d347b44259d47fb11 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sun, 9 Aug 2020 02:24:05 -0700 Subject: [PATCH 23/61] clean up dep lib vars --- CMakeLists.txt | 15 --------------- depends/CMakeLists.txt | 18 +++++++++++++----- 2 files changed, 13 insertions(+), 20 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index dbd028a36..e5e88ceb3 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -417,32 +417,17 @@ endif() # build the lib itself if(BUILD_LIBRARY) - ExternalProject_Get_Property(xlsxio_project INSTALL_DIR) - set(XLSXIO_INSTALL_DIR ${INSTALL_DIR}) - ExternalProject_Get_Property(xlsxio_project BUILD_BYPRODUCTS) - set(XLSXIO_LIB ${BUILD_BYPRODUCTS}) include_directories(${XLSXIO_INSTALL_DIR}/include) add_library(xlsxio_read STATIC IMPORTED) set_target_properties(xlsxio_read PROPERTIES IMPORTED_LOCATION ${XLSXIO_LIB}) - add_dependencies(xlsxio_read xlsxio_project) - ExternalProject_Get_Property(libzip_project INSTALL_DIR) - set(LIBZIP_INSTALL_DIR ${INSTALL_DIR}) - ExternalProject_Get_Property(libzip_project BUILD_BYPRODUCTS) - set(LIBZIP_LIB ${BUILD_BYPRODUCTS}) include_directories(${LIBZIP_INSTALL_DIR}/include) add_library(zip STATIC IMPORTED) set_target_properties(zip PROPERTIES IMPORTED_LOCATION ${LIBZIP_LIB}) - add_dependencies(zip libzip_project) - ExternalProject_Get_Property(libexpat_project INSTALL_DIR) - set(LIBEXPAT_INSTALL_DIR ${INSTALL_DIR}) - ExternalProject_Get_Property(libexpat_project BUILD_BYPRODUCTS) - set(LIBEXPAT_LIB ${BUILD_BYPRODUCTS}) include_directories(${LIBEXPAT_INSTALL_DIR}/include) add_library(expat STATIC IMPORTED) set_target_properties(expat PROPERTIES IMPORTED_LOCATION ${LIBEXPAT_LIB}) - add_dependencies(expat libexpat_project) add_subdirectory(library) install(FILES LICENSE.rst docs/changelog.txt DESTINATION ${DFHACK_USERDOC_DESTINATION}) diff --git a/depends/CMakeLists.txt b/depends/CMakeLists.txt index 9f2e98763..475166285 100644 --- a/depends/CMakeLists.txt +++ b/depends/CMakeLists.txt @@ -43,31 +43,39 @@ else() endif() set(LIBEXPAT_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libexpat) -set(LIBEXPAT_OUTPUT_LIB ${LIBEXPAT_INSTALL_DIR}/lib/libexpat${EXPAT_LIB_SUFFIX}) +set(LIBEXPAT_INSTALL_DIR ${LIBEXPAT_INSTALL_DIR} PARENT_SCOPE) +set(LIBEXPAT_LIB ${LIBEXPAT_INSTALL_DIR}/lib/libexpat${EXPAT_LIB_SUFFIX}) +set(LIBEXPAT_LIB ${LIBEXPAT_LIB} PARENT_SCOPE) ExternalProject_Add(libexpat_project PREFIX libexpat SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/libexpat/expat INSTALL_DIR ${LIBEXPAT_INSTALL_DIR} - BUILD_BYPRODUCTS ${LIBEXPAT_OUTPUT_LIB} + BUILD_BYPRODUCTS ${LIBEXPAT_LIB} CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${LIBEXPAT_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DEXPAT_BUILD_EXAMPLES=OFF -DEXPAT_BUILD_TESTS=OFF -DEXPAT_BUILD_TOOLS=OFF -DEXPAT_SHARED_LIBS=OFF -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON ) set(LIBZIP_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libzip) +set(LIBZIP_INSTALL_DIR ${LIBZIP_INSTALL_DIR} PARENT_SCOPE) +set(LIBZIP_LIB ${LIBZIP_INSTALL_DIR}/lib/libzip${LIB_SUFFIX}) +set(LIBZIP_LIB ${LIBZIP_LIB} PARENT_SCOPE) ExternalProject_Add(libzip_project PREFIX libzip SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/libzip INSTALL_DIR ${LIBZIP_INSTALL_DIR} - BUILD_BYPRODUCTS ${LIBZIP_INSTALL_DIR}/lib/libzip${LIB_SUFFIX} + BUILD_BYPRODUCTS ${LIBZIP_LIB} CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${LIBZIP_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_DOC=OFF -DBUILD_EXAMPLES=OFF -DBUILD_REGRESS=OFF -DBUILD_SHARED_LIBS=OFF -DBUILD_TOOLS=OFF -DENABLE_BZIP2=OFF -DENABLE_COMMONCRYPTO=OFF -DENABLE_GNUTLS=OFF -DENABLE_LZMA=OFF -DENABLE_MBEDTLS=OFF -DENABLE_OPENSSL=OFF -DENABLE_WINDOWS_CRYPTO=OFF -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON ) set(XLSXIO_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/xlsxio) +set(XLSXIO_INSTALL_DIR ${XLSXIO_INSTALL_DIR} PARENT_SCOPE) +set(XLSXIO_LIB ${XLSXIO_INSTALL_DIR}/lib/libxlsxio_read${LIB_SUFFIX}) +set(XLSXIO_LIB ${XLSXIO_LIB} PARENT_SCOPE) ExternalProject_Add(xlsxio_project PREFIX xlsxio SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/xlsxio INSTALL_DIR ${XLSXIO_INSTALL_DIR} - BUILD_BYPRODUCTS ${XLSXIO_INSTALL_DIR}/lib/libxlsxio_read${LIB_SUFFIX} - CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${XLSXIO_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_STATIC=ON -DBUILD_SHARED=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_TOOLS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_DOCUMENTATION=OFF -DWITH_LIBZIP=ON -DZLIB_DIR=${ZLIB_DIR} -DLIBZIP_DIR=${LIBZIP_INSTALL_DIR} -DEXPAT_DIR=${LIBEXPAT_INSTALL_DIR} -DEXPAT_LIBRARIES=${LIBEXPAT_OUTPUT_LIB} -DEXPAT_INCLUDE_DIR=${LIBEXPAT_INSTALL_DIR}/include -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON + BUILD_BYPRODUCTS ${XLSXIO_LIB} + CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${XLSXIO_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_STATIC=ON -DBUILD_SHARED=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_TOOLS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_DOCUMENTATION=OFF -DWITH_LIBZIP=ON -DZLIB_DIR=${ZLIB_DIR} -DLIBZIP_DIR=${LIBZIP_INSTALL_DIR} -DEXPAT_DIR=${LIBEXPAT_INSTALL_DIR} -DEXPAT_LIBRARIES=${LIBEXPAT_LIB} -DEXPAT_INCLUDE_DIR=${LIBEXPAT_INSTALL_DIR}/include -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON ) add_dependencies(xlsxio_project libexpat_project) add_dependencies(xlsxio_project libzip_project) From f04a9c3180b6a32f77e57a130a0636868a70ee7f Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sun, 9 Aug 2020 02:32:31 -0700 Subject: [PATCH 24/61] clean up logging; use xlsxio's misspelling of "DIRS" --- depends/CMakeLists.txt | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/depends/CMakeLists.txt b/depends/CMakeLists.txt index 475166285..808bb4304 100644 --- a/depends/CMakeLists.txt +++ b/depends/CMakeLists.txt @@ -25,11 +25,9 @@ foreach(var ${vars}) AND NOT var MATCHES "^CMAKE_CACHE" AND NOT var MATCHES "^CMAKE_HOME" AND NOT var MATCHES "^CMAKE_PROJECT") - message("${var} = [${${var}}]") list(APPEND CL_ARGS "-D${var}=${${var}}") endif() if(var MATCHES "^ZLIB") - message("${var} = [${${var}}]") list(APPEND CL_ARGS "-D${var}=${${var}}") endif() endforeach() @@ -51,7 +49,7 @@ ExternalProject_Add(libexpat_project SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/libexpat/expat INSTALL_DIR ${LIBEXPAT_INSTALL_DIR} BUILD_BYPRODUCTS ${LIBEXPAT_LIB} - CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${LIBEXPAT_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DEXPAT_BUILD_EXAMPLES=OFF -DEXPAT_BUILD_TESTS=OFF -DEXPAT_BUILD_TOOLS=OFF -DEXPAT_SHARED_LIBS=OFF -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON + CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${LIBEXPAT_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DEXPAT_BUILD_EXAMPLES=OFF -DEXPAT_BUILD_TESTS=OFF -DEXPAT_BUILD_TOOLS=OFF -DEXPAT_SHARED_LIBS=OFF -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_POSITION_INDEPENDENT_CODE=ON ) set(LIBZIP_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libzip) @@ -63,7 +61,7 @@ ExternalProject_Add(libzip_project SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/libzip INSTALL_DIR ${LIBZIP_INSTALL_DIR} BUILD_BYPRODUCTS ${LIBZIP_LIB} - CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${LIBZIP_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_DOC=OFF -DBUILD_EXAMPLES=OFF -DBUILD_REGRESS=OFF -DBUILD_SHARED_LIBS=OFF -DBUILD_TOOLS=OFF -DENABLE_BZIP2=OFF -DENABLE_COMMONCRYPTO=OFF -DENABLE_GNUTLS=OFF -DENABLE_LZMA=OFF -DENABLE_MBEDTLS=OFF -DENABLE_OPENSSL=OFF -DENABLE_WINDOWS_CRYPTO=OFF -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON + CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${LIBZIP_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_DOC=OFF -DBUILD_EXAMPLES=OFF -DBUILD_REGRESS=OFF -DBUILD_SHARED_LIBS=OFF -DBUILD_TOOLS=OFF -DENABLE_BZIP2=OFF -DENABLE_COMMONCRYPTO=OFF -DENABLE_GNUTLS=OFF -DENABLE_LZMA=OFF -DENABLE_MBEDTLS=OFF -DENABLE_OPENSSL=OFF -DENABLE_WINDOWS_CRYPTO=OFF -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_POSITION_INDEPENDENT_CODE=ON ) set(XLSXIO_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/xlsxio) @@ -75,7 +73,7 @@ ExternalProject_Add(xlsxio_project SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/xlsxio INSTALL_DIR ${XLSXIO_INSTALL_DIR} BUILD_BYPRODUCTS ${XLSXIO_LIB} - CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${XLSXIO_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_STATIC=ON -DBUILD_SHARED=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_TOOLS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_DOCUMENTATION=OFF -DWITH_LIBZIP=ON -DZLIB_DIR=${ZLIB_DIR} -DLIBZIP_DIR=${LIBZIP_INSTALL_DIR} -DEXPAT_DIR=${LIBEXPAT_INSTALL_DIR} -DEXPAT_LIBRARIES=${LIBEXPAT_LIB} -DEXPAT_INCLUDE_DIR=${LIBEXPAT_INSTALL_DIR}/include -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON + CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${XLSXIO_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_STATIC=ON -DBUILD_SHARED=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_TOOLS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_DOCUMENTATION=OFF -DWITH_LIBZIP=ON -DZLIB_DIR=${ZLIB_DIR} -DLIBZIP_DIR=${LIBZIP_INSTALL_DIR} -DEXPAT_DIR=${LIBEXPAT_INSTALL_DIR} -DEXPAT_LIBRARIES=${LIBEXPAT_LIB} -DEXPAT_INCLUDE_DIRS=${LIBEXPAT_INSTALL_DIR}/include -DCMAKE_POSITION_INDEPENDENT_CODE=ON ) add_dependencies(xlsxio_project libexpat_project) add_dependencies(xlsxio_project libzip_project) From 8913d43d6f8cfe8d3921ef3f05f7fae4ddce2947 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sun, 9 Aug 2020 02:59:22 -0700 Subject: [PATCH 25/61] ensure xlsxio_project gets built before dfhack --- CMakeLists.txt | 6 ++---- depends/CMakeLists.txt | 2 -- 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index e5e88ceb3..eeb8c8d12 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -420,17 +420,15 @@ if(BUILD_LIBRARY) include_directories(${XLSXIO_INSTALL_DIR}/include) add_library(xlsxio_read STATIC IMPORTED) set_target_properties(xlsxio_read PROPERTIES IMPORTED_LOCATION ${XLSXIO_LIB}) - - include_directories(${LIBZIP_INSTALL_DIR}/include) add_library(zip STATIC IMPORTED) set_target_properties(zip PROPERTIES IMPORTED_LOCATION ${LIBZIP_LIB}) - - include_directories(${LIBEXPAT_INSTALL_DIR}/include) add_library(expat STATIC IMPORTED) set_target_properties(expat PROPERTIES IMPORTED_LOCATION ${LIBEXPAT_LIB}) add_subdirectory(library) install(FILES LICENSE.rst docs/changelog.txt DESTINATION ${DFHACK_USERDOC_DESTINATION}) + + add_dependencies(dfhack xlsxio_project) endif() file(WRITE "${CMAKE_BINARY_DIR}/dfhack_setarch.txt" ${DFHACK_SETARCH}) diff --git a/depends/CMakeLists.txt b/depends/CMakeLists.txt index 808bb4304..0b747c157 100644 --- a/depends/CMakeLists.txt +++ b/depends/CMakeLists.txt @@ -41,7 +41,6 @@ else() endif() set(LIBEXPAT_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libexpat) -set(LIBEXPAT_INSTALL_DIR ${LIBEXPAT_INSTALL_DIR} PARENT_SCOPE) set(LIBEXPAT_LIB ${LIBEXPAT_INSTALL_DIR}/lib/libexpat${EXPAT_LIB_SUFFIX}) set(LIBEXPAT_LIB ${LIBEXPAT_LIB} PARENT_SCOPE) ExternalProject_Add(libexpat_project @@ -53,7 +52,6 @@ ExternalProject_Add(libexpat_project ) set(LIBZIP_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libzip) -set(LIBZIP_INSTALL_DIR ${LIBZIP_INSTALL_DIR} PARENT_SCOPE) set(LIBZIP_LIB ${LIBZIP_INSTALL_DIR}/lib/libzip${LIB_SUFFIX}) set(LIBZIP_LIB ${LIBZIP_LIB} PARENT_SCOPE) ExternalProject_Add(libzip_project From 7a0853025a0955190b9a53a87c85180d96cd627e Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sun, 9 Aug 2020 03:30:59 -0700 Subject: [PATCH 26/61] use non-standard xlsxio lib name on windows --- depends/CMakeLists.txt | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/depends/CMakeLists.txt b/depends/CMakeLists.txt index 0b747c157..db132a809 100644 --- a/depends/CMakeLists.txt +++ b/depends/CMakeLists.txt @@ -35,9 +35,11 @@ endforeach() if(WIN32) set(EXPAT_LIB_SUFFIX "MD.lib") set(LIB_SUFFIX ".lib") + set(XLSXIO_LIB_PREFIX "") else() set(EXPAT_LIB_SUFFIX ".a") set(LIB_SUFFIX ".a") + set(XLSXIO_LIB_PREFIX "lib") endif() set(LIBEXPAT_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libexpat) @@ -64,7 +66,7 @@ ExternalProject_Add(libzip_project set(XLSXIO_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/xlsxio) set(XLSXIO_INSTALL_DIR ${XLSXIO_INSTALL_DIR} PARENT_SCOPE) -set(XLSXIO_LIB ${XLSXIO_INSTALL_DIR}/lib/libxlsxio_read${LIB_SUFFIX}) +set(XLSXIO_LIB ${XLSXIO_INSTALL_DIR}/lib/${XLSXIO_LIB_PREFIX}xlsxio_read${LIB_SUFFIX}) set(XLSXIO_LIB ${XLSXIO_LIB} PARENT_SCOPE) ExternalProject_Add(xlsxio_project PREFIX xlsxio From 5fc627a16f26c7acae4d0bf8b858066440afa9d1 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sun, 9 Aug 2020 04:03:04 -0700 Subject: [PATCH 27/61] no lib prefix for libzip on windows too --- depends/CMakeLists.txt | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/depends/CMakeLists.txt b/depends/CMakeLists.txt index db132a809..bb5dc72dc 100644 --- a/depends/CMakeLists.txt +++ b/depends/CMakeLists.txt @@ -35,11 +35,11 @@ endforeach() if(WIN32) set(EXPAT_LIB_SUFFIX "MD.lib") set(LIB_SUFFIX ".lib") - set(XLSXIO_LIB_PREFIX "") + set(LIB_PREFIX "") else() set(EXPAT_LIB_SUFFIX ".a") set(LIB_SUFFIX ".a") - set(XLSXIO_LIB_PREFIX "lib") + set(LIB_PREFIX "lib") endif() set(LIBEXPAT_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libexpat) @@ -54,7 +54,7 @@ ExternalProject_Add(libexpat_project ) set(LIBZIP_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libzip) -set(LIBZIP_LIB ${LIBZIP_INSTALL_DIR}/lib/libzip${LIB_SUFFIX}) +set(LIBZIP_LIB ${LIBZIP_INSTALL_DIR}/lib/${LIB_PREFIX}zip${LIB_SUFFIX}) set(LIBZIP_LIB ${LIBZIP_LIB} PARENT_SCOPE) ExternalProject_Add(libzip_project PREFIX libzip @@ -66,7 +66,7 @@ ExternalProject_Add(libzip_project set(XLSXIO_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/xlsxio) set(XLSXIO_INSTALL_DIR ${XLSXIO_INSTALL_DIR} PARENT_SCOPE) -set(XLSXIO_LIB ${XLSXIO_INSTALL_DIR}/lib/${XLSXIO_LIB_PREFIX}xlsxio_read${LIB_SUFFIX}) +set(XLSXIO_LIB ${XLSXIO_INSTALL_DIR}/lib/${LIB_PREFIX}xlsxio_read${LIB_SUFFIX}) set(XLSXIO_LIB ${XLSXIO_LIB} PARENT_SCOPE) ExternalProject_Add(xlsxio_project PREFIX xlsxio From a41f7c099843d23286857f976ab962b336e679c5 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sun, 9 Aug 2020 08:54:42 -0700 Subject: [PATCH 28/61] convert XlsxReader module to xlsxreader plugin --- CMakeLists.txt | 12 -- depends/CMakeLists.txt | 2 + library/CMakeLists.txt | 4 +- library/LuaApi.cpp | 61 ---------- library/include/modules/XlsxReader.h | 45 ------- library/modules/XlsxReader.cpp | 115 ----------------- plugins/CMakeLists.txt | 11 ++ plugins/lua/xlsxreader.lua | 17 +++ plugins/xlsxreader.cpp | 176 +++++++++++++++++++++++++++ 9 files changed, 207 insertions(+), 236 deletions(-) delete mode 100644 library/include/modules/XlsxReader.h delete mode 100644 library/modules/XlsxReader.cpp create mode 100644 plugins/lua/xlsxreader.lua create mode 100644 plugins/xlsxreader.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index eeb8c8d12..cd06587eb 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -364,8 +364,6 @@ endif() #### expose depends #### -include(ExternalProject) - if(UNIX) # Rescan for pthread and zlib if the build arch changed if(NOT "${DFHACK_BUILD_ARCH}" STREQUAL "${DFHACK_BUILD_ARCH_PREV}") @@ -417,18 +415,8 @@ endif() # build the lib itself if(BUILD_LIBRARY) - include_directories(${XLSXIO_INSTALL_DIR}/include) - add_library(xlsxio_read STATIC IMPORTED) - set_target_properties(xlsxio_read PROPERTIES IMPORTED_LOCATION ${XLSXIO_LIB}) - add_library(zip STATIC IMPORTED) - set_target_properties(zip PROPERTIES IMPORTED_LOCATION ${LIBZIP_LIB}) - add_library(expat STATIC IMPORTED) - set_target_properties(expat PROPERTIES IMPORTED_LOCATION ${LIBEXPAT_LIB}) - add_subdirectory(library) install(FILES LICENSE.rst docs/changelog.txt DESTINATION ${DFHACK_USERDOC_DESTINATION}) - - add_dependencies(dfhack xlsxio_project) endif() file(WRITE "${CMAKE_BINARY_DIR}/dfhack_setarch.txt" ${DFHACK_SETARCH}) diff --git a/depends/CMakeLists.txt b/depends/CMakeLists.txt index bb5dc72dc..69ce40ae3 100644 --- a/depends/CMakeLists.txt +++ b/depends/CMakeLists.txt @@ -32,6 +32,8 @@ foreach(var ${vars}) endif() endforeach() +include(ExternalProject) + if(WIN32) set(EXPAT_LIB_SUFFIX "MD.lib") set(LIB_SUFFIX ".lib") diff --git a/library/CMakeLists.txt b/library/CMakeLists.txt index 583c8768f..2de293e15 100644 --- a/library/CMakeLists.txt +++ b/library/CMakeLists.txt @@ -144,7 +144,6 @@ set(MODULE_HEADERS include/modules/Translation.h include/modules/Units.h include/modules/World.h - include/modules/XlsxReader.h ) set(MODULE_SOURCES @@ -173,7 +172,6 @@ set(MODULE_SOURCES modules/Units.cpp modules/Windows.cpp modules/World.cpp - modules/XlsxReader.cpp ) set(STATIC_FIELDS_FILES) @@ -411,7 +409,7 @@ if(APPLE) set_target_properties(dfhack PROPERTIES SOVERSION 1.0.0) endif() -target_link_libraries(dfhack protobuf-lite clsocket lua jsoncpp_lib_static xlsxio_read zip expat dfhack-version ${PROJECT_LIBS}) +target_link_libraries(dfhack protobuf-lite clsocket lua jsoncpp_lib_static dfhack-version ${PROJECT_LIBS}) set_target_properties(dfhack PROPERTIES INTERFACE_LINK_LIBRARIES "") target_link_libraries(dfhack-client protobuf-lite clsocket jsoncpp_lib_static) diff --git a/library/LuaApi.cpp b/library/LuaApi.cpp index 0265eee20..78364083f 100644 --- a/library/LuaApi.cpp +++ b/library/LuaApi.cpp @@ -60,7 +60,6 @@ distribution. #include "modules/Translation.h" #include "modules/Units.h" #include "modules/World.h" -#include "modules/XlsxReader.h" #include "LuaWrapper.h" #include "LuaTools.h" @@ -2481,65 +2480,6 @@ static const LuaWrapper::FunctionReg dfhack_kitchen_module[] = { {NULL, NULL} }; -/***** XlsxReader module *****/ - -static const LuaWrapper::FunctionReg dfhack_xlsxreader_module[] = { - WRAPM(XlsxReader, open_xlsx_file), - WRAPM(XlsxReader, close_xlsx_file), - WRAPM(XlsxReader, open_sheet), - WRAPM(XlsxReader, close_sheet), - {NULL, NULL} -}; - -// internal function to factor out handle extraction -static void * get_xlsxreader_handle(lua_State *L) -{ - if (lua_gettop(L) < 1 || lua_isnil(L, 1)) - { - luaL_error(L, "invalid xlsxreader handle"); - } - luaL_checktype(L, 1, LUA_TLIGHTUSERDATA); - return lua_touserdata(L, 1); -} - -// takes a file handle and returns a table-list of sheet names -static int xlsxreader_list_sheets(lua_State *L) -{ - XlsxReader::xlsx_file_handle file_handle = get_xlsxreader_handle(L); - Lua::PushVector(L, XlsxReader::list_sheets(file_handle), true); - return 1; -} - -// takes the sheet handle and returns a table-list of strings, or nil if we -// already processed the last row in the file. -static int xlsxreader_get_row(lua_State *L) -{ - XlsxReader::xlsx_sheet_handle sheet_handle = get_xlsxreader_handle(L); - bool ok = XlsxReader::get_next_row(sheet_handle); - if (!ok) - { - lua_pushnil(L); - } - else - { - std::string value; - auto cells = std::vector(); - while (XlsxReader::get_next_cell(sheet_handle, value)) - { - cells.push_back(value); - } - Lua::PushVector(L, cells, true); - } - - return 1; -} - -static const luaL_Reg dfhack_xlsxreader_funcs[] = { - {"list_sheets", xlsxreader_list_sheets}, - {"get_row", xlsxreader_get_row}, - {NULL, NULL} -}; - /***** Console module *****/ namespace console { @@ -3093,7 +3033,6 @@ void OpenDFHackApi(lua_State *state) OpenModule(state, "filesystem", dfhack_filesystem_module, dfhack_filesystem_funcs); OpenModule(state, "designations", dfhack_designations_module, dfhack_designations_funcs); OpenModule(state, "kitchen", dfhack_kitchen_module); - OpenModule(state, "xlsxreader", dfhack_xlsxreader_module, dfhack_xlsxreader_funcs); OpenModule(state, "console", dfhack_console_module); OpenModule(state, "internal", dfhack_internal_module, dfhack_internal_funcs); } diff --git a/library/include/modules/XlsxReader.h b/library/include/modules/XlsxReader.h deleted file mode 100644 index 93ea1cf20..000000000 --- a/library/include/modules/XlsxReader.h +++ /dev/null @@ -1,45 +0,0 @@ -/* - * Wrapper for xlsxio_read library functions. - */ - -#pragma once - -#include -#include - -#include "Export.h" - -/** - * \defgroup grp_xlsx_reader Xlsx file reader - * @ingroup grp_modules - */ -namespace DFHack -{ -namespace XlsxReader -{ - -typedef void* xlsx_file_handle; -typedef void* xlsx_sheet_handle; - -// returns NULL on error -DFHACK_EXPORT xlsx_file_handle open_xlsx_file(std::string filename); -DFHACK_EXPORT void close_xlsx_file(xlsx_file_handle file_handle); -DFHACK_EXPORT std::vector list_sheets(xlsx_file_handle file_handle); - -// returns XLSXIOReaderSheet object or NULL on error -DFHACK_EXPORT xlsx_sheet_handle open_sheet( - xlsx_file_handle file_handle, std::string sheet_name); -DFHACK_EXPORT void close_sheet(xlsx_sheet_handle sheet_handle); - -// start reading the next row of data; must be called before GetNextCell . -// returns false if there is no next row to get. -DFHACK_EXPORT bool get_next_row(xlsx_sheet_handle sheet_handle); - -// fills the value param with the contents of the cell in the next column cell -// in the current row. -// returns false if there are no more cells in this row. -DFHACK_EXPORT bool get_next_cell( - xlsx_sheet_handle sheet_handle, std::string& value); - -} -} diff --git a/library/modules/XlsxReader.cpp b/library/modules/XlsxReader.cpp deleted file mode 100644 index 405369c85..000000000 --- a/library/modules/XlsxReader.cpp +++ /dev/null @@ -1,115 +0,0 @@ -/* - * Wrapper for xlsxio_read library functions. - * - * Sample usage: - * - * std::string filename = "sample_file.xlsx"; - * xlsxioreader xlsxfile = XlsxReader::open_xlsx_file(filename); - * if (xlsxfile == NULL) { - * printf("cannot open file: '%s'", filename.c_str()); - * return false; - * } - * auto sheetNames = XlsxReader::list_sheets(xlsxfile); - * for (auto sheetName = sheetNames.begin(); - * sheetName != sheetNames.end(); - * ++sheetName) { - * printf("reading sheet: %s\n", sheetName->c_str()); - * xlsxioreadersheet xlsxsheet = - * XlsxReader::open_sheet(xlsxfile, *sheetName); - * if (xlsxsheet == NULL) { - * printf("cannot open sheet: '%s'", sheetName->c_str()); - * continue; - * } - * std::string value; - * int row_num = 1; - * while (XlsxReader::GetNextRow(xlsxsheet)) { - * std::string s; - * printf("%d:\t", row_num); - * while (XlsxReader::GetNextCell(xlsxsheet, s)) { - * printf("%s\t", s.c_str()); - * } - * printf("\n"); - * ++row_num; - * } - * XlsxReader::close_sheet(xlsxsheet); - * } - * XlsxReader::close_xlsx_file(xlsxfile); - * return true; - */ - -#include - -#include "modules/XlsxReader.h" - -using namespace DFHack; - - -// returns NULL on error -DFHACK_EXPORT XlsxReader::xlsx_file_handle XlsxReader::open_xlsx_file( - std::string filename) -{ - return xlsxioread_open(filename.c_str()); -} - -DFHACK_EXPORT void XlsxReader::close_xlsx_file( - XlsxReader::xlsx_file_handle file_handle) -{ - xlsxioread_close((xlsxioreader)file_handle); -} - -static int list_callback(const XLSXIOCHAR* name, void* cbdata) -{ - auto sheetNames = (std::vector *)cbdata; - sheetNames->push_back(name); - return 0; -} - -DFHACK_EXPORT std::vector XlsxReader::list_sheets( - XlsxReader::xlsx_file_handle file_handle) -{ - auto sheetNames = std::vector(); - xlsxioread_list_sheets( - (xlsxioreader)file_handle, list_callback, &sheetNames); - return sheetNames; -} - -// returns XLSXIOReaderSheet object or NULL on error -DFHACK_EXPORT XlsxReader::xlsx_sheet_handle XlsxReader::open_sheet( - XlsxReader::xlsx_file_handle file_handle, std::string sheet_name) -{ - if (file_handle == NULL) - return NULL; - return xlsxioread_sheet_open( - (xlsxioreader)file_handle, sheet_name.c_str(), XLSXIOREAD_SKIP_NONE); -} - -DFHACK_EXPORT void XlsxReader::close_sheet( - XlsxReader::xlsx_sheet_handle sheet_handle) -{ - xlsxioread_sheet_close((xlsxioreadersheet)sheet_handle); -} - -// start reading the next row of data; must be called before GetNextCell . -// returns false if there is no next row to get. -DFHACK_EXPORT bool XlsxReader::get_next_row( - XlsxReader::xlsx_sheet_handle sheet_handle) -{ - return xlsxioread_sheet_next_row((xlsxioreadersheet)sheet_handle) != 0; -} - -// fills the value param with the contents of the cell in the next column cell -// in the current row. -// returns false if there are no more cells in this row. -DFHACK_EXPORT bool XlsxReader::get_next_cell( - XlsxReader::xlsx_sheet_handle sheet_handle, std::string& value) -{ - char* result; - if (!xlsxioread_sheet_next_cell_string((xlsxioreadersheet)sheet_handle, - &result)) { - value.clear(); - return false; - } - value.assign(result); - free(result); - return true; -} diff --git a/plugins/CMakeLists.txt b/plugins/CMakeLists.txt index 678b7eec2..6531165e1 100644 --- a/plugins/CMakeLists.txt +++ b/plugins/CMakeLists.txt @@ -77,6 +77,14 @@ set_source_files_properties( Brushes.h PROPERTIES HEADER_FILE_ONLY TRUE ) add_library(buildingplan-lib STATIC buildingplan-lib.cpp) target_link_libraries(buildingplan-lib dfhack) +# xlsxreader deps +add_library(xlsxio_read STATIC IMPORTED) +set_target_properties(xlsxio_read PROPERTIES IMPORTED_LOCATION ${XLSXIO_LIB}) +add_library(zip STATIC IMPORTED) +set_target_properties(zip PROPERTIES IMPORTED_LOCATION ${LIBZIP_LIB}) +add_library(expat STATIC IMPORTED) +set_target_properties(expat PROPERTIES IMPORTED_LOCATION ${LIBEXPAT_LIB}) + # Plugins option(BUILD_SUPPORTED "Build the supported plugins (reveal, probe, etc.)." ON) if(BUILD_SUPPORTED) @@ -175,8 +183,11 @@ if(BUILD_SUPPORTED) add_subdirectory(tweak) dfhack_plugin(workflow workflow.cpp LINK_LIBRARIES lua) dfhack_plugin(workNow workNow.cpp) + dfhack_plugin(xlsxreader xlsxreader.cpp LINK_LIBRARIES lua xlsxio_read zip expat z) dfhack_plugin(zone zone.cpp LINK_LIBRARIES lua) endif() +target_include_directories(xlsxreader PRIVATE ${XLSXIO_INSTALL_DIR}/include) +add_dependencies(xlsxreader xlsxio_project) # this is the skeleton plugin. If you want to make your own, make a copy and then change it option(BUILD_SKELETON "Build the skeleton plugin." OFF) diff --git a/plugins/lua/xlsxreader.lua b/plugins/lua/xlsxreader.lua new file mode 100644 index 000000000..3dbc1f2d8 --- /dev/null +++ b/plugins/lua/xlsxreader.lua @@ -0,0 +1,17 @@ +local _ENV = mkmodule('plugins.xlsxreader') + +--[[ + + Native functions: + + * file_handle open_xlsx_file(filename) + * close_xlsx_file(file_handle) + * sheet_names list_sheets(file_handle) + + * sheet_handle open_sheet(file_handle, sheet_name) + * close_sheet(sheet_handle) + * cell_strings get_row(sheet_handle) + +--]] + +return _ENV diff --git a/plugins/xlsxreader.cpp b/plugins/xlsxreader.cpp new file mode 100644 index 000000000..9c773718d --- /dev/null +++ b/plugins/xlsxreader.cpp @@ -0,0 +1,176 @@ +/* + * Wrapper for xlsxio_read library functions. + * + * Sample usage: + * + * std::string filename = "sample_file.xlsx"; + * xlsxioreader xlsxfile = XlsxReader::open_xlsx_file(filename); + * if (xlsxfile == NULL) { + * printf("cannot open file: '%s'", filename.c_str()); + * return false; + * } + * auto sheetNames = XlsxReader::list_sheets(xlsxfile); + * for (auto sheetName = sheetNames.begin(); + * sheetName != sheetNames.end(); + * ++sheetName) { + * printf("reading sheet: %s\n", sheetName->c_str()); + * xlsxioreadersheet xlsxsheet = + * XlsxReader::open_sheet(xlsxfile, *sheetName); + * if (xlsxsheet == NULL) { + * printf("cannot open sheet: '%s'", sheetName->c_str()); + * continue; + * } + * std::string value; + * int row_num = 1; + * while (XlsxReader::GetNextRow(xlsxsheet)) { + * std::string s; + * printf("%d:\t", row_num); + * while (XlsxReader::GetNextCell(xlsxsheet, s)) { + * printf("%s\t", s.c_str()); + * } + * printf("\n"); + * ++row_num; + * } + * XlsxReader::close_sheet(xlsxsheet); + * } + * XlsxReader::close_xlsx_file(xlsxfile); + * return true; + */ + +#include + +#include "DataFuncs.h" +#include "LuaTools.h" +#include "PluginManager.h" + +using namespace DFHack; + +DFHACK_PLUGIN("xlsxreader"); + +typedef void* xlsx_file_handle; +typedef void* xlsx_sheet_handle; + +// returns NULL on error +xlsx_file_handle open_xlsx_file(std::string filename) +{ + return xlsxioread_open(filename.c_str()); +} + +void close_xlsx_file(xlsx_file_handle file_handle) +{ + xlsxioread_close((xlsxioreader)file_handle); +} + +// returns XLSXIOReaderSheet object or NULL on error +xlsx_sheet_handle open_sheet(xlsx_file_handle file_handle, std::string sheet_name) +{ + if (file_handle == NULL) + return NULL; + return xlsxioread_sheet_open((xlsxioreader)file_handle, + sheet_name.c_str(), XLSXIOREAD_SKIP_NONE); +} + +void close_sheet(xlsx_sheet_handle sheet_handle) +{ + xlsxioread_sheet_close((xlsxioreadersheet)sheet_handle); +} + +static int list_callback(const XLSXIOCHAR* name, void* cbdata) +{ + auto sheetNames = (std::vector *)cbdata; + sheetNames->push_back(name); + return 0; +} + +// start reading the next row of data; must be called before get_next_cell. +// returns false if there is no next row to get. +static bool get_next_row(xlsx_sheet_handle sheet_handle) +{ + return xlsxioread_sheet_next_row((xlsxioreadersheet)sheet_handle) != 0; +} + +// fills the value param with the contents of the cell in the next column cell +// in the current row. +// returns false if there are no more cells in this row. +static bool get_next_cell(xlsx_sheet_handle sheet_handle, std::string& value) +{ + char* result; + if (!xlsxioread_sheet_next_cell_string( + (xlsxioreadersheet)sheet_handle, &result)) { + value.clear(); + return false; + } + value.assign(result); + free(result); + return true; +} + +// internal function to factor out handle extraction +static void * get_xlsxreader_handle(lua_State *L) +{ + if (lua_gettop(L) < 1 || lua_isnil(L, 1)) + { + luaL_error(L, "invalid xlsxreader handle"); + } + luaL_checktype(L, 1, LUA_TLIGHTUSERDATA); + return lua_touserdata(L, 1); +} + +// takes a file handle and returns a table-list of sheet names +int list_sheets(lua_State *L) +{ + xlsxioreader file_handle = (xlsxioreader)get_xlsxreader_handle(L); + auto sheetNames = std::vector(); + xlsxioread_list_sheets(file_handle, list_callback, &sheetNames); + Lua::PushVector(L, sheetNames, true); + return 1; +} + +// takes the sheet handle and returns a table-list of strings, or nil if we +// already processed the last row in the file. +int get_row(lua_State *L) +{ + xlsxioreadersheet sheet_handle = + (xlsxioreadersheet)get_xlsxreader_handle(L); + bool ok = get_next_row(sheet_handle); + if (!ok) + { + lua_pushnil(L); + } + else + { + std::string value; + auto cells = std::vector(); + while (get_next_cell(sheet_handle, value)) + { + cells.push_back(value); + } + Lua::PushVector(L, cells, true); + } + + return 1; +} + +DFHACK_PLUGIN_LUA_FUNCTIONS { + DFHACK_LUA_FUNCTION(open_xlsx_file), + DFHACK_LUA_FUNCTION(close_xlsx_file), + DFHACK_LUA_FUNCTION(open_sheet), + DFHACK_LUA_FUNCTION(close_sheet), + DFHACK_LUA_END +}; + +DFHACK_PLUGIN_LUA_COMMANDS{ + DFHACK_LUA_COMMAND(list_sheets), + DFHACK_LUA_COMMAND(get_row), + DFHACK_LUA_END +}; + +DFhackCExport command_result plugin_init(color_ostream &, std::vector &) +{ + return CR_OK; +} + +DFhackCExport command_result plugin_shutdown(color_ostream &) +{ + return CR_OK; +} From 4a04273fae91ff7aaa9c3bb2cb2bebed65220f3f Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sun, 9 Aug 2020 09:28:19 -0700 Subject: [PATCH 29/61] document xlsxreader functions --- docs/Lua API.rst | 31 +++++++++++++++++++++++++++++++ docs/Plugins.rst | 3 ++- plugins/lua/xlsxreader.lua | 18 ++++++++++++++++++ 3 files changed, 51 insertions(+), 1 deletion(-) diff --git a/docs/Lua API.rst b/docs/Lua API.rst index 45ccd46a9..e66989e35 100644 --- a/docs/Lua API.rst +++ b/docs/Lua API.rst @@ -4102,6 +4102,37 @@ Lua plugin classes - ``shuffle()``: shuffles the sequence of numbers - ``next()``: returns next number in the sequence +xlsxreader +========== + +Utility functions to facilitate reading .xlsx spreadsheets. It provides the +following API methods: + + - ``file_handle open_xlsx_file(filename)`` + - ``close_xlsx_file(file_handle)`` + - ``sheet_names list_sheets(file_handle)`` + - ``sheet_handle open_sheet(file_handle, sheet_name)`` + - ``close_sheet(sheet_handle)`` + - ``cell_strings get_row(sheet_handle)`` + + Example:: + + local xlsxreader = require('plugins.xlsxreader') + + local filepath = "path/to/some_file.xlsx" + local xlsx_file = xlsxreader.open_xlsx_file(filepath) + for _, sheet_name in ipairs(xlsxreader.list_sheets(xlsx_file)) do + print('reading sheet: '..sheet_name) + local xlsx_sheet = xlsxreader.open_sheet(xlsx_file, sheet_name) + local row_cells = xlsxreader.get_row(xlsx_sheet) + while row_cells do + printall(row_cells) + row_cells = xlsxreader.get_row(xlsx_sheet) + end + xlsxreader.close_sheet(xlsx_sheet) + end + xlsxreader.close_xlsx_file(xlsx_file) + ======= Scripts ======= diff --git a/docs/Plugins.rst b/docs/Plugins.rst index 6699163c2..9b5569e62 100644 --- a/docs/Plugins.rst +++ b/docs/Plugins.rst @@ -2939,10 +2939,11 @@ Lua API ======= Some plugins consist solely of native libraries exposed to Lua. They are listed -in the `lua-api` file under `lua-plugins`: +in the `lua-api` file under `lua-plugins`:: * `eventful` * `building-hacks` * `luasocket` * `map-render` * `cxxrandom` +* `xlsxreader` diff --git a/plugins/lua/xlsxreader.lua b/plugins/lua/xlsxreader.lua index 3dbc1f2d8..09ba01978 100644 --- a/plugins/lua/xlsxreader.lua +++ b/plugins/lua/xlsxreader.lua @@ -12,6 +12,24 @@ local _ENV = mkmodule('plugins.xlsxreader') * close_sheet(sheet_handle) * cell_strings get_row(sheet_handle) +Sample usage: + + local xlsxreader = require('plugins.xlsxreader') + + local filepath = "path/to/some_file.xlsx" + local xlsx_file = xlsxreader.open_xlsx_file(filepath) + for _, sheet_name in ipairs(xlsxreader.list_sheets(xlsx_file)) do + print('reading sheet: '..sheet_name) + local xlsx_sheet = xlsxreader.open_sheet(xlsx_file, sheet_name) + local row_cells = xlsxreader.get_row(xlsx_sheet) + while row_cells do + printall(row_cells) + row_cells = xlsxreader.get_row(xlsx_sheet) + end + xlsxreader.close_sheet(xlsx_sheet) + end + xlsxreader.close_xlsx_file(xlsx_file) + --]] return _ENV From 5294360908c462ea24a613230df18dc336f55d75 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sun, 9 Aug 2020 09:46:54 -0700 Subject: [PATCH 30/61] more windows library naming shenanigans --- plugins/CMakeLists.txt | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/plugins/CMakeLists.txt b/plugins/CMakeLists.txt index 6531165e1..2bc135d33 100644 --- a/plugins/CMakeLists.txt +++ b/plugins/CMakeLists.txt @@ -85,6 +85,12 @@ set_target_properties(zip PROPERTIES IMPORTED_LOCATION ${LIBZIP_LIB}) add_library(expat STATIC IMPORTED) set_target_properties(expat PROPERTIES IMPORTED_LOCATION ${LIBEXPAT_LIB}) +if(WIN32) + set(LIB_Z_LIB "zlib.lib") +else() + set(LIB_Z_LIB "z") +endif() + # Plugins option(BUILD_SUPPORTED "Build the supported plugins (reveal, probe, etc.)." ON) if(BUILD_SUPPORTED) @@ -183,7 +189,7 @@ if(BUILD_SUPPORTED) add_subdirectory(tweak) dfhack_plugin(workflow workflow.cpp LINK_LIBRARIES lua) dfhack_plugin(workNow workNow.cpp) - dfhack_plugin(xlsxreader xlsxreader.cpp LINK_LIBRARIES lua xlsxio_read zip expat z) + dfhack_plugin(xlsxreader xlsxreader.cpp LINK_LIBRARIES lua xlsxio_read zip expat ${LIB_Z_LIB}) dfhack_plugin(zone zone.cpp LINK_LIBRARIES lua) endif() target_include_directories(xlsxreader PRIVATE ${XLSXIO_INSTALL_DIR}/include) From e2129673c12750a89b5939d0d353cfc597fc8c46 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sun, 9 Aug 2020 10:11:24 -0700 Subject: [PATCH 31/61] fix zlib path on windows --- plugins/CMakeLists.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/CMakeLists.txt b/plugins/CMakeLists.txt index 2bc135d33..c2798cd7d 100644 --- a/plugins/CMakeLists.txt +++ b/plugins/CMakeLists.txt @@ -86,7 +86,7 @@ add_library(expat STATIC IMPORTED) set_target_properties(expat PROPERTIES IMPORTED_LOCATION ${LIBEXPAT_LIB}) if(WIN32) - set(LIB_Z_LIB "zlib.lib") + set(LIB_Z_LIB "depends/zlib/lib/zlib.lib") else() set(LIB_Z_LIB "z") endif() From 06e3f77e39d106fdc1633e3e6bfc207447a1d805 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sun, 9 Aug 2020 10:47:15 -0700 Subject: [PATCH 32/61] fix windows zlib path, hopefully for the last time --- plugins/CMakeLists.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/CMakeLists.txt b/plugins/CMakeLists.txt index c2798cd7d..bc897ba7d 100644 --- a/plugins/CMakeLists.txt +++ b/plugins/CMakeLists.txt @@ -86,7 +86,7 @@ add_library(expat STATIC IMPORTED) set_target_properties(expat PROPERTIES IMPORTED_LOCATION ${LIBEXPAT_LIB}) if(WIN32) - set(LIB_Z_LIB "depends/zlib/lib/zlib.lib") + set(LIB_Z_LIB "depends/zlib/lib/zlib") else() set(LIB_Z_LIB "z") endif() From 679ca41da8739aad216fd28735b30ef08903011d Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Sun, 9 Aug 2020 21:22:17 -0700 Subject: [PATCH 33/61] make sure xlsxio knows to link statically --- depends/CMakeLists.txt | 3 +-- plugins/xlsxreader.cpp | 1 + 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/depends/CMakeLists.txt b/depends/CMakeLists.txt index 69ce40ae3..a2eafb052 100644 --- a/depends/CMakeLists.txt +++ b/depends/CMakeLists.txt @@ -72,10 +72,9 @@ set(XLSXIO_LIB ${XLSXIO_INSTALL_DIR}/lib/${LIB_PREFIX}xlsxio_read${LIB_SUFFIX}) set(XLSXIO_LIB ${XLSXIO_LIB} PARENT_SCOPE) ExternalProject_Add(xlsxio_project PREFIX xlsxio + DEPENDS libexpat_project libzip_project SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/xlsxio INSTALL_DIR ${XLSXIO_INSTALL_DIR} BUILD_BYPRODUCTS ${XLSXIO_LIB} CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${XLSXIO_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_STATIC=ON -DBUILD_SHARED=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_TOOLS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_DOCUMENTATION=OFF -DWITH_LIBZIP=ON -DZLIB_DIR=${ZLIB_DIR} -DLIBZIP_DIR=${LIBZIP_INSTALL_DIR} -DEXPAT_DIR=${LIBEXPAT_INSTALL_DIR} -DEXPAT_LIBRARIES=${LIBEXPAT_LIB} -DEXPAT_INCLUDE_DIRS=${LIBEXPAT_INSTALL_DIR}/include -DCMAKE_POSITION_INDEPENDENT_CODE=ON ) -add_dependencies(xlsxio_project libexpat_project) -add_dependencies(xlsxio_project libzip_project) diff --git a/plugins/xlsxreader.cpp b/plugins/xlsxreader.cpp index 9c773718d..fb5039465 100644 --- a/plugins/xlsxreader.cpp +++ b/plugins/xlsxreader.cpp @@ -37,6 +37,7 @@ * return true; */ +#define BUILD_XLSXIO_STATIC #include #include "DataFuncs.h" From 6da364a60b628a078c45bea1870ca3991561e404 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Mon, 10 Aug 2020 00:01:47 -0700 Subject: [PATCH 34/61] ensure libexpat is used as a static lib --- depends/CMakeLists.txt | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/depends/CMakeLists.txt b/depends/CMakeLists.txt index a2eafb052..9d63f3c37 100644 --- a/depends/CMakeLists.txt +++ b/depends/CMakeLists.txt @@ -38,10 +38,12 @@ if(WIN32) set(EXPAT_LIB_SUFFIX "MD.lib") set(LIB_SUFFIX ".lib") set(LIB_PREFIX "") + set(XLSXIO_C_FLAGS "${CMAKE_C_FLAGS} /DXML_STATIC") else() set(EXPAT_LIB_SUFFIX ".a") set(LIB_SUFFIX ".a") set(LIB_PREFIX "lib") + set(XLSXIO_C_FLAGS "${CMAKE_C_FLAGS} -DXML_STATIC") endif() set(LIBEXPAT_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libexpat) @@ -76,5 +78,5 @@ ExternalProject_Add(xlsxio_project SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/xlsxio INSTALL_DIR ${XLSXIO_INSTALL_DIR} BUILD_BYPRODUCTS ${XLSXIO_LIB} - CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${XLSXIO_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_STATIC=ON -DBUILD_SHARED=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_TOOLS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_DOCUMENTATION=OFF -DWITH_LIBZIP=ON -DZLIB_DIR=${ZLIB_DIR} -DLIBZIP_DIR=${LIBZIP_INSTALL_DIR} -DEXPAT_DIR=${LIBEXPAT_INSTALL_DIR} -DEXPAT_LIBRARIES=${LIBEXPAT_LIB} -DEXPAT_INCLUDE_DIRS=${LIBEXPAT_INSTALL_DIR}/include -DCMAKE_POSITION_INDEPENDENT_CODE=ON + CMAKE_ARGS ${CL_ARGS} -DCMAKE_INSTALL_PREFIX=${XLSXIO_INSTALL_DIR} -DCMAKE_BUILD_TYPE=Release -DBUILD_STATIC=ON -DBUILD_SHARED=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_TOOLS=OFF -DBUILD_EXAMPLES=OFF -DBUILD_DOCUMENTATION=OFF -DWITH_LIBZIP=ON -DZLIB_DIR=${ZLIB_DIR} -DLIBZIP_DIR=${LIBZIP_INSTALL_DIR} -DEXPAT_DIR=${LIBEXPAT_INSTALL_DIR} -DEXPAT_LIBRARIES=${LIBEXPAT_LIB} -DEXPAT_INCLUDE_DIRS=${LIBEXPAT_INSTALL_DIR}/include -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_C_FLAGS=${XLSXIO_C_FLAGS} ) From 702228bf124ad656153c65eb17e2d9b93a107082 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Mon, 10 Aug 2020 01:01:13 -0700 Subject: [PATCH 35/61] use standard CMAKE vars for prefixes and suffixes --- depends/CMakeLists.txt | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/depends/CMakeLists.txt b/depends/CMakeLists.txt index 9d63f3c37..2437794bb 100644 --- a/depends/CMakeLists.txt +++ b/depends/CMakeLists.txt @@ -35,19 +35,15 @@ endforeach() include(ExternalProject) if(WIN32) - set(EXPAT_LIB_SUFFIX "MD.lib") - set(LIB_SUFFIX ".lib") - set(LIB_PREFIX "") + set(EXPAT_LIB_NAME_SUFFIX "MD") set(XLSXIO_C_FLAGS "${CMAKE_C_FLAGS} /DXML_STATIC") else() - set(EXPAT_LIB_SUFFIX ".a") - set(LIB_SUFFIX ".a") - set(LIB_PREFIX "lib") + set(EXPAT_LIB_NAME_SUFFIX "") set(XLSXIO_C_FLAGS "${CMAKE_C_FLAGS} -DXML_STATIC") endif() set(LIBEXPAT_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libexpat) -set(LIBEXPAT_LIB ${LIBEXPAT_INSTALL_DIR}/lib/libexpat${EXPAT_LIB_SUFFIX}) +set(LIBEXPAT_LIB ${LIBEXPAT_INSTALL_DIR}/lib/libexpat${EXPAT_LIB_NAME_SUFFIX}${CMAKE_STATIC_LIBRARY_SUFFIX}) set(LIBEXPAT_LIB ${LIBEXPAT_LIB} PARENT_SCOPE) ExternalProject_Add(libexpat_project PREFIX libexpat @@ -58,7 +54,7 @@ ExternalProject_Add(libexpat_project ) set(LIBZIP_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/libzip) -set(LIBZIP_LIB ${LIBZIP_INSTALL_DIR}/lib/${LIB_PREFIX}zip${LIB_SUFFIX}) +set(LIBZIP_LIB ${LIBZIP_INSTALL_DIR}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}zip${CMAKE_STATIC_LIBRARY_SUFFIX}) set(LIBZIP_LIB ${LIBZIP_LIB} PARENT_SCOPE) ExternalProject_Add(libzip_project PREFIX libzip @@ -70,7 +66,7 @@ ExternalProject_Add(libzip_project set(XLSXIO_INSTALL_DIR ${CMAKE_CURRENT_BINARY_DIR}/xlsxio) set(XLSXIO_INSTALL_DIR ${XLSXIO_INSTALL_DIR} PARENT_SCOPE) -set(XLSXIO_LIB ${XLSXIO_INSTALL_DIR}/lib/${LIB_PREFIX}xlsxio_read${LIB_SUFFIX}) +set(XLSXIO_LIB ${XLSXIO_INSTALL_DIR}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}xlsxio_read${CMAKE_STATIC_LIBRARY_SUFFIX}) set(XLSXIO_LIB ${XLSXIO_LIB} PARENT_SCOPE) ExternalProject_Add(xlsxio_project PREFIX xlsxio From 69322ec4299e09c19d0581654d933cd29a75d4b1 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Mon, 10 Aug 2020 01:13:16 -0700 Subject: [PATCH 36/61] use proper USERDATA instead of LIGHTUSERDATA --- plugins/xlsxreader.cpp | 108 +++++++++++++++++++++-------------------- 1 file changed, 56 insertions(+), 52 deletions(-) diff --git a/plugins/xlsxreader.cpp b/plugins/xlsxreader.cpp index fb5039465..02b83127d 100644 --- a/plugins/xlsxreader.cpp +++ b/plugins/xlsxreader.cpp @@ -48,36 +48,53 @@ using namespace DFHack; DFHACK_PLUGIN("xlsxreader"); -typedef void* xlsx_file_handle; -typedef void* xlsx_sheet_handle; +struct handle_identity : public compound_identity { + handle_identity(): compound_identity(0, nullptr, nullptr, "handle") {}; + DFHack::identity_type type() override { return IDTYPE_OPAQUE; } +}; + +struct xlsx_file_handle { + const xlsxioreader handle; + xlsx_file_handle(xlsxioreader handle): handle(handle) {} + static handle_identity _identity; +}; + +struct xlsx_sheet_handle { + const xlsxioreadersheet handle; + xlsx_sheet_handle(xlsxioreadersheet handle): handle(handle) {} + static handle_identity _identity; +}; + +handle_identity xlsx_file_handle::_identity; +handle_identity xlsx_sheet_handle::_identity; // returns NULL on error -xlsx_file_handle open_xlsx_file(std::string filename) -{ - return xlsxioread_open(filename.c_str()); +xlsx_file_handle* open_xlsx_file(std::string filename) { + return new xlsx_file_handle(xlsxioread_open(filename.c_str())); } -void close_xlsx_file(xlsx_file_handle file_handle) -{ - xlsxioread_close((xlsxioreader)file_handle); +void close_xlsx_file(xlsx_file_handle *file_handle) { + CHECK_NULL_POINTER(file_handle); + xlsxioread_close(file_handle->handle); + delete(file_handle); } // returns XLSXIOReaderSheet object or NULL on error -xlsx_sheet_handle open_sheet(xlsx_file_handle file_handle, std::string sheet_name) -{ - if (file_handle == NULL) - return NULL; - return xlsxioread_sheet_open((xlsxioreader)file_handle, - sheet_name.c_str(), XLSXIOREAD_SKIP_NONE); +xlsx_sheet_handle* open_sheet(xlsx_file_handle *file_handle, + std::string sheet_name) { + CHECK_NULL_POINTER(file_handle); + return new xlsx_sheet_handle( + xlsxioread_sheet_open(file_handle->handle, + sheet_name.c_str(), XLSXIOREAD_SKIP_NONE)); } -void close_sheet(xlsx_sheet_handle sheet_handle) -{ - xlsxioread_sheet_close((xlsxioreadersheet)sheet_handle); +void close_sheet(xlsx_sheet_handle *sheet_handle) { + CHECK_NULL_POINTER(sheet_handle); + xlsxioread_sheet_close(sheet_handle->handle); + delete(sheet_handle); } -static int list_callback(const XLSXIOCHAR* name, void* cbdata) -{ +static int list_callback(const XLSXIOCHAR* name, void* cbdata) { auto sheetNames = (std::vector *)cbdata; sheetNames->push_back(name); return 0; @@ -85,19 +102,16 @@ static int list_callback(const XLSXIOCHAR* name, void* cbdata) // start reading the next row of data; must be called before get_next_cell. // returns false if there is no next row to get. -static bool get_next_row(xlsx_sheet_handle sheet_handle) -{ - return xlsxioread_sheet_next_row((xlsxioreadersheet)sheet_handle) != 0; +static bool get_next_row(xlsx_sheet_handle *sheet_handle) { + return xlsxioread_sheet_next_row(sheet_handle->handle) != 0; } // fills the value param with the contents of the cell in the next column cell // in the current row. // returns false if there are no more cells in this row. -static bool get_next_cell(xlsx_sheet_handle sheet_handle, std::string& value) -{ +static bool get_next_cell(xlsx_sheet_handle *sheet_handle, std::string& value) { char* result; - if (!xlsxioread_sheet_next_cell_string( - (xlsxioreadersheet)sheet_handle, &result)) { + if (!xlsxioread_sheet_next_cell_string(sheet_handle->handle, &result)) { value.clear(); return false; } @@ -106,44 +120,35 @@ static bool get_next_cell(xlsx_sheet_handle sheet_handle, std::string& value) return true; } -// internal function to factor out handle extraction -static void * get_xlsxreader_handle(lua_State *L) -{ - if (lua_gettop(L) < 1 || lua_isnil(L, 1)) - { +// internal function to extract a handle from the first stack param +static void * get_xlsxreader_handle(lua_State *L) { + if (lua_gettop(L) < 1 || lua_isnil(L, 1)) { luaL_error(L, "invalid xlsxreader handle"); } - luaL_checktype(L, 1, LUA_TLIGHTUSERDATA); - return lua_touserdata(L, 1); + luaL_checktype(L, 1, LUA_TUSERDATA); + return LuaWrapper::get_object_ref(L, 1); } // takes a file handle and returns a table-list of sheet names -int list_sheets(lua_State *L) -{ - xlsxioreader file_handle = (xlsxioreader)get_xlsxreader_handle(L); +int list_sheets(lua_State *L) { + auto file_handle = (xlsx_file_handle *)get_xlsxreader_handle(L); auto sheetNames = std::vector(); - xlsxioread_list_sheets(file_handle, list_callback, &sheetNames); + xlsxioread_list_sheets(file_handle->handle, list_callback, &sheetNames); Lua::PushVector(L, sheetNames, true); return 1; } // takes the sheet handle and returns a table-list of strings, or nil if we // already processed the last row in the file. -int get_row(lua_State *L) -{ - xlsxioreadersheet sheet_handle = - (xlsxioreadersheet)get_xlsxreader_handle(L); +int get_row(lua_State *L) { + auto sheet_handle = (xlsx_sheet_handle *)get_xlsxreader_handle(L); bool ok = get_next_row(sheet_handle); - if (!ok) - { + if (!ok) { lua_pushnil(L); - } - else - { + } else { std::string value; auto cells = std::vector(); - while (get_next_cell(sheet_handle, value)) - { + while (get_next_cell(sheet_handle, value)) { cells.push_back(value); } Lua::PushVector(L, cells, true); @@ -166,12 +171,11 @@ DFHACK_PLUGIN_LUA_COMMANDS{ DFHACK_LUA_END }; -DFhackCExport command_result plugin_init(color_ostream &, std::vector &) -{ +DFhackCExport command_result plugin_init(color_ostream &, + std::vector &) { return CR_OK; } -DFhackCExport command_result plugin_shutdown(color_ostream &) -{ +DFhackCExport command_result plugin_shutdown(color_ostream &) { return CR_OK; } From 17c509a5961def16be02a4d1279944e89418a9b4 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Mon, 10 Aug 2020 09:18:49 -0700 Subject: [PATCH 37/61] update sample code for xlsxreader to use finalize --- plugins/lua/xlsxreader.lua | 32 ++++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/plugins/lua/xlsxreader.lua b/plugins/lua/xlsxreader.lua index 09ba01978..323231915 100644 --- a/plugins/lua/xlsxreader.lua +++ b/plugins/lua/xlsxreader.lua @@ -16,19 +16,31 @@ Sample usage: local xlsxreader = require('plugins.xlsxreader') - local filepath = "path/to/some_file.xlsx" - local xlsx_file = xlsxreader.open_xlsx_file(filepath) - for _, sheet_name in ipairs(xlsxreader.list_sheets(xlsx_file)) do + local function dump_sheet(xlsx_file, sheet_name) print('reading sheet: '..sheet_name) local xlsx_sheet = xlsxreader.open_sheet(xlsx_file, sheet_name) - local row_cells = xlsxreader.get_row(xlsx_sheet) - while row_cells do - printall(row_cells) - row_cells = xlsxreader.get_row(xlsx_sheet) - end - xlsxreader.close_sheet(xlsx_sheet) + dfhack.with_finalize( + function () xlsxreader.close_sheet(xlsx_sheet) end, + function () + local row_cells = xlsxreader.get_row(xlsx_sheet) + while row_cells do + printall(row_cells) + row_cells = xlsxreader.get_row(xlsx_sheet) + end + end + ) end - xlsxreader.close_xlsx_file(xlsx_file) + + local filepath = "path/to/some_file.xlsx" + local xlsx_file = xlsxreader.open_xlsx_file(filepath) + dfhack.with_finalize( + function () xlsxreader.close_xlsx_file(xlsx_file) end, + function () + for _, sheet_name in ipairs(xlsxreader.list_sheets(xlsx_file)) do + dump_sheet(xlsx_file, sheet_name) + end + end + ) --]] From a2279cc9a72b567812856f6e606bba38c55e6c2f Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Mon, 10 Aug 2020 14:37:59 -0700 Subject: [PATCH 38/61] update docs in Lua API.rst too --- docs/Lua API.rst | 32 ++++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/docs/Lua API.rst b/docs/Lua API.rst index e66989e35..45eeca78e 100644 --- a/docs/Lua API.rst +++ b/docs/Lua API.rst @@ -4119,19 +4119,31 @@ following API methods: local xlsxreader = require('plugins.xlsxreader') - local filepath = "path/to/some_file.xlsx" - local xlsx_file = xlsxreader.open_xlsx_file(filepath) - for _, sheet_name in ipairs(xlsxreader.list_sheets(xlsx_file)) do + local function dump_sheet(xlsx_file, sheet_name) print('reading sheet: '..sheet_name) local xlsx_sheet = xlsxreader.open_sheet(xlsx_file, sheet_name) - local row_cells = xlsxreader.get_row(xlsx_sheet) - while row_cells do - printall(row_cells) - row_cells = xlsxreader.get_row(xlsx_sheet) - end - xlsxreader.close_sheet(xlsx_sheet) + dfhack.with_finalize( + function () xlsxreader.close_sheet(xlsx_sheet) end, + function () + local row_cells = xlsxreader.get_row(xlsx_sheet) + while row_cells do + printall(row_cells) + row_cells = xlsxreader.get_row(xlsx_sheet) + end + end + ) end - xlsxreader.close_xlsx_file(xlsx_file) + + local filepath = "path/to/some_file.xlsx" + local xlsx_file = xlsxreader.open_xlsx_file(filepath) + dfhack.with_finalize( + function () xlsxreader.close_xlsx_file(xlsx_file) end, + function () + for _, sheet_name in ipairs(xlsxreader.list_sheets(xlsx_file)) do + dump_sheet(xlsx_file, sheet_name) + end + end + ) ======= Scripts From 0d9614154f1f85e508549941a7519b59e21135bf Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Mon, 10 Aug 2020 23:04:05 -0700 Subject: [PATCH 39/61] don't turn a rst list into code --- docs/Plugins.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Plugins.rst b/docs/Plugins.rst index 9b5569e62..e98e46073 100644 --- a/docs/Plugins.rst +++ b/docs/Plugins.rst @@ -2939,7 +2939,7 @@ Lua API ======= Some plugins consist solely of native libraries exposed to Lua. They are listed -in the `lua-api` file under `lua-plugins`:: +in the `lua-api` file under `lua-plugins`: * `eventful` * `building-hacks` From 438d264b3154cb95fbf6c66d0146202b8347016c Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Mon, 10 Aug 2020 23:09:03 -0700 Subject: [PATCH 40/61] use separate identity classes for each handle type --- plugins/xlsxreader.cpp | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/plugins/xlsxreader.cpp b/plugins/xlsxreader.cpp index 02b83127d..58321705d 100644 --- a/plugins/xlsxreader.cpp +++ b/plugins/xlsxreader.cpp @@ -48,25 +48,32 @@ using namespace DFHack; DFHACK_PLUGIN("xlsxreader"); -struct handle_identity : public compound_identity { - handle_identity(): compound_identity(0, nullptr, nullptr, "handle") {}; +struct xlsx_file_handle_identity : public compound_identity { + xlsx_file_handle_identity() + :compound_identity(0, nullptr, nullptr, "handle") {}; + DFHack::identity_type type() override { return IDTYPE_OPAQUE; } +}; + +struct xlsx_sheet_handle_identity : public compound_identity { + xlsx_sheet_handle_identity() + :compound_identity(0, nullptr, nullptr, "handle") {}; DFHack::identity_type type() override { return IDTYPE_OPAQUE; } }; struct xlsx_file_handle { const xlsxioreader handle; xlsx_file_handle(xlsxioreader handle): handle(handle) {} - static handle_identity _identity; + static xlsx_file_handle_identity _identity; }; struct xlsx_sheet_handle { const xlsxioreadersheet handle; xlsx_sheet_handle(xlsxioreadersheet handle): handle(handle) {} - static handle_identity _identity; + static xlsx_sheet_handle_identity _identity; }; -handle_identity xlsx_file_handle::_identity; -handle_identity xlsx_sheet_handle::_identity; +xlsx_file_handle_identity xlsx_file_handle::_identity; +xlsx_sheet_handle_identity xlsx_sheet_handle::_identity; // returns NULL on error xlsx_file_handle* open_xlsx_file(std::string filename) { From 1e9562bd76a83557ed4af7ac574e445f801366f8 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Mon, 10 Aug 2020 23:35:38 -0700 Subject: [PATCH 41/61] add target label for xlsxreader --- docs/Lua API.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/Lua API.rst b/docs/Lua API.rst index 45eeca78e..5cecfb7f2 100644 --- a/docs/Lua API.rst +++ b/docs/Lua API.rst @@ -4102,6 +4102,8 @@ Lua plugin classes - ``shuffle()``: shuffles the sequence of numbers - ``next()``: returns next number in the sequence +.. _xlsxreader: + xlsxreader ========== From c6f9eb519a166c49d6bb37aecf3948689a71788f Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Tue, 11 Aug 2020 13:44:07 -0700 Subject: [PATCH 42/61] remove outdated example code in xlsxreader.cpp there is already good example code in Lua API.rst and in xlsxreader.lua, so just remove this one --- plugins/xlsxreader.cpp | 35 ----------------------------------- 1 file changed, 35 deletions(-) diff --git a/plugins/xlsxreader.cpp b/plugins/xlsxreader.cpp index 58321705d..560d3cc2e 100644 --- a/plugins/xlsxreader.cpp +++ b/plugins/xlsxreader.cpp @@ -1,40 +1,5 @@ /* * Wrapper for xlsxio_read library functions. - * - * Sample usage: - * - * std::string filename = "sample_file.xlsx"; - * xlsxioreader xlsxfile = XlsxReader::open_xlsx_file(filename); - * if (xlsxfile == NULL) { - * printf("cannot open file: '%s'", filename.c_str()); - * return false; - * } - * auto sheetNames = XlsxReader::list_sheets(xlsxfile); - * for (auto sheetName = sheetNames.begin(); - * sheetName != sheetNames.end(); - * ++sheetName) { - * printf("reading sheet: %s\n", sheetName->c_str()); - * xlsxioreadersheet xlsxsheet = - * XlsxReader::open_sheet(xlsxfile, *sheetName); - * if (xlsxsheet == NULL) { - * printf("cannot open sheet: '%s'", sheetName->c_str()); - * continue; - * } - * std::string value; - * int row_num = 1; - * while (XlsxReader::GetNextRow(xlsxsheet)) { - * std::string s; - * printf("%d:\t", row_num); - * while (XlsxReader::GetNextCell(xlsxsheet, s)) { - * printf("%s\t", s.c_str()); - * } - * printf("\n"); - * ++row_num; - * } - * XlsxReader::close_sheet(xlsxsheet); - * } - * XlsxReader::close_xlsx_file(xlsxfile); - * return true; */ #define BUILD_XLSXIO_STATIC From cfc736caa1d627c800cee49221d30945bfc8567d Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Tue, 11 Aug 2020 14:18:09 -0700 Subject: [PATCH 43/61] give the identity classes distinctive name fields both would have been represented as "handle" to Lua --- plugins/xlsxreader.cpp | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/plugins/xlsxreader.cpp b/plugins/xlsxreader.cpp index 560d3cc2e..d8e6264f7 100644 --- a/plugins/xlsxreader.cpp +++ b/plugins/xlsxreader.cpp @@ -15,13 +15,13 @@ DFHACK_PLUGIN("xlsxreader"); struct xlsx_file_handle_identity : public compound_identity { xlsx_file_handle_identity() - :compound_identity(0, nullptr, nullptr, "handle") {}; + :compound_identity(0, nullptr, nullptr, "xlsx_file_handle") {}; DFHack::identity_type type() override { return IDTYPE_OPAQUE; } }; struct xlsx_sheet_handle_identity : public compound_identity { xlsx_sheet_handle_identity() - :compound_identity(0, nullptr, nullptr, "handle") {}; + :compound_identity(0, nullptr, nullptr, "xlsx_sheet_handle") {}; DFHack::identity_type type() override { return IDTYPE_OPAQUE; } }; From 0bda289b1769429fa17e12862626502498440529 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Tue, 11 Aug 2020 14:51:17 -0700 Subject: [PATCH 44/61] move xlsxreader identity statics to dfhack lib --- library/CMakeLists.txt | 3 +++ library/PluginStatics.cpp | 9 +++++++++ library/include/PluginStatics.h | 34 +++++++++++++++++++++++++++++++++ plugins/xlsxreader.cpp | 28 +-------------------------- 4 files changed, 47 insertions(+), 27 deletions(-) create mode 100644 library/PluginStatics.cpp create mode 100644 library/include/PluginStatics.h diff --git a/library/CMakeLists.txt b/library/CMakeLists.txt index 2de293e15..b551c03f9 100644 --- a/library/CMakeLists.txt +++ b/library/CMakeLists.txt @@ -40,6 +40,8 @@ set(MAIN_HEADERS include/Module.h include/Pragma.h include/MemAccess.h + include/PluginManager.h + include/PluginStatics.h include/Signal.hpp include/TileTypes.h include/Types.h @@ -72,6 +74,7 @@ set(MAIN_SOURCES MiscUtils.cpp Types.cpp PluginManager.cpp + PluginStatics.cpp TileTypes.cpp VersionInfoFactory.cpp RemoteClient.cpp diff --git a/library/PluginStatics.cpp b/library/PluginStatics.cpp new file mode 100644 index 000000000..d5abdb840 --- /dev/null +++ b/library/PluginStatics.cpp @@ -0,0 +1,9 @@ +#include "PluginStatics.h" + +namespace DFHack { + +// xlsxreader statics +DFHACK_EXPORT xlsx_file_handle_identity xlsx_file_handle::_identity; +DFHACK_EXPORT xlsx_sheet_handle_identity xlsx_sheet_handle::_identity; + +} diff --git a/library/include/PluginStatics.h b/library/include/PluginStatics.h new file mode 100644 index 000000000..b5ca4b468 --- /dev/null +++ b/library/include/PluginStatics.h @@ -0,0 +1,34 @@ +#pragma once + +#include "DataIdentity.h" + +namespace DFHack { + +// xlsxreader definitions +struct xlsx_file_handle_identity : public compound_identity { + xlsx_file_handle_identity() + :compound_identity(0, nullptr, nullptr, "xlsx_file_handle") {}; + DFHack::identity_type type() override { return IDTYPE_OPAQUE; } +}; + +struct xlsx_sheet_handle_identity : public compound_identity { + xlsx_sheet_handle_identity() + :compound_identity(0, nullptr, nullptr, "xlsx_sheet_handle") {}; + DFHack::identity_type type() override { return IDTYPE_OPAQUE; } +}; + +struct xlsx_file_handle { + typedef struct xlsxio_read_struct* xlsxioreader; + const xlsxioreader handle; + xlsx_file_handle(xlsxioreader handle): handle(handle) {} + static xlsx_file_handle_identity _identity; +}; + +struct xlsx_sheet_handle { + typedef struct xlsxio_read_sheet_struct* xlsxioreadersheet; + const xlsxioreadersheet handle; + xlsx_sheet_handle(xlsxioreadersheet handle): handle(handle) {} + static xlsx_sheet_handle_identity _identity; +}; + +} diff --git a/plugins/xlsxreader.cpp b/plugins/xlsxreader.cpp index d8e6264f7..6f76f5e97 100644 --- a/plugins/xlsxreader.cpp +++ b/plugins/xlsxreader.cpp @@ -8,38 +8,12 @@ #include "DataFuncs.h" #include "LuaTools.h" #include "PluginManager.h" +#include "PluginStatics.h" using namespace DFHack; DFHACK_PLUGIN("xlsxreader"); -struct xlsx_file_handle_identity : public compound_identity { - xlsx_file_handle_identity() - :compound_identity(0, nullptr, nullptr, "xlsx_file_handle") {}; - DFHack::identity_type type() override { return IDTYPE_OPAQUE; } -}; - -struct xlsx_sheet_handle_identity : public compound_identity { - xlsx_sheet_handle_identity() - :compound_identity(0, nullptr, nullptr, "xlsx_sheet_handle") {}; - DFHack::identity_type type() override { return IDTYPE_OPAQUE; } -}; - -struct xlsx_file_handle { - const xlsxioreader handle; - xlsx_file_handle(xlsxioreader handle): handle(handle) {} - static xlsx_file_handle_identity _identity; -}; - -struct xlsx_sheet_handle { - const xlsxioreadersheet handle; - xlsx_sheet_handle(xlsxioreadersheet handle): handle(handle) {} - static xlsx_sheet_handle_identity _identity; -}; - -xlsx_file_handle_identity xlsx_file_handle::_identity; -xlsx_sheet_handle_identity xlsx_sheet_handle::_identity; - // returns NULL on error xlsx_file_handle* open_xlsx_file(std::string filename) { return new xlsx_file_handle(xlsxioread_open(filename.c_str())); From 4f88bb4c3d3f33e44990778d41d1167bacba39fd Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Tue, 11 Aug 2020 16:54:53 -0700 Subject: [PATCH 45/61] attempt to solve compilation error on windows --- library/include/PluginStatics.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/library/include/PluginStatics.h b/library/include/PluginStatics.h index b5ca4b468..a88249eae 100644 --- a/library/include/PluginStatics.h +++ b/library/include/PluginStatics.h @@ -21,14 +21,14 @@ struct xlsx_file_handle { typedef struct xlsxio_read_struct* xlsxioreader; const xlsxioreader handle; xlsx_file_handle(xlsxioreader handle): handle(handle) {} - static xlsx_file_handle_identity _identity; + DFHACK_IMPORT static xlsx_file_handle_identity _identity; }; struct xlsx_sheet_handle { typedef struct xlsxio_read_sheet_struct* xlsxioreadersheet; const xlsxioreadersheet handle; xlsx_sheet_handle(xlsxioreadersheet handle): handle(handle) {} - static xlsx_sheet_handle_identity _identity; + DFHACK_IMPORT static xlsx_sheet_handle_identity _identity; }; } From 39ae2d95fc78d547dd84002d77770a862b794ea1 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Tue, 11 Aug 2020 17:06:28 -0700 Subject: [PATCH 46/61] 2nd attempt to solve compilation errors on windows --- library/include/PluginStatics.h | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/library/include/PluginStatics.h b/library/include/PluginStatics.h index a88249eae..d0835e90f 100644 --- a/library/include/PluginStatics.h +++ b/library/include/PluginStatics.h @@ -5,30 +5,30 @@ namespace DFHack { // xlsxreader definitions -struct xlsx_file_handle_identity : public compound_identity { +struct DFHACK_EXPORT xlsx_file_handle_identity : public compound_identity { xlsx_file_handle_identity() :compound_identity(0, nullptr, nullptr, "xlsx_file_handle") {}; DFHack::identity_type type() override { return IDTYPE_OPAQUE; } }; -struct xlsx_sheet_handle_identity : public compound_identity { +struct DFHACK_EXPORT xlsx_sheet_handle_identity : public compound_identity { xlsx_sheet_handle_identity() :compound_identity(0, nullptr, nullptr, "xlsx_sheet_handle") {}; DFHack::identity_type type() override { return IDTYPE_OPAQUE; } }; -struct xlsx_file_handle { +struct DFHACK_EXPORT xlsx_file_handle { typedef struct xlsxio_read_struct* xlsxioreader; const xlsxioreader handle; xlsx_file_handle(xlsxioreader handle): handle(handle) {} - DFHACK_IMPORT static xlsx_file_handle_identity _identity; + static xlsx_file_handle_identity _identity; }; -struct xlsx_sheet_handle { +struct DFHACK_EXPORT xlsx_sheet_handle { typedef struct xlsxio_read_sheet_struct* xlsxioreadersheet; const xlsxioreadersheet handle; xlsx_sheet_handle(xlsxioreadersheet handle): handle(handle) {} - DFHACK_IMPORT static xlsx_sheet_handle_identity _identity; + static xlsx_sheet_handle_identity _identity; }; } From bda678bcd4d1ca609b296bcf8effc6364ae88fa2 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Tue, 11 Aug 2020 19:47:04 -0700 Subject: [PATCH 47/61] add some explanatory text to the top of the header --- library/include/PluginStatics.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/library/include/PluginStatics.h b/library/include/PluginStatics.h index d0835e90f..061fd3a21 100644 --- a/library/include/PluginStatics.h +++ b/library/include/PluginStatics.h @@ -1,3 +1,11 @@ +/* + * This file and the companion PluginStatics.cpp contain static structures used + * by DFHack plugins. Linking them here, into the dfhack library, instead of + * into the plugins themselves allows the plugins to be freely unloaded and + * reloaded without fear of causing cached references to static data becoming + * corrupted. + */ + #pragma once #include "DataIdentity.h" From 42108a64b9e249a2e7d31cd5b3c9ebd706bc78fa Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Wed, 12 Aug 2020 15:12:53 -0700 Subject: [PATCH 48/61] expose basic buildingplan functionality to Lua --- plugins/CMakeLists.txt | 2 +- plugins/buildingplan-lib.cpp | 5 ----- plugins/buildingplan-lib.h | 5 +++++ plugins/buildingplan.cpp | 22 ++++++++++++++++++++++ plugins/lua/buildingplan.lua | 3 +++ 5 files changed, 31 insertions(+), 6 deletions(-) create mode 100644 plugins/lua/buildingplan.lua diff --git a/plugins/CMakeLists.txt b/plugins/CMakeLists.txt index 678b7eec2..aa2f70625 100644 --- a/plugins/CMakeLists.txt +++ b/plugins/CMakeLists.txt @@ -96,7 +96,7 @@ if(BUILD_SUPPORTED) dfhack_plugin(blueprint blueprint.cpp LINK_LIBRARIES lua) dfhack_plugin(burrows burrows.cpp LINK_LIBRARIES lua) dfhack_plugin(building-hacks building-hacks.cpp LINK_LIBRARIES lua) - dfhack_plugin(buildingplan buildingplan.cpp LINK_LIBRARIES buildingplan-lib) + dfhack_plugin(buildingplan buildingplan.cpp LINK_LIBRARIES lua buildingplan-lib) dfhack_plugin(changeitem changeitem.cpp) dfhack_plugin(changelayer changelayer.cpp) dfhack_plugin(changevein changevein.cpp) diff --git a/plugins/buildingplan-lib.cpp b/plugins/buildingplan-lib.cpp index 0c29284db..e21c843aa 100644 --- a/plugins/buildingplan-lib.cpp +++ b/plugins/buildingplan-lib.cpp @@ -623,11 +623,6 @@ bool Planner::allocatePlannedBuilding(df::building_type type) return false; } - for (auto iter = newinst->jobs.begin(); iter != newinst->jobs.end(); iter++) - { - (*iter)->flags.bits.suspend = true; - } - if (type == building_type::Door) { auto door = virtual_cast(newinst); diff --git a/plugins/buildingplan-lib.h b/plugins/buildingplan-lib.h index 0d66e3248..d8d276a55 100644 --- a/plugins/buildingplan-lib.h +++ b/plugins/buildingplan-lib.h @@ -403,6 +403,11 @@ public: void addPlannedBuilding(df::building *bld) { + for (auto iter = bld->jobs.begin(); iter != bld->jobs.end(); iter++) + { + (*iter)->flags.bits.suspend = true; + } + PlannedBuilding pb(bld, &default_item_filters[bld->getType()]); planned_buildings.push_back(pb); } diff --git a/plugins/buildingplan.cpp b/plugins/buildingplan.cpp index c19eb02e9..c42aeac53 100644 --- a/plugins/buildingplan.cpp +++ b/plugins/buildingplan.cpp @@ -1,3 +1,4 @@ +#include "LuaTools.h" #include "buildingplan-lib.h" DFHACK_PLUGIN("buildingplan"); @@ -414,3 +415,24 @@ DFhackCExport command_result plugin_onstatechange(color_ostream &out, state_chan return CR_OK; } + +// Lua API section + +static bool isPlannableBuilding(df::building_type type) { + return planner.isPlanableBuilding(type); +} + +static void addPlannedBuilding(df::building *bld) { + planner.addPlannedBuilding(bld); +} + +static void doCycle() { + planner.doCycle(); +} + +DFHACK_PLUGIN_LUA_FUNCTIONS { + DFHACK_LUA_FUNCTION(isPlannableBuilding), + DFHACK_LUA_FUNCTION(addPlannedBuilding), + DFHACK_LUA_FUNCTION(doCycle), + DFHACK_LUA_END +}; diff --git a/plugins/lua/buildingplan.lua b/plugins/lua/buildingplan.lua new file mode 100644 index 000000000..c1f86081c --- /dev/null +++ b/plugins/lua/buildingplan.lua @@ -0,0 +1,3 @@ +local _ENV = mkmodule('plugins.buildingplan') + +return _ENV From 26ce6fe199fa88b18b3463c6d71448f44f1f1b96 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Thu, 13 Aug 2020 00:51:34 -0700 Subject: [PATCH 49/61] document labels and multi-blueprint file formats --- data/blueprints/README.md | 70 +++++++++++++++++++++++++++++++-------- 1 file changed, 57 insertions(+), 13 deletions(-) diff --git a/data/blueprints/README.md b/data/blueprints/README.md index 6f11df32a..f54bbde97 100644 --- a/data/blueprints/README.md +++ b/data/blueprints/README.md @@ -13,7 +13,7 @@ The original idea and 1.0 codebase came from [Valdemar's](https://dwarffortressw This document focuses on DFHack Quickfort's capabilities and teaches players how to understand and build blueprint files. Much of the text was originally written by Joel Thornton, reused here with his permission. -For those just looking to apply blueprints, check out the [quickfort command syntax](https://docs.dfhack.org/en/stable/docs/_auto/base.html#quickfort) in the DFHack Scripts documentation. There are many ready-to-use blueprints available in the `blueprints/library` subfolder in your DFHack installation. Browse them on your computer or [online](https://github.com/DFHack/dfhack/tree/develop/data/blueprints/library), or run `quickfort list -l` at the `DFHack#` prompt to list them, and then `quickfort run` to apply them to your fort! +For those just looking to apply blueprints, check out the [quickfort command syntax](https://docs.dfhack.org/en/stable/docs/_auto/base.html#quickfort) in the DFHack Scripts documentation. There are also many ready-to-use blueprints available in the `blueprints/library` subfolder in your DFHack installation. Browse them on your computer or [online](https://github.com/DFHack/dfhack/tree/develop/data/blueprints/library), or run `quickfort list -l` at the `DFHack#` prompt to list them, and then `quickfort run` to apply them to your fort! See the [Links section](#links) for more information and online resources. @@ -25,7 +25,7 @@ Table of Contents * [Editing Blueprints](#editing-blueprints) * [Area expansion syntax](#area-expansion-syntax) * [Automatic area expansion](#automatic-area-expansion) - * [Specifying a starting position](#specifying-a-starting-position) + * [Blueprint labels and cursor offsets](#blueprint-labels-and-cursor-offsets) * [Multilevel blueprints](#multilevel-blueprints) * [Minecart tracks](#minecart-tracks) * [Packaging a set of blueprints](#packaging-a-set-of-blueprints) @@ -40,6 +40,7 @@ Features * General * Manages complete blueprints to handle the four main phases of DF construction * Supports .csv and multi-worksheet .xlsx blueprint files + * Supports multiple blueprints per .csv file/spreadsheet sheet * Near-instant application, even for very large and complex blueprints * Blueprints can span multiple z-levels * Undo functionality for dig, build, and place blueprints @@ -250,21 +251,28 @@ This style can be convenient for laying out multiple buildings of the same type. Quickfort will intelligently break large areas of the same designation into appropriately-sized chunks. -Specifying a starting position ------------------------------- +Blueprint labels and cursor offsets +----------------------------------- -You can optionally specify a cursor starting position for a particular blueprint, simplifying the task of blueprint alignment. This can be helpful for blueprints that are based on a central staircase, for example. +The modeline has some additional components that we haven't talked about yet. You can give a blueprint a label by adding a `label()` marker and a cursor offset by adding a `start()` marker. -To specify a cursor starting position, use the following modified format for the header line of your blueprint: +Labels are displayed in the `quickfort list` output and are used for addressing specific blueprints when there are multiple blueprints in a single file or spreadsheet sheet (see [Packaging a set of blueprints](#packaging-a-set-of-blueprints) below). If a blueprint has no label, the label becomes the ordinal of the blueprint's position in the file or sheet. For example, the label of the first blueprint will be "1" if it is not otherwise set, the label of the second blueprint will be "2" if it is not otherwise set, etc. Labels that are explicitly defined must start with a letter to ensure the auto-generated labels don't conflict with user-defined labels. - #mode start(X;Y;STARTCOMMENT) comment +Start positions specify a cursor offset for a particular blueprint, simplifying the task of blueprint alignment. This can be helpful for blueprints that are based on a central staircase, for example. -where X and Y specify the starting cursor position (1;1 is the top left cell) and STARTCOMMENT (optional) is information about where to position the cursor. This information appears in the `quickfort list` output. +The full modeline syntax is: + + #mode label(mylabel) start(X;Y;STARTCOMMENT) comment + +where X and Y specify the starting cursor position (1;1 is the top left cell) and STARTCOMMENT (optional) is information about where to position the cursor. This information also appears in the `quickfort list` output. + +Note that all elements are optional except for the initial `#mode`, and, of course, if `label` is specified, there must be a label string, and if `start()` is specified, values for X and Y must be present. A couple examples: #dig start(3; 3; Center tile of a 5-tile square) Regular blueprint comment - #build start(10;15) + #build label(noblebedroom) start(10;15) + #query label(configstockpiles) No explicit start() means cursor is at upper left corner Multilevel blueprints @@ -408,18 +416,54 @@ Which would result in a carved track simliar to a constructed track of the form: Packaging a set of blueprints ----------------------------- -A complete QF specification for a section of your fortress may contain 4 or more separate blueprints, one for each "phase" of construction (dig, build, place stockpiles, query building adjustments). +A complete specification for a section of your fortress may contain 4 or more separate blueprints, one for each "phase" of construction (dig, build, place stockpiles, query building adjustments). + +To manage all the separate blueprints, it is often convenient to keep related blueprints in a single file. For .xlsx spreadsheets, you can keep each blueprint in a separate sheet. Online spreadsheet applications like [Google Sheets](https://sheets.new) make it easy to work with multiple related blueprints, and, as a bonus, they retain any formatting you've set, like column sizes and coloring. + +For both .csv files and .xlsx spreadsheets you can also add as many blueprints as you want in a single file or sheet. Just add a modeline in the first column to indicate the start of a new blueprint. Instead of multiple .csv files, you can concatenate them into one single file. + +For example, you can store multiple blueprints together like this: + + #dig label(digbed) + d d d d # + d d d d # + d d d d # + d d d d # + # # # # # + #build label(buildbed) + b f h # + # + # + n # + # # # # # + #place label(placebed) + # + f(2x2) # + # + # + # # # # # + #query label(boozebed) + # + booze # + # + # + # # # # # + #query label(roombed) + r{+ 3}& # + # + # + # + # # # # # -To manage all the separate blueprints, it is often convenient to keep related blueprints in a single .xlsx file, with each blueprint in a separate sheet within the file. Online spreadsheet applications like [Google Sheets](https://sheets.new) make it easy to work with multiple related blueprints, and, as a bonus, they retain any formatting you've set, like column sizes and coloring. -Of course, you could still choose to keep your blueprints in single-sheet .csv files and just give related blueprints similar names. The blueprint modes suggest a convenient naming scheme for .csv-based blueprint "stacks": +Of course, you could still choose to keep your blueprints in single-sheet .csv files and just give related blueprints similar names: bedroom.1.dig.csv bedroom.2.build.csv bedroom.3.place.csv bedroom.4.query.csv -But the naming is completely up to you, of course. +But the naming and organization is completely up to you. Tips and Tricks From 7fd7bd2fab5709713463875096fc9aa424cb90c7 Mon Sep 17 00:00:00 2001 From: Myk Taylor Date: Thu, 13 Aug 2020 11:52:34 -0700 Subject: [PATCH 50/61] document meta blueprints and reorg sections --- data/blueprints/README.md | 185 ++++++++++++++++++++++++++++---------- 1 file changed, 140 insertions(+), 45 deletions(-) diff --git a/data/blueprints/README.md b/data/blueprints/README.md index f54bbde97..05f4c6926 100644 --- a/data/blueprints/README.md +++ b/data/blueprints/README.md @@ -25,10 +25,11 @@ Table of Contents * [Editing Blueprints](#editing-blueprints) * [Area expansion syntax](#area-expansion-syntax) * [Automatic area expansion](#automatic-area-expansion) - * [Blueprint labels and cursor offsets](#blueprint-labels-and-cursor-offsets) - * [Multilevel blueprints](#multilevel-blueprints) * [Minecart tracks](#minecart-tracks) -* [Packaging a set of blueprints](#packaging-a-set-of-blueprints) + * [Multilevel blueprints](#multilevel-blueprints) + * [Blueprint labels and cursor offsets](#blueprint-labels-and-cursor-offsets) + * [Packaging a set of blueprints](#packaging-a-set-of-blueprints) + * [Meta blueprints](#meta-blueprints) * [Troubleshooting](#troubleshooting) * [Tips and tricks](#tips-and-tricks) * [Links](#links) @@ -40,14 +41,15 @@ Features * General * Manages complete blueprints to handle the four main phases of DF construction * Supports .csv and multi-worksheet .xlsx blueprint files - * Supports multiple blueprints per .csv file/spreadsheet sheet * Near-instant application, even for very large and complex blueprints + * Supports multiple blueprints per .csv file/spreadsheet sheet + * "meta" blueprints that automate the application of sequences of blueprints * Blueprints can span multiple z-levels * Undo functionality for dig, build, and place blueprints * Automatic cropping of blueprints so you don't get errors if the blueprint extends off the map * Can generate manager orders for everything required to apply build blueprints * Library of ready-to-use blueprints included - * Verbose output mode for debugging blueprints + * Verbose output mode for debugging * Dig mode * Supports all types of designations, including dumping/forbidding items and setting traffic areas * Supports applying dig blueprints in marker mode @@ -87,6 +89,8 @@ The keyword "dig" tells Quickfort we are going to be using the Designations menu place Place stockpiles menu (p) query Set building tasks/prefs menu (q) +There are also "meta" blueprints, but we'll talk about those [later](#meta-blueprints). + Optionally following this keyword and a space, you may enter a comment. This comment will appear in the output of `quickfort list` when run from the `DFHack#` prompt. You can use this space for explanations, attribution, etc. Below this line begin entering the keys you want sent in each cell. For example, we could dig out a 4x4 room like so (spaces are used as column separators here for clarity, but a real .csv file would have commas): @@ -251,46 +255,6 @@ This style can be convenient for laying out multiple buildings of the same type. Quickfort will intelligently break large areas of the same designation into appropriately-sized chunks. -Blueprint labels and cursor offsets ------------------------------------ - -The modeline has some additional components that we haven't talked about yet. You can give a blueprint a label by adding a `label()` marker and a cursor offset by adding a `start()` marker. - -Labels are displayed in the `quickfort list` output and are used for addressing specific blueprints when there are multiple blueprints in a single file or spreadsheet sheet (see [Packaging a set of blueprints](#packaging-a-set-of-blueprints) below). If a blueprint has no label, the label becomes the ordinal of the blueprint's position in the file or sheet. For example, the label of the first blueprint will be "1" if it is not otherwise set, the label of the second blueprint will be "2" if it is not otherwise set, etc. Labels that are explicitly defined must start with a letter to ensure the auto-generated labels don't conflict with user-defined labels. - -Start positions specify a cursor offset for a particular blueprint, simplifying the task of blueprint alignment. This can be helpful for blueprints that are based on a central staircase, for example. - -The full modeline syntax is: - - #mode label(mylabel) start(X;Y;STARTCOMMENT) comment - -where X and Y specify the starting cursor position (1;1 is the top left cell) and STARTCOMMENT (optional) is information about where to position the cursor. This information also appears in the `quickfort list` output. - -Note that all elements are optional except for the initial `#mode`, and, of course, if `label` is specified, there must be a label string, and if `start()` is specified, values for X and Y must be present. - -A couple examples: - - #dig start(3; 3; Center tile of a 5-tile square) Regular blueprint comment - #build label(noblebedroom) start(10;15) - #query label(configstockpiles) No explicit start() means cursor is at upper left corner - - -Multilevel blueprints ---------------------- - -Multilevel blueprints are accommodated by separating Z-levels of the blueprint with #> (go down one z-level) or #< (go up one z-level) at the end of each floor. - - #dig Stairs leading down to a small room below - j ` ` # - ` ` ` # - ` ` ` # - #> # # # - u d d # - d d d # - d d d # - # # # # - - Minecart tracks --------------- @@ -413,6 +377,48 @@ Which would result in a carved track simliar to a constructed track of the form: # # # # +Multilevel blueprints +--------------------- + +Multilevel blueprints are accommodated by separating Z-levels of the blueprint with `#>` (go down one z-level) or `#<` (go up one z-level) at the end of each floor. + + #dig Stairs leading down to a small room below + j ` ` # + ` ` ` # + ` ` ` # + #> # # # + u d d # + d d d # + d d d # + # # # # + +The marker must appear in the first column of the row to be recognized, just like a modeline. + + +Blueprint labels and cursor offsets +----------------------------------- + +The modeline has some additional components that we haven't talked about yet. You can give a blueprint a label by adding a `label()` marker and a cursor offset by adding a `start()` marker. + +Labels are displayed in the `quickfort list` output and are used for addressing specific blueprints when there are multiple blueprints in a single file or spreadsheet sheet (see [Packaging a set of blueprints](#packaging-a-set-of-blueprints) below). If a blueprint has no label, the label becomes the ordinal of the blueprint's position in the file or sheet. For example, the label of the first blueprint will be "1" if it is not otherwise set, the label of the second blueprint will be "2" if it is not otherwise set, etc. Labels that are explicitly defined must start with a letter to ensure the auto-generated labels don't conflict with user-defined labels. + +Start positions specify a cursor offset for a particular blueprint, simplifying the task of blueprint alignment. This can be helpful for blueprints that are based on a central staircase, for example. + +The full modeline syntax is: + + #mode label(mylabel) start(X;Y;STARTCOMMENT) comment + +where X and Y specify the starting cursor position (1;1 is the top left cell) and STARTCOMMENT (optional) is information about where to position the cursor. This information also appears in the `quickfort list` output. + +Note that all elements are optional except for the initial `#mode`, and, of course, if `label` is specified, there must be a label string, and if `start()` is specified, values for X and Y must be present. + +A couple examples: + + #dig start(3; 3; Center tile of a 5-tile square) Regular blueprint comment + #build label(noblebedroom) start(10;15) + #query label(configstockpiles) No explicit start() means cursor is at upper left corner + + Packaging a set of blueprints ----------------------------- @@ -466,6 +472,95 @@ Of course, you could still choose to keep your blueprints in single-sheet .csv f But the naming and organization is completely up to you. +Meta blueprints +--------------- + +Meta blueprints are blueprints that script a series of other blueprints. Many blueprint packages follow this pattern: + +- Apply dig blueprint to designate dig areas +- Wait for miners to dig +- **Apply build buildprint** to designate buildings +- **Apply place buildprint** to designate stockpiles +- **Apply query blueprint** to configure stockpiles +- Wait for buildings to get built +- Apply a different query blueprint to configure rooms + +Those three "apply"s in the middle might as well get done in one command instead of three. A meta blueprint can encode that sequence. A meta blueprint refers to other blueprints by their label (see the [labels and offsets section](#blueprint-labels-and-cursor-offsets) above) in the same format used by the `DFHack#` quickfort command: "/