From 50925726792a5a29d01e00872b27f3fcb1b4732e Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Tue, 17 Sep 2024 14:21:20 +0100 Subject: [PATCH 01/54] remove the multiple blank lines --- docs/basics/funsql.md | 49 +++++++++++++++---------------------------- 1 file changed, 17 insertions(+), 32 deletions(-) diff --git a/docs/basics/funsql.md b/docs/basics/funsql.md index ab2072a2e..9121fe159 100644 --- a/docs/basics/funsql.md +++ b/docs/basics/funsql.md @@ -20,6 +20,8 @@ q)?[t;();0b;(enlist `n)!enlist `n] The q interpreter parses `delete`, `exec`, `select`, and `update` into their equivalent functional forms, so there is no performance difference. + + The functional forms are ```syntax @@ -34,46 +36,29 @@ The functional forms are where: -`t` -: is a table, or the name of a table as a symbol atom. - -`c` -: is the [Where phrase](qsql.md#where-phrase), a list of constraints. -: Every constraint in `c` is a [parse tree](parsetrees.md) representing an expression to be evaluated; the result of each being a boolean vector. The parse tree consists of a function followed by a list of its arguments, each an expression containing column names and other variables. (Represented by symbols; distinguish actual symbol constants by enlisting them.) The function is applied to the arguments, producing a boolean vector that selects the rows. The selection is performed in the order of the items in `c`, from left to right: only rows selected by one constraint are evaluated by the next. +* `t` is a table, or the name of a table as a symbol atom. +* `c` is the [Where phrase](qsql.md#where-phrase), a list of constraints. +Every constraint in `c` is a [parse tree](parsetrees.md) representing an expression to be evaluated; the result of each being a boolean vector. The parse tree consists of a function followed by a list of its arguments, each an expression containing column names and other variables. Represented by symbols, it distinguishes actual symbol constants by enlisting them. The function is applied to the arguments, producing a boolean vector that selects the rows. The selection is performed in the order of the items in `c`, from left to right: only rows selected by one constraint are evaluated by the next. - -`b` -: is the [By phrase](qsql.md#by-phrase), one of: - +* `b` is the [By phrase](qsql.md#by-phrase). +The domain of dictionary `b` is a list of symbols that are the key names for the grouping. Its range is a list of column expressions (parse trees) whose results are used to construct the groups. The grouping is ordered by the domain items, from major to minor. +`b` is one of: - the general empty list `()` - boolean atom: `0b` for no grouping; `1b` for distinct - a symbol atom or list naming table column/s - - a dictionary of group-by specifications - -: The domain of dictionary `b` is a list of symbols that are the key names for the grouping. Its range is a list of column expressions (parse trees) whose results are used to construct the groups. The grouping is ordered by the domain items, from major to minor. - -`a` -: is the [Select phrase](../ref/select.md#select-phrase), one of: - + - a dictionary of group-by specifications +* `a` is the [Select phrase](../ref/select.md#select-phrase). +The domain of dictionary `a` is a list of symbols containing the names of the produced columns. [QSQL query templates](qsql.md) assign default column names in the result, but here each result column must be named explicitly. +Each item of its range is an evaluation list consisting of a function and its argument(s), each of which is a column name or another such result list. For each evaluation list, the function is applied to the specified value(s) for each row and the result is returned. The evaluation lists are resolved recursively when operations are nested. +`a` is one of - the general empty list `()` - a symbol atom: the name of a table column - a parse tree - a dictionary of select specifications (aggregations) - -: The domain of dictionary `a` is a list of symbols containing the names of the produced columns. ([QSQL query templates](qsql.md) assign default column names in the result, but here each result column must be named explicitly.) -: Each item of its range is an evaluation list consisting of a function and its argument/s, each of which is a column name or another such result list. For each evaluation list, the function is applied to the specified value/s for each row and the result is returned. The evaluation lists are resolved recursively when operations are nested. - -`i` -: is a list of indexes - -`p` -: is a [parse tree](parsetrees.md) - -`n` -: is a non-negative integer or infinity, indicating the maximum number of records to be returned - -`g` -: is a unary grade function +* `i` is a list of indexes +* `p` is a [parse tree](parsetrees.md) +* `n` is a non-negative integer or infinity, indicating the maximum number of records to be returned +* `g` is a unary grade function ## Call by name From 74a7fef88a6135564308c29007224762157f890b Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Tue, 17 Sep 2024 14:23:32 +0100 Subject: [PATCH 02/54] remove bad anchor link --- docs/releases/ChangesIn3.4.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/releases/ChangesIn3.4.md b/docs/releases/ChangesIn3.4.md index 0418d69ab..301b64a08 100644 --- a/docs/releases/ChangesIn3.4.md +++ b/docs/releases/ChangesIn3.4.md @@ -27,7 +27,7 @@ Below is a summary of changes from V3.3. Commercially licensed users may obtain - can utilize the [snappy](http://google.github.io/snappy/) compression algorithm as algo \#3 for [File Compression](../kb/file-compression.md). - certain vector types can now be [updated efficiently](../ref/amend.md), directly on disk, rather than having to rewrite the whole file on change. - added async broadcast as [`-25!`(handles;msg)](../basics/internal.md#-25x-async-broadcast) which serializes the msg once, queuing it as async msg to each handle. -- [`parse`](../basics/parsetrees.md#parse) can now handle k in addition to q code. +- [`parse`](../basics/parsetrees.md) can now handle k in addition to q code. - `.Q.en` can now handle lists of sym vectors: [Enumerating nested varchar columns](../kb/splayed-tables.md#enumerating-nested-varchar-columns-in-a-table) ## Not upwardly compatible From 7569818d726a06f9435e6e8d571c4eb1d6ea7164 Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Thu, 19 Sep 2024 13:52:41 +0100 Subject: [PATCH 03/54] doc on using symlinks with segmented db --- docs/basics/cmdline.md | 2 +- docs/database/segment.md | 21 +++++++++++++++++++++ 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/docs/basics/cmdline.md b/docs/basics/cmdline.md index b278317b2..79da0a2b5 100644 --- a/docs/basics/cmdline.md +++ b/docs/basics/cmdline.md @@ -374,7 +374,7 @@ Timeout in seconds for client queries, i.e. maximum time a client call will exec `-u 1` disables - system commands from a remote (signals `'access`), including exit via `"\\"` -- access to files outside the current directory for any handle context ([`.z.w`](../ref/dotz.md#zw-handle)) other than 0 +- access to files outside the current directory for any handle context ([`.z.w`](../ref/dotz.md#zw-handle)) other than 0. Segmented database partitions using directories outside the current working directory can be enabled using the method described [here](../database/segment.md#using-symlinks). - hopen on a fifo (since 4.1t 2021.10.13, 4.0 2023.08.11) - hopen of a file (since 4.1t 2021.10.13, 4.0 2023.08.11) - the [`exit`](../ref/exit.md) keyword (since 4.1t 2021.07.12) diff --git a/docs/database/segment.md b/docs/database/segment.md index 84d977cd7..6971a1217 100644 --- a/docs/database/segment.md +++ b/docs/database/segment.md @@ -49,7 +49,28 @@ db db db `/0/db` is good, but `/0/db/` can be bad, depending on the filesystem. +### Using symlinks +Security related options such as [`reval`](../ref/eval.md#reval), or the command line option [`-u 1`](../basics/cmdline.md#-u-usr-pwd), restrict access to within the current directory. +This can prevent users from accessing a segmented database when `par.txt` contains references to partitions that are situated outside the current directory. + +In order to provide access, symlinks can be used. An example of using symlinks is as follows: + +```bash +$ ln -s /db1 db1$ +$ ln -s /db2 db2$ +$ ls -l +total 16 +lrwxr-xr-x 1 user kx 6 18 Sep 12:30 db1$ -> /db1 +lrwxr-xr-x 1 user kx 6 18 Sep 12:30 db2$ -> /db2 +-rw-r--r-- 1 user kx 10 18 Sep 12:30 par.txt +-rw-r--r-- 1 user kx 48 18 Sep 12:30 sym +$ cat par.txt +db1$ +db2$ +``` + +Using a trailing `$` in the directory name ensures the originating symlinks are not picked up by kdb+, instead using the directory referenced. ## Multithreading From 782fb46304ffc2e4f9dec9ec2a31db9aa5c67fae Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Fri, 20 Sep 2024 15:22:53 +0100 Subject: [PATCH 04/54] fix section headings --- docs/wp/parse-trees.md | 43 ++++++++++++++++-------------------------- 1 file changed, 16 insertions(+), 27 deletions(-) diff --git a/docs/wp/parse-trees.md b/docs/wp/parse-trees.md index 35b9b2cc3..59600f5f3 100644 --- a/docs/wp/parse-trees.md +++ b/docs/wp/parse-trees.md @@ -357,7 +357,7 @@ In this case, it is not obvious what the functional form of the above query shou There are three issues with this parse-and-“by eye” method to convert to the equivalent functional form. We will cover these in the next three subsections. -### Parse trees and eval +#### Parse trees and eval The first issue with passing a `select` query to `parse` is that each returned item is in unevaluated form. As [discussed above](#eval-and-value), simply applying `value` to a parse tree does not work. However, if we evaluate each one of the arguments fully, then there would be no nested parse trees. We could then apply `value` to the result: @@ -403,7 +403,7 @@ q)value[str] ~ value @[pTree;2 3 4;eval] ``` -### Variable representation in parse trees +#### Variable representation in parse trees Recall that in a parse tree a variable is represented by a symbol containing its name. So to represent a symbol or a list of symbols, you must use `enlist` on that expression. In k, `enlist` is the unary form of the comma operator in k: @@ -428,7 +428,7 @@ q)(#;3;,`a`b`c`d`e`f) In the parse tree we receive we need to somehow distinguish between k’s unary `,` (which we want to replace with `enlist`) and the binary Join operator, which we want to leave as it is. -### Explicit definitions in `.q` are shown in full +#### Explicit definitions in `.q` are shown in full The `fby` in the `select` query above is represented by its full k definition. @@ -441,7 +441,7 @@ k){@[(#y)#x[0]0#x 1;g;:;x[0]'x[1]g:.=y]} While using the k form isn’t generally a problem from a functionality perspective, it does however make the resulting functional statement difficult to read. -## The solution +### The solution We will write a function to automate the process of converting a `select` query into its equivalent functional form. @@ -579,7 +579,7 @@ q).q`reciprocal These instances are rare and a developer should be able to spot when they occur. Of course, the functional form will still work as expected but could confuse readers of the code. -### Fifth and sixth arguments +#### Fifth and sixth arguments Functional select also has ranks 5 and 6; i.e. fifth and sixth arguments. @@ -636,28 +636,7 @@ q)buildQuery "select[10 20;>price] from trade" "?[trade;();0b;();10 20;(idesc;`price)]" ``` -The full `buildQuery` function code is provided in the Appendix. - - -## Conclusion - -This paper has investigated how statements in q are evaluated by the parser. To do this we examined the relationship between q and k, and explained why parse trees generated by the parse function contain k code. We have then used this understanding to explain how qSQL statements can be converted into their equivalent and more powerful functional form. - -To further help those learning how to construct functional statements, and also as a useful development tool, we have written a function which will correctly convert a qSQL query into the q functional form. This would be useful for checking that a functional statement is formed correctly or to help with a particularly tricky query. - -All tests were run using kdb+ version 3.2 (2015.01.14). - - -## Authors - -**Peter Storeng** is a mathematician and software developer who has been using kdb+ since 2012. - -![Stephen Taylor](../img/faces/stephentaylor.png) -{: .small-face} - -**Stephen Taylor** FRSA has followed the evolution of the Iversonian languages through APL, J, k, and q, and is a former editor of [_Vector_](https://vector.org.uk), the journal of the British APL Association. - -## Appendix +The full `buildQuery` function code is as follows: ```q \c 30 200 @@ -698,3 +677,13 @@ inner:{[x] buildQuery:{inner parse x} ``` + +## Authors + +**Peter Storeng** is a mathematician and software developer who has been using kdb+ since 2012. + +![Stephen Taylor](../img/faces/stephentaylor.png) +{: .small-face} + +**Stephen Taylor** FRSA has followed the evolution of the Iversonian languages through APL, J, k, and q, and is a former editor of [_Vector_](https://vector.org.uk), the journal of the British APL Association. + From 82081959a2eeb8e34b859c241f73a5215fc4defc Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Fri, 20 Sep 2024 15:52:05 +0100 Subject: [PATCH 05/54] removed dup info --- docs/wp/capi/index.md | 5 ---- docs/wp/fix-messaging.md | 13 ---------- docs/wp/gui/index.md | 32 ------------------------- docs/wp/intraday-writedown/index.md | 12 ---------- docs/wp/lightning-tickerplants/index.md | 21 ---------------- docs/wp/multi-thread/index.md | 2 -- 6 files changed, 85 deletions(-) diff --git a/docs/wp/capi/index.md b/docs/wp/capi/index.md index 1982e855f..59d5a180d 100644 --- a/docs/wp/capi/index.md +++ b/docs/wp/capi/index.md @@ -1713,11 +1713,6 @@ int main() { } ``` -## Conclusion - -This document covered multiple aspects of the C API interface for connecting with the kdb+ database. Topics covered included the creation of socket connections, execution of queries, error handling, memory management, and the creation and extraction of data from `K` objects such as lists, dictionaries and tables. Practical examples formed the basis for the construction of a C subscriber process, capable of consuming a kdb+ data feed, and a feedhandler process designed to publish data to a kdb+ tickerplant. Finally, the use of shared C libraries to extend the functionality of kdb+ was also demonstrated. - - ## Author ![Jeremy Lucid](../../img/faces/jeremylucid.jpg) diff --git a/docs/wp/fix-messaging.md b/docs/wp/fix-messaging.md index 386b04beb..e81bb1ed1 100644 --- a/docs/wp/fix-messaging.md +++ b/docs/wp/fix-messaging.md @@ -571,19 +571,6 @@ OrderID SecurityID Side MsgType OrdStatus OrderQty CumQty AvgPx CommValue "00000005" VOD.L 2 8 2 3130 3130 229.7559 143.8272 ``` - -## Conclusion - -This paper has provided a guide to working with FIX messages in kdb+, focusing primarily on capturing messages from an OMS. We focused on some key FIX fields to provide an understanding of the valid entries and an insight into how they should be handled. - -:fontawesome-solid-globe: -[fixwiki.org/fixwiki](http://fixwiki.org/fixwiki/FIXwiki "fixwiki.org") -for a comprehensive list of all fields for each FIX version - -It is an essential requirement to be able to view the current and final state of each order received from the OMS. We provided an example to show how to generate an order state. This process can be extended to also derive the latest execution state. Additional fields need to be extracted to identity the execution type, amendments, cancellations etc, similar to the order state. An execution state enables complex analytics, including risk benchmarking and transaction cost analysis to be computed efficiently on every execution. - -All tests were run using kdb+ version 3.1 (2013.12.27) - ## Author **Damien Barker** is a financial engineer who has worked as a consultant for some of the world's largest financial institutions. Based in London, Damien is currently working on trading and analytics application at a US investment bank. diff --git a/docs/wp/gui/index.md b/docs/wp/gui/index.md index 0d03dd5ec..bd65059c9 100644 --- a/docs/wp/gui/index.md +++ b/docs/wp/gui/index.md @@ -841,38 +841,6 @@ more effort, nor would giving the user the ability to choose what period time they analyze. Furthermore, WebSockets could be used to deliver streaming data from the kdb+ back end to the C# GUI. - -## Conclusion - -Despite its popularity and the potential opportunities for -development, combining C# and kdb+ remains a relatively unexploited -area. We have demonstrated a few simple processes and applications -which could be useful in developing a trading application using kdb+ -and C#. - -The applications developed for this example are simple in nature and -in implementation but the potential for creating more sophisticated -and powerful tools exists. We have shown very limited analytical and -GUI elements but these could be expanded upon by the use of Cascading -Style Sheets and more extensive configuration. - -The paper has also demonstrated the potential versatility of kdb+, not -just in the banking and financial sectors but in all sectors where C# -is popular and a database is required for back-end data storage and -management. These examples could be pushed out to analytical or -performance-based sectors or markets inexperienced in kdb+ but -requiring tools to help utilize the rapidly growing Big Data -environment. - -All examples of kdb+ were run using version 3.2 (2015.01.14). All -tests of C# were run using .NET version 4.0. The example applications -were built with Visual Studio 2010. - -The C# source code for this paper -can be found on GitHub at :fontawesome-brands-github: -[kxcontrib/csharpgui](https://github.com/kxcontrib/csharpgui). - - ## Author **Michael Reynolds** works as a kdb+ consultant for one of the largest investment banks in the world. As part of his daily job, Michael is responsible for maintaining kdb+ databases as well as a C# APIs and plug-ins. diff --git a/docs/wp/intraday-writedown/index.md b/docs/wp/intraday-writedown/index.md index 5465df2c4..c91030a3c 100644 --- a/docs/wp/intraday-writedown/index.md +++ b/docs/wp/intraday-writedown/index.md @@ -637,18 +637,6 @@ All cols, 3 syms | partWrite.q | ``select from quote where int in sym?`eurusd`e As can be seen in Table 3 and Figure 3, the speed-up in query times is substantial and thus may be worth the more complex storage and querying method. - -## Conclusion - -This paper has outlined two different methods for dealing with insufficient RAM on a server, meaning a full day’s worth of data cannot be held in memory. It looked at `w.q`, available from GitHub, which writes to a splayed table on disk and examined a number of customizations that could be made to it. - -It also detailed an alternative method using a partitioned temporary directory and how to query this data. The paper also compared the query speed and speed of the end-of-day process of the two solutions and discussed the reasons for their relative performance. - -It is important to point out that any of these solutions come with their own drawbacks and as such the best solution for your individual project will depend on the specific requirements. No one solution will provide the performance and ease of use equal to just maintaining a full day’s worth of data in memory. The complexity of how that data is stored and queried will inevitably increase. However, by using some or a combination of the ideas presented in this paper, a workable solution that fits your needs may be possible. - -Tests performed using kdb+ version 3.1 (2014.02.08) - - ## Author **Colm McCarthy** is a senior kdb+ consultant who has worked for leading investment banks across a number of different asset classes. diff --git a/docs/wp/lightning-tickerplants/index.md b/docs/wp/lightning-tickerplants/index.md index 8a5f54dd9..ffab4a201 100644 --- a/docs/wp/lightning-tickerplants/index.md +++ b/docs/wp/lightning-tickerplants/index.md @@ -1089,22 +1089,6 @@ invoice settlement messages back to individual devices to release data to subscr In this way, individual devices do not need to run their own node or store Bitcoin private keys. -## Conclusion - -The technology of Bitcoin and layer-two solutions like Lightning open up the possibility for applications -to interact directly with a decentralized peer-to-peer payments layer through the use of simple APIs, where the value -transfer reduces to the exchange of encoded text messages over TCP/IP. - -This ability to easily send and receive payments in a peer-to-peer fashion, especially micropayments, has the potential -to enable the construction of new innovative applications not hindered by third-party friction. - -In the tickerplant example, a simple template was provided to demonstrate how market data, or any other form of streaming data, -could be monetized with the creation of a pay-per-request system utilizing Lightning micropayments. - -While Lightning remains an experimental and rapidly changing technology, with many outstanding challenges, -it is hoped that this paper has at least helped explain some of the key concepts and techniques, and also showcased some synergies between the technology and kdb+ for potential integrations. - - ## Author ![Jeremy Lucid](../../img/faces/jeremylucid.jpg) @@ -1112,11 +1096,6 @@ it is hoped that this paper has at least helped explain some of the key concepts **Jeremy Lucid** is a kdb+ consultant, based in Ireland, who has worked on real-time Best Execution projects for a major multinational banking institution and a _Kx for Surveillance_ implementation at a leading options and futures exchange. -
    -- :fontawesome-regular-map: [Storing and exploring the Bitcoin blockchain](../blockchain/index.md) -- :fontawesome-regular-map: [C API for kdb+](../capi/index.md) -
- ## Acknowledgments diff --git a/docs/wp/multi-thread/index.md b/docs/wp/multi-thread/index.md index be2dfba94..64c9e503d 100644 --- a/docs/wp/multi-thread/index.md +++ b/docs/wp/multi-thread/index.md @@ -479,8 +479,6 @@ Used incorrectly, parallel processing can be less efficient than the equivalent The use of multi-threading in kdb+ should therefore be treated on a case-by-case basis. It is a feature which should often be considered by developers, but performance testing is recommended before implementation. -System-management solutions are available to assist with administering multi-process environments, like [KX Control](../../devtools.md#kx-control) which provides a range of tools for visualizing process workflow, task scheduling and system-resource monitoring. These features are especially helpful at managing primary/replica configurations, where client queries or the loading of data is divided between processes in order to increase throughput. - All tests performed using kdb+ version 3.1 (2013.06.25) From 7544db36927e5cae96720b9467d13c47cee51a8a Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Fri, 20 Sep 2024 16:28:54 +0100 Subject: [PATCH 06/54] removed dup info --- docs/wp/astronomy.md | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/docs/wp/astronomy.md b/docs/wp/astronomy.md index a401efec9..38a8be786 100644 --- a/docs/wp/astronomy.md +++ b/docs/wp/astronomy.md @@ -253,22 +253,12 @@ We previously mentioned how astronomy data can often come with a time domain, wh We could apply an attribute to this data for optimization, such as the sorted attribute to the class column in the example data set. The benefits of this would become more apparent as more files were being loaded in. Ciaran Gorman’s [“Columnar database and query optimization”](columnar-database/index.md) gives an in-depth explanation as to how they can be applied. -## Conclusion - -In this paper, we have taken one sample of astronomy data that came in a format which was initially unfamiliar to us, and subsequently analyzed it in kdb+ with relative ease. We made use of the flexibility that kdb+ has in extending to other languages; in this case C, which effectively only required using the Dynamic Load operator. This approach can be replicated where C utilities have been created for interacting with data sources, or for other reasons. - -We were then able to build functionality to calculate the recessional velocity of different object classes, which took less than a tenth of a second to run on a table of one million rows. In doing this we made use of kdb+’s efficient `fby` syntax to make use of an aggregate function as a filter criteria. It is apparent that kdb+ is extremely performant, both in storing and analyzing this data. The nature and expected volumes of data, and the way it will be used, go hand in hand with kdb+ and we have found it to be an ideal fit. - -We believe this will be a very attractive solution to the field of astronomy going forward, coping with the predicted expansion of data volumes in the field due to the new projects that are scheduled to begin in the near future. - - ## Further reading - _The Atlantic_: [“How big data is changing astronomy (again)”](https://www.theatlantic.com/technology/archive/2012/04/how-big-data-is-changing-astronomy-again/255917/) - [Cloud Computing and the Square Kilometre Array](http://www.skatelescope.org/uploaded/8762_134_Memo_Newman.pdf) - ## Authors **Andrew Magowan** is a kdb+ consultant who has developed data and analytic systems for some of the world's largest financial institutions. Andrew is currently based in New York where he maintains a global tick capture application across a range of asset classes at a major investment bank. From 58ff878b86954e6beb65732053471976b0083504 Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Fri, 20 Sep 2024 17:11:29 +0100 Subject: [PATCH 07/54] removed dup --- docs/wp/iterators/index.md | 19 ------------------- docs/wp/symfiles.md | 23 ----------------------- 2 files changed, 42 deletions(-) diff --git a/docs/wp/iterators/index.md b/docs/wp/iterators/index.md index 5553a3437..e3a7d394b 100644 --- a/docs/wp/iterators/index.md +++ b/docs/wp/iterators/index.md @@ -1006,25 +1006,6 @@ BB 80.14611 CC 67.48254 ``` - -## Conclusion - -This white paper summarizes the q iterators, showing how they derive new functions from values – functions, file- and process handles, lists and dictionaries. - -It showed with examples how the effect of the iterator is determined sometimes by the rank of the applicable value, and sometimes also by the rank at which the derived function is applied. Even complicated examples, composing multiple iterators, can be analyzed to understand the behavior. - -Certain uses of iterators, such as the creation of recursive functions and applying iterators to functions within select statements, were examined in more detail, as these are often poorly understood, but useful in many situations. -Some common uses were looked at in to demonstrate the ability of iterators to reduce execution times. - -This white paper illustrates how iterators can be used easily to extend the -application of inbuilt and user-defined functions, allowing code to take full advantage of kdb+’s ability to process large volumes of data quickly. - -Using iterators correctly minimizes the amount of data manipulation and produces code that is concise and easier to maintain. - -All tests were run using kdb+ 3.6 (2018.10.23). - - - ## Authors **Conor Slattery** is a financial engineer who has designed kdb+ applications for a range of asset classes. Conor is currently working with a New York-based investment firm, developing kdb+ trading platforms for the US equity markets. diff --git a/docs/wp/symfiles.md b/docs/wp/symfiles.md index b214f87da..a6c5fd1c9 100644 --- a/docs/wp/symfiles.md +++ b/docs/wp/symfiles.md @@ -742,28 +742,6 @@ system"mv sym zym" /make backup of sym file } peach symFiles ``` - -## Conclusion - -While the q language provides tools to easily create and update HDBs -through functions in the `.Q` namespace, understanding how the underlying -code and concepts work is integral to creating and maintaining a -functioning database. - -In this paper, we demonstrated, through examples, enumeration in kdb+ -and various methods of using symbols within an HDB; using multiple sym -files and moving data between HDBs. In situations where you need new -columns `dbmaint.q` provides utility functions to manage this with ease. - -We discussed datatype considerations in order to utilize the performance -benefits of enumerations while maintaining a sym universe containing -only necessary symbols. However, if there is still a need to perform -maintenance on the sym file because of bloat we walked through two -methods to reduce the size of the sym file. - -All tests performed using kdb+ version 3.6 (2018.05.17). - - ## Author ![Paula Clarke](../img/faces/paulaclarke.jpg) @@ -771,7 +749,6 @@ All tests performed using kdb+ version 3.6 (2018.05.17). **Paula Clarke** is a senior kdb+ consultant for KX who has worked for some of the world’s largest financial institutions designing, developing and maintaining global tick-capture kdb+ systems. - ## Related articles :fontawesome-regular-map: From febc275949aea8a19f16d88400997408736c5513 Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Mon, 23 Sep 2024 15:33:17 +0100 Subject: [PATCH 08/54] merged 3 pages into 2, about parse/func sql. removed parse trees whitepaper --- docs/basics/funsql.md | 387 +++++++++++++++++++++ docs/basics/parsetrees.md | 299 +++++++++++++++-- docs/ref/parse.md | 2 +- docs/wp/index.md | 1 - docs/wp/parse-trees.md | 689 -------------------------------------- mkdocs.yml | 1 - 6 files changed, 665 insertions(+), 714 deletions(-) delete mode 100644 docs/wp/parse-trees.md diff --git a/docs/basics/funsql.md b/docs/basics/funsql.md index 9121fe159..ff850c11c 100644 --- a/docs/basics/funsql.md +++ b/docs/basics/funsql.md @@ -2,6 +2,7 @@ title: Functional qSQL | Basics | kdb+ and q documentation description: The functional forms of delete, exec, select and update are particularly useful for programmatically-generated queries, such as when column names are dynamically produced. keywords: delete, exec, functional, kdb+, q, select, sql, update +author: [Peter Storeng, Stephen Taylor, Simon Shanks] --- # Functional qSQL @@ -592,6 +593,392 @@ _Q for Mortals_ [§9.12.4 Functional delete](https://code.kx.com/q4m3/9_Queries_q-sql/#9124-functional-delete) +## Conversion using parse + +Applying [parse](parsetrees.md) to a qSQL statement written as a string will return the internal representation of the functional form. +With some manipulation this can then be used to piece together the functional form in q. +This generally becomes more difficult as the query becomes more complex and requires a deep understanding of what kdb+ is doing when it parses qSQL form. + +An example of using parse to convert qSQL to its corresponding functional form is as follows: +```q +q)t:([]c1:`a`b`c; c2:10 20 30) +q)parse "select c2:2*c2 from t where c1=`c" +? +`t +,,(=;`c1;,`c) +0b +(,`c2)!,(*;2;`c2) + +q)?[`t; enlist (=;`c1;enlist `c); 0b; (enlist `c2)!enlist (*;2;`c2)] +c2 +-- +60 +``` + +### Issues converting to functional form + +To convert a `select` query to a functional form one may attempt to +apply the `parse` function to the query string: + +```q +q)parse "select sym,price,size from trade where price>50" +? +`trade +,,(>;`price;50) +0b +`sym`price`size!`sym`price`size +``` + +As we know, `parse` produces a parse tree and since some of the elements may themselves be parse trees we can’t immediately take the output of parse and plug it into the form `?[t;c;b;a]`. After a little playing around with the result of `parse` you might eventually figure out that the correct functional form is as follows. + +```q +q)funcQry:?[`trade;enlist(>;`price;50);0b;`sym`price`size! `sym`price`size] + +q)strQry:select sym,price,size from trade where price>50 q) +q)funcQry~strQry +1b +``` + +This, however, becomes more difficult as the query statements become more complex: + +```q +q)parse "select count i from trade where 140>(count;i) fby sym" +? +`trade +,,(>;140;(k){@[(#y)#x[0]0#x +1;g;:;x[0]'x[1]g:.=y]};(enlist;#:;`i);`sym)) +0b +(,`x)!,(#:;`i) +``` + +In this case, it is not obvious what the functional form of the above query should be, even after applying `parse`. + +There are three issues with this parse-and-“by eye” method to convert to the equivalent functional form. We will cover these in the next three subsections. + + +#### Parse trees and eval + +The first issue with passing a `select` query to `parse` is that each returned item is in unevaluated form. As [discussed here](parsetrees.md#eval-and-value), simply applying `value` to a parse tree does not work. However, if we evaluate each one of the arguments fully, then there would be no nested parse trees. We could then apply `value` to the result: + +```q +q)eval each parse "select count i from trade where 140>(count;i) fby sym" +? ++`sym`time`price`size!(`VOD`IBM`BP`VOD`IBM`IBM`HSBC`VOD`MS.. +,(>;140;(k){@[(#y)#x[0]0#x +1;g;:;x[0]'x[1]g:.=y]};(enlist;#:;`i);`sym)) +0b +(,`x)!,(#:;`i) +``` + +The equivalence below holds for a general qSQL query provided as a string: + +```q +q)value[str]~value eval each parse str +1b +``` + +In particular: + +```q +q)str:"select count i from trade where 140>(count;i) fby sym" + +q)value[str]~value eval each parse str +1b +``` + +In fact, since within the functional form we can refer to the table by name we can make this even clearer. Also, the first item in the result of `parse` applied to a `select` query will always be `?` (or `!` for a `delete`or `update` query) which cannot be evaluated any further. So we don’t need to apply `eval` to it. + +```q +q)pTree:parse str:"select count i from trade where 140>(count;i) fby sym" +q)@[pTree;2 3 4;eval] +? +`trade +,(>;140;(k){@[(#y)#x[0]0#x +1;g;:;x[0]'x[1]g:.=y]};(enlist;#:;`i);`sym)) +0b +(,`x)!,(#:;`i) +q)value[str] ~ value @[pTree;2 3 4;eval] +1b +``` + + +#### Variable representation in parse trees + +Recall that in a parse tree a variable is represented by a symbol containing its name. So to represent a symbol or a list of symbols, you must use `enlist` on that expression. In k, `enlist` is the unary form of the comma operator in k: + +```q +q)parse"3#`a`b`c`d`e`f" +# +3 +,`a`b`c`d`e`f +q)(#;3;enlist `a`b`c`d`e`f)~parse"3#`a`b`c`d`e`f" +1b +``` + +This causes a difficulty. As [discussed above](#variadic-operators), q has no unary syntax for operators. + +Which means the following isn’t a valid q expression and so returns an error. + +```q +q)(#;3;,`a`b`c`d`e`f) +', +``` + +In the parse tree we receive we need to somehow distinguish between k’s unary `,` (which we want to replace with `enlist`) and the binary Join operator, which we want to leave as it is. + + +#### Explicit definitions in `.q` are shown in full + +The `fby` in the `select` query above is represented by its full k +definition. + +```q +q)parse "fby" +k){@[(#y)#x[0]0#x 1;g;:;x[0]'x[1]g:.=y]} +``` + +While using the k form isn’t generally a problem from a functionality perspective, it does however make the resulting functional statement difficult to read. + + +### The solution + +We will write a function to automate the process of converting a `select` query into its equivalent functional form. + +This function, `buildQuery`, will return the functional form as a string. + +```q +q)buildQuery "select count i from trade where 140>(count;i) fby sym" +"?[trade;enlist(>;140;(fby;(enlist;count;`i);`sym));0b; + (enlist`x)! enlist (count;`i)]" +``` + +When executed it will always return the same result as the `select` query from which it is derived: + +```q +q)str:"select count i from trade where 140>(count;i) fby sym" +q)value[str]~value buildQuery str +1b +``` + +And since the same logic applies to `exec`, `update` and `delete` it will be able to convert to their corresponding functional forms also. + +To write this function we will solve the three issues outlined above: + +1. parse-tree items may be parse trees +2. parse trees use k’s unary syntax for operators +3. q keywords from `.q.` are replaced by their k definitions + +The first issue, where some items returned by `parse` may themselves be parse trees is easily resolved by applying `eval` to the individual items. + +The second issue is with k’s unary syntax for `,`. We want to replace it with the q keyword `enlist`. To do this we define a function that traverses the parse tree and detects if any element is an enlisted list of symbols or an enlisted single symbol. If it finds one we replace it with a string representation of `enlist` instead of `,`. + +```q +ereptest:{ //returns a boolean + (1=count x) and ((0=type x) and 11=type first x) or 11=type x} +ereplace:{"enlist",.Q.s1 first x} +funcEn:{$[ereptest x;ereplace x;0=type x;.z.s each x;x]} +``` + +Before we replace the item we first need to check it has the +correct form. We need to test if it is one of: + +- An enlisted list of syms. It will have type `0h`, count 1 and the type of its first item will be `11h` if and only if it is an enlisted list of syms. +- An enlisted single sym. It will have type `11h` and count 1 if and only if it is an enlisted single symbol. + +The `ereptest` function above performs this check, with `ereplace` performing the replacement. + +!!! tip "Console size" + + `.Q.s1` is dependent on the size of the console so make it larger if necessary. + +Since we are going to be checking a parse tree which may contain parse trees nested to arbitrary depth, we need a way to check all the elements down to the base level. + +We observe that a parse tree is a general list, and therefore of type `0h`. This knowledge combined with the use of `.z.s` allows us to scan a parse tree recursively. The logic goes: if what you have passed into `funcEn` is a parse tree then reapply the function to each element. + +To illustrate we examine the following `select` query. + +```q +q)show pTree:parse "select from trade where sym like \"F*\",not sym=`FD" +? +`trade +,((like;`sym;"F*");(~:;(=;`sym;,`FD))) 0b +() + +q)x:eval pTree 2 //apply eval to Where clause +``` + +Consider the Where clause in isolation. + +```q +q)x //a 2-list of Where clauses +(like;`sym;"F*") +(~:;(=;`sym;,`FD)) + +q)funcEn x +(like;`sym;"F*") +(~:;(=;`sym;"enlist`FD")) +``` + +Similarly we create a function which will replace k functions with +their q equivalents in string form, thus addressing the third issue above. + +```q +q)kreplace:{[x] $[`=qval:.q?x;x;string qval]} +q)funcK:{$[0=t:type x;.z.s each x;t<100h;x;kreplace x]} +``` + +Running these functions against our Where clause, we see the k +representations being converted to q. + +```q +q)x +(like;`sym;"F*") +(~:;(=;`sym;,`FD)) + +q)funcK x //replaces ~: with “not” +(like;`sym;"F*") +("not";(=;`sym;,`FD)) +``` + +Next, we make a slight change to `kreplace` and `ereplace` and combine them. + +```q +kreplace:{[x] $[`=qval:.q?x;x;"~~",string[qval],"~~"]} +ereplace:{"~~enlist",(.Q.s1 first x),"~~"} +q)funcEn funcK x +(like;`sym;"F*") ("~~not~~";(=;`sym;"~~enlist`FD~~")) +``` + +The double tilde here is going to act as a tag to allow us to differentiate from actual string elements in the parse tree. This allows us to drop the embedded quotation marks at a later stage inside the `buildQuery` function: + +```q +q)ssr/[;("\"~~";"~~\"");("";"")] .Q.s1 funcEn funcK x +"((like;`sym;\"F*\");(not;(=;`sym;enlist`FD)))" +``` + +thus giving us the correct format for the Where clause in a functional select. By applying the same logic to the rest of the parse tree we can write the `buildQuery` function. + +```q +q)buildQuery "select from trade where sym like \"F*\",not sym=`FD" +"?[trade;((like;`sym;\"F*\");(not;(=;`sym;enlist`FD)));0b;()]" +``` + +One thing to take note of is that since we use reverse lookup on the `.q` namespace and only want one result we occasionally get the wrong keyword back. + +```q +q)buildQuery "update tstamp:ltime tstamp from z" +"![z;();0b;(enlist`tstamp)!enlist (reciprocal;`tstamp)]" + +q).q`ltime +%: +q).q`reciprocal +%: +``` + +These instances are rare and a developer should be able to spot when they occur. Of course, the functional form will still work as expected but could confuse readers of the code. + + +#### Fifth and sixth arguments + +Functional select also has ranks 5 and 6; i.e. fifth and sixth arguments. + +:fontawesome-regular-hand-point-right: +_Q for Mortals_: [§9.12.1 Functional queries](/q4m3/9_Queries_q-sql/#9121-functional-select) + +We also cover these with the `buildQuery` function. + +```q +q)buildQuery "select[10 20] from trade" +"?[trade;();0b;();10 20]" +q)//5th parameter included +``` + +The 6th argument is a column and a direction to order the results by. Use `<` for ascending and `>` for descending. + +```q +q)parse"select[10;:) +`hopen`hclose + +q)qfind each ("<:";">:") //qfind defined above +hopen +hclose +``` + +We see that the k function for the 6th argument of the functional form is `<:` (ascending) or `>:` (descending). At first glance this appears to be `hopen` or `hclose`. In fact in earlier versions of q, `iasc` and `hopen` were equivalent (as were `idesc` and `hclose`). The definitions of `iasc` and `idesc` were later altered to signal a rank error if not applied to a list. + +```q +q)iasc +k){$[0h>@x;'`rank;@x;'`rank;>x]} + +q)iasc 7 +'rank +``` + +Since the columns of a table are lists, it is irrelevant whether the functional form uses the old or new version of `iasc` or `idesc`. + +The `buildQuery` function handles the 6th argument as a special case so will produce `iasc` or `idesc` as appropriate. + +```q +q)buildQuery "select[10 20;>price] from trade" +"?[trade;();0b;();10 20;(idesc;`price)]" +``` + +The full `buildQuery` function code is as follows: + +```q +\c 30 200 +tidy:{ssr/[;("\"~~";"~~\"");("";"")] $[","=first x;1_x;x]} +strBrk:{y,(";" sv x),z} + +//replace k representation with equivalent q keyword +kreplace:{[x] $[`=qval:.q?x;x;"~~",string[qval],"~~"]} +funcK:{$[0=t:type x;.z.s each x;t<100h;x;kreplace x]} + +//replace eg ,`FD`ABC`DEF with "enlist`FD`ABC`DEF" +ereplace:{"~~enlist",(.Q.s1 first x),"~~"} +ereptest:{(1=count x) and ((0=type x) and 11=type first x) or 11=type x} +funcEn:{$[ereptest x;ereplace x;0=type x;.z.s each x;x]} + +basic:{tidy .Q.s1 funcK funcEn x} + +addbraks:{"(",x,")"} + +//Where clause needs to be a list of Where clauses, +//so if only one Where clause, need to enlist. +stringify:{$[(0=type x) and 1=count x;"enlist ";""],basic x} + +//if a dictionary, apply to both keys and values +ab:{ + $[(0=count x) or -1=type x; .Q.s1 x; + 99=type x; (addbraks stringify key x ),"!",stringify value x; + stringify x] } + +inner:{[x] + idxs:2 3 4 5 6 inter ainds:til count x; + x:@[x;idxs;'[ab;eval]]; + if[6 in idxs;x[6]:ssr/[;("hopen";"hclose");("iasc";"idesc")] x[6]]; + //for select statements within select statements + x[1]:$[-11=type x 1;x 1;[idxs,:1;.z.s x 1]]; + x:@[x;ainds except idxs;string]; + x[0],strBrk[1_x;"[";"]"] } + +buildQuery:{inner parse x} +``` + + + ---- :fontawesome-solid-book-open: [qSQL](qsql.md) diff --git a/docs/basics/parsetrees.md b/docs/basics/parsetrees.md index 01cd366ed..98d4a6f7e 100644 --- a/docs/basics/parsetrees.md +++ b/docs/basics/parsetrees.md @@ -1,13 +1,19 @@ --- title: Parse trees – Basics – kdb+ and q documentation description: A parse tree represents an expression, not immediately evaluated. Its virtue is that the expression can be evaluated whenever and in whatever context it is needed. The two main functions dealing with parse trees are eval, which evaluates a parse tree, and parse, which returns one from a string containing a valid q expression. -keywords: kdb+, parse, parse tree, q +keywords: kdb+, parse, parse tree, q, k4, k +author: [Peter Storeng, Stephen Taylor, Simon Shanks] --- # Parse trees +## Overview +[`parse`](../ref/parse.md) is a useful tool for seeing how a statement in q is evaluated. Pass the `parse` keyword a q statement as a string and it will return the parse tree of that expression. -A _parse tree_ represents an expression, not immediately evaluated. Its virtue is that the expression can be evaluated whenever and in whatever context it is needed. The two main functions dealing with parse trees are [`eval`](../ref/eval.md), which evaluates a parse tree, and [`parse`](../ref/parse.md), which returns one from a string containing a valid q expression. +A _parse tree_ represents an expression, not immediately evaluated. Its virtue is that the expression can be evaluated whenever and in whatever context it is needed. The two main functions dealing with parse trees are: + +1. [`eval`](../ref/eval.md), which evaluates a parse tree. +2. [`parse`](../ref/parse.md), which returns one from a string containing a valid q expression. Parse trees may be the result of applying `parse`, or constructed explicitly. The simplest parse tree is a single constant expression. Note that, in a parse tree, a variable is represented by a symbol containing its name. To represent a symbol or a list of symbols, you will need to use [`enlist`](../ref/enlist.md) on that expression. @@ -32,34 +38,283 @@ q)eval ((/;+);(til;(+;2;2))) 6 ``` +## k4, q and `q.k` -## Functional form of a qSQL query +kdb+ is a database management system which ships with the general-purpose and database language q. Q is an embedded domain-specific language implemented in the k programming language, sometimes known as k4. The q interpreter can switch between q and k modes and evaluate expressions written in k as well as q. -Sometimes you need to translate a [qSQL query](qsql.md) into its [functional form](funsql.md), for example, so you can pass column names as arguments. -Translation can be non-trivial. +The `parse` keyword can expose the underlying implementation in `k`. + +The k language is for KX implementors. +It is not documented or supported for use outside KX. +All the same functionality is available in the much more readable q language. However in certain cases, such as debugging, a basic understanding of some k syntax can be useful. + +The `q.k` file is part of the standard installation of q and loads into each q session on startup. It defines many of the q keywords in terms of k. To see how a q keyword is defined in terms of k we could check the `q.k` file or simply enter it into the q prompt: + +```q +q)type +@: +``` + +The `parse` keyword on an operation involving the example above will expose the `k` code. Using the underlying code, it can be run using kdb+ in-build k interpreter to show that it produces the same result: +```q +q)type 6 +-7h +q)parse "type 6" +@: +6 +q)k)@6 +-7h +``` + +A few q keywords are defined natively from C and do not have a k representation: + +```q +q)like +like +``` + +## Parse trees + +A parse tree is a q construct which represents an expression but which is not immediately evaluated. It takes the form of a list where the first item is a function and the remaining items are the arguments. Any of the items of the list can be parse trees themselves. + +Note that in a parse tree a variable is represented by a symbol containing its name. Thus to distinguish a symbol or a list of symbols from a variable it is necessary to enlist that expression. When we apply the `parse` function to create a parse tree, explicit definitions in `.q` are shown in their full k form. In particular, an enlisted element is represented by a preceding comma. + +```q +q)parse"5 6 7 8 + 1 2 3 4" ++ //the function/operator +5 6 7 8 //first argument +1 2 3 4 //second argument +``` +```q +q)parse"2+4*7" ++ //the function/operator +2 //first argument +(*;4;7) //second argument, itself a parse tree +``` +```q +q)v:`e`f +q)`a`b`c,`d,v +`a`b`c`d`e`f +q)parse"`a`b`c,`d,v" +, // join operator +,`a`b`c //actual symbols/lists of symbols are enlisted +(,;,`d;`v) //v a variable represented as a symbol +``` + +We can also manually construct a parse tree: + +```q +q)show pTree:parse "(aggr;data) fby grp" +k){@[(#y)#x[0]0#x 1;g;:;x[0]'x[1]g:.=y]} //fby in k form +(enlist;`aggr;`data) +`grp + +q)pTree~(fby;(enlist;`aggr;`data);`grp) //manually constructed +1b //parse tree +``` + +As asserted previously every statement in q parses into the form: + +```txt +(function; arg 1; …; arg n) +``` + +where every item could itself be a parse tree. In this way we see that every action in q is essentially a function evaluation. + + +## `eval` and `value` + +[`eval`](../ref/eval.md) can be thought of as the dual to `parse`. The following holds for all valid q statements (without side effects) put into a string. (Recall that `value` executes the command inside a string.) + +```q +//a tautology (for all valid q expressions str) +q)value[str]~eval parse str +1b +q)value["2+4*7"]~eval parse"2+4*7" //simple example +1b +``` + +When passed a list, `value` applies the first item (which contains a function) to the rest of the list (the arguments). + +```q +q)function[arg 1;..;arg n] ~ value(function;arg 1;..;arg n) +1b +``` + +When `eval` and `value` operate on a parse tree with no nested parse trees they return the same result. However it is not true that `eval` and `value` are equivalent in general. `eval` operates on parse trees, evaluating any nested parse trees, whereas `value` operates on the literals. + +```q +q)value(+;7;3) //parse tree, with no nested trees +10 +q)eval(+;7;3) +10 +q)eval(+;7;(+;2;1)) //parse tree with nested trees +10 +q)value(+;7;(+;2;1)) +'type +``` +```q +q)value(,;`a;`b) +`a`b +q)eval(,;`a;`b) //no variable b defined +'b +q)eval(,;enlist `a;enlist `b) +`a`b +``` + + +## Variadic operators + +Many operators and some keywords in k and q are [variadic](glossary.md#variadic): they are overloaded so that the behavior of the operator changes depending on the number and type of arguments. In q (not k) the unary form of operators such as (`+`, `$`, `.`, `&` etc.) is disabled: keywords are provided instead. -!!! tip "Use `parse` to reveal the functional form of a qSQL query" +For example, in k the unary form of the `$` operator equates to the `string` +keyword in q. -> The result will often include [k code](exposed-infrastructure.md) but it is usually recognizable and you can use it in functional form. — _Q for Mortals_ §A.67 +```q +q)k)$42 +"42" +q)$42 //$ unary form disabled in q +'$ +q)string 42 +"42" +``` + +!!! info "A parenthesized variadic function applied prefix is parsed as its unary form." + +```q +q)($)42 +"42" +``` + +A familiar example of a variadic function is the Add Over function `+/` derived by applying the Over iterator to the Add operator. + +```q +q)+/[1000;2 3 4] // +/ applied binary +1009 +q)+/[2 3 4] // +/ applied unary +9 +q)(+/)2 3 4 // +/ applied unary +9 +``` + +In k, the unary form of an operator can also be specified explicitly by suffixing it with a colon. + +```q +q)k)$:42 +"42" +``` + +`+:` is a unary operator; the unary form of `+`. We can see this in the parse tree: + +```q +q)parse"6(+)4" +6 +(+:;4) +``` + +The items of a `parse` result use k syntax. Since (most of) the q keywords are defined in the `.q` namespace, we can use dictionary reverse lookup to find the meaning. + +```q +q).q?(+:) +`flip +``` + +So we can see that in k, the unary form of `+` corresponds to `flip` in q. + +```q +q)d:`c1`c2`c3!(1 2;3 4;5 6) +q)d +c1| 1 2 +c2| 3 4 +c3| 5 6 +q)k)+d +c1 c2 c3 +-------- +1 3 5 +2 4 6 +q)k)+:d +c1 c2 c3 +-------- +1 3 5 +2 4 6 +``` -Remove one level for the functional form. +!!! warning "Exposed infrastructure" + + The unary forms of operators are [exposed infrastructure](). + Their use in q expressions is **strongly discouraged**. + Use the corresponding q keywords instead. + + For example, write `flip d` rather than `(+:)d`. + + The unary forms are reviewed here to enable an understanding of parse trees, in which k syntax is visible. + + + +When using reverse lookup on the `.q` context we are slightly hampered by the fact that it is not an injective mapping. The Find `?` operator returns only the first q keyword matching the k expression. In some cases there is more than one. Instead use the following function: ```q -q)t:([]c1:`a`b`c; c2:10 20 30) -q)parse "select c2:2*c2 from t where c1=`c" -? -`t -,,(=;`c1;,`c) -0b -(,`c2)!,(*;2;`c2) +q)qfind:{key[.q]where x~/:string value .q} -q)?[`t; enlist (=;`c1;enlist `c); 0b; (enlist `c2)!enlist (*;2;`c2)] -c2 --- -60 +q)qfind"k){x*y div x:$[16h=abs[@x];\"j\"$x;x]}" +,`xbar +q)qfind"~:" +`not`hdel +``` + +We see `not` and `hdel` are equivalent. Writing the following could be confusing: + +```q +q)hdel 01001b +10110b ``` ----- -:fontawesome-regular-map: -[Parse trees and functional forms](../wp/parse-trees.md) +So q provides two different names for clarity. + + +## Iterators as higher-order functions + +An iterator applies to a value (function, list, or dictionary) to produce a related function. This is again easy to see by inspecting the parse tree: + +```q +q)+/[1 2 3 4] +10 +q)parse "+/[1 2 3 4]" +(/;+) +1 2 3 4 +``` + +The first item of the parse tree is `(/;+)`, which is itself a parse +tree. We know the first item of a parse tree is to be applied to the +remaining items. Here `/` (the Over iterator) is applied to `+` to +produce a new function which sums the items of a list. + +:fontawesome-regular-hand-point-right: +[Iterators](../wp/iterators/index.md) + + +## Functional form of a qSQL query + +Sometimes you need to translate a [qSQL query](qsql.md) into its [functional form](funsql.md), for example, so you can pass column names as arguments. +Details are provided [here](funsql.md#conversion-using-parse). diff --git a/docs/ref/parse.md b/docs/ref/parse.md index 8460c46e2..0a975c561 100644 --- a/docs/ref/parse.md +++ b/docs/ref/parse.md @@ -120,4 +120,4 @@ q)views[] [Parse trees](../basics/parsetrees.md)
:fontawesome-regular-map: -[Parse trees and functional forms](../wp/parse-trees.md) +[Convert qSQL to functional forms using parse](../basics/parsetrees.md#conversion-using-parse) diff --git a/docs/wp/index.md b/docs/wp/index.md index 9c8ed10fc..c59600e9b 100644 --- a/docs/wp/index.md +++ b/docs/wp/index.md @@ -73,7 +73,6 @@ White papers are flagged in the navigation menus. ## :fontawesome-solid-code: Programming in q - [**Iterators**](iterators/index.md)
Conor Slattery & Stephen Taylor, 2019.03 -- [**Parse trees and functional forms**](parse-trees.md)
Peter Storeng & Stephen Taylor, 2019.03 - [**kdb+ query scaling**](query-scaling.md)
Ian Lester, 2014.01 - [**The application of foreign keys and linked columns in kdb+**](foreign-keys.md)
Kevin Smyth, 2013.04 - [**Columnar database and query optimization**](columnar-database/index.md)
Ciáran Gorman, 2012.06 diff --git a/docs/wp/parse-trees.md b/docs/wp/parse-trees.md deleted file mode 100644 index 59600f5f3..000000000 --- a/docs/wp/parse-trees.md +++ /dev/null @@ -1,689 +0,0 @@ ---- -title: Parse trees and functional forms | kdb+ and q documentation -description: How to understand parse trees and use functional forms in q queries; how to convert qSQL expressions to functional form. -author: [Peter Storeng, Stephen Taylor] -keywords: functional, kdb+, parse, parse tree, q, qSQL, query, SQL ---- -# Parse trees and functional forms - -by [Peter Storeng & Stephen Taylor](#authors) -{: .wp-author} - - -The importance of understanding and using the functional form of qSQL statements in kdb+ cannot be overstated. The functional form has many advantages over the qSQL approach, including the ability to select columns and build Where clauses dynamically. It is important for any q programmer to understand the functional form fully and how to convert to it from qSQL. - -Applying `parse` to a qSQL statement written as a string will return the internal representation of the functional form. With some manipulation this can then be used to piece together the functional form in q. This generally becomes more difficult as the query becomes more complex and requires a deep understanding of what kdb+ is doing when it parses qSQL form. - -The main goal of this paper is to show in detail how this conversion works, so that it is understood how to build the functional form of qSQL statements correctly. In order to do this, we will need to look at how q and k commands relate to each other, as the parse function often returns the cryptic k code for functions. An understanding of what parse trees are, and how to use them, is also vital in the building of functional queries. - -Finally, this paper will look at creating a function which will automate the process of converting qSQL statements into functional form. This is to be used as a helpful development tool when facing the task of writing a tricky functional statement. - -All tests were run using kdb+ version 3.2 (2015.01.14). - - -## k4, q and `q.k` - -kdb+ is a database management system which ships with the general-purpose and database language q. Q is an embedded domain-specific language implemented in the k programming language, sometimes known as k4. The q interpreter can switch between q and k modes and evaluate expressions written in k as well as q. - -The k language is for KX implementors. -It is not documented or supported for use outside KX. -All the same functionality is available in the much more readable q language. However in certain cases, such as debugging, a basic understanding of some k syntax can be useful. - -The `q.k` file is part of the standard installation of q and loads into each q session on startup. It defines many of the q keywords in terms of k. To see how a q keyword is defined in terms of k we could check the `q.k` file or simply enter it into the q prompt: - -```q -q)key -!: -``` - -A few q keywords are defined natively from C and do not have a k representation: - -```q -q)like -like -``` - - -## The `parse` keyword - -`parse` is a useful tool for seeing how a statement in q is evaluated. Pass the `parse` keyword a q statement as a string and it will return the parse tree of that expression. - - -### Parse trees - -A parse tree is a q construct which represents an expression but which is not immediately evaluated. It takes the form of a list where the first item is a function and the remaining items are the arguments. Any of the items of the list can be parse trees themselves. - -Note that in a parse tree a variable is represented by a symbol containing its name. Thus to distinguish a symbol or a list of symbols from a variable it is necessary to enlist that expression. When we apply the `parse` function to create a parse tree, explicit definitions in `.q` are shown in their full k form. In particular, an enlisted element is represented by a preceding comma. - -```q -q)parse"5 6 7 8 + 1 2 3 4" -+ //the function/operator -5 6 7 8 //first argument -1 2 3 4 //second argument -``` -```q -q)parse"2+4*7" -+ //the function/operator -2 //first argument -(*;4;7) //second argument, itself a parse tree -``` -```q -q)v:`e`f -q)`a`b`c,`d,v -`a`b`c`d`e`f -q)parse"`a`b`c,`d,v" -, // join operator -,`a`b`c //actual symbols/lists of symbols are enlisted -(,;,`d;`v) //v a variable represented as a symbol -``` - -We can also manually construct a parse tree: - -```q -q)show pTree:parse "(aggr;data) fby grp" -k){@[(#y)#x[0]0#x 1;g;:;x[0]'x[1]g:.=y]} //fby in k form -(enlist;`aggr;`data) -`grp - -q)pTree~(fby;(enlist;`aggr;`data);`grp) //manually constructed -1b //parse tree -``` - -As asserted previously every statement in q parses into the form: - -```txt -(function; arg 1; …; arg n) -``` - -where every item could itself be a parse tree. In this way we see that every action in q is essentially a function evaluation. - - -### `eval` and `value` - -`eval` can be thought of as the dual to `parse`. The following holds for all valid q statements (without side effects) put into a string. (Recall that `value` executes the command inside a string.) - -```q -//a tautology (for all valid q expressions str) -q)value[str]~eval parse str -1b -q)value["2+4*7"]~eval parse"2+4*7" //simple example -1b -``` - -When passed a list, `value` applies the first item (which contains a function) to the rest of the list (the arguments). - -```q -q)function[arg 1;..;arg n] ~ value(function;arg 1;..;arg n) -1b -``` - -When `eval` and `value` operate on a parse tree with no nested parse trees they return the same result. However it is not true that `eval` and `value` are equivalent in general. `eval` operates on parse trees, evaluating any nested parse trees, whereas `value` operates on the literals. - -```q -q)value(+;7;3) //parse tree, with no nested trees -10 -q)eval(+;7;3) -10 -q)eval(+;7;(+;2;1)) //parse tree with nested trees -10 -q)value(+;7;(+;2;1)) -'type -``` -```q -q)value(,;`a;`b) -`a`b -q)eval(,;`a;`b) //no variable b defined -'b -q)eval(,;enlist `a;enlist `b) -`a`b -``` - - -### Variadic operators - -Many operators and some keywords in k and q are [variadic](../basics/glossary.md#variadic): they are overloaded so that the behavior of the operator changes depending on the number and type of arguments. In q (not k) the unary form of operators such as (`+`, `$`, `.`, `&` etc.) is disabled: keywords are provided instead. - -For example, in k the unary form of the `$` operator equates to the `string` -keyword in q. - -```q -q)k)$42 -"42" -q)$42 //$ unary form disabled in q -'$ -q)string 42 -"42" -``` - -!!! info "A parenthesized variadic function applied prefix is parsed as its unary form." - -```q -q)($)42 -"42" -``` - -A familiar example of a variadic function is the Add Over function `+/` derived by applying the Over iterator to the Add operator. - -```q -q)+/[1000;2 3 4] // +/ applied binary -1009 -q)+/[2 3 4] // +/ applied unary -9 -q)(+/)2 3 4 // +/ applied unary -9 -``` - -In k, the unary form of an operator can also be specified explicitly by suffixing it with a colon. - -```q -q)k)$:42 -"42" -``` - -`+:` is a unary operator; the unary form of `+`. We can see this in the parse tree: - -```q -q)parse"6(+)4" -6 -(+:;4) -``` - -The items of a `parse` result use k syntax. Since (most of) the q keywords are defined in the `.q` namespace, we can use dictionary reverse lookup to find the meaning. - -```q -q).q?(+:) -`flip -``` - -So we can see that in k, the unary form of `+` corresponds to `flip` in q. - -```q -q)d:`c1`c2`c3!(1 2;3 4;5 6) -q)d -c1| 1 2 -c2| 3 4 -c3| 5 6 -q)k)+d -c1 c2 c3 --------- -1 3 5 -2 4 6 -q)k)+:d -c1 c2 c3 --------- -1 3 5 -2 4 6 -``` - -!!! warning "Exposed infrastructure" - - The unary forms of operators are [exposed infrastructure](). - Their use in q expressions is **strongly discouraged**. - Use the corresponding q keywords instead. - - For example, write `flip d` rather than `(+:)d`. - - The unary forms are reviewed here to enable an understanding of parse trees, in which k syntax is visible. - - - -When using reverse lookup on the `.q` context we are slightly hampered by the fact that it is not an injective mapping. The Find `?` operator returns only the first q keyword matching the k expression. In some cases there is more than one. Instead use the following function: - -```q -q)qfind:{key[.q]where x~/:string value .q} - -q)qfind"k){x*y div x:$[16h=abs[@x];\"j\"$x;x]}" -,`xbar -q)qfind"~:" -`not`hdel -``` - -We see `not` and `hdel` are equivalent. Writing the following could be confusing: - -```q -q)hdel 01001b -10110b -``` - -So q provides two different names for clarity. - - -### Iterators as higher-order functions - -An iterator applies to a value (function, list, or dictionary) to produce a related function. This is again easy to see by inspecting the parse tree: - -```q -q)+/[1 2 3 4] -10 -q)parse "+/[1 2 3 4]" -(/;+) -1 2 3 4 -``` - -The first item of the parse tree is `(/;+)`, which is itself a parse -tree. We know the first item of a parse tree is to be applied to the -remaining items. Here `/` (the Over iterator) is applied to `+` to -produce a new function which sums the items of a list. - -:fontawesome-regular-hand-point-right: -[Iterators](iterators/index.md) - - -## Functional queries - -Alongside each qSQL query we also have the equivalent functional -forms. These are especially useful for programmatically-generated -queries, such as when column names are dynamically queried. - -```q -?[t;c;b;a] // select and exec -![t;c;b;a] // update and delete -``` - -Here - -- `t` is a table -- `c` is a list of constraints in the portable parse tree format -- `b` is a dictionary of group-bys -- `a` is a dictionary of aggregates - -:fontawesome-regular-hand-point-right: -_Q for Mortals_: [§9.12 Functional forms of queries](/q4m3/9_Queries_q-sql/#912-functional-forms) - -The q interpreter parses the syntactic forms of `select`, `exec`, `update` -and `delete` into their equivalent functional forms. Therefore there is -no performance difference between a qSQL query and a functional -one. - - -### Issues converting to functional form - -To convert a `select` query to a functional form one may attempt to -apply the `parse` function to the query string: - -```q -q)parse "select sym,price,size from trade where price>50" -? -`trade -,,(>;`price;50) -0b -`sym`price`size!`sym`price`size -``` - -As we know, `parse` produces a parse tree and since some of the elements may themselves be parse trees we can’t immediately take the output of parse and plug it into the form `?[t;c;b;a]`. After a little playing around with the result of `parse` you might eventually figure out that the correct functional form is as follows. - -```q -q)funcQry:?[`trade;enlist(>;`price;50);0b;`sym`price`size! `sym`price`size] - -q)strQry:select sym,price,size from trade where price>50 q) -q)funcQry~strQry -1b -``` - -This, however, becomes more difficult as the query statements become more complex: - -```q -q)parse "select count i from trade where 140>(count;i) fby sym" -? -`trade -,,(>;140;(k){@[(#y)#x[0]0#x -1;g;:;x[0]'x[1]g:.=y]};(enlist;#:;`i);`sym)) -0b -(,`x)!,(#:;`i) -``` - -In this case, it is not obvious what the functional form of the above query should be, even after applying `parse`. - -There are three issues with this parse-and-“by eye” method to convert to the equivalent functional form. We will cover these in the next three subsections. - - -#### Parse trees and eval - -The first issue with passing a `select` query to `parse` is that each returned item is in unevaluated form. As [discussed above](#eval-and-value), simply applying `value` to a parse tree does not work. However, if we evaluate each one of the arguments fully, then there would be no nested parse trees. We could then apply `value` to the result: - -```q -q)eval each parse "select count i from trade where 140>(count;i) fby sym" -? -+`sym`time`price`size!(`VOD`IBM`BP`VOD`IBM`IBM`HSBC`VOD`MS.. -,(>;140;(k){@[(#y)#x[0]0#x -1;g;:;x[0]'x[1]g:.=y]};(enlist;#:;`i);`sym)) -0b -(,`x)!,(#:;`i) -``` - -The equivalence below holds for a general qSQL query provided as a string: - -```q -q)value[str]~value eval each parse str -1b -``` - -In particular: - -```q -q)str:"select count i from trade where 140>(count;i) fby sym" - -q)value[str]~value eval each parse str -1b -``` - -In fact, since within the functional form we can refer to the table by name we can make this even clearer. Also, the first item in the result of `parse` applied to a `select` query will always be `?` (or `!` for a `delete`or `update` query) which cannot be evaluated any further. So we don’t need to apply `eval` to it. - -```q -q)pTree:parse str:"select count i from trade where 140>(count;i) fby sym" -q)@[pTree;2 3 4;eval] -? -`trade -,(>;140;(k){@[(#y)#x[0]0#x -1;g;:;x[0]'x[1]g:.=y]};(enlist;#:;`i);`sym)) -0b -(,`x)!,(#:;`i) -q)value[str] ~ value @[pTree;2 3 4;eval] -1b -``` - - -#### Variable representation in parse trees - -Recall that in a parse tree a variable is represented by a symbol containing its name. So to represent a symbol or a list of symbols, you must use `enlist` on that expression. In k, `enlist` is the unary form of the comma operator in k: - -```q -q)parse"3#`a`b`c`d`e`f" -# -3 -,`a`b`c`d`e`f -q)(#;3;enlist `a`b`c`d`e`f)~parse"3#`a`b`c`d`e`f" -1b -``` - -This causes a difficulty. As [discussed above](#variadic-operators), q has no unary syntax for operators. - -Which means the following isn’t a valid q expression and so returns an error. - -```q -q)(#;3;,`a`b`c`d`e`f) -', -``` - -In the parse tree we receive we need to somehow distinguish between k’s unary `,` (which we want to replace with `enlist`) and the binary Join operator, which we want to leave as it is. - - -#### Explicit definitions in `.q` are shown in full - -The `fby` in the `select` query above is represented by its full k -definition. - -```q -q)parse "fby" -k){@[(#y)#x[0]0#x 1;g;:;x[0]'x[1]g:.=y]} -``` - -While using the k form isn’t generally a problem from a functionality perspective, it does however make the resulting functional statement difficult to read. - - -### The solution - -We will write a function to automate the process of converting a `select` query into its equivalent functional form. - -This function, `buildQuery`, will return the functional form as a string. - -```q -q)buildQuery "select count i from trade where 140>(count;i) fby sym" -"?[trade;enlist(>;140;(fby;(enlist;count;`i);`sym));0b; - (enlist`x)! enlist (count;`i)]" -``` - -When executed it will always return the same result as the `select` query from which it is derived: - -```q -q)str:"select count i from trade where 140>(count;i) fby sym" -q)value[str]~value buildQuery str -1b -``` - -And since the same logic applies to `exec`, `update` and `delete` it will be able to convert to their corresponding functional forms also. - -To write this function we will solve the three issues outlined above: - -1. parse-tree items may be parse trees -2. parse trees use k’s unary syntax for operators -3. q keywords from `.q.` are replaced by their k definitions - -The first issue, where some items returned by `parse` may themselves be parse trees is easily resolved by applying `eval` to the individual items. - -The second issue is with k’s unary syntax for `,`. We want to replace it with the q keyword `enlist`. To do this we define a function that traverses the parse tree and detects if any element is an enlisted list of symbols or an enlisted single symbol. If it finds one we replace it with a string representation of `enlist` instead of `,`. - -```q -ereptest:{ //returns a boolean - (1=count x) and ((0=type x) and 11=type first x) or 11=type x} -ereplace:{"enlist",.Q.s1 first x} -funcEn:{$[ereptest x;ereplace x;0=type x;.z.s each x;x]} -``` - -Before we replace the item we first need to check it has the -correct form. We need to test if it is one of: - -- An enlisted list of syms. It will have type `0h`, count 1 and the type of its first item will be `11h` if and only if it is an enlisted list of syms. -- An enlisted single sym. It will have type `11h` and count 1 if and only if it is an enlisted single symbol. - -The `ereptest` function above performs this check, with `ereplace` performing the replacement. - -!!! tip "Console size" - - `.Q.s1` is dependent on the size of the console so make it larger if necessary. - -Since we are going to be checking a parse tree which may contain parse trees nested to arbitrary depth, we need a way to check all the elements down to the base level. - -We observe that a parse tree is a general list, and therefore of type `0h`. This knowledge combined with the use of `.z.s` allows us to scan a parse tree recursively. The logic goes: if what you have passed into `funcEn` is a parse tree then reapply the function to each element. - -To illustrate we examine the following `select` query. - -```q -q)show pTree:parse "select from trade where sym like \"F*\",not sym=`FD" -? -`trade -,((like;`sym;"F*");(~:;(=;`sym;,`FD))) 0b -() - -q)x:eval pTree 2 //apply eval to Where clause -``` - -Consider the Where clause in isolation. - -```q -q)x //a 2-list of Where clauses -(like;`sym;"F*") -(~:;(=;`sym;,`FD)) - -q)funcEn x -(like;`sym;"F*") -(~:;(=;`sym;"enlist`FD")) -``` - -Similarly we create a function which will replace k functions with -their q equivalents in string form, thus addressing the third issue above. - -```q -q)kreplace:{[x] $[`=qval:.q?x;x;string qval]} -q)funcK:{$[0=t:type x;.z.s each x;t<100h;x;kreplace x]} -``` - -Running these functions against our Where clause, we see the k -representations being converted to q. - -```q -q)x -(like;`sym;"F*") -(~:;(=;`sym;,`FD)) - -q)funcK x //replaces ~: with “not” -(like;`sym;"F*") -("not";(=;`sym;,`FD)) -``` - -Next, we make a slight change to `kreplace` and `ereplace` and combine them. - -```q -kreplace:{[x] $[`=qval:.q?x;x;"~~",string[qval],"~~"]} -ereplace:{"~~enlist",(.Q.s1 first x),"~~"} -q)funcEn funcK x -(like;`sym;"F*") ("~~not~~";(=;`sym;"~~enlist`FD~~")) -``` - -The double tilde here is going to act as a tag to allow us to differentiate from actual string elements in the parse tree. This allows us to drop the embedded quotation marks at a later stage inside the `buildQuery` function: - -```q -q)ssr/[;("\"~~";"~~\"");("";"")] .Q.s1 funcEn funcK x -"((like;`sym;\"F*\");(not;(=;`sym;enlist`FD)))" -``` - -thus giving us the correct format for the Where clause in a functional select. By applying the same logic to the rest of the parse tree we can write the `buildQuery` function. - -```q -q)buildQuery "select from trade where sym like \"F*\",not sym=`FD" -"?[trade;((like;`sym;\"F*\");(not;(=;`sym;enlist`FD)));0b;()]" -``` - -One thing to take note of is that since we use reverse lookup on the `.q` namespace and only want one result we occasionally get the wrong keyword back. - -```q -q)buildQuery "update tstamp:ltime tstamp from z" -"![z;();0b;(enlist`tstamp)!enlist (reciprocal;`tstamp)]" - -q).q`ltime -%: -q).q`reciprocal -%: -``` - -These instances are rare and a developer should be able to spot when they occur. Of course, the functional form will still work as expected but could confuse readers of the code. - - -#### Fifth and sixth arguments - -Functional select also has ranks 5 and 6; i.e. fifth and sixth arguments. - -:fontawesome-regular-hand-point-right: -_Q for Mortals_: [§9.12.1 Functional queries](/q4m3/9_Queries_q-sql/#9121-functional-select) - -We also cover these with the `buildQuery` function. - -```q -q)buildQuery "select[10 20] from trade" -"?[trade;();0b;();10 20]" -q)//5th parameter included -``` - -The 6th argument is a column and a direction to order the results by. Use `<` for ascending and `>` for descending. - -```q -q)parse"select[10;:) -`hopen`hclose - -q)qfind each ("<:";">:") //qfind defined above -hopen -hclose -``` - -We see that the k function for the 6th argument of the functional form is `<:` (ascending) or `>:` (descending). At first glance this appears to be `hopen` or `hclose`. In fact in earlier versions of q, `iasc` and `hopen` were equivalent (as were `idesc` and `hclose`). The definitions of `iasc` and `idesc` were later altered to signal a rank error if not applied to a list. - -```q -q)iasc -k){$[0h>@x;'`rank;@x;'`rank;>x]} - -q)iasc 7 -'rank -``` - -Since the columns of a table are lists, it is irrelevant whether the functional form uses the old or new version of `iasc` or `idesc`. - -The `buildQuery` function handles the 6th argument as a special case so will produce `iasc` or `idesc` as appropriate. - -```q -q)buildQuery "select[10 20;>price] from trade" -"?[trade;();0b;();10 20;(idesc;`price)]" -``` - -The full `buildQuery` function code is as follows: - -```q -\c 30 200 -tidy:{ssr/[;("\"~~";"~~\"");("";"")] $[","=first x;1_x;x]} -strBrk:{y,(";" sv x),z} - -//replace k representation with equivalent q keyword -kreplace:{[x] $[`=qval:.q?x;x;"~~",string[qval],"~~"]} -funcK:{$[0=t:type x;.z.s each x;t<100h;x;kreplace x]} - -//replace eg ,`FD`ABC`DEF with "enlist`FD`ABC`DEF" -ereplace:{"~~enlist",(.Q.s1 first x),"~~"} -ereptest:{(1=count x) and ((0=type x) and 11=type first x) or 11=type x} -funcEn:{$[ereptest x;ereplace x;0=type x;.z.s each x;x]} - -basic:{tidy .Q.s1 funcK funcEn x} - -addbraks:{"(",x,")"} - -//Where clause needs to be a list of Where clauses, -//so if only one Where clause, need to enlist. -stringify:{$[(0=type x) and 1=count x;"enlist ";""],basic x} - -//if a dictionary, apply to both keys and values -ab:{ - $[(0=count x) or -1=type x; .Q.s1 x; - 99=type x; (addbraks stringify key x ),"!",stringify value x; - stringify x] } - -inner:{[x] - idxs:2 3 4 5 6 inter ainds:til count x; - x:@[x;idxs;'[ab;eval]]; - if[6 in idxs;x[6]:ssr/[;("hopen";"hclose");("iasc";"idesc")] x[6]]; - //for select statements within select statements - x[1]:$[-11=type x 1;x 1;[idxs,:1;.z.s x 1]]; - x:@[x;ainds except idxs;string]; - x[0],strBrk[1_x;"[";"]"] } - -buildQuery:{inner parse x} -``` - -## Authors - -**Peter Storeng** is a mathematician and software developer who has been using kdb+ since 2012. - -![Stephen Taylor](../img/faces/stephentaylor.png) -{: .small-face} - -**Stephen Taylor** FRSA has followed the evolution of the Iversonian languages through APL, J, k, and q, and is a former editor of [_Vector_](https://vector.org.uk), the journal of the British APL Association. - diff --git a/mkdocs.yml b/mkdocs.yml index 5a7a96ebb..a1656bc2c 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -419,7 +419,6 @@ nav: - Metadata: basics/metadata.md - Namespaces: basics/namespaces.md - Parse trees: basics/parsetrees.md - - Parse trees, functional SQL: wp/parse-trees.md - qSQL: - qSQL queries: basics/qsql.md - Functional qSQL: basics/funsql.md From ac191a3f865198c14e97f86f1ccc94f8b9f8bd9b Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Tue, 24 Sep 2024 10:24:36 +0100 Subject: [PATCH 09/54] fix broken link in github page to wind --- docs/github.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/github.md b/docs/github.md index 5de692442..7af49604e 100644 --- a/docs/github.md +++ b/docs/github.md @@ -808,7 +808,7 @@ GitHub topic queries:   :fontawesome-brands-github: [timeseries/twitter-kdb](https://github.com/timeseries/twitter-kdb) -[Wind资讯](https://www.wind.com.cn/en/):fontawesome-brands-github: [FlyingOE/q_Wind](https://github.com/FlyingOE/q_Wind) +[Wind资讯](https://www.wind.com.cn/):fontawesome-brands-github: [FlyingOE/q_Wind](https://github.com/FlyingOE/q_Wind) :fontawesome-brands-yahoo: Yahoo!:fontawesome-brands-github: [fdeleze/tickYahoo](https://github.com/fdeleze/tickYahoo) From fdfa7fb231ad532ec278c16e6e52bafa31c18f2a Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Tue, 24 Sep 2024 10:48:03 +0100 Subject: [PATCH 10/54] fix broken link to A+ --- docs/kb/lp.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/kb/lp.md b/docs/kb/lp.md index 41e5c1c8a..4f424a643 100644 --- a/docs/kb/lp.md +++ b/docs/kb/lp.md @@ -13,7 +13,7 @@ _Linear Programming is a large topic, of which this article reviews just a few a [![Ken Iverson](../img/kei01.jpg)](https://en.wikipedia.org/wiki/Kenneth_E._Iverson "Wikipedia: Kenneth E. Iverson") Q is a descendant of the notation devised at Harvard by the [Turing Award](https://en.wikipedia.org/wiki/Turing_Award) winner, mathematician [Ken Iverson](https://en.wikipedia.org/wiki/Kenneth_E._Iverson), when he worked with [Howard Aiken](https://en.wikipedia.org/wiki/Howard_H._Aiken) and Nobel Prize winner [Wassily Leontief](https://en.wikipedia.org/wiki/Wassily_Leontief) on the computation of economic input-output tables. At Harvard, Ken Iverson and fellow Turing Award winner [Fred Brooks](https://en.wikipedia.org/wiki/Fred_Brooks) gave the world’s first course in what was then called ‘data processing’. - Like other descendants of Iverson Notation (e.g. [A+](http://www.aplusdev.org/index.html), [APL](https://en.wikipedia.org/wiki/APL_(programming_language)), [J](https://en.wikipedia.org/wiki/J_(programming_language))), q inherits compact and powerful expression of linear algebra. + Like other descendants of Iverson Notation (e.g. [A+](https://en.wikipedia.org/wiki/A%2B_(programming_language)), [APL](https://en.wikipedia.org/wiki/APL_(programming_language)), [J](https://en.wikipedia.org/wiki/J_(programming_language))), q inherits compact and powerful expression of linear algebra. Q Math Library: :fontawesome-brands-github: [zholos/qml](https://github.com/zholos/qml) From 284933512a8b398a4fa74e3cf742944008ba9337 Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Tue, 24 Sep 2024 17:18:39 +0100 Subject: [PATCH 11/54] 4.1 2024.07.08 .z.H returns sorted list of handles --- docs/ref/dotz.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/ref/dotz.md b/docs/ref/dotz.md index 4935a7f8b..b9bba81ce 100644 --- a/docs/ref/dotz.md +++ b/docs/ref/dotz.md @@ -297,9 +297,9 @@ q).z.f ## `.z.H` (active sockets) -Active sockets as a list. (A low-cost method.) +Active sockets as a list (a low-cost method). Since v4.0 2020.06.01. -Since v4.0 2020.06.01. +List has [sorted attribute](set-attribute.md#sorted) applied since v4.1 2024.07.08. ```q q).z.H~key .z.W From 787bbb5bbc400096148f7b6339f2ba0d1f517268 Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Tue, 24 Sep 2024 17:39:08 +0100 Subject: [PATCH 12/54] 4.1 2024.09.13 .Q.id now supports dictionaries --- docs/ref/dotq.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/ref/dotq.md b/docs/ref/dotq.md index 3570a49fb..5cba0f7f4 100644 --- a/docs/ref/dotq.md +++ b/docs/ref/dotq.md @@ -1107,7 +1107,7 @@ Where `x` is `ab`ab`twowords`a2drifters`a22 ``` -- a **table**, returns `x` with column names sanitized by removing characters that interfere with `select/exec/update` and adding `"1"` to column names which clash with commands in the `.q` namespace. (Updated in V3.2 to include `.Q.res` for checking collisions.) +- a **table**, returns `x` with column names sanitized by removing characters that interfere with `select/exec/update` and adding `"1"` to column names which clash with commands in the `.q` namespace. Updated in V3.2 to include [`.Q.res`](#res-keywords) for checking collisions. ```q q).Q.id flip (5#.Q.res)!(5#()) @@ -1118,6 +1118,17 @@ Where `x` is ---- ``` +- a **dictionary** (since v4.1 2024.09.13), supports the same rules as `table` above + + ```q + q).Q.id (5#.Q.res)!(5#()) + abs1 | + acos1| + asin1| + atan1| + avg1 | + ``` + Since 4.1t 2022.03.25,4.0 2022.10.26 produces a symbol `a` when the input contains a single character that is not in [.Q.an](#all-alphanumerics) (it previously produced an empty sym) e.g. ```q From 38a309e2983c1a770b00efebc96ff64019e0c3bf Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Tue, 24 Sep 2024 18:05:39 +0100 Subject: [PATCH 13/54] add links to ref material --- docs/ref/dotq.md | 45 +++++++++++++++++++++++++-------------------- 1 file changed, 25 insertions(+), 20 deletions(-) diff --git a/docs/ref/dotq.md b/docs/ref/dotq.md index 5cba0f7f4..f02e73e04 100644 --- a/docs/ref/dotq.md +++ b/docs/ref/dotq.md @@ -218,12 +218,16 @@ Since V3.6 2018.05.18. .Q.bv[`] ``` -In partitioned DBs, construct the dictionary `.Q.vp` of table schemas for tables with missing partitions. Optionally allow tables to be missing from partitions, by scanning partitions for missing tables and taking the tables’ prototypes from the last partition. After loading/re-loading from the filesystem, invoke `.Q.bv[]` to (re)populate `.Q.vt`/`.Q.vp`, which are used inside `.Q.p1` during the partitioned select `.Q.ps`. +In partitioned DBs, construct the dictionary [`.Q.vp`](#vp-missing-partitions) of table schemas for tables with missing partitions. Optionally allow tables to be missing from partitions, by scanning partitions for missing tables and taking the tables’ prototypes from the last partition. + +After loading/re-loading from the filesystem, invoke `.Q.bv[]` to (re)populate `.Q.vt`/`.Q.vp`, which are used inside `.Q.p1` during the partitioned select `.Q.ps`. (Since V2.8 2012.01.20, modified V3.0 2012.01.26) -If your table exists at least in the latest partition (so there is a prototype for the schema), you could use `.Q.bv[]` to create empty tables on the fly at run-time without having to create those empties on disk. ``.Q.bv[`]`` (with argument) will use prototype from first partition instead of last. (Since V3.2 2014.08.22.) +If your table exists at least in the latest partition (so there is a prototype for the schema), you could use `.Q.bv[]` to create empty tables on the fly at run-time without having to create those empties on disk. + +``.Q.bv[`]`` (with argument) will use prototype from first partition instead of last. (Since V3.2 2014.08.22.) -Some admins prefer to see errors instead of auto-manufactured empties for missing data, which is why `.Q.bv` is not the default behavior. +!!! note "Some admins prefer to see errors instead of auto-manufactured empties for missing data, which is why `.Q.bv` is not the default behavior." ```q q)n:100 @@ -255,7 +259,7 @@ q)@[get;"select from tt";-2@]; / no error .Q.Cf x ``` -A projection of `.Q.Xf`: i.e. ``.Q.Xf[`char;]`` +A projection of [`.Q.Xf`](#xf-create-file): i.e. ``.Q.Xf[`char;]`` ## `chk` (fill HDB) @@ -295,7 +299,7 @@ _Q for Mortals_ .Q.cn x ``` -Where `x` is a partitioned table, passed by value, returns its count. Populates `.Q.pn` cache. +Where `x` is a partitioned table, passed by value, returns its count. Populates [`.Q.pn`](#pn-partition-counts) cache. ## `D` (partitions) @@ -304,7 +308,7 @@ Where `x` is a partitioned table, passed by value, returns its count. Populates .Q.D ``` -In segmented DBs, contains a list of the partitions – conformant to `.Q.P` – that are present in each segment. +In segmented DBs, contains a list of the partitions – conformant to [`.Q.P`](#p-segments) – that are present in each segment. `.Q.P!.Q.D` can be used to create a dictionary of partition-to-segment information. @@ -554,7 +558,7 @@ Where returns `y` as a string formatted as a float to `x` decimal places. -Because of the limits of precision in a double, for `y` above `1e13` or the limit set by `\P`, formats in scientific notation. +Because of the limits of precision in a double, for `y` above `1e13` or the limit set by [`\P`](../basics/syscmds.md#p-precision), formats in scientific notation. ```q q)\P 0 @@ -599,7 +603,7 @@ q)\t .Q.fc[f]vec 6 ``` -In this case the overhead of creating threads in `peach` significantly outweighs the computational benefit of parallel execution. +In this case the overhead of creating threads in [`peach`](each.md) significantly outweighs the computational benefit of parallel execution. ```q q)\t f peach vec @@ -1344,7 +1348,7 @@ q)0W~.Q.M / defaults to long infinity Keeps partitions mapped to avoid the overhead of repeated file system calls during a `select`. (Since V3.1.) -For use with partitioned HDBS, used in tandem with `\l dir` +For use with partitioned HDBS, used in tandem with [`\l dir`](../basics/syscmds.md#l-load-file-or-directory) ```q q)\l . @@ -1452,7 +1456,7 @@ q)all{`p=attr .Q.par[`:.;x;`quote]`sym}each date .Q.PD ``` -In partitioned DBs, a list of partition locations – conformant to `.Q.PV` – which represents the partition location for each partition. +In partitioned DBs, a list of partition locations – conformant to [`.Q.PV`](#pv-partition-values) – which represents the partition location for each partition. (In non-segmented DBs, this will be simply ``count[.Q.PV]#`:.``.) `.Q.PV!.Q.PD` can be used to create a dictionary of partition-to-location information. @@ -1478,7 +1482,7 @@ q).Q.PV!.Q.PD .Q.pd ``` -In partitioned DBs, `.Q.PD` as modified by `.Q.view`. +In partitioned DBs, [`.Q.PD`](#pd-partition-locations) as modified by [`.Q.view`](#view-subview). ## `pf` (partition field) @@ -1497,8 +1501,9 @@ Possible values are `` `date`month`year`int``. .Q.pn ``` -In partitioned DBs, returns a dictionary of cached partition counts – conformant to `.Q.pt`, each conformant to `.Q.pv` – as populated by `.Q.cn`. -Cleared by `.Q.view`. +In partitioned DBs, returns a dictionary of cached partition counts – conformant to [`.Q.pt`](#pt-partitioned-tables), each conformant to [`.Q.pv`](#pv-modified-partition-values) – as populated by [`.Q.cn`](#cn-count-partitioned-table). + +Cleared by [`.Q.view`](#view-subview). `.Q.pv!flip .Q.pn` can be used to create a crosstab of table-to-partition-counts once `.Q.pn` is fully populated. @@ -1582,7 +1587,7 @@ Returns a list of partitioned tables. A list of the values of the partition domain: the values corresponding to the slice directories actually found in the root. -In partitioned DBs, `.Q.PV` as modified by `.Q.view`. +In partitioned DBs, [`.Q.PV`](#pv-partition-values) as modified by [`.Q.view`](#view-subview). :fontawesome-solid-street-view: _Q for Mortals_ @@ -1595,8 +1600,8 @@ _Q for Mortals_ .Q.PV ``` -In partitioned DBs, returns a list of partition values – conformant to `.Q.PD` – which represents the partition value for each partition. -(In a date-partitioned DB, unless the date has been modified by `.Q.view`, this will be simply date.) +In partitioned DBs, returns a list of partition values – conformant to [`.Q.PD`](#pd-partition-locations) – which represents the partition value for each partition. +(In a date-partitioned DB, unless the date has been modified by [`.Q.view`](#view-subview), this will be simply date.) ```q q).Q.PD @@ -1760,7 +1765,7 @@ extends [Trap At](apply.md#trap-at) (`@[f;x;g]`) to collect backtrace: `g` gets 1. the error string 2. the backtrace object -You can format the backtrace object with `.Q.sbt`. +You can format the backtrace object with [`.Q.sbt`](#sbt-string-backtrace). ```q q)f:{`hello+x} @@ -1831,7 +1836,7 @@ extends [Trap](apply.md#trap) (`.[f;x;g]`) to collect backtrace: `g` is called w 1. the error string 2. the backtrace object -You can format the backtrace object with `.Q.sbt`. +You can format the backtrace object with [`.Q.sbt`](#sbt-string-backtrace). ```q q).Q.trpd[{x+y};(1;2);{2"error: ",x,"\nbacktrace:\n",.Q.sbt y;-1}] @@ -1983,7 +1988,7 @@ _Q for Mortals_ .Q.vp ``` -In partitioned DBs, returns a dictionary of table schemas for tables with missing partitions, as populated by `.Q.bv`. +In partitioned DBs, returns a dictionary of table schemas for tables with missing partitions, as populated by [`.Q.bv`](#bv-build-vp). (Since V3.0 2012.01.26.) ```q @@ -2063,7 +2068,7 @@ q)type get`:emptyNestedCharVector .Q.x ``` -Set by `.Q.opt`: a list of _non-command_ parameters from the command line, where _command parameters_ are prefixed by `-`. +Set by [`.Q.opt`](#opt-command-parameters): a list of _non-command_ parameters from the command line, where _command parameters_ are prefixed by `-`. ```bash ~$ q taq.k path/to/source path/to/destn From c208dbeb86672559ef8cf637fddaa7e9e21f1b8e Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Wed, 25 Sep 2024 10:21:57 +0100 Subject: [PATCH 14/54] Q.bvi and Q.li addition from 4.1 --- docs/ref/dotq.md | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/docs/ref/dotq.md b/docs/ref/dotq.md index f02e73e04..592e0c5c1 100644 --- a/docs/ref/dotq.md +++ b/docs/ref/dotq.md @@ -249,6 +249,11 @@ q)@[get;"select from tt";-2@]; / no error ``` +## `bvi` (build incremental vp) + +Same functionality as [`.Q.bv`](#bv-build-vp), but scans only new partitions loaded in the hdb since the last time `.Q.bv` or `.Q.bvi` was run. Since v4.1 2024.09.13. + + ## `Cf` (create empty nested char file) !!! warning "Deprecated" @@ -1290,6 +1295,32 @@ q).Q.ld read0`:funcs.q "/ multi line func" "f:{\n x+y\n }" "/ single line func" "g:{x*y}" ``` +## `li` (load partitions) + +```syntax +.Q.li[partitions] +``` + +In the current hdb, adds any partition(s) which are both in the list supplied and on disk. Partitions can be a list or atomic variable. For example: + +```q +q)`:/tmp/db/2001.01.01/t/ set tt:.Q.en[`:/tmp/db]([]sym:10?`A`B`C;time:10?.z.T;price:10?10f) +q)\l /tmp/db +q)`:2001.01.02/t/`:2001.01.03/t/ set\:tt +q)date +,2001.01.01 +q).Q.li[2001.01.02];date +2001.01.01 2001.01.02 +q).Q.li[2001.01.02 2001.01.03];select count i by date from t +date | x +----------| -- +2001.01.01| 10 +2001.01.02| 10 +2001.01.03| 10 +``` + +Since v4.1 2024.09.20. + ## `lo` (load without) From 95371a0988bda6700ea1d09968d0640ecff36572 Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Wed, 25 Sep 2024 11:03:18 +0100 Subject: [PATCH 15/54] seperate out command-line tools in q namespace menu --- docs/ref/dotq.md | 77 ++++++++++++++++++++++++------------------------ 1 file changed, 39 insertions(+), 38 deletions(-) diff --git a/docs/ref/dotq.md b/docs/ref/dotq.md index 592e0c5c1..78968a444 100644 --- a/docs/ref/dotq.md +++ b/docs/ref/dotq.md @@ -13,51 +13,52 @@ _Tools_ [addmonths](#addmonths) [btoa b64 encode](#btoa-b64-encode) [bt backtrace](#bt-backtrace) [j10 encode binhex](#j10-encode-binhex) [dd join symbols](#dd-join-symbols) [j12 encode base 36](#j12-encode-base-36) - [def parse options](#def-parse-options) [ty type](#ty-type) - [f format](#f-format) [x10 decode binhex](#x10-decode-binhex) - [fc parallel on cut](#fc-parallel-on-cut) [x12 decode base 36](#x12-decode-base-36) - [ff append columns](#ff-append-columns) - [fmt format](#fmt-format) **Database** - [ft apply simple](#ft-apply-simple) [chk fill HDB](#chk-fill-hdb) - [fu apply unique](#fu-apply-unique) [dpft dpfts save table](#dpft-save-table) - [gc garbage collect](#gc-garbage-collect) [dpt dpts save table unsorted](#dpt-save-table-unsorted) - [gz GZip](#gz-gzip) [dsftg load process save](#dsftg-load-process-save) - [id sanitize](#id-sanitize) [en enumerate varchar cols](#en-enumerate-varchar-cols) - [prf0 code profiler](#prf0-code-profiler) [ens enumerate against domain](#ens-enumerate-against-domain) - [qt is table](#qt-is-table) [fk foreign key](#fk-foreign-key) - [res keywords](#res-keywords) [hdpf save tables](#hdpf-save-tables) - [s plain text](#s-plain-text) [l load](#l-load) - [s1 string representation](#s1-string-representation) [ld load and group](#ld-load-and-group) - [sbt string backtrace](#sbt-string-backtrace) [lo load without](#lo-load-without) - [sha1 SHA-1 encode](#sha1-sha-1-encode) [M chunk size](#m-chunk-size) - [trp extend trap at](#trp-extend-trap-at) [qp is partitioned](#qp-is-partitioned) - [trpd extend trap](#trpd-extend-trap) [qt is table](#qt-is-table) - [ts time and space](#ts-time-and-space) - [u date based](#u-date-based) **Partitioned database state** - [V table to dict](#v-table-to-dict) [bv build vp](#bv-build-vp) - [v value](#v-value) [cn count partitioned table](#cn-count-partitioned-table) - [view subview](#view-subview) [D partitions](#d-partitions) - [ind partitioned index](#ind-partitioned-index) -**Constants** [MAP maps partitions](#map-maps-partitions) - [A a an alphabets](#a-upper-case-alphabet) [par locate partition](#par-locate-partition) - [b6 bicameral alphanums](#b6-bicameral-alphanums) [PD partition locations](#pd-partition-locations) - [n nA nums & alphanums](#n-nums) [pd modified partition locns](#pd-modified-partition-locations) - [pf partition field](#pf-partition-field) -**Environment** [pn partition counts](#pn-partition-counts) - [K k version](#k-version-date) [pt partitioned tables](#pt-partitioned-tables) - [opt command parameters](#opt-command-parameters) [PV partition values](#pv-partition-values) - [w memory stats](#w-memory-stats) [pv modified partition values](#pv-modified-partition-values) + [f format](#f-format) [ty type](#ty-type) + [fc parallel on cut](#fc-parallel-on-cut) [x10 decode binhex](#x10-decode-binhex) + [ff append columns](#ff-append-columns) [x12 decode base 36](#x12-decode-base-36) + [fmt format](#fmt-format) + [ft apply simple](#ft-apply-simple) **Database** + [fu apply unique](#fu-apply-unique) [chk fill HDB](#chk-fill-hdb) + [gc garbage collect](#gc-garbage-collect) [dpft dpfts save table](#dpft-save-table) + [gz GZip](#gz-gzip) [dpt dpts save table unsorted](#dpt-save-table-unsorted) + [id sanitize](#id-sanitize) [dsftg load process save](#dsftg-load-process-save) + [prf0 code profiler](#prf0-code-profiler) [en enumerate varchar cols](#en-enumerate-varchar-cols) + [qt is table](#qt-is-table) [ens enumerate against domain](#ens-enumerate-against-domain) + [res keywords](#res-keywords) [fk foreign key](#fk-foreign-key) + [s plain text](#s-plain-text) [hdpf save tables](#hdpf-save-tables) + [s1 string representation](#s1-string-representation) [l load](#l-load) + [sbt string backtrace](#sbt-string-backtrace) [ld load and group](#ld-load-and-group) + [sha1 SHA-1 encode](#sha1-sha-1-encode) [li load partitions](#li-load-partitions) + [trp extend trap at](#trp-extend-trap-at) [lo load without](#lo-load-without) + [trpd extend trap](#trpd-extend-trap) [M chunk size](#m-chunk-size) + [ts time and space](#ts-time-and-space) [qp is partitioned](#qp-is-partitioned) + [u date based](#u-date-based) [qt is table](#qt-is-table) + [V table to dict](#v-table-to-dict) + [v value](#v-value) **Partitioned database state** + [view subview](#view-subview) [bv build vp](#bv-build-vp) + [bvi build incremental vp](#bvi-build-incremental-vp) + **Constants** [cn count partitioned table](#cn-count-partitioned-table) + [A a an alphabets](#a-upper-case-alphabet) [D partitions](#d-partitions) + [b6 bicameral alphanums](#b6-bicameral-alphanums) [ind partitioned index](#ind-partitioned-index) + [n nA nums & alphanums](#n-nums) [MAP maps partitions](#map-maps-partitions) + [par locate partition](#par-locate-partition) + **Environment** [PD partition locations](#pd-partition-locations) + [K k version](#k-version-date) [pd modified partition locns](#pd-modified-partition-locations) + [w memory stats](#w-memory-stats) [pf partition field](#pf-partition-field) + [pn partition counts](#pn-partition-counts) + **Environment (Command-line)** [pt partitioned tables](#pt-partitioned-tables) + [def parse options](#def-parse-options) [PV partition values](#pv-partition-values) + [opt command parameters](#opt-command-parameters) [pv modified partition values](#pv-modified-partition-values) [x non-command parameters](#x-non-command-parameters) [qp is partitioned](#qp-is-partitioned) - [vp missing partitions](#vp-missing-partitions) -**IPC** + [vp missing partitions](#vp-missing-partitions) + **IPC** [addr IP address](#addr-ip-address) **Segmented database state** [fps fpn streaming algorithm](#fpn-streaming-algorithm) [P segments](#p-segments) [fs fsn streaming algorithm](#fs-streaming-algorithm) [u date based](#u-date-based) - [hg HTTP get](#hg-http-get) + [hg HTTP get](#hg-http-get) [host hostname](#host-hostname) **File I/O** [hp HTTP post](#hp-http-post) [Cf create empty nested char file](#cf-create-empty-nested-char-file) [Xf create file](#xf-create-file) - From 87bc3fc200dbbbc96a0da361083562f2d54c8e06 Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Wed, 25 Sep 2024 11:10:34 +0100 Subject: [PATCH 16/54] change misleading title for .Q.def --- docs/ref/dotq.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/ref/dotq.md b/docs/ref/dotq.md index 78968a444..e6384a09f 100644 --- a/docs/ref/dotq.md +++ b/docs/ref/dotq.md @@ -36,7 +36,7 @@ _Tools_ [V table to dict](#v-table-to-dict) [v value](#v-value) **Partitioned database state** [view subview](#view-subview) [bv build vp](#bv-build-vp) - [bvi build incremental vp](#bvi-build-incremental-vp) + [bvi build incremental vp](#bvi-build-incremental-vp) **Constants** [cn count partitioned table](#cn-count-partitioned-table) [A a an alphabets](#a-upper-case-alphabet) [D partitions](#d-partitions) [b6 bicameral alphanums](#b6-bicameral-alphanums) [ind partitioned index](#ind-partitioned-index) @@ -47,7 +47,7 @@ _Tools_ [w memory stats](#w-memory-stats) [pf partition field](#pf-partition-field) [pn partition counts](#pn-partition-counts) **Environment (Command-line)** [pt partitioned tables](#pt-partitioned-tables) - [def parse options](#def-parse-options) [PV partition values](#pv-partition-values) + [def command defaults](#def-command-defaults) [PV partition values](#pv-partition-values) [opt command parameters](#opt-command-parameters) [pv modified partition values](#pv-modified-partition-values) [x non-command parameters](#x-non-command-parameters) [qp is partitioned](#qp-is-partitioned) [vp missing partitions](#vp-missing-partitions) @@ -356,8 +356,8 @@ AAPL O AAPL.O IBM N IBM.N ``` - -## `def` (parse options) +[](){#def-parse-options} +## `def` (command defaults) _Default values for command-line arguments_ From 6404e8c5e8bddfef69ae6ba5c3f90e7db9cb8d72 Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Wed, 25 Sep 2024 11:56:16 +0100 Subject: [PATCH 17/54] have examples of .Q.opt and .Q.def with their definitions --- docs/ref/dotq.md | 40 ++++++++++++++++++++++++++++++++++++---- docs/ref/dotz.md | 42 ++++-------------------------------------- 2 files changed, 40 insertions(+), 42 deletions(-) diff --git a/docs/ref/dotq.md b/docs/ref/dotq.md index e6384a09f..2188a8298 100644 --- a/docs/ref/dotq.md +++ b/docs/ref/dotq.md @@ -365,7 +365,28 @@ _Default values for command-line arguments_ .Q.def[x;y] ``` -Provides defaults and types for command-line arguments parsed with [``.Q.opt``](#opt-command-parameters). +Provides defaults and types for command-line arguments parsed with [`.Q.opt`](#opt-command-parameters). + +```bash +$ q -abc 123 -xyz 321 +``` +```q +q).Q.def[`abc`xyz`efg!(1;2.;`a)].Q.opt .z.x +abc| 123 +xyz| 321f +efg| `a +q)\\ +``` + +```bash +$ q -abc 123 -xyz 321 -efg foo +``` +```q +q).Q.def[`abc`xyz`efg!(1;2.;`a)].Q.opt .z.x +abc| 123 +xyz| 321f +efg| `foo +``` :fontawesome-solid-book: [`.z.x`](dotz.md#zx-argv) @@ -1433,7 +1454,19 @@ q).Q.nA .Q.opt .z.x ``` -Returns a dictionary, so you can easily see if a key was defined (flag set or not) or, if a value is passed, to refer to it by its key. +Presents command-line arguments as a dictionary, using the output of [`.z.x`](dotz.md#zx-argv). Defaults can be added using [`.Q.def`](#def-command-defaults). + +```bash +$ q -param1 val1 -param2 val2 +``` +```q +q)params:.Q.opt .z.x +q)show params +param1| "val1" +param2| "val2" +q)params`param1 +"val1" +``` :fontawesome-solid-book: [`.z.x`](dotz.md#zx-argv) @@ -2103,9 +2136,8 @@ q)type get`:emptyNestedCharVector Set by [`.Q.opt`](#opt-command-parameters): a list of _non-command_ parameters from the command line, where _command parameters_ are prefixed by `-`. ```bash -~$ q taq.k path/to/source path/to/destn +$ q taq.k path/to/source path/to/destn ``` - ```q q)cla:.Q.opt .z.X /command-line arguments q).Q.x diff --git a/docs/ref/dotz.md b/docs/ref/dotz.md index b9bba81ce..74a3b6a23 100644 --- a/docs/ref/dotz.md +++ b/docs/ref/dotz.md @@ -1077,47 +1077,13 @@ Command-line arguments as a list of strings ```q $ q test.q -P 0 -abc 123 q).z.x -("-abc";"123") +"-abc" +"123" ``` -Note that the script name and the single-letter options used by q itself are not included. +!!! Note "The script name and the single-letter options used by q itself are not included." -Command-line options can be converted to a dictionary using the convenient `.Q.opt` function. - -```bash -$ q -abc 123 -xyz 321 -``` - -```q -q).Q.opt .z.x -abc| "123" -xyz| "321" -``` - -Defaults and types can be provided with `.Q.def`. - -```bash -$ q -abc 123 -xyz 321 -``` - -```q -q).Q.def[`abc`xyz`efg!(1;2.;`a)].Q.opt .z.x -abc| 123 -xyz| 321f -efg| `a -q)\\ -``` - -```bash -$ q -abc 123 -xyz 321 -efg foo -``` - -```q -q).Q.def[`abc`xyz`efg!(1;2.;`a)].Q.opt .z.x -abc| 123 -xyz| 321f -efg| `foo -``` +Command-line options can be converted to a dictionary using the convenient [`.Q.opt`](dotq.md#opt-command-parameters) function. :fontawesome-solid-hand-point-right: [`.z.X` raw command line](#zx-raw-command-line), [`.z.f` file](#zf-file), [`.z.q` quiet mode](#zq-quiet-mode) From 86e90f1a17874c0c2bdd5aa41be6318718c277cb Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Wed, 25 Sep 2024 12:59:21 +0100 Subject: [PATCH 18/54] split menu into debug/profile section --- docs/ref/dotq.md | 96 ++++++++++++++++++++++++------------------------ 1 file changed, 48 insertions(+), 48 deletions(-) diff --git a/docs/ref/dotq.md b/docs/ref/dotq.md index 2188a8298..46befda40 100644 --- a/docs/ref/dotq.md +++ b/docs/ref/dotq.md @@ -11,54 +11,54 @@ _Tools_
**General** **Datatype** [addmonths](#addmonths) [btoa b64 encode](#btoa-b64-encode) - [bt backtrace](#bt-backtrace) [j10 encode binhex](#j10-encode-binhex) - [dd join symbols](#dd-join-symbols) [j12 encode base 36](#j12-encode-base-36) - [f format](#f-format) [ty type](#ty-type) - [fc parallel on cut](#fc-parallel-on-cut) [x10 decode binhex](#x10-decode-binhex) - [ff append columns](#ff-append-columns) [x12 decode base 36](#x12-decode-base-36) - [fmt format](#fmt-format) - [ft apply simple](#ft-apply-simple) **Database** - [fu apply unique](#fu-apply-unique) [chk fill HDB](#chk-fill-hdb) - [gc garbage collect](#gc-garbage-collect) [dpft dpfts save table](#dpft-save-table) - [gz GZip](#gz-gzip) [dpt dpts save table unsorted](#dpt-save-table-unsorted) - [id sanitize](#id-sanitize) [dsftg load process save](#dsftg-load-process-save) - [prf0 code profiler](#prf0-code-profiler) [en enumerate varchar cols](#en-enumerate-varchar-cols) - [qt is table](#qt-is-table) [ens enumerate against domain](#ens-enumerate-against-domain) - [res keywords](#res-keywords) [fk foreign key](#fk-foreign-key) - [s plain text](#s-plain-text) [hdpf save tables](#hdpf-save-tables) - [s1 string representation](#s1-string-representation) [l load](#l-load) - [sbt string backtrace](#sbt-string-backtrace) [ld load and group](#ld-load-and-group) - [sha1 SHA-1 encode](#sha1-sha-1-encode) [li load partitions](#li-load-partitions) - [trp extend trap at](#trp-extend-trap-at) [lo load without](#lo-load-without) - [trpd extend trap](#trpd-extend-trap) [M chunk size](#m-chunk-size) - [ts time and space](#ts-time-and-space) [qp is partitioned](#qp-is-partitioned) - [u date based](#u-date-based) [qt is table](#qt-is-table) - [V table to dict](#v-table-to-dict) - [v value](#v-value) **Partitioned database state** - [view subview](#view-subview) [bv build vp](#bv-build-vp) - [bvi build incremental vp](#bvi-build-incremental-vp) - **Constants** [cn count partitioned table](#cn-count-partitioned-table) - [A a an alphabets](#a-upper-case-alphabet) [D partitions](#d-partitions) - [b6 bicameral alphanums](#b6-bicameral-alphanums) [ind partitioned index](#ind-partitioned-index) - [n nA nums & alphanums](#n-nums) [MAP maps partitions](#map-maps-partitions) - [par locate partition](#par-locate-partition) - **Environment** [PD partition locations](#pd-partition-locations) - [K k version](#k-version-date) [pd modified partition locns](#pd-modified-partition-locations) - [w memory stats](#w-memory-stats) [pf partition field](#pf-partition-field) - [pn partition counts](#pn-partition-counts) - **Environment (Command-line)** [pt partitioned tables](#pt-partitioned-tables) - [def command defaults](#def-command-defaults) [PV partition values](#pv-partition-values) - [opt command parameters](#opt-command-parameters) [pv modified partition values](#pv-modified-partition-values) - [x non-command parameters](#x-non-command-parameters) [qp is partitioned](#qp-is-partitioned) - [vp missing partitions](#vp-missing-partitions) - **IPC** - [addr IP address](#addr-ip-address) **Segmented database state** - [fps fpn streaming algorithm](#fpn-streaming-algorithm) [P segments](#p-segments) - [fs fsn streaming algorithm](#fs-streaming-algorithm) [u date based](#u-date-based) - [hg HTTP get](#hg-http-get) - [host hostname](#host-hostname) **File I/O** - [hp HTTP post](#hp-http-post) [Cf create empty nested char file](#cf-create-empty-nested-char-file) - [Xf create file](#xf-create-file) + [dd join symbols](#dd-join-symbols) [j10 encode binhex](#j10-encode-binhex) + [f format](#f-format) [j12 encode base 36](#j12-encode-base-36) + [fc parallel on cut](#fc-parallel-on-cut) [ty type](#ty-type) + [ff append columns](#ff-append-columns) [x10 decode binhex](#x10-decode-binhex) + [fmt format](#fmt-format) [x12 decode base 36](#x12-decode-base-36) + [ft apply simple](#ft-apply-simple) + [fu apply unique](#fu-apply-unique) **Database** + [gc garbage collect](#gc-garbage-collect) [chk fill HDB](#chk-fill-hdb) + [gz GZip](#gz-gzip) [dpft dpfts save table](#dpft-save-table) + [id sanitize](#id-sanitize) [dpt dpts save table unsorted](#dpt-save-table-unsorted) + [qt is table](#qt-is-table) [dsftg load process save](#dsftg-load-process-save) + [res keywords](#res-keywords) [en enumerate varchar cols](#en-enumerate-varchar-cols) + [s plain text](#s-plain-text) [ens enumerate against domain](#ens-enumerate-against-domain) + [s1 string representation](#s1-string-representation) [fk foreign key](#fk-foreign-key) + [sha1 SHA-1 encode](#sha1-sha-1-encode) [hdpf save tables](#hdpf-save-tables) + [V table to dict](#v-table-to-dict) [l load](#l-load) + [v value](#v-value) [ld load and group](#ld-load-and-group) + [view subview](#view-subview) [li load partitions](#li-load-partitions) + [lo load without](#lo-load-without) + **Constants** [M chunk size](#m-chunk-size) + [A a an alphabets](#a-upper-case-alphabet) [qp is partitioned](#qp-is-partitioned) + [b6 bicameral alphanums](#b6-bicameral-alphanums) [qt is table](#qt-is-table) + [n nA nums & alphanums](#n-nums) + **Partitioned database state** + **Debug/Profile** [bv build vp](#bv-build-vp) + [bt backtrace](#bt-backtrace) [bvi build incremental vp](#bvi-build-incremental-vp) + [prf0 code profiler](#prf0-code-profiler) [cn count partitioned table](#cn-count-partitioned-table) + [sbt string backtrace](#sbt-string-backtrace) [D partitions](#d-partitions) + [trp extend trap at](#trp-extend-trap-at) [ind partitioned index](#ind-partitioned-index) + [trpd extend trap](#trpd-extend-trap) [MAP maps partitions](#map-maps-partitions) + [ts time and space](#ts-time-and-space) [par locate partition](#par-locate-partition) + [PD partition locations](#pd-partition-locations) + **Environment** [pd modified partition locns](#pd-modified-partition-locations) + [K k version](#k-version-date) [pf partition field](#pf-partition-field) + [w memory stats](#w-memory-stats) [pn partition counts](#pn-partition-counts) + [pt partitioned tables](#pt-partitioned-tables) + **Environment (Command-line)** [PV partition values](#pv-partition-values) + [def command defaults](#def-command-defaults) [pv modified partition values](#pv-modified-partition-values) + [opt command parameters](#opt-command-parameters) [qp is partitioned](#qp-is-partitioned) + [x non-command parameters](#x-non-command-parameters) [vp missing partitions](#vp-missing-partitions) + + **IPC** **Segmented database state** + [addr IP address](#addr-ip-address) [P segments](#p-segments) + [fps fpn streaming algorithm](#fpn-streaming-algorithm) [u date based](#u-date-based) + [fs fsn streaming algorithm](#fs-streaming-algorithm) + [hg HTTP get](#hg-http-get) **File I/O** + [host hostname](#host-hostname) [Cf create empty nested char file](#cf-create-empty-nested-char-file) + [hp HTTP post](#hp-http-post) [Xf create file](#xf-create-file)
From e96ca7e39157fd0a31a90498537bdeb0d58b3c01 Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Wed, 25 Sep 2024 13:25:10 +0100 Subject: [PATCH 19/54] changed title of .z.b --- docs/ref/dotz.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/ref/dotz.md b/docs/ref/dotz.md index 74a3b6a23..5e44d6662 100644 --- a/docs/ref/dotz.md +++ b/docs/ref/dotz.md @@ -13,7 +13,7 @@ _Environment and callbacks_
Environment Callbacks [.z.a IP address](#za-ip-address) [.z.ac HTTP auth](#zac-http-auth) - [.z.b dependencies](#zb-dependencies) [.z.bm msg validator](#zbm-msg-validator) + [.z.b view dependencies](#zb-view-dependencies) [.z.bm msg validator](#zbm-msg-validator) [.z.c cores](#zc-cores) [.z.exit action on exit](#zexit-action-on-exit) [.z.D/d date shortcuts](#zt-zt-zd-zd-timedate-shortcuts) [.z.pc close](#zpc-close) [.z.e TLS connection status](#ze-tls-connection-status) [.z.pd peach handles](#zpd-peach-handles) @@ -130,8 +130,8 @@ If the user is not permitted, the client is sent a default 401 HTTP unauthorized :fontawesome-solid-graduation-cap:[HTTP](../kb/http.md) - -## `.z.b` (dependencies) +[](){#zb-dependencies} +## `.z.b` (view dependencies) The dependency dictionary. From b7ed061fee4856e2525613f93dc10922d3eb96bf Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Wed, 25 Sep 2024 13:27:27 +0100 Subject: [PATCH 20/54] add link to view details --- docs/ref/dotz.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/ref/dotz.md b/docs/ref/dotz.md index 5e44d6662..9f9d35e55 100644 --- a/docs/ref/dotz.md +++ b/docs/ref/dotz.md @@ -145,6 +145,9 @@ y| ,`a :fontawesome-solid-book-open: [`\b`](../basics/syscmds.md#b-views) +
+:fontawesome-solid-graduation-cap: +[Views](../learn/views.md) ## `.z.bm` (msg validator) From cb8299674346033d7556ed44383cebd67e5565fb Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Wed, 25 Sep 2024 14:23:52 +0100 Subject: [PATCH 21/54] seperate time/date tools into own section --- docs/ref/dotz.md | 52 +++++++++++++++++++++++++----------------------- 1 file changed, 27 insertions(+), 25 deletions(-) diff --git a/docs/ref/dotz.md b/docs/ref/dotz.md index 9f9d35e55..76cfa5137 100644 --- a/docs/ref/dotz.md +++ b/docs/ref/dotz.md @@ -11,32 +11,34 @@ keywords: callbacks, environment, kdb+, q _Environment and callbacks_
-Environment Callbacks - [.z.a IP address](#za-ip-address) [.z.ac HTTP auth](#zac-http-auth) - [.z.b view dependencies](#zb-view-dependencies) [.z.bm msg validator](#zbm-msg-validator) - [.z.c cores](#zc-cores) [.z.exit action on exit](#zexit-action-on-exit) - [.z.D/d date shortcuts](#zt-zt-zd-zd-timedate-shortcuts) [.z.pc close](#zpc-close) - [.z.e TLS connection status](#ze-tls-connection-status) [.z.pd peach handles](#zpd-peach-handles) - [.z.ex failed primitive](#zex-failed-primitive) [.z.pg get](#zpg-get) - [.z.ey arg to failed primitive](#zey-argument-to-failed-primitive) [.z.ph HTTP get](#zph-http-get) - [.z.f file](#zf-file) [.z.pi input](#zpi-input) - [.z.H active sockets](#zh-active-sockets) [.z.pm HTTP methods](#zpm-http-methods) - [.z.h host](#zh-host) [.z.po open](#zpo-open) - [.z.i PID](#zi-pid) [.z.pp HTTP post](#zpp-http-post) - [.z.K version](#zk-version) [.z.pq qcon](#zpq-qcon) - [.z.k release date](#zk-release-date) [.z.r blocked](#zr-blocked) - [.z.l license](#zl-license) [.z.ps set](#zps-set) - [.z.N/n local/UTC timespan](#zn-local-timespan) [.z.pw validate user](#zpw-validate-user) - [.z.o OS version](#zo-os-version) [.z.ts timer](#zts-timer) - [.z.P/p local/UTC timestamp](#zp-local-timestamp) [.z.vs value set](#zvs-value-set) - [.z.q quiet mode](#zq-quiet-mode) [.z.wc WebSocket close](#zwc-websocket-close) - [.z.s self](#zs-self) [.z.wo WebSocket open](#zwo-websocket-open) - [.z.T/t time shortcuts](#zt-zt-zd-zd-timedate-shortcuts) [.z.ws WebSockets](#zws-websockets) - [.z.u user ID](#zu-user-id) - [.z.W/w handles/handle](#zw-handles) - [.z.X/x raw/parsed command line](#zx-raw-command-line) +Environment Callbacks + [.z.a IP address](#za-ip-address) [.z.ac HTTP auth](#zac-http-auth) + [.z.b view dependencies](#zb-view-dependencies) [.z.bm msg validator](#zbm-msg-validator) + [.z.c cores](#zc-cores) [.z.exit action on exit](#zexit-action-on-exit) + [.z.e TLS connection status](#ze-tls-connection-status) [.z.pc close](#zpc-close) + [.z.ex failed primitive](#zex-failed-primitive) [.z.pd peach handles](#zpd-peach-handles) + [.z.ey arg to failed primitive](#zey-argument-to-failed-primitive) [.z.pg get](#zpg-get) + [.z.f file](#zf-file) [.z.ph HTTP get](#zph-http-get) + [.z.H active sockets](#zh-active-sockets) [.z.pi input](#zpi-input) + [.z.h host](#zh-host) [.z.pm HTTP methods](#zpm-http-methods) + [.z.i PID](#zi-pid) [.z.po open](#zpo-open) + [.z.K version](#zk-version) [.z.pp HTTP post](#zpp-http-post) + [.z.k release date](#zk-release-date) [.z.pq qcon](#zpq-qcon) + [.z.l license](#zl-license) [.z.r blocked](#zr-blocked) + [.z.o OS version](#zo-os-version) [.z.ps set](#zps-set) + [.z.q quiet mode](#zq-quiet-mode) [.z.pw validate user](#zpw-validate-user) + [.z.s self](#zs-self) [.z.ts timer](#zts-timer) + [.z.u user ID](#zu-user-id) [.z.vs value set](#zvs-value-set) + [.z.W/w handles/handle](#zw-handles) [.z.wc WebSocket close](#zwc-websocket-close) + [.z.X/x raw/parsed command line](#zx-raw-command-line) [.z.wo WebSocket open](#zwo-websocket-open) + [.z.zd compression/encryption defaults](#zzd-compressionencryption-defaults) [.z.ws WebSockets](#zws-websockets) + + Environment (Time/Date) + [.z.D/d date shortcuts](#zt-zt-zd-zd-timedate-shortcuts) + [.z.N/n local/UTC timespan](#zn-local-timespan) + [.z.P/p local/UTC timestamp](#zp-local-timestamp) + [.z.T/t time shortcuts](#zt-zt-zd-zd-timedate-shortcuts) [.z.Z/z local/UTC datetime](#zz-local-datetime) - [.z.zd compression/encryption defaults](#zzd-compressionencryption-defaults)
The `.z` [namespace](../basics/namespaces.md) contains environment variables and functions, and hooks for callbacks. From 096f237e8429ea5ca518409915bbe53d7a3e4621 Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Wed, 25 Sep 2024 14:46:14 +0100 Subject: [PATCH 22/54] split http/websockets callbacks into own section --- docs/ref/dotz.md | 48 ++++++++++++++++++++++++------------------------ 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/docs/ref/dotz.md b/docs/ref/dotz.md index 76cfa5137..a151dbe67 100644 --- a/docs/ref/dotz.md +++ b/docs/ref/dotz.md @@ -12,30 +12,30 @@ _Environment and callbacks_
Environment Callbacks - [.z.a IP address](#za-ip-address) [.z.ac HTTP auth](#zac-http-auth) - [.z.b view dependencies](#zb-view-dependencies) [.z.bm msg validator](#zbm-msg-validator) - [.z.c cores](#zc-cores) [.z.exit action on exit](#zexit-action-on-exit) - [.z.e TLS connection status](#ze-tls-connection-status) [.z.pc close](#zpc-close) - [.z.ex failed primitive](#zex-failed-primitive) [.z.pd peach handles](#zpd-peach-handles) - [.z.ey arg to failed primitive](#zey-argument-to-failed-primitive) [.z.pg get](#zpg-get) - [.z.f file](#zf-file) [.z.ph HTTP get](#zph-http-get) - [.z.H active sockets](#zh-active-sockets) [.z.pi input](#zpi-input) - [.z.h host](#zh-host) [.z.pm HTTP methods](#zpm-http-methods) - [.z.i PID](#zi-pid) [.z.po open](#zpo-open) - [.z.K version](#zk-version) [.z.pp HTTP post](#zpp-http-post) - [.z.k release date](#zk-release-date) [.z.pq qcon](#zpq-qcon) - [.z.l license](#zl-license) [.z.r blocked](#zr-blocked) - [.z.o OS version](#zo-os-version) [.z.ps set](#zps-set) - [.z.q quiet mode](#zq-quiet-mode) [.z.pw validate user](#zpw-validate-user) - [.z.s self](#zs-self) [.z.ts timer](#zts-timer) - [.z.u user ID](#zu-user-id) [.z.vs value set](#zvs-value-set) - [.z.W/w handles/handle](#zw-handles) [.z.wc WebSocket close](#zwc-websocket-close) - [.z.X/x raw/parsed command line](#zx-raw-command-line) [.z.wo WebSocket open](#zwo-websocket-open) - [.z.zd compression/encryption defaults](#zzd-compressionencryption-defaults) [.z.ws WebSockets](#zws-websockets) - - Environment (Time/Date) - [.z.D/d date shortcuts](#zt-zt-zd-zd-timedate-shortcuts) - [.z.N/n local/UTC timespan](#zn-local-timespan) + [.z.a IP address](#za-ip-address) [.z.bm msg validator](#zbm-msg-validator) + [.z.b view dependencies](#zb-view-dependencies) [.z.exit action on exit](#zexit-action-on-exit) + [.z.c cores](#zc-cores) [.z.pc close](#zpc-close) + [.z.e TLS connection status](#ze-tls-connection-status) [.z.pd peach handles](#zpd-peach-handles) + [.z.ex failed primitive](#zex-failed-primitive) [.z.pg get](#zpg-get) + [.z.ey arg to failed primitive](#zey-argument-to-failed-primitive) [.z.pi input](#zpi-input) + [.z.f file](#zf-file) [.z.po open](#zpo-open) + [.z.H active sockets](#zh-active-sockets) [.z.pq qcon](#zpq-qcon) + [.z.h host](#zh-host) [.z.r blocked](#zr-blocked) + [.z.i PID](#zi-pid) [.z.ps set](#zps-set) + [.z.K version](#zk-version) [.z.pw validate user](#zpw-validate-user) + [.z.k release date](#zk-release-date) [.z.ts timer](#zts-timer) + [.z.l license](#zl-license) [.z.vs value set](#zvs-value-set) + [.z.o OS version](#zo-os-version) + [.z.q quiet mode](#zq-quiet-mode) Callbacks (HTTP) + [.z.s self](#zs-self) [.z.ac HTTP auth](#zac-http-auth) + [.z.u user ID](#zu-user-id) [.z.ph HTTP get](#zph-http-get) + [.z.W/w handles/handle](#zw-handles) [.z.pm HTTP methods](#zpm-http-methods) + [.z.X/x raw/parsed command line](#zx-raw-command-line) [.z.pp HTTP post](#zpp-http-post) + [.z.zd compression/encryption defaults](#zzd-compressionencryption-defaults) + Callbacks (WebSockets) + Environment (Time/Date) [.z.wc WebSocket close](#zwc-websocket-close) + [.z.D/d date shortcuts](#zt-zt-zd-zd-timedate-shortcuts) [.z.wo WebSocket open](#zwo-websocket-open) + [.z.N/n local/UTC timespan](#zn-local-timespan) [.z.ws WebSockets](#zws-websockets) [.z.P/p local/UTC timestamp](#zp-local-timestamp) [.z.T/t time shortcuts](#zt-zt-zd-zd-timedate-shortcuts) [.z.Z/z local/UTC datetime](#zz-local-datetime) From 61187662061c0f2e527eca95e65a7ab7d2bd1d8d Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Wed, 25 Sep 2024 17:09:12 +0100 Subject: [PATCH 23/54] .z namespace, seperated out connection utils in menu --- docs/ref/dotz.md | 44 +++++++++++++++++++++++--------------------- 1 file changed, 23 insertions(+), 21 deletions(-) diff --git a/docs/ref/dotz.md b/docs/ref/dotz.md index a151dbe67..7c3129f7f 100644 --- a/docs/ref/dotz.md +++ b/docs/ref/dotz.md @@ -15,27 +15,29 @@ Environment Callbacks [.z.a IP address](#za-ip-address) [.z.bm msg validator](#zbm-msg-validator) [.z.b view dependencies](#zb-view-dependencies) [.z.exit action on exit](#zexit-action-on-exit) [.z.c cores](#zc-cores) [.z.pc close](#zpc-close) - [.z.e TLS connection status](#ze-tls-connection-status) [.z.pd peach handles](#zpd-peach-handles) - [.z.ex failed primitive](#zex-failed-primitive) [.z.pg get](#zpg-get) - [.z.ey arg to failed primitive](#zey-argument-to-failed-primitive) [.z.pi input](#zpi-input) - [.z.f file](#zf-file) [.z.po open](#zpo-open) - [.z.H active sockets](#zh-active-sockets) [.z.pq qcon](#zpq-qcon) - [.z.h host](#zh-host) [.z.r blocked](#zr-blocked) - [.z.i PID](#zi-pid) [.z.ps set](#zps-set) - [.z.K version](#zk-version) [.z.pw validate user](#zpw-validate-user) - [.z.k release date](#zk-release-date) [.z.ts timer](#zts-timer) - [.z.l license](#zl-license) [.z.vs value set](#zvs-value-set) - [.z.o OS version](#zo-os-version) - [.z.q quiet mode](#zq-quiet-mode) Callbacks (HTTP) - [.z.s self](#zs-self) [.z.ac HTTP auth](#zac-http-auth) - [.z.u user ID](#zu-user-id) [.z.ph HTTP get](#zph-http-get) - [.z.W/w handles/handle](#zw-handles) [.z.pm HTTP methods](#zpm-http-methods) - [.z.X/x raw/parsed command line](#zx-raw-command-line) [.z.pp HTTP post](#zpp-http-post) - [.z.zd compression/encryption defaults](#zzd-compressionencryption-defaults) - Callbacks (WebSockets) - Environment (Time/Date) [.z.wc WebSocket close](#zwc-websocket-close) - [.z.D/d date shortcuts](#zt-zt-zd-zd-timedate-shortcuts) [.z.wo WebSocket open](#zwo-websocket-open) - [.z.N/n local/UTC timespan](#zn-local-timespan) [.z.ws WebSockets](#zws-websockets) + [.z.ex failed primitive](#zex-failed-primitive) [.z.pd peach handles](#zpd-peach-handles) + [.z.ey arg to failed primitive](#zey-argument-to-failed-primitive) [.z.pg get](#zpg-get) + [.z.f file](#zf-file) [.z.pi input](#zpi-input) + [.z.h host](#zh-host) [.z.po open](#zpo-open) + [.z.i PID](#zi-pid) [.z.pq qcon](#zpq-qcon) + [.z.K version](#zk-version) [.z.r blocked](#zr-blocked) + [.z.k release date](#zk-release-date) [.z.ps set](#zps-set) + [.z.l license](#zl-license) [.z.pw validate user](#zpw-validate-user) + [.z.o OS version](#zo-os-version) [.z.ts timer](#zts-timer) + [.z.q quiet mode](#zq-quiet-mode) [.z.vs value set](#zvs-value-set) + [.z.s self](#zs-self) + [.z.u user ID](#zu-user-id) Callbacks (HTTP) + [.z.X/x raw/parsed command line](#zx-raw-command-line) [.z.ac HTTP auth](#zac-http-auth) + [.z.zd compression/encryption defaults](#zzd-compressionencryption-defaults) [.z.ph HTTP get](#zph-http-get) + [.z.pm HTTP methods](#zpm-http-methods) +Environment (Connections) [.z.pp HTTP post](#zpp-http-post) + [.z.e TLS connection status](#ze-tls-connection-status) + [.z.H active sockets](#zh-active-sockets) Callbacks (WebSockets) + [.z.W/w handles/handle](#zw-handles) [.z.wc WebSocket close](#zwc-websocket-close) + [.z.wo WebSocket open](#zwo-websocket-open) +Environment (Time/Date) [.z.ws WebSockets](#zws-websockets) + [.z.D/d date shortcuts](#zt-zt-zd-zd-timedate-shortcuts) + [.z.N/n local/UTC timespan](#zn-local-timespan) [.z.P/p local/UTC timestamp](#zp-local-timestamp) [.z.T/t time shortcuts](#zt-zt-zd-zd-timedate-shortcuts) [.z.Z/z local/UTC datetime](#zz-local-datetime) From ba31ab745ea05ad22db0beba71452957a3abae97 Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Wed, 25 Sep 2024 17:33:05 +0100 Subject: [PATCH 24/54] cross reference cmd-line utils with each other (links added) --- docs/ref/dotq.md | 13 ++++++------- docs/ref/dotz.md | 4 ++-- 2 files changed, 8 insertions(+), 9 deletions(-) diff --git a/docs/ref/dotq.md b/docs/ref/dotq.md index 46befda40..6cce08ab4 100644 --- a/docs/ref/dotq.md +++ b/docs/ref/dotq.md @@ -388,8 +388,8 @@ xyz| 321f efg| `foo ``` -:fontawesome-solid-book: -[`.z.x`](dotz.md#zx-argv) +:fontawesome-solid-hand-point-right: +[`.z.x`](dotz.md#zx-argv) (argv), [`.z.X`](dotz.md#zx-raw-command-line) (raw command line), [`.z.f`](dotz.md#zf-file) (file), [`.z.q`](dotz.md#zq-quiet-mode) (quiet mode), [`.Q.opt`](#opt-command-parameters) (command parameters), [`.Q.x`](#x-non-command-parameters) (non-command parameters) ## `dpft` (save table) @@ -1468,8 +1468,8 @@ q)params`param1 "val1" ``` -:fontawesome-solid-book: -[`.z.x`](dotz.md#zx-argv) +:fontawesome-solid-hand-point-right: +[`.z.x`](dotz.md#zx-argv) (argv), [`.z.X`](dotz.md#zx-raw-command-line) (raw command line), [`.z.f`](dotz.md#zf-file) (file), [`.z.q`](dotz.md#zq-quiet-mode) (quiet mode), [`.Q.def`](#def-command-defaults) (command defaults), [`.Q.x`](#x-non-command-parameters) (non-command parameters) ## `P` (segments) @@ -2146,8 +2146,7 @@ q).Q.x "path/to/destn" ``` -:fontawesome-solid-book: -[`.z.x`](dotz.md#zx-argv), -[`.z.X`](dotz.md#zx-raw-command-line) +:fontawesome-solid-hand-point-right: +[`.z.x`](dotz.md#zx-argv) (argv), [`.z.X`](dotz.md#zx-raw-command-line) (raw command line), [`.z.f`](dotz.md#zf-file) (file), [`.z.q`](dotz.md#zq-quiet-mode) (quiet mode), [`.Q.opt`](#opt-command-parameters) (command parameters), [`.Q.def`](#def-command-defaults) (command defaults) diff --git a/docs/ref/dotz.md b/docs/ref/dotz.md index 7c3129f7f..c2cc503c6 100644 --- a/docs/ref/dotz.md +++ b/docs/ref/dotz.md @@ -1074,7 +1074,7 @@ q).z.X ``` :fontawesome-solid-hand-point-right: -[`.z.x` argv](#zx-argv), [`.z.f` file](#zf-file), [`.z.q` quiet mode](#zq-quiet-mode) +[`.z.x`](#zx-argv) (argv), [`.z.f`](#zf-file) (file), [`.z.q`](#zq-quiet-mode) (quiet mode), [`.Q.opt`](dotq.md#opt-command-parameters) (command parameters), [`.Q.def`](dotq.md#def-command-defaults) (command defaults), [`.Q.x`](dotq.md#x-non-command-parameters) (non-command parameters) ## `.z.x` (argv) @@ -1093,7 +1093,7 @@ q).z.x Command-line options can be converted to a dictionary using the convenient [`.Q.opt`](dotq.md#opt-command-parameters) function. :fontawesome-solid-hand-point-right: -[`.z.X` raw command line](#zx-raw-command-line), [`.z.f` file](#zf-file), [`.z.q` quiet mode](#zq-quiet-mode) +[`.z.X`](#zx-raw-command-line) (raw command line), [`.z.f`](#zf-file) (file), [`.z.q`](#zq-quiet-mode) (quiet mode), [`.Q.opt`](dotq.md#opt-command-parameters) (command parameters), [`.Q.def`](dotq.md#def-command-defaults) (command defaults), [`.Q.x`](dotq.md#x-non-command-parameters) (non-command parameters) ## `.z.Z` (local datetime) From 88b37eacbeb24510843a3c814466751c39870b98 Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Thu, 26 Sep 2024 10:17:48 +0100 Subject: [PATCH 25/54] add example to .Q.opt --- docs/ref/dotq.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/ref/dotq.md b/docs/ref/dotq.md index 6cce08ab4..b1a08cbd0 100644 --- a/docs/ref/dotq.md +++ b/docs/ref/dotq.md @@ -1467,6 +1467,16 @@ param2| "val2" q)params`param1 "val1" ``` +Example of a command-line parameter with no value and a parameter with multiple values: +```bash +$ q -param1 -param2 as asd -param3 +``` +```q +q).Q.opt .z.x +param1| () +param2| ("as";"asd") +param3| () +``` :fontawesome-solid-hand-point-right: [`.z.x`](dotz.md#zx-argv) (argv), [`.z.X`](dotz.md#zx-raw-command-line) (raw command line), [`.z.f`](dotz.md#zf-file) (file), [`.z.q`](dotz.md#zq-quiet-mode) (quiet mode), [`.Q.def`](#def-command-defaults) (command defaults), [`.Q.x`](#x-non-command-parameters) (non-command parameters) From 6c815ce903fd46c10777f1a1e41fad74cbb6d0ed Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Thu, 26 Sep 2024 10:46:05 +0100 Subject: [PATCH 26/54] fix alignment --- docs/ref/dotz.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ref/dotz.md b/docs/ref/dotz.md index c2cc503c6..bf01884e2 100644 --- a/docs/ref/dotz.md +++ b/docs/ref/dotz.md @@ -32,7 +32,7 @@ Environment Callbacks [.z.pm HTTP methods](#zpm-http-methods) Environment (Connections) [.z.pp HTTP post](#zpp-http-post) [.z.e TLS connection status](#ze-tls-connection-status) - [.z.H active sockets](#zh-active-sockets) Callbacks (WebSockets) + [.z.H active sockets](#zh-active-sockets) Callbacks (WebSockets) [.z.W/w handles/handle](#zw-handles) [.z.wc WebSocket close](#zwc-websocket-close) [.z.wo WebSocket open](#zwo-websocket-open) Environment (Time/Date) [.z.ws WebSockets](#zws-websockets) From ce611fb802df573e2683fa0558e0a676dbc0cce8 Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Thu, 26 Sep 2024 11:11:45 +0100 Subject: [PATCH 27/54] add details for .Q.def --- docs/ref/dotq.md | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/docs/ref/dotq.md b/docs/ref/dotq.md index b1a08cbd0..22c7b33df 100644 --- a/docs/ref/dotq.md +++ b/docs/ref/dotq.md @@ -359,13 +359,15 @@ IBM N IBM.N [](){#def-parse-options} ## `def` (command defaults) -_Default values for command-line arguments_ +_Default values and type checks for command-line arguments parsed with [`.Q.opt`](#opt-command-parameters)_ ```syntax .Q.def[x;y] ``` -Provides defaults and types for command-line arguments parsed with [`.Q.opt`](#opt-command-parameters). +Where `x` is a dictionary of default parameter names and values, and `y` is the output of `.Q.opt`. + +Types are inferred from the default values provided, which must be an atom type. ```bash $ q -abc 123 -xyz 321 @@ -375,17 +377,18 @@ q).Q.def[`abc`xyz`efg!(1;2.;`a)].Q.opt .z.x abc| 123 xyz| 321f efg| `a -q)\\ ``` +If a command-line value cannot be [converted to the data type](tok.md) of the default value, a [null](../basics/datatypes.md) will be produced + ```bash -$ q -abc 123 -xyz 321 -efg foo +$ q -param1 11 -param2 2000.01.01 -param3 wrong ``` ```q -q).Q.def[`abc`xyz`efg!(1;2.;`a)].Q.opt .z.x -abc| 123 -xyz| 321f -efg| `foo +q).Q.def[`param1`param2`param3!(1;1999.01.01;23.1)].Q.opt .z.x +param1| 11 +param2| 2000.01.01 +param3| 0n ``` :fontawesome-solid-hand-point-right: From eea171cb914f45da9f65f6d0e01f6b4709ee6750 Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Thu, 26 Sep 2024 12:26:45 +0100 Subject: [PATCH 28/54] add name of links to csv utils --- docs/ref/csv.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/ref/csv.md b/docs/ref/csv.md index f81c7687d..d4d264b2f 100644 --- a/docs/ref/csv.md +++ b/docs/ref/csv.md @@ -18,8 +18,8 @@ A synonym for `","` for use in preparing text for CSV files, or reading them. :fontawesome-solid-book: [Prepare Text](file-text.md#prepare-text), -[`.h.cd`](doth.md#hcd-csv-from-data), -[`.h.td`](doth.md#htd-tsv) +[`.h.cd`](doth.md#hcd-csv-from-data) (csv from data), +[`.h.td`](doth.md#htd-tsv) (tsv from data)
:fontawesome-solid-book-open: [File system](../basics/files.md) From e90e543b8c9f8df6659bd3096d46fb0429d587d0 Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Thu, 26 Sep 2024 13:28:34 +0100 Subject: [PATCH 29/54] add title to 2 sections in h namespace --- docs/ref/doth.md | 55 +++++++++++++++++++++++++----------------------- 1 file changed, 29 insertions(+), 26 deletions(-) diff --git a/docs/ref/doth.md b/docs/ref/doth.md index 53a0c84d0..b7e908e30 100644 --- a/docs/ref/doth.md +++ b/docs/ref/doth.md @@ -12,32 +12,35 @@ _Markup tools_
-[`.h.br`](#hbr-linebreak) linebreak [`.h.cd`](#hcd-csv-from-data) CSV from data -[`.h.code`](#hcode-code-after-tab) code after Tab [`.h.d`](#hd-delimiter) delimiter -[`.h.fram`](#hfram-frame) frame [`.h.ed`](#hed-excel-from-data) Excel from data -[`.h.ha`](#hha-anchor) anchor [`.h.edsn`](#hedsn-excel-from-tables) Excel from tables -[`.h.hb`](#hhb-anchor-target) anchor target [`.h.hc`](#hhc-escape-lt) escape lt -[`.h.ht`](#hht-marqdown-to-html) Marqdown to HTML [`.h.hr`](#hhr-horizontal-rule) horizontal rule -[`.h.hta`](#hhta-start-tag) start tag [`.h.iso8601`](#hiso8601-iso-timestamp) ISO timestamp -[`.h.htac`](#hhtac-element) element [`.h.jx`](#hjx-table) table -[`.h.htc`](#hhtc-element) element [`.h.td`](#htd-tsv-from-data) TSV from data -[`.h.html`](#hhtml-document) document [`.h.tx`](#htx-filetypes) filetypes -[`.h.http`](#hhttp-hyperlinks) hyperlinks [`.h.xd`](#hxd-xml) XML -[`.h.nbr`](#hnbr-no-break) no break [`.h.xs`](#hxs-xml-escape) XML escape -[`.h.pre`](#hpre-pre) pre [`.h.xt`](#hxt-json) JSON -[`.h.text`](#htext-paragraphs) paragraphs -[`.h.xmp`](#hxmp-xmp) XMP - -[`.h.he`](#hhe-http-400) HTTP 400 [`.h.c0`](#hc0-web-color) web color -[`.h.hn`](#hhn-http-response) HTTP response [`.h.c1`](#hc1-web-color) web color -[`.h.hp`](#hhp-http-response-pre) HTTP response pre [`.h.HOME`](#hhome-webserver-root) webserver root -[`.h.hy`](#hhy-http-response-content) HTTP response content [`.h.logo`](#hlogo-kx-logo) KX logo -[`.h.ka`](#hka-http-keepalive) HTTP keep-alive [`.h.sa`](#hsa-anchor-style) anchor style - [`.h.sb`](#hsb-body-style) body style -[`.h.hu`](#hhu-uri-escape) URI escape [`.h.ty`](#hty-mime-types) MIME types -[`.h.hug`](#hhug-uri-map) URI map [`.h.val`](#hval-value) value -[`.h.sc`](#hsc-uri-safe) URI-safe -[`.h.uh`](#huh-uri-unescape) URI unescape + [`.h.br`](#hbr-linebreak) linebreak [`.h.cd`](#hcd-csv-from-data) CSV from data + [`.h.code`](#hcode-code-after-tab) code after Tab [`.h.d`](#hd-delimiter) delimiter + [`.h.fram`](#hfram-frame) frame [`.h.ed`](#hed-excel-from-data) Excel from data + [`.h.ha`](#hha-anchor) anchor [`.h.edsn`](#hedsn-excel-from-tables) Excel from tables + [`.h.hb`](#hhb-anchor-target) anchor target [`.h.hc`](#hhc-escape-lt) escape lt + [`.h.ht`](#hht-marqdown-to-html) Marqdown to HTML [`.h.hr`](#hhr-horizontal-rule) horizontal rule + [`.h.hta`](#hhta-start-tag) start tag [`.h.iso8601`](#hiso8601-iso-timestamp) ISO timestamp + [`.h.htac`](#hhtac-element) element [`.h.jx`](#hjx-table) table + [`.h.htc`](#hhtc-element) element [`.h.td`](#htd-tsv-from-data) TSV from data + [`.h.html`](#hhtml-document) document [`.h.tx`](#htx-filetypes) filetypes + [`.h.http`](#hhttp-hyperlinks) hyperlinks [`.h.xd`](#hxd-xml) XML + [`.h.nbr`](#hnbr-no-break) no break [`.h.xs`](#hxs-xml-escape) XML escape + [`.h.pre`](#hpre-pre) pre [`.h.xt`](#hxt-json) JSON + [`.h.text`](#htext-paragraphs) paragraphs + [`.h.xmp`](#hxmp-xmp) XMP [`.h.c0`](#hc0-web-color) web color + [`.h.c1`](#hc1-web-color) web color +HTTP [`.h.HOME`](#hhome-webserver-root) webserver root + [`.h.he`](#hhe-http-400) HTTP 400 [`.h.logo`](#hlogo-kx-logo) KX logo + [`.h.hn`](#hhn-http-response) HTTP response [`.h.sa`](#hsa-anchor-style) anchor style + [`.h.hp`](#hhp-http-response-pre) HTTP response pre [`.h.sb`](#hsb-body-style) body style + [`.h.hy`](#hhy-http-response-content) HTTP response content [`.h.val`](#hval-value) value + [`.h.ka`](#hka-http-keepalive) HTTP keep-alive + [`.h.ty`](#hty-mime-types) MIME types + +URI formatting + [`.h.hu`](#hhu-uri-escape) URI escape + [`.h.hug`](#hhug-uri-map) URI map + [`.h.sc`](#hsc-uri-safe) URI-safe + [`.h.uh`](#huh-uri-unescape) URI unescape
The `.h` [namespace](../basics/namespaces.md) contains objects for From 53675bb640a27bf0b4d3c5db372d78e6b5a31fb9 Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Thu, 26 Sep 2024 13:51:24 +0100 Subject: [PATCH 30/54] split .h reference into appropriate categories --- docs/ref/doth.md | 40 ++++++++++++++++++++++------------------ 1 file changed, 22 insertions(+), 18 deletions(-) diff --git a/docs/ref/doth.md b/docs/ref/doth.md index b7e908e30..ac72d8e76 100644 --- a/docs/ref/doth.md +++ b/docs/ref/doth.md @@ -12,27 +12,31 @@ _Markup tools_
- [`.h.br`](#hbr-linebreak) linebreak [`.h.cd`](#hcd-csv-from-data) CSV from data +Markup (HTML and XML) Data Serialization + [`.h.br`](#hbr-linebreak) linebreak [`.h.cd`](#hcd-csv-from-data) CSV from data [`.h.code`](#hcode-code-after-tab) code after Tab [`.h.d`](#hd-delimiter) delimiter [`.h.fram`](#hfram-frame) frame [`.h.ed`](#hed-excel-from-data) Excel from data [`.h.ha`](#hha-anchor) anchor [`.h.edsn`](#hedsn-excel-from-tables) Excel from tables - [`.h.hb`](#hhb-anchor-target) anchor target [`.h.hc`](#hhc-escape-lt) escape lt - [`.h.ht`](#hht-marqdown-to-html) Marqdown to HTML [`.h.hr`](#hhr-horizontal-rule) horizontal rule - [`.h.hta`](#hhta-start-tag) start tag [`.h.iso8601`](#hiso8601-iso-timestamp) ISO timestamp - [`.h.htac`](#hhtac-element) element [`.h.jx`](#hjx-table) table - [`.h.htc`](#hhtc-element) element [`.h.td`](#htd-tsv-from-data) TSV from data - [`.h.html`](#hhtml-document) document [`.h.tx`](#htx-filetypes) filetypes - [`.h.http`](#hhttp-hyperlinks) hyperlinks [`.h.xd`](#hxd-xml) XML - [`.h.nbr`](#hnbr-no-break) no break [`.h.xs`](#hxs-xml-escape) XML escape - [`.h.pre`](#hpre-pre) pre [`.h.xt`](#hxt-json) JSON - [`.h.text`](#htext-paragraphs) paragraphs - [`.h.xmp`](#hxmp-xmp) XMP [`.h.c0`](#hc0-web-color) web color - [`.h.c1`](#hc1-web-color) web color -HTTP [`.h.HOME`](#hhome-webserver-root) webserver root - [`.h.he`](#hhe-http-400) HTTP 400 [`.h.logo`](#hlogo-kx-logo) KX logo - [`.h.hn`](#hhn-http-response) HTTP response [`.h.sa`](#hsa-anchor-style) anchor style - [`.h.hp`](#hhp-http-response-pre) HTTP response pre [`.h.sb`](#hsb-body-style) body style - [`.h.hy`](#hhy-http-response-content) HTTP response content [`.h.val`](#hval-value) value + [`.h.hb`](#hhb-anchor-target) anchor target [`.h.ht`](#hht-marqdown-to-html) Marqdown to HTML + [`.h.hc`](#hhc-escape-lt) escape lt [`.h.iso8601`](#hiso8601-iso-timestamp) ISO timestamp + [`.h.hr`](#hhr-horizontal-rule) horizontal rule [`.h.jx`](#hjx-table) table + [`.h.hta`](#hhta-start-tag) start tag [`.h.td`](#htd-tsv-from-data) TSV from data + [`.h.htac`](#hhtac-element) element [`.h.tx`](#htx-filetypes) filetypes + [`.h.htc`](#hhtc-element) element [`.h.xd`](#hxd-xml) XML + [`.h.html`](#hhtml-document) document [`.h.xt`](#hxt-json) JSON + [`.h.http`](#hhttp-hyperlinks) hyperlinks + [`.h.logo`](#hlogo-kx-logo) KX logo Web Console + [`.h.nbr`](#hnbr-no-break) no break [`.h.c0`](#hc0-web-color) web color + [`.h.pre`](#hpre-pre) pre [`.h.c1`](#hc1-web-color) web color + [`.h.text`](#htext-paragraphs) paragraphs [`.h.HOME`](#hhome-webserver-root) webserver root + [`.h.xmp`](#hxmp-xmp) XMP [`.h.sa`](#hsa-anchor-style) anchor style + [`.h.xs`](#hxs-xml-escape) XML escape [`.h.sb`](#hsb-body-style) body style + [`.h.val`](#hval-value) value +HTTP + [`.h.he`](#hhe-http-400) HTTP 400 + [`.h.hn`](#hhn-http-response) HTTP response + [`.h.hp`](#hhp-http-response-pre) HTTP response pre + [`.h.hy`](#hhy-http-response-content) HTTP response content [`.h.ka`](#hka-http-keepalive) HTTP keep-alive [`.h.ty`](#hty-mime-types) MIME types From 8ef17e85ea655bf1d0248f6725cda2433e04a466 Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Thu, 26 Sep 2024 14:13:56 +0100 Subject: [PATCH 31/54] add link to .j from .h.xt --- docs/ref/doth.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/ref/doth.md b/docs/ref/doth.md index ac72d8e76..a325bea41 100644 --- a/docs/ref/doth.md +++ b/docs/ref/doth.md @@ -22,7 +22,7 @@ Markup (HTML and XML) Data Serialization [`.h.hr`](#hhr-horizontal-rule) horizontal rule [`.h.jx`](#hjx-table) table [`.h.hta`](#hhta-start-tag) start tag [`.h.td`](#htd-tsv-from-data) TSV from data [`.h.htac`](#hhtac-element) element [`.h.tx`](#htx-filetypes) filetypes - [`.h.htc`](#hhtc-element) element [`.h.xd`](#hxd-xml) XML + [`.h.htc`](#hhtc-element) element [`.h.xd`](#hxd-xml) XML from data [`.h.html`](#hhtml-document) document [`.h.xt`](#hxt-json) JSON [`.h.http`](#hhttp-hyperlinks) hyperlinks [`.h.logo`](#hlogo-kx-logo) KX logo Web Console @@ -868,3 +868,5 @@ q)first .h.xt[`json;("{\"foo\":\"bar\"}";"{\"this\":\"that\"}")] foo| "bar" ``` +:fontawesome-regular-hand-point-right: +[`.j` namespace](dotj.md) (JSON serialization) \ No newline at end of file From c9826b1276e020cd7d25760f26d6d8dae1bb810a Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Thu, 26 Sep 2024 14:35:36 +0100 Subject: [PATCH 32/54] reformat main menu of .h --- docs/ref/doth.md | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/docs/ref/doth.md b/docs/ref/doth.md index a325bea41..065c4fbca 100644 --- a/docs/ref/doth.md +++ b/docs/ref/doth.md @@ -33,18 +33,12 @@ Markup (HTML and XML) Data Serialization [`.h.xs`](#hxs-xml-escape) XML escape [`.h.sb`](#hsb-body-style) body style [`.h.val`](#hval-value) value HTTP - [`.h.he`](#hhe-http-400) HTTP 400 - [`.h.hn`](#hhn-http-response) HTTP response - [`.h.hp`](#hhp-http-response-pre) HTTP response pre - [`.h.hy`](#hhy-http-response-content) HTTP response content - [`.h.ka`](#hka-http-keepalive) HTTP keep-alive - [`.h.ty`](#hty-mime-types) MIME types - -URI formatting - [`.h.hu`](#hhu-uri-escape) URI escape - [`.h.hug`](#hhug-uri-map) URI map - [`.h.sc`](#hsc-uri-safe) URI-safe - [`.h.uh`](#huh-uri-unescape) URI unescape + [`.h.he`](#hhe-http-400) HTTP 400 URI formatting + [`.h.hn`](#hhn-http-response) HTTP response [`.h.hu`](#hhu-uri-escape) URI escape + [`.h.hp`](#hhp-http-response-pre) HTTP response pre [`.h.hug`](#hhug-uri-map) URI map + [`.h.hy`](#hhy-http-response-content) HTTP response content [`.h.sc`](#hsc-uri-safe) URI-safe + [`.h.ka`](#hka-http-keepalive) HTTP keep-alive [`.h.uh`](#huh-uri-unescape) URI unescape + [`.h.ty`](#hty-mime-types) MIME types
The `.h` [namespace](../basics/namespaces.md) contains objects for From 1c7c5ef9e5e74a871fae55fddeb3b824b91f56de Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Fri, 27 Sep 2024 09:39:51 +0100 Subject: [PATCH 33/54] add details to system namespaces --- docs/ref/index.md | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/docs/ref/index.md b/docs/ref/index.md index f294ca568..fdbae7959 100644 --- a/docs/ref/index.md +++ b/docs/ref/index.md @@ -257,24 +257,23 @@ milliseconds: time mod 1000 ## Namespaces -### [`.h`](doth.md) +### [`.h`](doth.md) (markup) -Markup output for HTTP +HTTP, markup and data conversion. -### [`.j`](dotj.md) +### [`.j`](dotj.md) (JSON) -De/serialize as JSON +De/serialize as JSON. -### `.m` +### [`.m`](dotm.md) (memory backed files) -:fontawesome-regular-hand-point-right: -[Memory backed by files](dotm.md) +Memory backed by files. -### [`.Q`](dotq.md) +### [`.Q`](dotq.md) (utils) -Utilities: general, environment, IPC, datatype, database, partitioned database state, segmented database state, file I/O +Utilities: general, environment, IPC, datatype, database, partitioned database state, segmented database state, file I/O, debugging, profiling. -### [`.z`](dotz.md) +### [`.z`](dotz.md) (system, callbacks) System variables, callbacks From 88cde8590861d8bf1fd113f122fa5dbd4f0474ea Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Fri, 27 Sep 2024 09:54:25 +0100 Subject: [PATCH 34/54] add links from .h namespace --- docs/ref/doth.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/ref/doth.md b/docs/ref/doth.md index 065c4fbca..895dc15db 100644 --- a/docs/ref/doth.md +++ b/docs/ref/doth.md @@ -93,6 +93,8 @@ q).h.cd (`a`b`c;1 2 3;"xyz") Columns can be nested vectors, in which case [`.h.d`](#hd-delimiter) is used to separate subitems. (Since V4.0 2020.03.17.) +:fontawesome-regular-hand-point-right: +[0: load csv](file-text.md#load-csv) ## `.h.code` (code after Tab) @@ -863,4 +865,4 @@ foo| "bar" ``` :fontawesome-regular-hand-point-right: -[`.j` namespace](dotj.md) (JSON serialization) \ No newline at end of file +[`.j` namespace](dotj.md) (JSON de/serialization) From 652fcd5e92c0b31779d6d05e420b2a9be674f2eb Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Mon, 30 Sep 2024 13:26:30 +0100 Subject: [PATCH 35/54] improved cross referencing to socket/handle utils --- docs/basics/internal.md | 12 ++++++++++-- docs/ref/dotz.md | 24 ++++++++++++------------ docs/ref/index.md | 4 ++-- 3 files changed, 24 insertions(+), 16 deletions(-) diff --git a/docs/basics/internal.md b/docs/basics/internal.md index 723ff9d38..9402ca5d3 100644 --- a/docs/basics/internal.md +++ b/docs/basics/internal.md @@ -466,6 +466,13 @@ where `x` is a list of socket handles, returns a table with columns Since v4.0 2020.06.01. ```q +q)h:hopen 5000 +q)-38!h +p| "q" +f| "t" +z| 0b +n| 0 +m| 0 q){([]h)!-38!h:.z.H}[] h| p f z n m -| --------- @@ -473,8 +480,9 @@ h| p f z n m 9| q t 0 0 0 ``` -:fontawesome-solid-book: -[`.z.H` active sockets](../ref/dotz.md#zh-active-sockets) +:fontawesome-solid-hand-point-right: +[`.z.H` active sockets](../ref/dotz.md#zh-active-sockets), [`.z.W` handles](../ref/dotz.md#zw-handles), [`.z.w` handle](../ref/dotz.md#zw-handle) + ## `-120!x` (memory domain) ```syntax diff --git a/docs/ref/dotz.md b/docs/ref/dotz.md index bf01884e2..5e6b71c7e 100644 --- a/docs/ref/dotz.md +++ b/docs/ref/dotz.md @@ -314,10 +314,7 @@ q).z.H~key .z.W ``` :fontawesome-solid-hand-point-right: -[`.z.W` handles](#zw-handles), [`.z.w` handle](#zw-handle) -
-:fontawesome-solid-book-open: -[`-38!` socket table](../basics/internal.md#-38x-socket-table) +[`.z.W` handles](#zw-handles), [`.z.w` handle](#zw-handle), [`-38!` socket table](../basics/internal.md#-38x-socket-table) ## `.z.h` (host) @@ -958,11 +955,17 @@ q)neg[h]({};til 1000000); neg[h]({};til 10); sum each .z.W 6| 8000140 ``` +Querying known handles can also be performed using [`-38!`](../basics/internal.md#-38x-socket-table), which can be more performant than using `.z.W` to return the entire dataset of handles. +```q +q)h:hopen 5000 +q)neg[h]"11+1111111";.z.W h +24 +q)neg[h]"11+1111111";(-38!h)`m +24 +``` + :fontawesome-solid-hand-point-right: -[`.z.h` active sockets](#zh-active-sockets), [`.z.w` handle](#zw-handle) -
-:fontawesome-solid-book-open: -[`-38!` socket table](../basics/internal.md#-38x-socket-table) +[`.z.h` active sockets](#zh-active-sockets), [`.z.w` handle](#zw-handle), [`-38!` socket table](../basics/internal.md#-38x-socket-table) ## `.z.w` (handle) @@ -977,10 +980,7 @@ q).z.w !!! warning "Inside a `.z.p`* callback it returns the handle of the client session, not the current session." :fontawesome-solid-hand-point-right: -[`.z.h` active sockets](#zh-active-sockets), [`.z.W` handles](#zw-handles) -
-:fontawesome-solid-book-open: -[`-38!` socket table](../basics/internal.md#-38x-socket-table) +[`.z.h` active sockets](#zh-active-sockets), [`.z.W` handles](#zw-handles), [`-38!` socket table](../basics/internal.md#-38x-socket-table) ## `.z.wc` (websocket close) diff --git a/docs/ref/index.md b/docs/ref/index.md index fdbae7959..8f336ca73 100644 --- a/docs/ref/index.md +++ b/docs/ref/index.md @@ -273,7 +273,7 @@ Memory backed by files. Utilities: general, environment, IPC, datatype, database, partitioned database state, segmented database state, file I/O, debugging, profiling. -### [`.z`](dotz.md) (system, callbacks) +### [`.z`](dotz.md) (environment, callbacks) -System variables, callbacks +Environment, callbacks From 05745f73795f09e87aa929e202f8427726734991 Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Mon, 30 Sep 2024 13:43:34 +0100 Subject: [PATCH 36/54] add link to -38! from ipc guide (view pending queue) --- docs/basics/ipc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/basics/ipc.md b/docs/basics/ipc.md index 603a4db08..711a06792 100644 --- a/docs/basics/ipc.md +++ b/docs/basics/ipc.md @@ -217,7 +217,7 @@ You may consider increasing the size of TCP send/receive buffers on your system Messages can be queued for sending to a remote process through using async messaging. kdb+ queues the serialized message in user space, later writing it to the socket as the remote end drains the message queue. -You can see how many messages are queued on a handle and their sizes as a dictionary using the command variable [`.z.W`](../ref/dotz.md#zw-handles "handles"). +You can see the queue size using [-38!](internal.md#-38x-socket-table), or [`.z.W`](../ref/dotz.md#zw-handles "handles") for all handles. Sometimes it is useful to send a large number of aysnc messages, but then to block until they have all been sent. This can be achieved through using async flush – invoked as `neg[h][]` or `neg[h](::)`. From 4a6276c47f8b9ff109aa4feffa1186bf4e5c5520 Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Tue, 1 Oct 2024 17:35:06 +0100 Subject: [PATCH 37/54] remove dup info, add link to orig info --- docs/basics/ipc.md | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/docs/basics/ipc.md b/docs/basics/ipc.md index 711a06792..9e01465e5 100644 --- a/docs/basics/ipc.md +++ b/docs/basics/ipc.md @@ -154,22 +154,18 @@ q)h(`add;2;3) / execute the 'add' function as defined on the server, pas #### One-shot message -A sync message can also be sent on a short-lived connection. +A sync message can also be sent on a short-lived connection (called a [one-shot](../ref/hopen.md#one-shot-request)). When sending multiple messages, this is less efficient than [using a pre-existing connection](#sync-request-get) due to the effort of repeated connections/disconnections. A useful shorthand for a one-shot get is: ```q -q)`::5001 "1+1" +q)`::5001 "1+1" 2 ``` - -Since V4.0 2020.03.09, a one-shot query can be run with a timeout (in milliseconds), as in the second example below: - +which is equivalent to ```q -q)`::4567"2+2" -4 -q)`::[(`::4567;100);"1+1"] +q)`:localhost:5001 "1+1" 2 ``` From 207fa21d4e8c5a679cf5c4cde5c832115fa05112 Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Wed, 2 Oct 2024 12:21:24 +0100 Subject: [PATCH 38/54] note on remembering to use hclose for connnections --- docs/basics/ipc.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/basics/ipc.md b/docs/basics/ipc.md index 9e01465e5..5681f5119 100644 --- a/docs/basics/ipc.md +++ b/docs/basics/ipc.md @@ -54,6 +54,8 @@ Sync messages can also be sent without a pre-existing connection using [one-shot The maximum number of connections is defined by the system limit for protocol (operating system configurable). Prior to 4.1t 2023.09.15, the limit was hardcoded to 1022. After the limit is reached, you see the error `'conn` on the server process. All successfully opened connections remain open. +!!! note "It is important to use [`hclose`](../ref/hopen.md#hclose) once finished with any connection. Connections will not automatically close if their associated handle is deleted." + ## Closing connections Client or server connections can be closed using [`hclose`](../ref/hopen.md#hclose). From 584eafa28cb721237efd528a79d69e24f420c6ad Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Fri, 4 Oct 2024 09:13:44 +0100 Subject: [PATCH 39/54] fix format --- docs/wp/data-loaders/index.md | 27 ++++++++------------------- 1 file changed, 8 insertions(+), 19 deletions(-) diff --git a/docs/wp/data-loaders/index.md b/docs/wp/data-loaders/index.md index b9aa16976..2fbd53f7e 100644 --- a/docs/wp/data-loaders/index.md +++ b/docs/wp/data-loaders/index.md @@ -184,7 +184,7 @@ Once the orchestrator recognizes that a batch has arrived (through either file n !!! tip "What functions to apply to specific batches/files can be managed with configuration based on the filename." -### Step 1: Orchestrator distributes read and save task per file +#### Step 1: Orchestrator distributes read and save task per file Upon the orchestrator recognizing that a batch with its required files is available (files 1, 2, and 3), the following is executed within the `.mi.sendToFreeWorker` function. @@ -248,7 +248,7 @@ and updates in memory tables for tracking. ``` -### Step 2: Worker receives task to read and write a file +#### Step 2: Worker receives task to read and write a file The worker receives a run task command `.mi.runTask` from the orchestrator: @@ -332,7 +332,7 @@ Once the read and save task is complete, the tracked information i.e. the name o The column memory statistics are available so memory required for any further jobs on these columns could be estimated as part of distributing tasks to extend memory management. -### Step 3: Orchestrator appends to sym and sends next task +#### Step 3: Orchestrator appends to sym and sends next task After a success message is received from a worker via `.mi.workerResponse`, the orchestrator updates the tasks and workers tables. @@ -449,7 +449,7 @@ In order to maintain sym file integrity the following method is used to ensure a [Working with symfiles](../symfiles.md) -### Step 4: Indexing +#### Step 4: Indexing The orchestrator sends another `.mi.runTask` to index the data by the chosen sorting columns (`` `sym`time`` in this example) to an available worker. @@ -501,7 +501,7 @@ set[` sv mdb,`.d;key x`colSizes] ``` -###Step 5: Merge +#### Step 5: Merge Once the index task is complete, the orchestrator assigns each worker a distinct subset of columns to merge one by one based on the index created in Step 4. @@ -531,14 +531,14 @@ set[toPath;data] ``` -### Step 6: Move table/s +#### Step 6: Move table/s After receiving a callback message from each worker that the merge has been completed, a worker is assigned via `.mi.runTask` to move the table/s to the relevant HDB directory. The merged table is moved using system `mv` (`MOVE` on Windows) command and during the move, the main HDB can be temporarily locked from queries. Once the orchestrator receives a success message for the move task the batch is considered complete and the processing of the next batch can commence. -### Post-batch tasks +#### Post-batch tasks In order to reduce downtime between batches, each worker is killed and restarted by the orchestrator process after the batch. Killing workers and restarting them has been found to free up memory faster rather than each worker running garbage collection, which can be time-consuming. @@ -546,6 +546,7 @@ Once the batch successfully completes, any post-ingestion event-driven tasks can :fontawesome-regular-map: [Surveillance techniques to effectively monitor algo- and high-frequency trading](../surveillance/index.md)
+:fontawesome-regular-map: [Transaction-cost analysis using kdb+](../transaction-cost.md) @@ -582,18 +583,6 @@ Key elements and benefits of the proposed framework are: - Post-ingestion actions can be added – e.g. trigger the running regulatory reports, benchmarks or alerts -## Conclusion - -For use cases where real-time feeds are unfeasible (due to cost, technical limitations or time), this style of batch ingestion framework is a great solution which can handle and scale to ever increasing data volumes. The need to ingest batch data as fast as possible due to reporting, regulatory obligations, post-trade requirements or other business needs means kdb+ is perfectly suited for the task. - -In this paper we discussed batch processing and use cases where it may be most appropriate: high volumes and throughput, large number of files, hardware constraints. We then described a proposed mass ingestion framework using kdb+, discussing the technical aspects of the framework such as the number of workers to utilize, maintaining sym file integrity and task allocation. Finally, we went through an example ingestion scenario and concluded by outlining the benefits of the framework. - -:fontawesome-brands-github: -[kxcontrib/massIngestionDataloader](https://github.com/kxcontrib/massIngestionDataloader) - -The framework outlined in this paper is a simplified version of a framework that has been used in several KX implementations for batch ingestion. It has proven to be fast, efficient and scalable. - - ## Author ![Enda Gildea](../../img/faces/endeagildea.jpg) From b66856e30d7373d1a5aafbdb41d29e695a215fb4 Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Fri, 4 Oct 2024 09:17:14 +0100 Subject: [PATCH 40/54] remove dup info --- docs/wp/compress/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/wp/compress/index.md b/docs/wp/compress/index.md index 9410664f3..c07ec8dfc 100644 --- a/docs/wp/compress/index.md +++ b/docs/wp/compress/index.md @@ -16,7 +16,7 @@ overall performance. Early versions of kdb+ achieved compression via file systems with inbuilt compression, such as ZFS. V2.7 introduced built-in OS-agnostic compression, which allowed on-disk data to be converted to compressed format using a range of algorithms and compression levels. This was expanded (V2.8), with the ability to stream in-memory data directly to compressed format on disk. ZFS compression is still useful for some kdb+ applications, as it keeps cached data available for multiple processes. However, this paper will focus on the inbuilt data-compression options provided by kdb+, available on all supported architectures. -Each system will have its own characteristics which determine the appropriate compression configurations to use. This paper will cover an introduction to compression in kdb+, a discussion of contributing factors to compression ratios and performance, and an analysis of how the use of compressed data can affect performance of some sample use cases. +Each system will have its own characteristics which determine the appropriate compression configurations to use. All tests were run using kdb+ version 3.1 (2013.09.05) From 42e60069c2560bb19b6fd1c179c00dd8b941155a Mon Sep 17 00:00:00 2001 From: Simon Shanks Date: Fri, 4 Oct 2024 09:48:54 +0100 Subject: [PATCH 41/54] remove dup info --- docs/wp/blockchain/index.md | 16 ---------------- docs/wp/option-pricing/index.md | 3 --- docs/wp/permissions/index.md | 8 ++++++-- docs/wp/socket-sharding/index.md | 4 +--- docs/wp/surveillance-latency/index.md | 2 +- 5 files changed, 8 insertions(+), 25 deletions(-) diff --git a/docs/wp/blockchain/index.md b/docs/wp/blockchain/index.md index 7718bfbc4..4e78a81da 100644 --- a/docs/wp/blockchain/index.md +++ b/docs/wp/blockchain/index.md @@ -972,22 +972,6 @@ time. ![](img/media/image28.png) -## Conclusion - -This paper described how a q process can be built to interact with a -Bitcoin full node to extract and store blockchain transaction data in -kdb+ format. By making use of appropriate schemas, partitioned database -structures and query performance techniques it was possible to create a -simple blockchain explorer process to look up transaction information and -perform some interesting blockchain analytics. This application of kdb+ -to the area of blockchain explorer technology is a further step toward -better understanding blockchain data, the unique challenges associated -with data retrieval and storage, and potential for application -development using the technology within the domain. - -[:fontawesome-solid-print:](/download/wp/blockchain-a4.pdf) - - ## Authors ![Daniel Irwin](../../img/faces/danielirwin.png) diff --git a/docs/wp/option-pricing/index.md b/docs/wp/option-pricing/index.md index 642c07c24..e29b4a3cb 100644 --- a/docs/wp/option-pricing/index.md +++ b/docs/wp/option-pricing/index.md @@ -682,9 +682,6 @@ In this paper we demonstrated that it is possible to calculate option prices usi Looking at the results produced, it is clear that both the option price produced and the resulting RMSE/log RMSE converged fastest when compared with the Black-Scholes price for the Quasi-Monte Carlo approach, with Sobol’ sequence number generation and Brownian-bridge construction. -Additionally, by plotting results we have shown that the q implementation replicates the original results produced in C++, presented in the paper by [S. Kucherenko et al. 2007](http://www.broda.co.uk/gsa/wilmott_GSA_SK.pdf "Wilmott"). - - ## Author **Deanna Morgan** joined First Derivatives in June 2018 as a data scientist in the Capital Markets Training Program and currently works as a machine-learning engineer in London. diff --git a/docs/wp/permissions/index.md b/docs/wp/permissions/index.md index c934cd7da..42dad8cd8 100644 --- a/docs/wp/permissions/index.md +++ b/docs/wp/permissions/index.md @@ -11,7 +11,9 @@ by [Tom Martin](#author) {: .wp-author} -Due to its efficiency in storing and retrieving large volumes of data, kdb+ is the data-storage technology of choice for many financial institutions. kdb+ processes thus often contain sensitive, proprietary information in the form of data or proprietary code and so it is important to restrict who can and cannot access this information. kdb+ offers a number of in-built access functions, though in a default kdb+ instance these are not activated. This paper discusses various methods in which a permissioning and entitlements system can be implemented in kdb+ by extending these in-built functions, allowing access to sensitive information to be controlled and restricted, exposing data to some clients but not to others. +kdb+ processes often contain sensitive, proprietary information in the form of data or proprietary code and so it is important to restrict who can and cannot access this information. + +kdb+ offers a number of in-built access functions. This paper discusses various methods in which a permissioning and entitlements system can be implemented in kdb+ by extending these in-built functions, allowing access to sensitive information to be controlled and restricted, exposing data to some clients but not to others. !!! tip "Commercial-grade products" @@ -1092,7 +1094,9 @@ poweruser1 "select from quote" 1 "" ## Conclusion -This paper was an introduction to permissioning in kdb+ without using LDAP or any other external entitlements system. As kdb+ is the market leader in the storage and retrieval of big data, it is the database technology of choice for the world’s top financial institutions and as such it is used to store sensitive, proprietary information. In order to pass a security audit, access to this data should be controlled and logged to ensure that only those who are entitled to view the information are able to do so. +This paper was an introduction to permissioning in kdb+ without using LDAP or any other external entitlements system. + +In order to pass a security audit, access to this data should be controlled and logged to ensure that only those who are entitled to view the information are able to do so. We have described a number of methods of securing a kdb+ process. We examined the concept of splitting clients into separate groups or classes, each with different permission levels. We examined how to block write access on a kdb+ process, and how to restrict certain users from viewing proprietary code. While the system described in the paper offers a broad coverage, including blocking some forms of code injection, it is not intended to be complete. diff --git a/docs/wp/socket-sharding/index.md b/docs/wp/socket-sharding/index.md index 390781647..f2a92b2fe 100644 --- a/docs/wp/socket-sharding/index.md +++ b/docs/wp/socket-sharding/index.md @@ -329,9 +329,7 @@ The reconnect attempt immediately connected resulting in minimal downtime. ## Conclusion -In this white paper, we demonstrated the set-up and use of socket sharding in kdb+ as well as investigating several example scenarios. - -From results generated throughout this paper, it is evident that there are both pros and cons of using socket sharding in kdb+. If processes listening on the sharded socket are all running with the same performance, socket sharding can reduce the response time for requests to clients and processing load on a listener process. +It is evident that there are both pros and cons of using socket sharding in kdb+. If processes listening on the sharded socket are all running with the same performance, socket sharding can reduce the response time for requests to clients and processing load on a listener process. As the processes are all listening on the same port, clients do not need configuration changes in order for connection requests to be assigned to new servers. diff --git a/docs/wp/surveillance-latency/index.md b/docs/wp/surveillance-latency/index.md index 68a7a667b..ed5ad0c31 100644 --- a/docs/wp/surveillance-latency/index.md +++ b/docs/wp/surveillance-latency/index.md @@ -330,7 +330,7 @@ As time progresses up to 12 hours, we can see that runtime memory usage fluctuat These results show that the use of event ID windows instead of timestamp windows causes a marginal but acceptable hit on the performance of the window-join approach. -## Conclusions +## Conclusion There are other factors that we have not experimented with – the distribution of the twenty runs of the alert analytics out to different secondary threads, implementing intraday memory management techniques in the alert engines or varying the hard-coded lookback threshold of five minutes on top of varying the intraday execution point frequency. From 3ad352b32523d3883a4b32d1d2512400db940edb Mon Sep 17 00:00:00 2001 From: Simon Shanks <59612559+sshanks-kx@users.noreply.github.com> Date: Fri, 4 Oct 2024 13:46:33 +0100 Subject: [PATCH 42/54] Update docs/basics/ipc.md Co-authored-by: Veronica Calescu --- docs/basics/ipc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/basics/ipc.md b/docs/basics/ipc.md index 5681f5119..2a9d80986 100644 --- a/docs/basics/ipc.md +++ b/docs/basics/ipc.md @@ -54,7 +54,7 @@ Sync messages can also be sent without a pre-existing connection using [one-shot The maximum number of connections is defined by the system limit for protocol (operating system configurable). Prior to 4.1t 2023.09.15, the limit was hardcoded to 1022. After the limit is reached, you see the error `'conn` on the server process. All successfully opened connections remain open. -!!! note "It is important to use [`hclose`](../ref/hopen.md#hclose) once finished with any connection. Connections will not automatically close if their associated handle is deleted." +!!! note "It is important to use [`hclose`](../ref/hopen.md#hclose) once finished with any connection. Connections do not automatically close if their associated handle is deleted." ## Closing connections From b59e242d1dfad5db497fcac12a40e2411fe2462f Mon Sep 17 00:00:00 2001 From: Simon Shanks <59612559+sshanks-kx@users.noreply.github.com> Date: Fri, 4 Oct 2024 13:46:46 +0100 Subject: [PATCH 43/54] Update docs/basics/parsetrees.md Co-authored-by: Veronica Calescu --- docs/basics/parsetrees.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/basics/parsetrees.md b/docs/basics/parsetrees.md index 98d4a6f7e..0ceff5d05 100644 --- a/docs/basics/parsetrees.md +++ b/docs/basics/parsetrees.md @@ -8,7 +8,7 @@ author: [Peter Storeng, Stephen Taylor, Simon Shanks] ## Overview -[`parse`](../ref/parse.md) is a useful tool for seeing how a statement in q is evaluated. Pass the `parse` keyword a q statement as a string and it will return the parse tree of that expression. +[`parse`](../ref/parse.md) is a useful tool for seeing how a statement in q is evaluated. Pass the `parse` keyword a q statement as a string and it returns the parse tree of that expression. A _parse tree_ represents an expression, not immediately evaluated. Its virtue is that the expression can be evaluated whenever and in whatever context it is needed. The two main functions dealing with parse trees are: From 1200fffcbbb7973981060dd87040efee4edec890 Mon Sep 17 00:00:00 2001 From: Simon Shanks <59612559+sshanks-kx@users.noreply.github.com> Date: Fri, 4 Oct 2024 13:47:01 +0100 Subject: [PATCH 44/54] Update docs/basics/parsetrees.md Co-authored-by: Veronica Calescu --- docs/basics/parsetrees.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/basics/parsetrees.md b/docs/basics/parsetrees.md index 0ceff5d05..65e9af09b 100644 --- a/docs/basics/parsetrees.md +++ b/docs/basics/parsetrees.md @@ -55,7 +55,7 @@ q)type @: ``` -The `parse` keyword on an operation involving the example above will expose the `k` code. Using the underlying code, it can be run using kdb+ in-build k interpreter to show that it produces the same result: +The `parse` keyword on an operation involving the example above exposes the `k` code. Using the underlying code, it can be run using kdb+ in-build k interpreter to show that it produces the same result: ```q q)type 6 -7h From 308b738a7ffb16bf9f0b24d26521af7ee271d451 Mon Sep 17 00:00:00 2001 From: Simon Shanks <59612559+sshanks-kx@users.noreply.github.com> Date: Fri, 4 Oct 2024 13:47:14 +0100 Subject: [PATCH 45/54] Update docs/basics/parsetrees.md Co-authored-by: Veronica Calescu --- docs/basics/parsetrees.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/basics/parsetrees.md b/docs/basics/parsetrees.md index 65e9af09b..1b753fcb1 100644 --- a/docs/basics/parsetrees.md +++ b/docs/basics/parsetrees.md @@ -77,7 +77,7 @@ like A parse tree is a q construct which represents an expression but which is not immediately evaluated. It takes the form of a list where the first item is a function and the remaining items are the arguments. Any of the items of the list can be parse trees themselves. -Note that in a parse tree a variable is represented by a symbol containing its name. Thus to distinguish a symbol or a list of symbols from a variable it is necessary to enlist that expression. When we apply the `parse` function to create a parse tree, explicit definitions in `.q` are shown in their full k form. In particular, an enlisted element is represented by a preceding comma. +Note that, in a parse tree, a variable is represented by a symbol containing its name. Thus, to distinguish a symbol or a list of symbols from a variable, it is necessary to enlist that expression. When we apply the `parse` function to create a parse tree, explicit definitions in `.q` are shown in their full k form. In particular, an enlisted element is represented by a preceding comma. ```q q)parse"5 6 7 8 + 1 2 3 4" From e45fc958b07dc973e347ca1c744aee09dbe6ead0 Mon Sep 17 00:00:00 2001 From: Simon Shanks <59612559+sshanks-kx@users.noreply.github.com> Date: Fri, 4 Oct 2024 13:47:27 +0100 Subject: [PATCH 46/54] Update docs/basics/parsetrees.md Co-authored-by: Veronica Calescu --- docs/basics/parsetrees.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/basics/parsetrees.md b/docs/basics/parsetrees.md index 1b753fcb1..47038eb73 100644 --- a/docs/basics/parsetrees.md +++ b/docs/basics/parsetrees.md @@ -141,7 +141,7 @@ q)function[arg 1;..;arg n] ~ value(function;arg 1;..;arg n) 1b ``` -When `eval` and `value` operate on a parse tree with no nested parse trees they return the same result. However it is not true that `eval` and `value` are equivalent in general. `eval` operates on parse trees, evaluating any nested parse trees, whereas `value` operates on the literals. +When `eval` and `value` operate on a parse tree with no nested parse trees, they return the same result. However it is not true that `eval` and `value` are equivalent in general. `eval` operates on parse trees, evaluating any nested parse trees, whereas `value` operates on the literals. ```q q)value(+;7;3) //parse tree, with no nested trees From 21d159f823004c45445087318afc18c31833316d Mon Sep 17 00:00:00 2001 From: Simon Shanks <59612559+sshanks-kx@users.noreply.github.com> Date: Fri, 4 Oct 2024 13:47:41 +0100 Subject: [PATCH 47/54] Update docs/ref/dotq.md Co-authored-by: Veronica Calescu --- docs/ref/dotq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ref/dotq.md b/docs/ref/dotq.md index 22c7b33df..63679527d 100644 --- a/docs/ref/dotq.md +++ b/docs/ref/dotq.md @@ -379,7 +379,7 @@ xyz| 321f efg| `a ``` -If a command-line value cannot be [converted to the data type](tok.md) of the default value, a [null](../basics/datatypes.md) will be produced +If a command-line value cannot be [converted to the data type](tok.md) of the default value, a [null](../basics/datatypes.md) is produced ```bash $ q -param1 11 -param2 2000.01.01 -param3 wrong From 22a311106e94d578e40c9cb351c4d1daffbc6dd2 Mon Sep 17 00:00:00 2001 From: Simon Shanks <59612559+sshanks-kx@users.noreply.github.com> Date: Fri, 4 Oct 2024 13:48:05 +0100 Subject: [PATCH 48/54] Update docs/ref/dotq.md Co-authored-by: Veronica Calescu --- docs/ref/dotq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ref/dotq.md b/docs/ref/dotq.md index 63679527d..6ee99a648 100644 --- a/docs/ref/dotq.md +++ b/docs/ref/dotq.md @@ -1679,7 +1679,7 @@ _Q for Mortals_ ``` In partitioned DBs, returns a list of partition values – conformant to [`.Q.PD`](#pd-partition-locations) – which represents the partition value for each partition. -(In a date-partitioned DB, unless the date has been modified by [`.Q.view`](#view-subview), this will be simply date.) +(In a date-partitioned DB, unless the date has been modified by [`.Q.view`](#view-subview), this is simply date.) ```q q).Q.PD From 7cb57dfd4fc3d9a7da5ff641add707cc20bb5d79 Mon Sep 17 00:00:00 2001 From: Simon Shanks <59612559+sshanks-kx@users.noreply.github.com> Date: Fri, 4 Oct 2024 13:48:17 +0100 Subject: [PATCH 49/54] Update docs/ref/dotq.md Co-authored-by: Veronica Calescu --- docs/ref/dotq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ref/dotq.md b/docs/ref/dotq.md index 6ee99a648..600e84328 100644 --- a/docs/ref/dotq.md +++ b/docs/ref/dotq.md @@ -252,7 +252,7 @@ q)@[get;"select from tt";-2@]; / no error ## `bvi` (build incremental vp) -Same functionality as [`.Q.bv`](#bv-build-vp), but scans only new partitions loaded in the hdb since the last time `.Q.bv` or `.Q.bvi` was run. Since v4.1 2024.09.13. +It offers the same functionality as [`.Q.bv`](#bv-build-vp), but scans only new partitions loaded in the hdb since the last time `.Q.bv` or `.Q.bvi` was run. Since v4.1 2024.09.13. ## `Cf` (create empty nested char file) From 55cc458fa9d72e0a20d8c2eece75a4ec7cd5da1e Mon Sep 17 00:00:00 2001 From: Simon Shanks <59612559+sshanks-kx@users.noreply.github.com> Date: Fri, 4 Oct 2024 13:48:30 +0100 Subject: [PATCH 50/54] Update docs/wp/compress/index.md Co-authored-by: Veronica Calescu --- docs/wp/compress/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/wp/compress/index.md b/docs/wp/compress/index.md index c07ec8dfc..85e644c16 100644 --- a/docs/wp/compress/index.md +++ b/docs/wp/compress/index.md @@ -16,7 +16,7 @@ overall performance. Early versions of kdb+ achieved compression via file systems with inbuilt compression, such as ZFS. V2.7 introduced built-in OS-agnostic compression, which allowed on-disk data to be converted to compressed format using a range of algorithms and compression levels. This was expanded (V2.8), with the ability to stream in-memory data directly to compressed format on disk. ZFS compression is still useful for some kdb+ applications, as it keeps cached data available for multiple processes. However, this paper will focus on the inbuilt data-compression options provided by kdb+, available on all supported architectures. -Each system will have its own characteristics which determine the appropriate compression configurations to use. +Each system has its own characteristics, which determine the appropriate compression configurations to use. All tests were run using kdb+ version 3.1 (2013.09.05) From 01c6b110961907281c5daab5a61ad69a83716b82 Mon Sep 17 00:00:00 2001 From: Simon Shanks <59612559+sshanks-kx@users.noreply.github.com> Date: Fri, 4 Oct 2024 13:48:53 +0100 Subject: [PATCH 51/54] Update docs/wp/permissions/index.md Co-authored-by: Veronica Calescu --- docs/wp/permissions/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/wp/permissions/index.md b/docs/wp/permissions/index.md index 42dad8cd8..072a6cb73 100644 --- a/docs/wp/permissions/index.md +++ b/docs/wp/permissions/index.md @@ -11,7 +11,7 @@ by [Tom Martin](#author) {: .wp-author} -kdb+ processes often contain sensitive, proprietary information in the form of data or proprietary code and so it is important to restrict who can and cannot access this information. +kdb+ processes often contain sensitive, proprietary information in the form of data or proprietary code. Thus, it is important to restrict the access to this information. kdb+ offers a number of in-built access functions. This paper discusses various methods in which a permissioning and entitlements system can be implemented in kdb+ by extending these in-built functions, allowing access to sensitive information to be controlled and restricted, exposing data to some clients but not to others. From 4bd7d6d68b8db6dfe4260d01f0d70c9c2d5cbc9c Mon Sep 17 00:00:00 2001 From: Simon Shanks <59612559+sshanks-kx@users.noreply.github.com> Date: Fri, 4 Oct 2024 13:49:22 +0100 Subject: [PATCH 52/54] Update docs/basics/parsetrees.md Co-authored-by: Veronica Calescu --- docs/basics/parsetrees.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/basics/parsetrees.md b/docs/basics/parsetrees.md index 47038eb73..1bb1d168e 100644 --- a/docs/basics/parsetrees.md +++ b/docs/basics/parsetrees.md @@ -165,7 +165,7 @@ q)eval(,;enlist `a;enlist `b) ## Variadic operators -Many operators and some keywords in k and q are [variadic](glossary.md#variadic): they are overloaded so that the behavior of the operator changes depending on the number and type of arguments. In q (not k) the unary form of operators such as (`+`, `$`, `.`, `&` etc.) is disabled: keywords are provided instead. +Many operators and some keywords in k and q are [variadic](glossary.md#variadic). That means they are overloaded so that the behavior of the operator changes depending on the number and type of arguments. In q (not k), the unary form of operators such as (`+`, `$`, `.`, `&` etc.) is disabled, and keywords are provided instead. For example, in k the unary form of the `$` operator equates to the `string` keyword in q. From dfbc71f865d44fefbf8e85fa04449b7d8a29e4f0 Mon Sep 17 00:00:00 2001 From: Simon Shanks <59612559+sshanks-kx@users.noreply.github.com> Date: Fri, 4 Oct 2024 13:49:36 +0100 Subject: [PATCH 53/54] Update docs/basics/parsetrees.md Co-authored-by: Veronica Calescu --- docs/basics/parsetrees.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/basics/parsetrees.md b/docs/basics/parsetrees.md index 1bb1d168e..a4b79840b 100644 --- a/docs/basics/parsetrees.md +++ b/docs/basics/parsetrees.md @@ -212,7 +212,7 @@ q)parse"6(+)4" (+:;4) ``` -The items of a `parse` result use k syntax. Since (most of) the q keywords are defined in the `.q` namespace, we can use dictionary reverse lookup to find the meaning. +The items of a `parse` result use k syntax. Since (most of) the q keywords are defined in the `.q` namespace, you can use dictionary reverse lookup to find the meaning. ```q q).q?(+:) From 96a29b738d1c23a8b850403469ddebe44cc542ed Mon Sep 17 00:00:00 2001 From: Simon Shanks <59612559+sshanks-kx@users.noreply.github.com> Date: Fri, 4 Oct 2024 13:49:52 +0100 Subject: [PATCH 54/54] Update docs/basics/parsetrees.md Co-authored-by: Veronica Calescu --- docs/basics/parsetrees.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/basics/parsetrees.md b/docs/basics/parsetrees.md index a4b79840b..ea02d6019 100644 --- a/docs/basics/parsetrees.md +++ b/docs/basics/parsetrees.md @@ -315,6 +315,6 @@ produce a new function which sums the items of a list. ## Functional form of a qSQL query -Sometimes you need to translate a [qSQL query](qsql.md) into its [functional form](funsql.md), for example, so you can pass column names as arguments. +Sometimes you need to translate a [qSQL query](qsql.md) into its [functional form](funsql.md). For example, so you can pass column names as arguments. Details are provided [here](funsql.md#conversion-using-parse).