Release data.table version 1.16.0

[MichaelChirico]

We are nearing our planned 6-month cadence for a non-patch release.

Filing this issue to track progress and as a call to flag issues for inclusion in / exclusion from the milestone:

https://github.com/Rdatatable/data.table/milestone/30

Let's see what we can finish by the end of June. Lots of goodies in the release already! Thanks for everyone's work :)

TODO

• include patch release NEWS to master for 1.15.{2,4}
• final pass over NEWS file for readability

[TysonStanley]

The list for 1.16.0 is still pretty large, are we wanting to get all of those included or is it time to move some to 1.17.0?

[tdhock]

the revdeps have been notified a long time ago so those issues should be ok

[tdhock]

probably the one that will block submission until resolved is the new R API compliance #6180

[MichaelChirico]

it's not clear they will block submission -- they are NOTEs after all. I am OK trying to submit after making some effort on all but SET_TRUELENGTH, this time around.

[TysonStanley]

@MichaelChirico sounds good, don't think they'll stop it either and can make it clear our intention over time.

[TysonStanley]

A little off topic but relevant for everyone working on data.table. I received this message today, it's a good reminder.

Today, I was working with data.table in R as I do almost daily. And I wanted to personally thank you and everyone involved in the creation and maintenance of data.table. Your work has been invaluable to me. You have all done an excellent job creating, maintaining, and improving the package. I know from experience that you seldom hear from anyone when things are working and outstanding; instead, only the complaints and notes that something doesn’t work. So, “Thank You!” And, as a side note, data.table (and R) was critical to completing my dissertation five years ago.

[MichaelChirico]

@jangorecki could you PTAL at your open PRs and flag which of those you're keen to have in the imminent release vs. are OK to let slip to 1.17.0?

[jangorecki]

It won't happen anytime soon, till end of the month I am without my workstation.

[MichaelChirico]

Thanks... are you OK to delay those PRs to 1.17.0 then, or should we delay the release until maybe end-August instead?

[MichaelChirico]

OK I think we are ready to start release procedure @tdhock 🎉

[TysonStanley]

Awesome! I'll start it this weekend. Will keep the group in the loop

[TysonStanley]

@tdhock @MichaelChirico looks like we have all of the revdeps covered except for lgrExtra. The error looks related to the new showProgress argument.

[TysonStanley]

      Running ‘testthat.R’
     ERROR
    Running the tests in ‘tests/testthat.R’ failed.
    Last 13 lines of output:
       7.     └─data.table:::stopf("%s must be TRUE or FALSE", "showProgress")
       8.       └─data.table:::raise_condition(...)
      ── Error ('test_SerializerJson.R:48:3'): unpack_json_cols works ────────────────
      Error in ``[.data.table`(dd, , sel, with = FALSE)`: showProgress must be TRUE or FALSE
      Backtrace:
          ▆
       1. └─l$appenders$db$flush() at test_SerializerJson.R:48:3
       2.   ├─dd[, sel, with = FALSE]
       3.   └─data.table:::`[.data.table`(dd, , sel, with = FALSE)
       4.     └─data.table:::stopf("%s must be TRUE or FALSE", "showProgress")
       5.       └─data.table:::raise_condition(...)
      
      [ FAIL 6 | WARN 3 | SKIP 6 | PASS 56 ]
      Error: Test failures
      Execution halted
    ```

[tdhock]

The revdep check system does not report any issue for lgrExtra, https://rcdata.nau.edu/genomic-ml/data.table-revdeps/analyze/2024-08-25/lgrExtra.txt which tested lgrExtra 0.0.8 published 2023-01-06.
maybe there is some issue with the revdep check system that we need to fix.
how did you get that report of testthat ERROR?

[TysonStanley]

Looks like it is triggered by this line: https://github.com/s-fleck/lgrExtra/blob/caa19377cc4e0c15ad4fda96bc439a1f1b6d0a01/R/AppenderDbi.R#L238

But when I use their approach with fictitious data, I can't replicate it.

dd = data.frame(x = 1:5, y = 5:1, z = LETTERS[1:5])
dd = data.table::as.data.table(dd)
sel = c(1, 3)
dd[, sel, with = FALSE]
#       x      z
#   <int> <char>
#1:     1      A
#2:     2      B
#3:     3      C
#4:     4      D
#5:     5      E

What am I missing? They use R6 classes that I don't know very well but don't think it would be triggered with that.

[TysonStanley]

The revdep check system does not report any issue for lgrExtra, https://rcdata.nau.edu/genomic-ml/data.table-revdeps/analyze/2024-08-25/lgrExtra.txt which tested lgrExtra 0.0.8 published 2023-01-06. maybe there is some issue with the revdep check system that we need to fix. how did you get that report of testthat ERROR?

I used revdepcheck package (it's no longer maintained so I've adapted for my use cases).

[TysonStanley]

@tdhock your system caught all of the others, this was the only one that seems to be missed

[TysonStanley]

It is also possible that both CRAN and my check are a false positive based on similar assumptions.

[MichaelChirico]

I don't reproduce the issue on CRAN or their dev version. I think we can ignore & proceed.

The only way that error could show up is if someone was running something like options(datatable.showProgress = 8) that's newly caught & breaks [ (but wasn't already caught by fread() or fwrite()):

options(datatable.showProgress=8)
DT=data.table(a=1)
DT[,1,a]
# Error in `[.data.table`(DT, , 1, a) : showProgress must be TRUE or FALSE

I do see {lgrExtra} setting that option, but it's to an explicit FALSE value:

https://github.com/cran/lgrExtra/blob/e7bc81e49a77062ddedfb70e89930d2379e89a21/tests/testthat/test_AppenderDbi.R#L3-L8

[TysonStanley]

Sounds good. I'd expect it to be on CRAN by tonight/tomorrow then.

[MichaelChirico]

Tag is up and it's on CRAN:

https://github.com/Rdatatable/data.table/releases/tag/1.16.0