-
Notifications
You must be signed in to change notification settings - Fork 591
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add first_value
and last_value
metrics with and "ignore nulls" option for use in a window
#9539
Comments
I see also FirstValue which according to the comment is specifically for windowed contexts. Neither is parametrized, so I'm not sure about |
Filed tobymao/sqlglot#3749 as I am having trouble figuring out how to create the SQLGlot expression. Other examples I found in the compiler use |
Update: def visit_FirstNonNullValue(self, op, *, arg):
return sge.IgnoreNulls(this=sge.FirstValue(this=arg)) This works for the |
Most of the team is at SciPy through the end of this week. This makes sense to me, but would love to get thoughts from @cpcloud and the other maintainers. |
We should definitely add this functionality, but likely as a flag on the ops that support, which is more than just first/last, so that we don't end up with a set of classes duplicated only to allow them to set @tswast Any chance someone from the BigFrames team wants to take a crack at it? |
Note that recently we made |
🤔 Perhaps we only expose |
Is your feature request related to a problem?
I would like to implement an efficient, pandas-compatible bfill and ffill in BigQuery DataFrames without custom operators.
What is the motivation behind your request?
To implement bfill and ffill in BigQuery DataFrames (see: #3652 for an issue requesting bfill and ffill in ibis), we add two new operations to Ibis: FirstNonNullValue and LastNonNullValue.
This is then used like
FirstNonNullValue(column).over(window)
, which when the window isibis.range_window(preceding=0, following=limit)
, implements a pandas-like bfill withlimit
parameter support.Likewise, use
LastNonNullValue(column).over(window)
with a window ofibis.range_window(preceding=limit, following=0)
to implement a pandas-like ffill withlimit
parameter support.Describe the solution you'd like
I propose the following two new "metrics", which require a window to operate:
Related: IGNORE NULLS in general is tracked here: #7649
Previously, with string-based compilation, this was accomplished with a custom BigQuery operator like:
https://github.com/googleapis/python-bigquery-dataframes/blob/74170dabd323f1b08ad76241e37ff9f2a5b67ab5/third_party/bigframes_vendored/ibis/backends/bigquery/registry.py#L15-L17
In SQLGlot, it does appear there is a First expression with at least some IGNORE NULLS support. My work is in-progress to see if this operator is compatible with the BigQuery dialect.
What version of ibis are you running?
8.x, but working on an upgrade to 9.x right now.
What backend(s) are you using, if any?
BigQuery
Code of Conduct
The text was updated successfully, but these errors were encountered: