TOPN_BYTES_LIMIT Query Option (Impala 3.1 or higher only)
The TOPN_BYTES_LIMIT query option places a limit on the amount of
estimated memory that Impala can process for top-N queries.
Top-N queries are the queries that include both ORDER BY and
LIMIT clauses. Top-N queries don't spill to disk so they have to
keep all rows they process in memory, and those queries can cause out-of-memory issues
when running with a large limit and an offset. If the Impala planner estimates that a
top-N operator will process more bytes than the TOPN_BYTES_LIMIT
value, it will replace the top-N operator with the sort operator. Switching
to the sort operator allows Impala to spill to disk, thus requiring less memory
than top-N, but potentially with performance penalties.
The option has no effect when set to 0 or -1.
Syntax:
SET TOPN_BYTES_LIMIT=limitType: Number
Default: 536870912 (512 MB)
Added in: Impala 3.1