Sets the degree of intra-node parallelism used for certain operations that can benefit from multithreaded execution. You can specify values higher than zero to find the ideal balance of response time, memory usage, and CPU usage during statement processing.
The Impala execution engine is being revamped incrementally to add
additional parallelism within a single host for certain statements and
kinds of operations. The setting MT_DOP=0 uses the
"old" code path with limited intra-node parallelism.
Currently, MT_DOP support varies by statement type:
COMPUTE [INCREMENTAL] STATS. Impala automatically sets
MT_DOP=4 for COMPUTE STATS and
COMPUTE INCREMENTAL STATS statements on Parquet tables.
SELECT statements. MT_DOP is 0 by default
for SELECT statements but can be set to a value greater
than 0 to control intra-node parallelism. This may be useful to tune
query performance and in particular to reduce execution time of
long-running, CPU-intensive queries.
DML statements. MT_DOP values greater
than zero are not currently supported for DML statements. DML statements
will produce an error if MT_DOP is set to a non-zero value.
In Impala 3.4 and earlier, not all SELECT
statements support setting MT_DOP. Specifically, only
scan and aggregation operators, and
local joins that do not need data exchanges (such as for nested types) are
supported. Other SELECT statements produce an error if
MT_DOP is set to a non-zero value.
Type: integer
Default: 0
Because COMPUTE STATS and COMPUTE INCREMENTAL STATS
statements for Parquet tables benefit substantially from extra intra-node
parallelism, Impala automatically sets MT_DOP=4 when computing stats
for Parquet tables.
Range: 0 to 64
Examples:
Any timing figures in the following examples are on a small, lightly loaded development cluster. Your mileage may vary. Speedups depend on many factors, including the number of rows, columns, and partitions within each table.
The following example shows how to run a COMPUTE STATS
statement against a Parquet table with or without an explicit MT_DOP
setting:
-- Explicitly setting MT_DOP to 0 selects the old code path.
set mt_dop = 0;
MT_DOP set to 0
-- The analysis for the billion rows is distributed among hosts,
-- but uses only a single core on each host.
compute stats billion_rows_parquet;
+-----------------------------------------+
| summary |
+-----------------------------------------+
| Updated 1 partition(s) and 2 column(s). |
+-----------------------------------------+
drop stats billion_rows_parquet;
-- Using 4 logical processors per host is faster.
set mt_dop = 4;
MT_DOP set to 4
compute stats billion_rows_parquet;
+-----------------------------------------+
| summary |
+-----------------------------------------+
| Updated 1 partition(s) and 2 column(s). |
+-----------------------------------------+
drop stats billion_rows_parquet;
-- Unsetting the option reverts back to its default.
-- Which for COMPUTE STATS and a Parquet table is 4,
-- so again it uses the fast path.
unset MT_DOP;
Unsetting option MT_DOP
compute stats billion_rows_parquet;
+-----------------------------------------+
| summary |
+-----------------------------------------+
| Updated 1 partition(s) and 2 column(s). |
+-----------------------------------------+
The following example shows the effects of setting MT_DOP
for a query on a Parquet table:
set mt_dop = 0;
MT_DOP set to 0
-- COUNT(DISTINCT) for a unique column is CPU-intensive.
select count(distinct id) from billion_rows_parquet;
+--------------------+
| count(distinct id) |
+--------------------+
| 1000000000 |
+--------------------+
Fetched 1 row(s) in 67.20s
set mt_dop = 16;
MT_DOP set to 16
-- Introducing more intra-node parallelism for the aggregation
-- speeds things up, and potentially reduces memory overhead by
-- reducing the number of scanner threads.
select count(distinct id) from billion_rows_parquet;
+--------------------+
| count(distinct id) |
+--------------------+
| 1000000000 |
+--------------------+
Fetched 1 row(s) in 17.19s
The following example shows how queries that are not compatible with non-zero
MT_DOP settings produce an error when MT_DOP
is set:
set mt_dop=1;
MT_DOP set to 1
insert into a1
select * from a2;
ERROR: NotImplementedException: MT_DOP not supported for DML statements.
Related information: