Impala Web User Interface for Debugging
Each of the Impala daemons (impalad, statestored, and catalogd) includes a built-in web server that displays diagnostic and status information.
- impalad Web UI
-
The impalad Web UI includes information about configuration settings, running and completed queries, and associated performance and resource usage for queries. In particular, the Details link for each query displays alternative views of the query including a graphical representation of the plan, and the output of the
EXPLAIN
,SUMMARY
, andPROFILE
statements from impala-shell. Each host that runs the impalad daemon has its own instance of the Web UI, with details about those queries for which that host served as the coordinator. The impalad Web UI is primarily used for diagnosing query problems that can be traced to a particular node. - statestored Web UI
-
The statestored Web UI includes information about memory usage, configuration settings, and ongoing health checks performed by
statestored
. Because there is only a single instance of thestatestored
within any Impala cluster, you access the Web UI only on the particular host that serves as the Impala StateStore. - catalogd Web UI
-
The catalogd Web UI includes information about the databases, tables, and other objects managed by Impala, in addition to the resource usage and configuration settings of the
catalogd
. Because there is only a single instance of thecatalogd
within any Impala cluster, you access the Web UI only on the particular host that serves as the Impala Catalog Server.
Configuring Debug Web UIs
Turning off the Web UI
To disable Web UI for the impalad
, statestored
, or
catalogd
:
-
Stop the Impala role you want to turn off the Web UI for,
impalad
,statestored
, orcatalogd
. -
Restart the role with the
‑‑enable_webserver=false
startup flag.
Using Kerberos HTTP SPNEGO Authentication for Web UI
To provide security through Kerberos, Impala Web UIs support SPNEGO, a protocol for securing HTTP requests with Kerberos by passing negotiation tokens through HTTP headers.
To enable authorization using SPNEGO, set the
--webserver_require_spnego=true
startup flag and restart the Impala
roles, impalad
, statestored
, or
catalogd
.
Debug Web UI for impalad
To debug and troubleshoot an impalad using a web-based interface,
open the URL
http://impala‑server‑hostname:25000/
in
a browser. (For secure clusters, use the prefix https://
instead of
http://
.)
Because each Impala node produces its own set of debug information, you should choose a specific node that you want to investigate an issue on.
Main Page
The main impalad
Web UI page at / lists the following
information about the impalad
:
-
The version of the impalad daemon
The Version section also contains other information, such as when Impala was built and what build flags were used.
- Process start time
- Hardware information
- OS information
- Process information
- CGroup information
Admission Controller Page
The Admission Controller impalad
debug Web UI is at
/admission page under the main impalad
Web UI.
Use the /admission page to troubleshoot queued queries and the admission control.
The admission page provides the following information about each resource pool to which queries have been submitted at least once:
-
Time since the
statestored
received the last update -
A warning if this
impalad
is considered disconnected from thestatestored
and thus the information on this page could be stale. - Pool configuration
- Queued queries submitted to this coordinator, in the order of submission
- Running queries submitted to this coordinator
-
Pool stats
- Average of time in queue: An exponential moving average which represents the average time in queue over the last 10 to 12 queries. If a query is admitted immediately, the wait time of 0 is used in calculating this average wait time.
-
Histogram of the distribution of peak memory used by queries admitted to the pool
Use the histogram to figure out settings for the minimum and maximum query
MEM_LIMIT
ranges for this pool.The histogram displays data for all queries admitted to the pool, including the queries that finished, got canceled, or hit an error.
Click on the pool name to only display information relevant to that pool. You can then refresh the debug page to see only the information for that specific pool.
Click Reset informational stats for all pools to reset the stats that keep track of historical data, such as Totals stats, Time in queue (exponential moving average), and the histogram.
http://impala-server-hostname:port/admission?json
See Admission Control and Query Queuing for the description of the properties in admission control.
Known Backends Page
The Known backends page of the impalad
debug Web UI is at
/backends under the main impalad
Web UI.
This page lists the following info for each of the impalad nodes in the cluster. Because each impalad daemon knows about every other impalad daemon through the StateStore, this information should be the same regardless of which node you select.
- Address of the node: Host name and port
-
KRPC address: The KRPC address of the
impalad
. Use this address when you issue theSHUTDOWN
command for thisimpalad
. - Whether acting as a coordinator
- Whether acting as an executor
-
Quiescing status: Specify whether the graceful shutdown process has been initiated
on this
impalad
. - Memory limit for admission: The amount of memory that can be admitted to this backend by the admission controller.
-
Memory reserved: The amount of memory reserved by queries that are active, either
currently executing or finished but not yet closed, on this backend.
The memory reserved for a query that is currently executing is its memory limit, if set. Otherwise, if the query has no limit or if the query finished executing, the current consumption is used.
- Memory of the queries admitted to this coordinator: The memory submitted to this particular host by the queries admitted by this coordinator.
Catalog Page
The Catalog page of the impalad
debug Web UI is at
/catalog under the main impalad
Web UI.
This page displays a list of databases and associated tables recognized by this
instance of impalad. You can use this page to locate which database
a table is in, check the exact spelling of a database or table name, look for
identical table names in multiple databases. The primary debugging use case would be
to check if an impalad
instance has knowledge of a particular table
that someone expects to be in a particular database.
Hadoop Configuration
The Hadoop Configuration page of the impalad
debug Web UI is
at /hadoop-varz under the main impalad
Web UI.
This page displays the Hadoop common configurations that Impala is running with.
JMX
The JMX page of the impalad
debug Web UI is at /jmx
under the main impalad
Web UI.
This page displays monitoring information about various JVM subsystems, such as memory pools, thread management, runtime. etc.
Java Log Level
The Change log level page of the impalad
debug Web UI is at
/log_level under the main impalad
Web UI.
This page displays the current Java and backend log levels, and it allows you to
change the log levels dynamically without having to restart the
impalad
.
Logs Page
The INFO logs page of the impalad
debug Web UI is at
/logs under the main impalad
Web UI.
This page shows the last portion of the impalad.INFO
log file,
including the info, warning, and error logs for the impalad. You
can see the details of the most recent operations, whether the operations succeeded or
encountered errors. This page provides one central place for the log files and saves
you from looking around the filesystem for the log files, which could be in different
locations on clusters that use cluster management software.
Memz Page
The Memory Usage page of the impalad
debug Web UI is at
/memz under the main impalad
Web UI.
This page displays the summary and detailed information about memory usage by the
impalad
.
Metrics Page
The Metrics page of the impalad
debug Web UI is at
/metrics under the main impalad
Web UI.
This page displays the current set of metrics, counters and flags representing various
aspects of impalad
internal operations.
Queries Page
The Queries page of the impalad
debug Web UI is at
/queries under the main impalad
Web UI.
- Currently running queries
- Queries that have completed their execution, but have not been closed yet
- Completed queries whose details still reside in memory
-‑-‑query_log_size
and -‑-‑query_log_size_in_bytes
startup options for
impalad
.
-
How many SQL statements are failing (
State
value ofEXCEPTION
) -
How large the result sets are (
# rows fetched
) -
How long each statement took (
Start Time
andEnd Time
)
Click the Details link for a query to display the detailed performance characteristics of that query, such as the profile output.
On the query detail page, in the Profile tab, you have options to export the query profile output to the Thrift, text, or Json format.
The Queries page also includes the Query Locations section that lists the number of running queries with fragments on this host.
RPC Services Page
The RPC durations page of the impalad
debug Web UI is at
/rpcz under the main impalad
Web UI.
This page displays information, such as the duration, about the RPC communications of
this impalad
with other Impala daemons.
Sessions Page
The Sessions page of the impalad
debug Web UI is at
/session under the main impalad
Web UI.
This page displays information about the sessions currently connected to this
impalad instance. For example, sessions could include connections
from the impala-shell
command, JDBC or ODBC applications, or the
Impala Query UI in the Hue web interface.
Threadz Page
The Threads page of the impalad
debug Web UI is at
/threadz under the main impalad
Web UI.
This page displays information about the threads used by this instance of
impalad
, and it shows which categories they are grouped into. Making
use of this information requires substantial knowledge about Impala internals.
Varz Page
The Varz page of the impalad
debug Web UI is at /varz
under the main impalad
Web UI.
This page shows the configuration settings in effect when this instance of
impalad
communicates with other Hadoop components such as HDFS and
YARN. These settings are collected from a set of configuration files.
The bottom of this page also lists all the command-line settings in effect for this
instance of impalad
. See
Modifying Impala Startup Options for information
about modifying these values.
Prometheus Metrics Page
At /metrics_prometheus under the main impalad
Web UI, the metrics are generated in Prometheus exposition format that
Prometheus can consume for event monitoring and alerting. The
/metrics_prometheus is not shown in the Web UI list of
pages.
Debug Web UI for statestored
To debug and troubleshoot the statestored
daemon using a web-based
interface, open the URL
http://impala‑server‑hostname:25010/
in
a browser. (For secure clusters, use the prefix https://
instead of
http://
.)
Main Page
The main statestored
Web UI page at / lists the following
information about the statestored
:
-
The version of the
statestored
daemon - Process start time
- Hardware information
- OS information
- Process information
- CGroup information
Logs Page
The INFO logs page of the debug Web UI is at /logs under the main
statestored
Web UI.
This page shows the last portion of the statestored.INFO log
file, including the info, warning, and error logs for the
statestored
. You can refer here to see the details of the most recent
operations, whether the operations succeeded or encountered errors. This page provides
one central place for the log files and saves you from looking around the filesystem
for the log files, which could be in different locations on clusters that use cluster
management software.
Memz Page
The Memory Usage page of the debug Web UI is at /memz under the main
statestored
Web UI.
This page displays summary and detailed information about memory usage by the
statestored
. You can see the memory limit in effect for the node, and
how much of that memory Impala is currently using.
Metrics Page
The Metrics page of the debug Web UI is at /metrics under the main
statestored
Web UI.
This page displays the current set of metrics: counters and flags representing various
aspects of statestored
internal operation.
RPC Services Page
The RPC durations page of the statestored
debug Web UI is at
/rpcz under the main statestored
Web UI.
This page displays information, such as the durations, about the RPC communications of
this statestored
with other Impala daemons.
Subscribers Page
The Subscribers page of the debug Web UI is at /subscribers under the
main statestored
Web UI.
This page displays information about the other Impala daemons that have registered
with the statestored
to receive and send updates.
Threadz Page
The Threads page of the debug Web UI is at /threadz under the main
statestored
Web UI.
This page displays information about the threads used by this instance of
statestored
, and shows which categories they are grouped into. Making
use of this information requires substantial knowledge about Impala internals.
Topics Page
The Topics page of the debug Web UI is at /topics under the main
statestored
Web UI.
This page displays information about the topics to which the other Impala daemons have registered to receive updates.
Varz Page
The Varz page of the debug Web UI is at /varz under the main
statestored
Web UI.
This page shows the configuration settings in effect when this instance of
statestored
communicates with other Hadoop components such as HDFS
and YARN. These settings are collected from a set of configuration files.
The bottom of this page also lists all the command-line settings in effect for this
instance of statestored
. See
Modifying Impala Startup Options for information
about modifying these values.
Prometheus Metrics Page
At /metrics_prometheus under the main
statestored
Web UI, the metrics are generated in
Prometheus exposition format that Prometheus can consume for event
monitoring and alerting. The /metrics_prometheus is not shown
in the Web UI list of pages.
Debug Web UI for catalogd
The main page of the debug Web UI is at
http://impala‑server‑hostname:25020/
(non-secure cluster) or
https://impala‑server‑hostname:25020/
(secure cluster).
Main Page
The main catalogd
Web UI page at / lists the
following information about the catalogd
:
- The version of the
catalogd
daemon - Process start time
- Hardware information
- OS information
- Process information
- CGroup information
Catalog Page
The Catalog page of the debug Web UI is at /catalog
under the main catalogd
Web UI.
This page displays a list of databases and associated tables
recognized by this instance of catalogd
. You can use
this page to locate which database a table is in, check the exact
spelling of a database or table name, look for identical table names
in multiple databases. The catalog information is represented as the
underlying Thrift data structures.
JMX
The JMX page of the catalogd
debug Web UI is
at /jmx under the main catalogd
Web UI.
This page displays monitoring information about various JVM subsystems, such as memory pools, thread management, runtime. etc.
Java Log Level
The Change log level page of the catalogd
debug Web UI is at /log_level under the main
catalogd
Web UI.
The page displays the current Java and backend log levels and allows
you to change the log levels dynamically without having to restart the
catalogd
Logs Page
The INFO logs page of the debug Web UI is at /logs
under the main catalogd
Web UI.
This page shows the last portion of the
catalogd.INFO
log file, including the info,
warning, and error logs for the catalogd
daemon. You
can refer here to see the details of the most recent operations,
whether the operations succeeded or encountered errors. This page
provides one central place for the log files and saves you from
looking around the filesystem for the log files, which could be in
different locations on clusters that use cluster management software.
Memz Page
The Memory Usage page of the debug Web UI is at /memz
under the main catalogd
Web UI.
This page displays summary and detailed information about memory
usage by the catalogd
. You can see the memory limit
in effect for the node, and how much of that memory Impala is
currently using.
Metrics Page
The Metrics page of the debug Web UI is at /metrics
under the main catalogd
Web UI.
This page displays the current set of metrics: counters and flags
representing various aspects of catalogd
internal
operation.
RPC Services Page
The RPC durations page of the catalogd
debug
Web UI is at /rpcz under the main catalogd
Web
UI.
This page displays information, such as the durations, about the RPC
communications of this catalogd
with other Impala
daemons.
Threadz Page
The Threads page of the debug Web UI is at /threadz
under the main catalogd
Web UI.
This page displays information about the threads used by this
instance of catalogd
, and shows which categories they
are grouped into. Making use of this information requires substantial
knowledge about Impala internals.
Varz Page
The Varz page of the debug Web UI is at /varz under
the main catalogd
Web UI.
This page shows the configuration settings in effect when this
instance of catalogd
communicates with other Hadoop
components such as HDFS and YARN. These settings are collected from a
set of configuration files.
The bottom of this page also lists all the command-line settings in
effect for this instance of catalogd
. See Modifying Impala Startup Options for information
about modifying these values.
Prometheus Metrics Page
At /metrics_prometheus under the main
catalogd
Web UI, the metrics are generated in
Prometheus exposition format that Prometheus can consume for event
monitoring and alerting. The /metrics_prometheus is not shown
in the Web UI list of pages.