Using Impala with Apache Ozone Storage

You can use Impala to query data files that reside on Apache Ozone distributed storage, rather than in HDFS. The combination of the Impala query engine and Apache Ozone storage is certified on Impala 4.2 or higher.

For more information on Ozone, see the Apache Ozone site.

The typical use case for Impala and Ozone together is to use Ozone for the default filesystem, replacing HDFS entirely. In this configuration, when you create a database, table, or partition, the data always resides on Ozone storage and you do not need to specify any special LOCATION attribute. If you do specify a LOCATION attribute, its value refers to a path within the Ozone filesystem. For example:

-- If the default filesystem is Ozone, all Impala data resides there
-- and all Impala databases and tables are located there.
CREATE TABLE t1 (x INT, s STRING);

-- You can specify LOCATION for database, table, or partition,
-- using values from the Ozone filesystem.
CREATE DATABASE d1 LOCATION '/some/path/on/ozone/server/d1.db';
CREATE TABLE d1.t2 (a TINYINT, b BOOLEAN);

Impala can write to, delete, and rename data files and database, table, and partition directories on Ozone storage. Therefore, Impala statements such as CREATE TABLE, DROP TABLE, CREATE DATABASE, DROP DATABASE, ALTER TABLE, and INSERT work the same with Ozone storage as with HDFS.

Ozone supports multiple protocols: ofs, o3fs, and s3a. Impala supports reading ofs and o3fs. Impala can also read s3a (see Using Impala with Amazon S3 Object Store). However ofs is their newer protocol, and the only one Impala supports as a default filesystem. We recommend using it for DDL Statements to avoid access limitations, and for DML Statements and SELECT Statement for performance.

Because Apache Ozone storage buckets use a global value for the block size rather than a configurable value for each file, the PARQUET_FILE_SIZE query option has no effect when Impala inserts data into a table or partition residing on Ozone storage.

Impala's spill-to-disk feature may be configured to use Ozone storage by specifying a full URI (e.g. ofs://host:port/volume/bucket/key) for the spill location. See Managing Disk Space for Impala Data for details on configuring remote spill-to-disk.