Known Issues

The latest set of known issues can be found here (https://github.com/cloudera-labs/hms-mirror/issues?q=is%3Aissue+is%3Aopen+label%3Abug)

Enhancement Requests

The latest set of enhancement requests can be found here (https://github.com/cloudera-labs/hms-mirror/issues?q=is%3Aissue+is%3Aopen+label%3Aenhancement).

If there is something you'd like to see, add a new issue here (https://github.com/cloudera-labs/hms-mirror/issues)

3.0.0.1

This release is based on the 2.3.1.5 release and includes all the features and bug fixes from that release.

This is a Security and CVE release that has upgrading all dependencies to the latest possible versions to eliminate as many of the community CVEs as possible. This also required us to upgrade the minimum JDK version to 17.

What's New

• JDK 17 Minimum Version Requirement. Addresses dependencies with CVE issues.

Bug Fixes --sync not dropping table on right when left is missing. (https://github.com/cloudera-labs/hms-mirror/issues/189)

2.3.1.5

Bug Fixes

• Fixed Web UI session status preventing progress.

• Handle npe from SQL in-place downgrade of ACID tables.

• Fixed locale issue with set statements that used numeric values.

• Fixed floorDiv(long,int) to floorDiv(long,long) for Java8 compatibility.

What's New

• Add support to 'in-place' removal of bucket definitions from an ACID table

Note This will be the last release in the 2.3.x branch with any feature enhancements. Future releases will be in the 3.x branch.

2.3.0.13

What's New

Enhance logging to show which instance is handling a connection/job in case of using multiple HS2 instances (https://github.com/cloudera-labs/hms-mirror/issues/183)

When using multiple HS2 instances, it can be difficult to determine which instance is handling a connection or job. This enhancement will allow you to add a set statement that identifies which instance is handling the connection or job. Use the -po[r|l] property override values if you'd like to include this information in the log output. -po hive.server2.thrift.bind.host will add the statement to both LEFT and RIGHT output. The option is also available in the Web UI through "Property Overrides".

2.3.0.12

Bug Fixes

The Hikari Connection Pool settings are causing intermittent connection failures, cause table transfer failures. (https://github.com/cloudera-labs/hms-mirror/issues/182)

2.3.0.10

Bug Fixes

Second run in WebUI Fails (https://github.com/cloudera-labs/hms-mirror/issues/181)

dbRegEx not being processed. Throws MISC_ERROR because it can't find any databases. (https://github.com/cloudera-labs/hms-mirror/issues/180)

Legacy DBPROPERTIES are causing ERROR when attempting to set on CDP (https://github.com/cloudera-labs/hms-mirror/issues/179)

Issue with jobs not completing when some schema's were already present.

Address lingering connections after run completes.

Fixed counters for CLI screen output.

Fixed an issue with tables being processed multiple times under some conditions.

Partition discovery for SHADOW table when source is a Managed table shouldn't try to build partitions with ALTER (https://github.com/cloudera-labs/hms-mirror/issues/178)

LEFT side SQL when running 'execute' mode for SQL data strategy isn't being run. (https://github.com/cloudera-labs/hms-mirror/issues/177)

CLI App version fails when attempting to set 'concurrency' option (https://github.com/cloudera-labs/hms-mirror/issues/176)

MSCK for Shadow table not generated when 'metastore_direct' on the LEFT isn't defined (https://github.com/cloudera-labs/hms-mirror/issues/175)

2.3.0.4

What's New

BETA Iceberg conversion support for the STORAGE_MIGRATION data strategy. See Iceberg Conversion (ICEBERG_MIGRATION) for more details. To activate this beta feature for the WebUI, add --hms-mirror. config.beta=true to the startup command. EG: hms-mirror --service --hms-mirror.config.beta=true

Bugs (Fixed)

Add to connection init the ability to set the queue AND trigger an engine resource (https://github.com/cloudera-labs/hms-mirror/issues/173)

Validate SQL elements before making changes to the Cluster (https://github.com/cloudera-labs/hms-mirror/issues/168)

Extend the HS2 connection validation with Tez task validation (https://github.com/cloudera-labs/hms-mirror/issues/167)

The reset-to-default-location doesn't seem to be working in v2 (https://github.com/cloudera-labs/hms-mirror/issues/165)

2.2.0.19.6 (pre-release)

What's New

BEHAVIOR CHANGE - Drop any shadow table definitions created during migration automatically (https://github.com/cloudera-labs/hms-mirror/issues/163)

This adjustment will remove shadow tables that were created during the migration process. This will help to keep the source and target clusters clean of any artifacts created during the migration process. An option saveWorkingTables has been added to the configuration to allow you to keep these tables if you need them for any audits or other reasons. The default is false, which means that the tables will be dropped automatically. An audit 'cleanup' file with the drop commands will be created in the report directory, regardless of the setting.

2.2.0.19.2 (pre-release)

What's New

Error event is not logged for table skipped because RIGHT already has the table (https://github.com/cloudera-labs/hms-mirror/issues/162) Would be great to be able to omit warnings from the end of logs (https://github.com/cloudera-labs/hms-mirror/issues/161) Be able to reduce spring framework related log entries (https://github.com/cloudera-labs/hms-mirror/issues/160)

2.2.0.19.1

Bug Fixes

• Doubling of mapping locations for partitions in distcp for STORAGE_MIGRATION.

• STORAGE_MIGRATION is setting ACID to ON, regardless. (https://github.com/cloudera-labs/hms-mirror/issues/158)

What's New

• Validate JDBC Jar Files in config. (https://github.com/cloudera-labs/hms-mirror/issues/159)

• Ability to turn-on strict mode for Storage Migration. This will cause distcp to fail when non-standard locations are used. To turn off, use the -sms|--storage-migration-strict flag via the CLI.

Behavior Changes

The default behavior for Storage Migration 'strict' has changed from true to false. The intent behind the strict mode was to ensure distcp would fail when non-standard locations are used. The combination of metastore_direct and knowing the partition location details gives us a better chance on making these mappings work for distcp. When the scenario arises, we do HIGHLY recommend that you validate the plans created. The new default behavior will allow distcp to continue when non-standard locations are encountered, while throwing a warning. This will allow the migration to continue, but you should validate the results.

You will need to reset the `strict` mode flag in you configuration yaml. It will be set to `true` if the configuration was created before this release. You will need to set it to `false` to maintain this new 'preferred' behavior.

transfer:
  storageMigration:
    strict: false

2.2.0.18.1

What's New

Beta Flag to be used for future beta features. To activate:

hms-mirror --beta

hms-mirror --service --hms-mirror.config.beta=true

Bugs (Fixed)

• Property overrides - po|pol|por - are not honored in v2, were working in 1.6.x (https://github.com/cloudera-labs/hms-mirror/issues/156)

• Partition auto-discovery config value is misconfigured during setup (https://github.com/cloudera-labs/hms-mirror/issues/155)

• Enhancement for hms-mirror setup (https://github.com/cloudera-labs/hms-mirror/issues/154)

• Remove automatic DB location as db_name.db in db location (https://github.com/cloudera-labs/hms-mirror/issues/153)

• Add support for Hive table backup in DISTCP mode (https://github.com/cloudera-labs/hms-mirror/issues/152)

2.2.0.17.1

What's New

• Extend / Split 'transfer-ownership' to apply to database and/or table (https://github.com/cloudera-labs/hms-mirror/issues/151)

• Improved Logging output

• Support alternate db name for STORAGE_MIGRATION (https://github.com/cloudera-labs/hms-mirror/issues/149)

• CLI Option to control data-movement-strategy

Bugs (Fixed)

• DB Prefix and DB Rename not working (https://github.com/cloudera-labs/hms-mirror/issues/150)

• When Metastore isn't configured for ALIGNED/DISTCP, it's not writing MSCK (https://github.com/cloudera-labs/hms-mirror/issues/131)

• Fixed missing CR/LF is clean up scripts.

2.2.0.15.1

What's New

Feature that allows you to 'skip' modifying the database location during a storage migration. This is useful if you're trying to archive tables in a database to another storage system, but want to leave the database location as is for new tables in the database. For STORAGE_MIGRATION, add option that would skip any Database Location Adjustments (https://github.com/cloudera-labs/hms-mirror/issues/147)

2.2.0.15

What's New

• Add support for Oracle Metastore Direct Connection

• SQL Strategy only uses MSCK for partition discovery for Shadow Table (https://github.com/cloudera-labs/hms-mirror/issues/145)

Bugs (Fixed)

• Output and Report Directory Consistency between CLI and Web UI. See docs for more details.

• Postgres Metastore Direct Connection Fixes

• SQL Data Strategy Validation Blockers for Acid tables-

2.2.0.12

What's New

• For STORAGE_MIGRATION to Ozone, ensure valid Volume/Bucket names (https://github.com/cloudera-labs/hms-mirror/issues/142)

Bugs (Fixed)

• Expose Stats in Web UI Reports (https://github.com/cloudera-labs/hms-mirror/issues/141)

• Fixed the ability to Encrypt/Decrypt password in the UI.

• Execute run for Strategies without RIGHT cluster definition fail to write reports (https://github.com/cloudera-labs/hms-mirror/issues/144)

• STORAGE_MIGRATION distcp efficiency option (https://github.com/cloudera-labs/hms-mirror/issues/143)

2.2.0.10

What's New

• [Hive 4 DB OWNER DDL syntax for ALTERing DB ONWER requires 'USER'](https://github. com/cloudera-labs/hms-mirror/issues/139)

This changed resulted in a simplification of how we determine what the cluster platform is. Previously we used two attributes (legacyHive and hdpHive3) to determine the platform. This information would direct logic around translations and other features.

Unfortunately, this isn't enough for us to determine all the scenarios we're encountering. These attributes have been replaced with a new attribute call platformType.

We will make automatic translations of legacy configurations to the new platformType attribute. The translation will be pretty basic and result in either the platform type being defined as HDP2 or CDP_7.1. If you have a more complex configuration, you'll need to adjust the platformType attribute manually. Future persisted configurations will use the new platformType attribute and drop the legacyHive and hdpHive3 attributes.

• Add "Property Overrides" to Web Interface (https://github.com/cloudera-labs/hms-mirror/issues/111)

A feature that was late in making it into the Web UI is now here.

• For Web UI Service, default to prefer IPV4 (https://github.com/cloudera-labs/hms-mirror/issues/134)

To ensure the right IP stack is used when the Web UI starts up, we're forcing this JDK configuration with the Web UI.

• Forcibly set Java Home via -Duser.home (https://github.com/cloudera-labs/hms-mirror/issues/136)

We had a few requests and issues with implementations were the target environment isn't always setup with normal user 'home' standards that we can rely on. This change allows us to set the 'home' directory for the user running the application and ensure its translated correctly in hms-mirror for storing and reading configurations, reports, and logs.

If you are in an environment that doesn't follow user $HOME standards, you can set the HOME environment variable to a custom directory BEFORE starting hms-mirror to alter the default behavior.

2.2.0.9

Bugs (Fixed)

• Allow process path that doesn't require Metastore Direct Connection (https://github.com/cloudera-labs/hms-mirror/issues/128)

• Fix Kerberos Connection Issues to HS2 (https://github.com/cloudera-labs/hms-mirror/issues/129)

• Job Progress getting stuck at the end while writing reports (https://github.com/cloudera-labs/hms-mirror/issues/130)

Enhancements

Increase build dependencies to CDP 7.1.9 SP1. Rework Pass Key Management. Additional details in Connection Validation.

2.2.0.8

Bugs (Fixed)

• Not parsing abfs protocol locations correctly (https://github.com/cloudera-labs/hms-mirror/issues/124)

• Database input duplication (https://github.com/cloudera-labs/hms-mirror/issues/125)

2.2.0.7

Bugs (Fixed)

• Non Configs blocking WEB UI Create (https://github.com/cloudera-labs/hms-mirror/issues/122)

• SCHEMA_ONLY with ALIGNED, DISTCP and Partitions is too granular on partition distcp (https://github.com/cloudera-labs/hms-mirror/issues/123)

• Expose rid in web ui (https://github.com/cloudera-labs/hms-mirror/issues/121)

2.2.0.5

Bugs (Fixed)

• CLI Setup Issues (https://github.com/cloudera-labs/hms-mirror/issues/120)

• Stale Config after several runs (https://github.com/cloudera-labs/hms-mirror/issues/118)

• SQL Strategy Transfer Fixes (https://github.com/cloudera-labs/hms-mirror/issues/117)

• Partition Reductions for Distcp (https://github.com/cloudera-labs/hms-mirror/issues/116)

2.2.0.4

Bugs (Fixed)

• Report Output Directory Issue (https://github.com/cloudera-labs/hms-mirror/issues/115)

• Filters Optimizations and Fixes (https://github.com/cloudera-labs/hms-mirror/issues/116)

2.2.0.2

This is a big release for hms-mirror. We've added a Web interface to hms-mirror that makes it easier to configure and run varies scenarios.

Along with the Web interface, we've made some significant adjustments to the hms-mirror engine which is much more complete than the previous release. The engine now supports a wider range of strategies and has a more robust configuration system.

We do our best to guide you through configurations that make sense, help you build plans and manage complex scenarios.

globalLocationMap:
  /tpcds_base_dir: "/alt/ext/location"
  /tpcds_base_dir2/web: "/alt/ext/location"

userGlobalLocationMap:
  /tpcds_base_dir:
    EXTERNAL_TABLE: "/alt/ext/location"
  /tpcds_base_dir2/web:
    EXTERNAL_TABLE: "/alt/ext/location"

Video Series

Impact on Source and Target Systems

hms-mirror uses HiveServer2 connections to 'extract' and 'replay' schema's between systems. It also utilizes Metastore Direct connections to pull partition level details that can't be efficiently collected through the HiveServer2 SQL interface.

The concurrency (default of 10), will act like 10 users making a whole lot of DDL requests. The impact to existing workloads is a possibility. If you have a large amount of data(metadata) to migrate/extract and you're concerned about the impact to the user base, you should 'isolate' the HiveServer2 AND Hive Metastore for this process. That could mean setting up an additional HiveServer2/Metastore pair that is used specifically by 'hms-mirror'.

Databases with a Large Number of Tables

As we mentioned, migrations are a database at a time. If the table in the database exceed some of these limits, consider using various table filters to process filtered lists of tables in the database at a time.

Reports

Reports are created at the database level as well. When the database has a large number of tables, these reports can be extremely long.

Accessing the Web Interface in Secure Environments

Ports for the Web Interface may not be available in secure enviroments. By default, the Web Interface is available on port '8090'.

hms-mirror --service --server.port=<new_port>

ssh -L 8090:<remote_host>:8090 <user>@<remote_host>

ssh -D <choose_a_port> <user>@<remote_host>

Starting

The hms-mirror web application is started by running:

hms-mirror --service

This will start the web application on the default port of 8090. The port can be changed by using the --server.port option during startup.

hms-mirror --service --server.port=8080

Point your browser to http://server-host:8090/hms-mirror to access the web application.

The web application will use the user home directory to store configuration files, logs, and reports. This isn't as much a concern as it is for the cli version since the web application will manage all this from the user's browser.

The web application works on the premise of a 'session'. Although the application is stateless, the session is used to store the user's configuration and state while they are using the application. This allows the user to navigate the application and make changes without losing their work. Currently, the session is global throughout the application and is not tied to a specific user browser session. An 'hms-mirror' session is NOT synonymous with a browser session.

An instance of the web application can only run ONE session at a time.

Stopping

hms-mirror --stop

Security

Coming Soon...

Where to Start?

There are three methods to building/loading a configuration. From the main page, select 'Initialize'.

This will bring up the 'Initialize' page where one of the following options can be selected.

Configurations are specific to a Data Strategy (Strategies). Once created, you can NOT change the Data Strategy but, you can clone the configuration and change the Data Strategy (last option).

Managing the Session Configuration

There are 2 states for a session. The in-memory state and the persisted state.

When you choose to 'Edit' a session configuration, you'll need to 'Save' the configuration for those changes to be applied to the session. 'Saving' the configuration will ONLY update the in-memory state of the session.

You will need to 'Persist' the session to save it for future use.

Application Report

The output report is in markdown format. You can use a markdown renderer to view the report. If you don't have a renderer, you can still read the report, it will just be harder to read.

The report include various stats regarding the run and details for each tables migration process. In this report, you'll find details on "why" a particular table was skipped, or what actions were taken to migrate the table. The report will even list issue encountered during the process.

SQL Scripts

hms-mirror will produce the SQL scripts used to migrate the data. These scripts are written to the output directory. The scripts are prefixed with the database name.

• <db_name>_LEFT_Clueanup_execute.sql - When present, this scripts represents SQL statements that should be run on the LEFT cluster to cleanup artifacts from the migration process.

• <db_name>_LEFT_execute.sql - When present, this scripts represents SQL statements that should be run on the LEFT cluster to migrate the data. If the -e option was used, the contents of this script will be executed on the LEFT cluster by hms-mirror. If the -e option was NOT specified, these script can be verified and executed manually on the LEFT cluster.

• <db_name>_RIGHT_execute.sql - When present, this scripts represents SQL statements that should be run on the RIGHT cluster to migrate the data. If the -e option was used, the contents of this script will be executed on the RIGHT cluster by hms-mirror. If the -e option was NOT specified, these script can be verified and executed manually on the RIGHT cluster.

• <db_name>_RIGHT_Clueanup_execute.sql - When present, this scripts represents SQL statements that should be run on the RIGHT cluster to cleanup artifacts from the migration process.

YAML Output

The <db_name>_hms-mirror.yaml file is a full listing of the migration process as a document. Use this file to programmatically determine what actions were taken during the migration process.

Runbook

The <db_name>_runbook.md is a markdown file that is a workbook of 'what' to do. It lays out the steps taken and the steps to be taken to complete the migration process.

distcp Scripts and Workbook

When you include the -dc|--distcp option when running hms-mirror, we'll build a template distcp job for each database that has data to be migrated. The result is a set of bash scripts and source files listing the contents to be used in the migration.

Depending other influencing options, there may be a distcp script for the LEFT and RIGHT clusters. The scripts will be prefixed with the database name.

The various distcp reports include:

• <db_name>_RIGHT_n_distcp_source.txt - A list of the source directories to be copied to the RIGHT cluster. The n will increment for each one of the jobs created for the database being migrated. These files must be copied to the RIGHT clusters HDFS filesystem. When running the distcp bash shell script, set the bash environment variable $HCFS_BASE_DIR to set the 'directory' these are copied to.

• <db_name>_RIGHT_distcp_script.sh - The bash script created that will run the distcp jobs. This script will be run on the RIGHT cluster. Review the comments in the script for details on how to run it.

• <db_name>_RIGHT_distcp_workbook.md - A markdown report table that breakdown what will be moved by the process.

** Example distcp Workbook **

Logs

Logs, as of 1.6.5.6 are now in the same output directory as the reports.

Output Directory

You can control where the reports are generated by using the -o <directory> option. Default behavior is to use the $HOME/.hms-mirror/reports directory. If you specify a different directory, the reports will be generated in the specific directory. If you use the -o <dir> option with the Web Interface, the reports will be in that 'base directory' with a subdirectory of the timestamp of the report.

For the CLI, the report will be placed in the directory. If the directory exists, the report will be placed in an increment of that directory to avoid overwriting an existing report. EG: /my/report -> /my/report_1

Options

There are several views for the reports that include a detailed view of the process, workbooks for 'distcp', scripts for 'distcp' , job 'yaml' files, the configuration used to run the job, etc.

You have options to view all these reports within the Web Interface. You can further 'download' a zip file of the report contents and view them locally.

To keep things organized, there is an 'archive' option that will move the reports to an 'archive' directory. This is useful to help keep the reports directory clean and organized. If you need to view those reports again, you can always move them back to the reports directory. Although, that process is manual.

If you run the process via the CLI and are using defaults, you'll be able to view the reports through the Web Interface since they are both using the same location.

User Home Directory

A lot of what happens in hms-mirror depends on a 'user' HOME environment variable to store configurations, logs, and reports.

If your environment doesn't follow a typical home environment setup, you will NEED to set the $HOME environment variable before running the application to ensure that the application can find and store the necessary files.

Hive/TEZ Properties Whitelist Requirements

HiveServer2 has restrictions on what properties can be set by the user in a session. To ensure that hms-mirror will be able to set the properties it needs, add hive.security.authorization.sqlstd.confwhitelist.append (https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.security.authorization.sqlstd.confwhitelist.append) property in the HiveServer2 Advanced Configuration Snippet (Safety Valve) for hive-site.xml with at least the following value(s) so hms-mirror can set the properties it needs:

tez\.grouping\..*

Backups

DO NOT SKIP THIS!!!

The hms-mirror process DOES 'DROP' tables when asked. If those tables manage data like a legacy managed, ACID, or external.table.purge=true scenario, we do our best NOT to DROP those and ERROR out. But, protect yourself and make backups of the areas you'll be working in.

Shared Authentication

The clusters must share a common authentication model to support cross-cluster HDFS access when HDFS is the underlying storage layer for the datasets. This means that a kerberos ticket used in the RIGHT cluster must be valid for the LEFT cluster.

For cloud storage, the two clusters must have rights to the target storage bucket.

If you can distcp between the clusters (Linking Cluster Storage Layers), you have the basic connectivity required to start working with hms-mirror.

Building METADATA

Rebuilding METADATA can be an expensive scenario. Especially when you are trying to reconstruct the entire metastore in a short time period, consider this in your planning. Know the number of partitions and buckets you will be moving and account for this. Test on smaller datasets (volume and metadata elements). Progress to testing higher volumes/partition counts to find limits and make adjustments to your strategy.

Using the SQL and EXPORT_IMPORT strategies will move metadata AND data, but rebuilding the metastore elements can be pretty expensive. So consider migrating the metadata separately from the data (distcp) and use MSCK on the RIGHT cluster to discover the data. This will be considerably more efficient.

If you will be doing a lot of metadata work on the RIGHT cluster. That cluster also serves a current user base; consider setting up separate HS2 pods for the migration to minimize the impact on the current user community. Isolate Migration Activities ("Isolate Migration Activities" in "Optimizations")

Partition Handling for Data Transfers

There are three settings in the configuration to control how and to what extent we'll attempt to migrate DATA for tables with partitions.

For non-ACID/transactional tables the setting in:

hybrid:
  exportImportPartitionLimit: 100
  sqlPartitionLimit: 500

Control both the HYBRID strategy for selecting either EXPORT_IMPORT or SQL and the SQL LIMIT for how many partitions we'll attempt. When the SQL limit is exceeded, you will need to use SCHEMA_ONLY to migrate the schema followed by distcp to move the data.

For ACID/transactional tables, the setting in:

migrateACID:
  partitionLimit: 500

Effectively draws the same limit as above.

Why do we have these limits? Mass migration of datasets via SQL and EXPORT_IMPORT with many partitions is costly and NOT very efficient. It's best that when these limits are reached that you separate the METADATA and DATA migration to DDL and distcp.

Permissions

We use a cross-cluster technique to back metadata in the RIGHT cluster with datasets in the LEFT cluster for data strategies: LINKED, HYBRID, EXPORT_IMPORT, SQL, and SCHEMA_ONLY (with -ams AVRO Migrate Schema).

See Linking Clusters Storage Layers (Linking Cluster Storage Layers) for details on configuring this state.

Goal

What does it take to support HDFS visibility between these two clusters?

Can that integration be used to support the Higher Clusters' use of the Lower Clusters HDFS Layer for distcp AND Hive External Table support?

Scenario #1

rmr /hadoop-ha

hdfs zkfc -formatZK

hadoop.rpc.protection=true
dfs.encrypt.data.transfer=true
dfs.encrypt.data.transfer.algorithm=3des
dfs.encrypt.data.transfer.cipher.key.bitlength=256

ipc.client.fallback-to-simple-auth-allowed=true

# For this Clusters Name Service
dfs.internal.nameservices=HOME90

# For the target (lower) environment HA NN Services
dfs.ha.namenodes.HDP50=nn1,nn2
dfs.namenode.rpc-address.HDP50.nn1=k01.streever.local:8020
dfs.namenode.rpc-address.HDP50.nn2=k02.streever.local:8020
dfs.namenode.http-address.HDP50.nn1=k01.streever.local:50070
dfs.namenode.http-address.HDP50.nn2=k02.streever.local:50070
dfs.namenode.https
 address.HDP50.nn1=k01.streever.local:50471
dfs.namenode.https-address.HDP50.nn2=k02.streever.local:50470
dfs.client.failover.proxy.provider.HDP50=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

# For Available Name Services
dfs.nameservices=HOME90,HDP50

hadoop distcp hdfs://HDP50/user/dstreev/sstats/queues/2020-10.txt /user/dstreev/temp

hadoop distcp /warehouse/tablespace/external/hive/cvs_hathi_workload.db/queue/2020-10.txt hdfs://HDP50/user/dstreev/temp

hadoop.proxyuser.hive.hosts=*

LEFT and RIGHT Clusters

hms-mirror defines clusters as LEFT and RIGHT. The LEFT cluster is the source of the metadata and the RIGHT cluster is the target. The LEFT cluster is usually the older cluster version. Regardless, under specific scenario's, hms-mirror will use an HDFS client to check directories and move small amounts of data (AVRO schema files). hms-mirror will depend on the configuration of the node it's running on to locate the 'hcfs filesystem'. This means that the /etc/hadoop/conf directory should contain all the environments settings to successfully connect to hcfs (Hadoop Compatible File System.

The configuration is done via a 'yaml' file, details below.

There are two ways to get started:

• The first time you run hms-mirror and it can't find a configuration, it will walk you through building one and save it to $HOME/.hms-mirror/cfg/default.yaml. Here's what you'll need to complete the setup:

  • URI's for each clusters HiveServer2

  • STANDALONE jar files for EACH Hive version.

    • We support the Apache Hive based drivers for Hive 1 and Hive2/3.

    • Recently added support for the Cloudera JDBC driver for CDP.

  • Username and Password for non-kerberized connections.

    • Note: hms-mirror will only support one kerberos connection. For the other, use another AUTH method.

  • The hcfs (Hadoop Compatible FileSystem) protocol and prefix used for the hive table locations in EACH cluster.

• Use the template yaml (Default Configuration Template) for reference and create a default.yaml in the running users $HOME/.hms-mirror/cfg directory.

You'll need JDBC driver jar files that are * specific to the clusters you'll integrate. If the LEFT cluster isn't the same version as the RIGHT cluster, don't use the same JDBC jar file, especially when integrating Hive 1 and Hive 3 services. The Hive 3 driver is NOT backwardly compatible with Hive 1.

See the running (Running) section for examples on running hms-mirror for various environment types and connections.

Binary Package

Download the latest version of hms-mirror-<version>-dist.tar.gz (https://github.com/cloudera-labs/hms-mirror/releases)

HMS-Mirror Setup from Binary Distribution

On the edgenode:

• Remove previous install directory rm -rf hms-mirror-install-<version>

  • If you don't remove the previous install directory, the default tar behaviour will NOT overwrite the existing directory, hence you won't get the new version.

• Expand the tarball tar zxvf hms-mirror-<version>-dist.tar.gz.

  This produces a child hms-mirror-install-<version> directory.

• Make sure you STOP any previous release of hms-mirror before attempting install.

  • hms-mirror --stop

• Two options for installation:

  • As the root user (or sudo), run hms-mirror-install-<version>/setup.sh. This will install the hms-mirror packages in /usr/local/hms-mirror and create symlinks for the executables in /usr/local/bin. At this point, hms-mirror should be available to all user and in the default path.

  • As the local user, run hms-mirror-install-<version>/setup.sh. This will install the hms-mirror packages in $HOME/.hms-mirror and create symlink in $HOME/bin. Ensure $HOME/bin is in the users path and run hms-mirror.

DO NOT RUN hms-mirror from the installation directory.

If you install both options, your environment PATH will determine which one is run. Make note of this because an upgrade may not be reachable.

Quick Start

After installation (above), run hms-mirror for the particular interface that interests you.

If either or both clusters are Kerberized, please review the detailed configuration guidance here ("Running Against a LEGACY (Non-CDP) Kerberized HiveServer2" in "Running") and here ("Kerberized Connections" in "Running").

General Guidance

• Run hms-mirror from the RIGHT cluster on an Edge Node.

hms-mirror is built (default setting) with CDP libraries and will connect natively using those libraries. The edge node should have the hdfs client installed and configured for that cluster. The application will use this connection for some migration strategies.

• If running from the LEFT cluster, note that the -ro/--read-only feature examines HDFS on the RIGHT cluster. The HDFS client on the LEFT cluster may not support this access.

• Connecting to HS2 through KNOX (in both clusters, if possible) reduces the complexities of the connection by removing Kerberos from the picture.

• The libraries will only support a Kerberos connection to a 'single' version of Hadoop at a time. This is relevant for 'Kerberized' connections to Hive Server 2. The default libraries will support a kerberized connection to a CDP clusters HS2 and HDFS. If the LEFT (source) cluster is Kerberized, including HS2, you will need to make some adjustments.

  • The LEFT clusters HS2 needs to support any auth mechanism BUT Kerberos.

  • Use an Ambari Group to setup an independent HS2 instance for this exercise or use KNOX.

Kerberized HS2 Connections

We currently have validated kerberos HS2 connections to CDP clusters using the Hive JDBC driver you'll find in your target CDP distribution.

Connections to Kerberized HS2 endpoints on NON-CDP clusters is NOT currently supported. You will need to use KNOX in HDP to connect to a kerberized HS2 endpoint. For CDH, you can setup a non-kerberized HS2 endpoint to support the migration.

This process has CHANGED compared to v1.x of `hms-mirror`. Please adjust your configurations accordingly.

We NO LONGER need to have the hive JDBC driver in the aux_libs directory ($HOME/.hms-mirror/aux_libs). The driver should be stored locally on the machine running hms-mirror and referenced in the configuration file via the `jarFile' attribute. Follow the same procedure as above for Kerberized connections as is done for non-kerberized connections.

At this point, just like the previous version of hms-mirror, you'll need to have a valid kerberos ticket on the machine running hms-mirror. This is required to authenticate to the kerberized HS2 endpoint.

REMOVE all 'hive' related JDBC jar files from aux_libs. Leaving them there WILL cause conflicts during the service startup.

Validating HS2 Connectivity

Once you have everything configured, you can validate all connections required by hms-mirror through the 'CONNECTIONS --> Validate' left menu option in the UI. This will test the connectivity to the various endpoints required by hms-mirror.

HDP 3 Connections

The JDBC driver for HDP Hive 3 has some embedded classes for log4j that conflict with the log4j classes in the hms-mirror application. To resolve this, you can use the Cloudera Apache JDBC driver for HDP 3 Hive. This driver is compatible with HDP 3 and does not have the log4j conflict.

Hive JDBC Driver Connection Pool Settings

Hive 1/2, which includes HDP 2.x and CDH 5.x / 6.x environments will use the Apache Commons DBCP2 Connection Pool libraries. Hive 3/4, which includes HDP 3.x and CDH 7.x environments will use the HikariCP Connection Pool libraries.

There are a few settings that you can adjust in the hms-mirror configuration file to tune the connection pool settings.

For the DBCP2 connection pool, you can set the maxWaitMillis setting, which has a default of 5000 milliseconds.

For the HikariCP connection pool, the following connection properties can be adjusted: connectionTimeout (default: 60000 milliseconds) validationTimeout (default: 30000 milliseconds) initializationFailTimeout (default: 10000 milliseconds)

The default setting have been pretty successful in our testing, but you can adjust these settings to meet your needs.

CLI

The DBCP2 connection pool settings can be adjustment through the CLI via the -pt|--pass-through option using one of more of the following setting:

• dbcp2.maxWaitMillis

EG: -pt dbcp2.maxWaitMillis=10000

The Hikari connection pool settings can be adjusted through the CLI via the -pt|--pass-through option using one of more of the following setting:

• hikari.connectionTimeout

• hikari.validationTimeout

• hikari.initializationFailTimeout

You can have multiple `-pt` options on the command line

EG: -pt hikari.connectionTimeout=60000 -pt hikari.validationTimeout=30000 -pt hikari.initializationFailTimeout=10000

Web UI

Config File

clusters:
  LEFT:
    environment: "LEFT|RIGHT"
    hiveServer2:
      uri: "..."
      connectionProperties:
        maxWaitMillis: "5000" # DBCP2 connection pool setting
        hikari.validationTimeout: "30000"
        hikari.initializationFailTimeout: "10000"
        hikari.connectionTimeout: "60000"

CDP Hive via Knox Gateway

Doesn't require Kerberos. Knox is SSL, so depending on whether you've self-signed your certs you may need to make adjustments.

• Apache Hive and CDP Packaged Apache Hive JDBC Driver

jdbc:hive2://s03.streever.local:8443/;ssl=true;transportMode=http;httpPath=gateway/cdp-proxy-api/hive;sslTrustStore=/Users/dstreev/bin/certs/gateway-client-trust.jks;trustStorePassword=changeit

• Cloudera JDBC Driver

jdbc:hive2://s03.streever.local:8443;transportMode=http;AuthMech=3;httpPath=gateway/cdp-proxy-api/hive;SSL=1;AllowSelfSignedCerts=1

CDP Hive direct with Kerberos

When connecting to via Kerberos, configure the jar files the same way as non-kerberized connections.

• Apache Hive and CDP Packaged Apache Hive JDBC Driver

jdbc:hive2://s04.streever.local:10001/;ssl=true;transportMode=http;httpPath=cliservice;sslTrustStore=/home/dstreev/bin/certs/gateway-client-trust.jks;trustStorePassword=changeit;principal=hive/_HOST@STREEVER.LOCAL

NOTE: This configuration includes a certificate reference for SSL. If you're using self-signed certs, you'll need to adjust the sslTrustStore and trustStorePassword values.

• Cloudera JDBC Driver

jdbc:hive2://s04.streever.local:10001;transportMode=http;AuthMech=1;KrbRealm=STREEVER.LOCAL;KrbHostFQDN=s04.streever.local;KrbServiceName=hive;KrbAuthType=2;httpPath=cliservice;SSL=1;AllowSelfSignedCerts=1

HDP2 HS2 with No Auth

Since CDP is usually kerberized AND hms-mirror doesn't support the simultanous connections to 2 different kerberos environments, I've setup an HS2 on HDP2 specifically for this effort. NOTE: You need to specify a username when connecting to let Hive know what the user is. No password required.

• Apache Hive Standalone Driver shipped with HDP2.

jdbc:hive2://k02.streever.local:10000

Direct Metastore DB Access

The LEFT and RIGHT configurations also suppport 'direct' metastore access to collect detailed partition information. The support this feature, get the JDBC driver that is appropriate for your metastore(s) backend dbs and place it in $HOME/.hms-mirror/aux_libs directory.

Amazon S3

This obviously includes Amazon S3, but also includes other S3 compatible storage systems like Minio, etc. that are compatible with the S3 API.

hadoop-aws-<platform-version>.jar
aws-java-sdk-bundle-<platform-version>.jar
ranger-raz-hook-s3-<platform-version>.jar

Microsoft Azure

abfs

hadoop-azure-<platform-version>9.jar
ranger-raz-hook-abfs-<platform-version>.jar

Google Cloud Storage (GFS)

google-cloud-storage-<platform-version>.jar

Secure Passwords in Configuration

There are multiple passwords stored in the configuration files. By default, the passwords are in clear text in the configuration file. This usually isn't an issue since the file can be protected at the UNIX level from peering eyes. But if you need to protect those passwords, hms-mirror supports storing an encrypted version of the password in the configuration.

When you're using this feature, you need to have a password-key. This is a key used to encrypt and decrypt the password in the configuration. The same password-key must be used for ALL passwords in the configuration file.

WEB UI

Passwords are saved in the configuration and can easily be encrypted and decrypted using the Web UI. If the password (s) are encrypted, the 'Passwords Encrypted' checkbox will be checked.

If the passwords are encrypted, you'll need to specify the 'Encrypt Key' before running or connection to any endpoints. Set the 'Encrypt Key' and click 'Save' to save the key for the session.

The 'Encrypt Key' is only used for the current session and will NOT be saved if you 'persist' the session.

Once encrypted, you'll need to specify the 'password key' with that session to decrypt them for use.

CLI

=== Errors ===
	38:Password en/de crypt

=== Warnings ===
	56:Encrypted password: HD1eNF8NMFahA2smLM9c4g==

=== Errors ===
	38:Password en/de crypt

=== Warnings ===
	57:Decrypted password: have-a-nice-day

Assumptions

1. This process will only 'migrate' EXTERNAL and MANAGED (non-ACID/Transactional) table METADATA (not data, except with SQL (SQL) and EXPORT_IMPORT (EXPORT_IMPORT)).

2. MANAGED tables replicated to the **RIGHT ** cluster will be converted to "EXTERNAL" tables for the 'metadata' stage. They will be tagged as 'legacy managed' in the **RIGHT ** cluster. They will be assigned the external.table.purge=true flag, to continue the behaviors of the legacy managed tables.

3. The RIGHT cluster has 'line of sight' to the LEFT cluster.

4. The RIGHT cluster has been configured to access the **LEFT ** cluster storage. See link clusters (Linking Cluster Storage Layers). This is the same configuration required to support distcp from the RIGHT cluster to the LEFT cluster.

5. The movement of metadata/data is from the LEFT cluster to the RIGHT cluster.

6. With Kerberos, each cluster must share the same trust mechanism.

• The RIGHT cluster must be Kerberized IF the LEFT cluster is.

• The LEFT cluster does NOT need to be kerberized if the RIGHT cluster is kerberized.

7. The * LEFT cluster does NOT have access to the RIGHT cluster.

8. The credentials use by 'hive' (doas=false) in the **RIGHT ** cluster must have access to the required storage (hdfs) locations on the lower cluster.

• If the **RIGHT ** cluster is running impersonation (doas=true), that user must have access to the required storage (hdfs) locations on the lower cluster.

******* ERRORS *********
20:STORAGE_MIGRATION requires you to specify PATH location for 'managed' and 'external' tables (-wd, -ewd) to migrate storage.  These will be appended to the -smn (storage-migration-namespace) parameter and used to set the 'database' LOCATION and MANAGEDLOCATION properties
61:You're using the same namespace in STORAGE_MIGRATION, without `-rdl` you'll need to ensure you have `-glm` set to map locations.

Running Against a LEGACY (Non-CDP) Kerberized HiveServer2

hms-mirror is pre-built with CDP libraries and WILL NOT be compatible with LEGACY kerberos environments. A Kerberos connection can only be made to ONE cluster when the clusters are NOT running the same 'major' version of Hadoop.

To attach to a LEGACY HS2, run hms-mirror with the --hadoop-classpath command-line option. This will strip the CDP libraries from hms-mirror and use the hosts Hadoop libraries by calling hadoop classpath to locate the binaries needed to do this.

On-Prem to Cloud Migrations

On-Prem to Cloud Migrations should run hms-mirror from the LEFT cluster since visibility in this scenario is usually restricted to LEFT->RIGHT.

If the cluster is an older version of Hadoop (HDP 2, CDH 5), your connection to the LEFT HS2 should NOT be kerberized. Use LDAP or NO_AUTH.

The clusters LEFT hcfsNamespace (clusters:LEFT:hcfsNamespace) should be the LEFT clusters HDFS service endpoint. The RIGHT hcfsNamespace (clusters:RIGHT:hcfsNamespace) should be the target root cloud storage location. The LEFT clusters configuration (/etc/hadoop/conf) should have all the necessary credentials to access this location. Ensure that the cloud storage connectors are available in the LEFT environment.

There are different strategies available for migrations between on-prem and cloud environments.

Connections

hms-mirror connects to 3 endpoints. The hive jdbc endpoints for each cluster (2) and the hdfs environment configured on the running host. This means you'll need:

• JDBC drivers to match the JDBC endpoints

• For non CDP 7.x environments and Kerberos connections, an edge node with the current Hadoop libraries.

See the config (Configuration) section to setup the config file for hms-mirror.

hiveServer2:
  uri: "<jdbc-url>"
  connectionProperties:
    user: "*****"
    password: "*****"
  jarFile: "<environment-specific-jdbc-standalone-driver>"

Troubleshooting

If each JDBC endpoint is Kerberized and the connection to the LEFT or RIGHT is successful, both NOT both, and the program seems to hang with no exception... it's most likely that the Kerberos ticket isn't TRUSTED across the two environments. You will only be able to support a Kerberos connection to the cluster where the ticket is trusted. The other cluster connection will need to be anything BUT Kerberos.

Add --show-cp to the hms-mirror command line to see the classpath used to run.

The argument --hadoop-classpath allows us to replace the embedded Hadoop Libs (v3.1) with the libs of the current platform via a call to hadoop classpath. This is necessary to connect to kerberized Hadoop v2/Hive v1 environments.

Check the location and references to the JDBC jar files. General rules for Kerberos Connections:

• The JDBC jar file should be in the $HOME/.hms-mirror/aux_libs. For Kerberos connections, we've seen issues attempting to load this jar in a sandbox, so this makes it available to the global classpath/loader.

• Get a Kerberos ticket for the running user before launching hms-mirror.

Controlling the YARN Queue that runs the SQL queries from hms-mirror

Use the jdbc url defined in default.yaml to set a queue.

jdbc:hive2://host:10000/.....;...?tez.queue.name=batch

The commandline properties -po, -pol, and -por can be used to override the queue name as well. For example: -pol tez.queue.name=batch will set the queue for the "LEFT" cluster while -por tez.queue.name=migration will set the queue for the "RIGHT" cluster.

Make Backups before running hms-mirror

Take snapshots of areas you'll touch:

• The HMS database on the LEFT and RIGHT clusters

• A snapshot of the HDFS directories on BOTH the LEFT and RIGHT clusters will be used/touched.

  NOTE: If you are testing and "DROPPING" dbs, Snapshots of those data directories could protect you from accidental deletions if you don't manage purge options correctly. Don't skip this... A snapshot of the db directory on HDFS will prevent DROP DATABASE x CASCADE from removing the DB directory (observed in CDP 7.1.4+ as tested, check your version) and all sub-directories even though tables were NOT configured with purge options.

Isolate Migration Activities

The migration of schemas can put a heavy load on HS2 and the HMS server it's using. That impact can manifest itself as 'pauses' for other clients trying to run queries. Extended schema/discovery operations have a 'blocking' tendency in HS2.

To prevent average user operational impact, I suggest establishing an isolated HMS and HS2 environment for the migration process.

Speed up CREATE/ALTER Table Statements - with existing data

Set ranger.plugin.hive.urlauth.filesystem.schemes=file in the Hive Server 2(hive_on_tez) Ranger Plugin Safety Value, via Cloudera Manager.

Add this to the HS2 instance on the RIGHT cluster when Ranger is used for Auth. This skips the check done against every directory at the table location (for CREATE or ALTER LOCATION). It is allowing the process of CREATE/ALTER to run much faster.

The default (true) behavior works well for the interactive use case. Still, bulk operations like this can take a long time if this validation needs to happen for every new partition during creation or discovery.

I recommend turning this back after the migration is complete. This setting exposes permissions issues at the time of CREATE/ALTER. So by skipping this, future access issues may arise if the permissions aren't aligned, which isn't a Ranger/Hive issue, it's a permissions issue.

Turn ON HMS partition discovery

In CDP 7.1.4 and below, the housekeeping threads in HMS used to discover partitions are NOT running. Add metastore.housekeeping.threads.on=true to the HMS Safety Value to activate the partition discovery thread. Once this has been set, the following parameters can be used to modify the default behavior.

hive.metastore.partition.management.task.frequency
hive.exec.input.listing.max.threads
hive.load.dynamic.partitions.thread
hive.metastore.fshandler.threads

METASTORE_HOUSEKEEPING_LEADER_HOSTNAME("metastore.housekeeping.leader.hostname",
            "hive.metastore.housekeeping.leader.hostname", "",
"If multiple Thrift metastore services are running, the hostname of Thrift metastore " +
        "service to run housekeeping tasks at. By default, this value is empty, which " +
        "means that the current metastore will run the housekeeping tasks. If configuration" +
        "metastore.thrift.bind.host is set on the intended leader metastore, this value should " +
        "match that configuration. Otherwise it should be same as the hostname returned by " +
        "InetAddress#getLocalHost#getHostName(). Given the uncertainty in the later " +
        "it is desirable to configure metastore.thrift.bind.host on the intended leader HMS."),
    METASTORE_HOUSEKEEPING_THREADS_ON("metastore.housekeeping.threads.on",
        "hive.metastore.housekeeping.threads.on", false,
        "Whether to run the tasks under metastore.task.threads.remote on this metastore instance or not.\n" +
            "Set this to true on one instance of the Thrift metastore service as part of turning\n" +
            "on Hive transactions. For a complete list of parameters required for turning on\n" +
            "transactions, see hive.txn.manager."),

Order of Evaluation

Order of Evaluation means that we will evaluate the attribute in the describe order and once a valid mapping is found, we will stop the evaluation. Evaluation order depends on the translation type as well.

Translation Types

Translation types are used to determine how the location should be transformed. The following are the translation types are ALIGNED and RELATIVE.

Legend

Translation Scenarios

Add 'database' names to the runtime configuration.

This is the simplest way to select databases. Additional filtered can be applied to tables through the table RegEx filters.

Web UI

Ensure you've selected 'Edit' from the Left Navigation Menu. Enter a comma separated list of databases and press 'Add'.

CLI

The `-db|--database` option allows you to list the databases you want to process.

Use the Database RegEx filter to include matching databases found in the source cluster.

Create Warehouse Plans for the databases you want to include.

Standard Locations

When you choose to ALIGN the datasets, you are choosing to collect everything in the dataset/database under the same location as you've defined in the warehouse plan. When you choose DISTCP as the movement plan for ALIGNED, we'll build a distcp plan that will make those translations. BUT, there are some restrictions to this.

When you choose to use non-standard locations for 'partition specs', we can't build a proper distcp plan. In this case we will throw an 'error' for the offending table and describe to imbalance. You can either fix/adjust the dataset OR choose to use the SQL data movement strategy.

Warehouse Plans Explained

Hive 1/2 Table Types History

Tables in Hive 1/2 are typically 'external' tables. This means that the data is managed independently of the Hive Metastore. The table definition in the Hive Metastore points to the location of the data but does NOT manage the data. In this case, if you drop the table (or a partition), the data remains on the filesystem. These are created with the CREATE EXTERNAL TABLE command.

Another table type in Hive 1/2 is the 'managed' table. This is where Hive manages the data. If you drop the table (or partition), Hive will clean up the data on the filesystem. These are created with the CREATE TABLE command.

There is also a variant of the 'managed' table that is ACID compliant. These are created with the CREATE TABLE command with table properties that enable ACID characteristics transactional=true. These tables have special characteristics and are not friendly to the 'schema on read' paradigm, since these tables are 'schema on write' and embed special columns in the data files.

Starting with Hive 3, the default behavior of the CREATE TABLE command is to create a 'managed' ACID table. The additional table properties for a 'transactional' table are no longer required to create an ACID table. This is a significant change from Hive 1/2.

There is a web application that allows you to experiment with various commands and settings, showing the resulting table structures.

Regardless, use the following web application to help you quickly through the trial and error phase of understanding these new structural and session settings.

Hive Create Path (https://dstreev.github.io/hive/create_path.html)

Iceberg Table Migration via Hive

See Iceberg Migration (ICEBERG_MIGRATION) for details.

File System Stats

SQL based operations, hms-mirror will attempt to gather file system stats for the tables being migrated. This is done by running hdfs dfs -count on the table location. This is done to help determine the best strategy for moving data and allows us to set certain hive session values and distribution strategies in SQL to optimize the data movement.

But some FileSystems may not be very efficient at gathering stats. For example, S3. In these cases, you can disable the stats gathering by adding -ssc|--skip-stats-collection to your command line.

When you have a LOT of tables, collecting stats can have a significant impact on the time it takes to run hms-mirror and the general pressure on the FileSystem to gather this information. In this case, you have to option to disable stats collection through -scc.

CREATE [EXTERNAL] TABLE IF NOT EXISTS Option

Default behavior for hms-mirror is to NOT include the IF NOT EXISTS clause in the CREATE TABLE statements. This is because we want to ensure that the table is created and that the schema is correct. If the table already exists, we want to fail.

But there are some scenarios where the table is present and we don't want the process to fail on the CREATE statement to ensure the remaining SQL statements are executed. In this case, you can modify add the commandline option -cine or add to the configuration:

clusters:
  RIGHT|LEFT:
    createIfNotExists: "true"

Using this option has the potential to create a table with a different schema than the source. This is not recommended. This option is applied when using the SCHEMA_ONLY data strategy.

Auto Gathering Stats (disabled by default)

CDP Default settings have enabled hive.stats.autogather and hive.stats.column.autogather. This impacts the speed of INSERT statements (used by hms-mirror to migrate data) and for large/numerous tables, the impact can be significant.

The default configuration for hms-mirror is to disable these settings. You can re-enable this in hms-mirror for each side (LEFT|RIGHT) separately. Add the following to your configuration.

clusters:
  LEFT|RIGHT:
    enableAutoTableStats: true
    enableAutoColumnStats: true

To disable, remove the enableAutoTableStats and enableAutoColumnStats entries or set them to false.

Non-Standard Partition Locations

Partitions created by 'hive' with default locations follow a file system naming convention that allows other partitions of 'hive' to discovery/manage those location and partition associations.

The standard is for partitions to exist as sub-directories of the table location. For example: Table Location is hdfs://my-cluster/warehouse/tablespace/external/hive/my_test.db/my_table and the partition location is hdfs://my-cluster/warehouse/tablespace/external/hive/my_test.db/my_table/dt=2020-01-01, assuming the partition column name is dt.

When this convention is not followed, additional steps are required to build the partition metadata. You can't use MSCK REPAIR because it will not find the partitions. You can use ALTER TABLE ADD PARTITION but you'll need to provide the location of the partition. hms-mirror will do this for you when using the data strategies -d DUMP|SCHEMA_ONLY and the commandline flag -epl|--evaluate-partition-location.

In order to make this evaluation efficient, we do NOT use standard HiveSQL to discover the partition details. It is possible to use HiveSQL for this, it's just not meant for operations at scale or tables with a lot of partitions.

Hence, we tap directly into the hive metastore database. In order to use this feature, you will need to add the following configuration definition to your hms-mirror configuration file (default.yaml).

clusters:
  LEFT|RIGHT:
    ...
    metastore_direct:
      uri: "<db_url>"
      type: MYSQL|POSTGRES|ORACLE
      connectionProperties:
        user: "<db_user>"
        password: "<db_password>"
      connectionPool:
        min: 3
        max: 5

You will also need to place a copy of the RDBMS JDBC driver in $HOME/.hms-mirror/aux_libs. The driver must match the type defined in the configuration file.

Note: Non-Standard Partition Location will affect other strategies like SQL where the LEFT clusters storage is accessible to the RIGHT and is used by the RIGHT to source data. The 'mirror' table used for the transfer will NOT discover the partitions and will NOT transfer data. See: Issue #63 (https://github.com/cloudera-labs/hms-mirror/issues/63) for updates on addressing this scenario. If this is affecting you, I highly recommend you comment on the issue to help us set priorities.

Optimizations

The following configuration settings control the various optimizations taken by hms-mirror. These settings are mutually exclusive.

• -at|--auto-tune

• -so|--skip-optimizations

• -sdpi|--sort-dynamic-partition-inserts

HDP3 MANAGEDLOCATION Database Property

HDP3 doesn't support MAANGEDLOCATION (https://github.com/cloudera-labs/hms-mirror/issues/52) so we've added a property to the cluster configuration to allow the system to SKIP setting the MANAGEDLOCATION database property in HDP 3 / Hive 3 environments.

clusters:
  LEFT:
    platformType: 'HDP3'

Compress Text Output

-cto will control the session level setting for `hive.exec.compress.output'.

VIEWS

hms-mirror now supports the migration of VIEWs between two environments. Use the -v|--views-only option to execute this path. VIEW creation requires dependent tables to exist.

Run hms-mirror to create all the target tables before running it with the -v option.

This flag is an OR for processing VIEW's OR TABLE's. They are NOT processed together.

Requirements

• The dependent tables must exist in the RIGHT cluster

• When using -dbp|--db-prefix option, VIEW definitions are NOT modified and will most likely cause VIEW creation to fail.

ACID Tables

hms-mirror supports the migration of ACID tables using the -d HYBRID|SQL|EXPORT_IMPORT data strategy in combination with the -ma|--migrate-acid or -mao|--migrate-acid-only flag. You can also simply 'replay' the schema definition (without data) using -d SCHEMA_ONLY -ma|-mao. The -ma|-mao flag takes an optional integer value that sets an 'Artificial Bucket Threshold'. When no parameter is specified, the default is 2.

Use this value to set a bucket limit where we'll remove the bucket definition during the translation. This is helpful for legacy ACID tables which required a bucket definition but weren't a part of the intended design. The migration provides an opportunity to correct this artificial design element.

With the default value 2, we will remove CLUSTERING from any ACID table definitions with 2 or fewer buckets defined. If you wish to keep ALL CLUSTERED definitions, regardless of size, set this value to 0.

There is now an option to 'downgrade' ACID tables to EXTERNAL/PURGE during migration using the -da option.

Intermediate/Common Storage Options

When bridging the gap between two clusters, you may find they can't share/link storage. In this case, using one of these options will help you with the transfer.

The -is or --intermediate-storage option is consider a transient location that both cluster can share, see, and have access to. The strategies for transferring data (EXPORT_IMPORT, SQL, HYBRID) will use this location to facilitate the transfer. This is a common strategy when migrating from on-prem environments to the cloud.

The -cs or --common-storage option is similar to -is but this option ends up being the final resting place for the data, not just the transfer location. And with this option, we can streamline the jumps required to migrate data. Again, this location needs to be accessible to both clusters.

Non-Native Hive Tables (Hbase, KAFKA, JDBC, Druid, etc..)

Any table definition without a LOCATION element is typically a reference to an external system like: HBase, Kafka, Druid, and/or (but not limited to) JDBC.

Requirements

These references require the environment to be:

• Correctly configured to use these resources

• Include the required libraries in the default hive environment.

• The referenced resource must exist already BEFORE the 'hive' DDL will successfully run.

AVRO Tables

AVRO tables can be designed with a 'reference' to a schema file in TBLPROPERTIES with avro.schema.url. The referenced file needs to be 'copied' to the RIGHT cluster BEFORE the CREATE statement for the AVRO table will succeed.

Add the -asm|--avro-schema-move option at the command line to copy the file from the LEFT cluster to the RIGHT cluster.

As long as the clusters are linked (Linking Cluster Storage Layers) and the cluster hcfsNamespace values are accurate, the user's credentials running hms-mirror will attempt to copy the schema file to the RIGHT cluster BEFORE executing the CREATE statement.

Requirements

• Link Clusters (Linking Cluster Storage Layers) for Data Strategies: SCHEMA_ONLY, SQL, EXPORT_IMPORT, and HYBRID

• Running user must have 'namespace' access to the directories identified in the TBLPROPERTIES key avro.schema.url.

• The user running hms-mirror will need enough storage level permissions to copy the file.

• When hive is running with doas=false, hive will need access to this file.

Table Translations

distcp Planning Workbook and Scripts

hms-mirror will create source files and a shell script that can be used as the basis for the 'distcp' job(s) used to support the databases and tables requested in -db. hms-mirror will NOT run these jobs. It will provide the basic job constructs that match what it did for the schemas. Use these constructs to build your execution plan and run these separately.

The constructs created are intended as a one-time transfer. If you are using SNAPSHOTS or --update flags in distcp to support incremental updates, you will have to make additional modifications to the scripts/process. Note: For these scenarios, hms-mirror supports options like -ro|--read-only and -sync.

Each time hms-mirror is run, source files for each database are created. These source files need to be copied to the distributed filesystem and reference with an -f option in distcp. We also create a basic shell script that can be used as a template to run the actual distcp jobs.

Depending on the job size and operational expectations, you may want to use SNAPSHOTS to ensure an immutable source or use a diff strategy for more complex migrations. Regardless, you'll need to make modifications to these scripts to suit your purposes.

If your process requires the data to exist BEFORE you migrate the schemas, run hms-mirror in the dry-run mode (default) and use the distcp planning workbook and scripts to transfer the datasets. Then run hms-mirror with the -e|--execute option to migrate the schemas.

These workbooks will NOT include elements for ACID/Transactional tables. Simply copying the dataset for transactional tables will NOT work. Use the HYBRID data strategy migration transactional table schemas and datasets.

ACID Table Downgrades

The default table creation scenario for Hive 3 (CDP and HDP 3) installations is to create ACID transactional tables. If you're moving from a legacy platform like HDP 2 or CDH, this may have caught you off guard and resulted in a lot of ACID tables you did NOT intend on.

The -da|--downgrade-acid option can be used to convert these ACID tables to EXTERNAL/PURGE tables.

If you have ACID tables on the current platform and would like to downgrade them, but you're not doing a migration, try the -ip|--in-place option. This will archive to existing ACID table and build a new table (with the original table name) that is EXTERNAL/PURGE.

Reset to Default Locations

Migrations present an opportunity to clean up a lot of history. While hms-mirror was originally designed to migration data and maintain the relative locations of the data, some may want to reorganize the data during the migration.

The option -rdl|--reset-default-location will overwrite the locations originally used and place the datasets in the 'default' locations as defined by -wd|--warehouse-directory and -ewd|--external-warehouse-directory.

The -rdl option requires -wd and -ewd to be specified. These locations will be used to ALTER the databases LOCATION and MANAGEDLOCATION values. After which, all new CREATE \[EXTERNAL\] TABLE definitions don't specify a LOCATION, which means table locations will use the default.

Legacy Row Serde Translations

By default, tables using old row serde classes will be converted to the newer serde as the definition is processed by hms-mirror. See here (https://github.com/cloudera-labs/hms-mirror/blob/6f6c309f24fbb8133e5bd52e5b18274094ff5be8/src/main/java/com/cloudera/utils/hadoop/hms/mirror/feature/LegacyTranslations.java#L28) for a list of serdes we look for.

If you do NOT want to apply this translation, add the option -slt|--skip-legacy-translation to the commandline.

Filtering Tables to Process

There are options to filter tables included in hms-mirror process. You can select -tf|--table-filter to "include" only tables that match this 'regular-expression'. Inversely, use -etf|--exclude-table-filter to omit tables from the list. These options are mutually exclusive.

The filters for -tf and -tef are expressed as a 'regular-expression'. Complex expressions should be enclosed in quotes to ensure the commandline interpreter doesn't split them.

Additional table filters (-tfs|--table-filter-size-limit and -tfp|--table-filter-partition-count-limit) that check a tables data size and partition count limits can also be applied to narrow the range of tables you'll process.

The filter does NOT override the requirement for options like -ma|-mao. It is used as an additional filter.

Migrations between Clusters WITHOUT line of Site

There will be cases where clusters can't be linked (Linking Cluster Storage Layers). And without this line of sight, data movement needs to happen through some other means.

Shared Storage Models (Isilon, Spectrum-Scale, etc.)

There are cases where 'HDFS' isn't the primary data source. So the only thing the cluster share is storage in these 'common' storage units. You want to transfer the schema, but the data doesn't need to move (at least for 'EXTERNAL' (non-transactional) tables). In this case, try the -d|--data-strategy COMMON. The schema's will go through all the needed conversions while the data remains in the same location.

Disconnected Mode

Use the -rid|--right-is-disconnected mode when you need to build (and/or) transfer schema/datasets from one cluster to another, but you can't connect to both at the same time. See the issues log for details regarding the cases here issue #17 (https://github.com/cloudera-labs/hms-mirror/issues/17).

Use cases:

• Schema Only Transfers

• SQL, EXPORT_IMPORT, and HYBRID only when -is or -cs is used. This might be the case when the clusters are secure (kerberized), but don't share a common kerberos domain/user auth. So an intermediate or common storage location will be used to migrate the data.

• Both clusters (and HS2 endpoints) are Kerberized, but the clusters are NOT the same major hadoop version. In this case, hms-mirror doesn't support connecting to both of these endpoints at the same time. Running in the disconnected mode will help push through with the conversion.

hms-mirror will run as normal, with the exception of examining and running scripts against the right cluster. It will be assumed that the RIGHT cluster elements do NOT exist.

The RIGHT_ 'execution' scripts and distcp commands will need to be run MANUALLY via Beeline on the RIGHT cluster.

Note: This will be know as the "right-is-disconnected" option. Which means the process should be run from a node that has access to the "left" cluster. This is 'counter' to our general recommendation that the process should be run from the 'right' cluster.

No-Purge Option

-np

Feature Request #25 (https://github.com/cloudera-labs/hms-mirror/issues/25) was introduced in v1.5.4.2 and gives the user to option to remove the external.table.purge option that is added when converting legacy managed tables to external table (Hive 1/2 to 3). This does affect the behavior of the table from the older platforms.

Property Overrides

-po[l|r] <key=value>[,<key=value>]...

Feature Request #27 (https://github.com/cloudera-labs/hms-mirror/issues/27) introduced in v1.5.4.2 provides the ability to set a hive properties at the beginning of each migration part. This is a comma separated list of key=value pairs with no space. If spaces are needed, quote the parameter on the commandline.

You can use -po to set the properties for BOTH clusters or -pol|-por to set them specifically for the 'left' and/or 'right' cluster.

For example: -po hive.exec.orc.split.strategy=BI,hive.compute.query.using.stats=false

To provide a consistent list of settings for ALL jobs, add/modify the following section in the configuration file ie: default.yaml used for processing. In this case you do NOT need to use the commandline option. Although, you can set basic values in the configuration file and add other via the commandline.

Notice that there are setting for the LEFT and RIGHT clusters.

optimization:
  overrides:
    left:
      tez.queue.name: "compaction"
    right:
      tez.queue.name: "migration"

Global Location Map

-glm|--global-location-map <from=to>[,...]

This is an opportunity to make some specific directory mappings during the migration. You can supply a comma separated list of directory pairs to be use for evaluation.

-glm /data/my_original_location=/corp/finance_new_loc,/user/hive=/warehouse/hive

These directory mappings ONLY apply to 'EXTERNAL' and 'DOWNGRADED ACID' tables. You can supply 'n' number of mappings to review through the commandline interface as describe above. To provide a consistent set of mappings to ALL jobs, add/modify the following section in the configuration file ie: default.yaml used for processing.

translator:
  globalLocationMap:
    /data/my_original_location: "/corp/finance_new_loc"
    /user/hive: "/warehouse/hive"

The list will be sorted by the length of the string, then alpha-numerically. This will ensure the deepest nested paths are evaluated FIRST. When a path is matched during evaluation, the process will NOT look and any more paths in the list. Therefore, avoiding possible double evaluations that may result when there are nested paths in the list.

Paths are evaluated with 'startsWith' on the original path (minus the original namespace). When a match is found, the path 'part' will be replaced with the value specified. The remaining path will remain intact and regardless of the -rdl setting, the LOCATION element will be included in the tables new CREATE statement.

Force External Locations

-fel|--force-external-location

Under some conditions, the default warehouse directory hierarchy is not honored. We've seen this in HDP 3. The -rdl option collects the external tables in the default warehouse directory by omitting the LOCATION element in the CREATE statement, relying on the default location. The default location is set at the DATABASE level by hms-mirror.

In HDP3, the CREATE statement doesn't honor the 'database' LOCATION element and reverts to the system wide warehouse directory configurations. The -fel flag will simply include the 'properly' adjusted LOCATION element in the CREATE statement to ensure the tables are created in the desired location. This setting overrides the effects intended by the -rdl option which intend to place the tables under the stated warehouse locations by omitting the location from the tables definition and relying on the default location specified in the database.

-fel will use the original location as a starting point. If -wd|-ewd are specified, they aren't not used in the translation, but warnings may be issued if the final location doesn't align with the warehouse directory. The effect change in the location when using -fel, add mappings via -glm.

HDP 3 Hive

HDP 3 (Hive 3) was an incomplete implementation with regards to complex table 'location' management. When you are working in this environment, add to the cluster configuration (LEFT or RIGHT) the setting: hdpHive3: true. There is NOT a commandline switch for this. JUst add it to the configuration file you're using to run the application.

clusters:
  LEFT:
    environment: "LEFT"
    legacyHive: false
    hdpHive3: true

HDP3 Hive did NOT have a MANAGEDLOCATION attribute for Databases.

The LOCATION element tracked the Manage ACID tables and will control where they go.

This LOCATION will need to be transferred to MANAGEDLOCATION 'after' upgrading to CDP to ensure ACID tables maintain the same behavior. EXTERNAL tables will explicity set there LOCATION element to match the setting in -ewd.

Future external tables, when no location is specified, will be created in the hive.metastore.warehouse.external.dir. This value is global in HDP Hive3 and can NOT be set for individual databases.

Post upgrade to CDP, you should add a specific directory value at the database level for better control.

Schema Fix Features

Schema Fix Features are a way to inject special considerations into the replay of a schema between clusters. Each schema is automatically check is a particular 'feature' applies.

If you find that this features check is causing issues, add the flag -sf to the application parameters and the feature checks will be skipped.

ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '\t'
  LINES TERMINATED BY '\n'
STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'

STORED AS ORC

ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '\t'
  LINES TERMINATED BY '\n'

ROW FORMAT SERDE
  'org.apache.hadoop.hive.ql.io.orc.OrcSerde'

ROW FORMAT DELIMITED,
    FIELDS TERMINATED BY '|'
 STORED AS INPUTFORMAT 
  'org.apache.hadoop.hive.ql.io.RCFileInputFormat'
 OUTPUTFORMAT 
  'org.apache.hadoop.hive.ql.io.RCFileOutputFormat'

STORED AS RCFILE

ROW FORMAT DELIMITED,                              
    FIELDS TERMINATED BY '|'                       
 STORED AS INPUTFORMAT

STORED AS RCFILE

ROW FORMAT DELIMITED
     FIELDS TERMINATED BY '|'
     LINES TERMINATED BY '\n'
WITH SERDEPROPERTIES (
     'escape.delim'='\\')
STORED AS INPUTFORMAT
     'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
     'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'

ROW FORMAT SERDE
  'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'

Data Movement

When the strategy is responsible for moving data as well, you have two options: SQL and distcp. You can set this option here:

SQL will rely on the engine and the clusters ability to see across data storage environments to move the data.

For distcp (Distributed Copy) (https://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html), we'll build a plan that matches what you've asked to migrate.

For a matrix of when this might be available, see Location Alignment (Location Alignment).

Options

-asm|--avro-schema-migration

AVRO tables have a TBLPROPERTIES key avro.schema.url that identifies the supporting avro schemas location on the current cluster. When migrating AVRO tables, use this option to check for this setting. As a result, hms-mirror will parse out the location and replace it with the same relative location.

hms-mirror will use the configured environments hdfs|core-site.xml files to launch an hdfs client that will copy the supporting avro schema between clusters. For this to work, the clusters MUST be visible to each other (Linking Cluster Storage Layers).

The DUMP strategy is like SCHEMA_ONLY; it just doesn't require the RIGHT cluster to be connected. Although, it does need the following settings for the RIGHT cluster to make the proper adjustments:

legacyHive
hcfsNamespace
hiveServer2 -> partitionDiscovery

When the option -ma (migrate acid) is specified, the ACID schema's will be migrated/dumped. It is essential to know that the data for ACID tables can NOT simply be copied from one clusters hive instance to another. The data needs to be extracted to a none ACID table, then use an external table definition to read and INSERT the data into the new ACID table on the RIGHT cluster. For those that insist on trying to simply copy the data.... you've been warned ;).

With the DUMP strategy, you'll have a 'translated' (for legacy hive) table DDL that can be run on the new cluster independently.

-f|--flip

The configuration yaml has two clusters defined: LEFT and RIGHT. By default, all work moves from LEFT to RIGHT. If you need to perform actions from RIGHT to LEFT, use the -f option to switch these clusters. The final report will show that these clusters have been reversed.

-ma|--migrate-acid or -mao|--migrate-acid-only

[optional artificial bucket limit] (default is 2)

The default behaviour is NOT to handle ACID tables. ACID tables require some additional handling. When you need to address ACID tables, use these flags to either include them OR migrate ONLY ACID tables.

Migrations from legacy hive environments may have had to create tables with an unnecessary CLUSTERED BY clause to support bucketing for these ACID tables. The artificial bucket limit is an opportunity to remove those definitions for the migrated tables. For example: If you've defined a table with 2 buckets, the default artificial bucket limit is 2, so the table CLUSTERED BY ... INTO 2 BUCKETS will be removed from the definition.

[-mnno|--migrate-non-native-only]

See Non-Native Tables ("Non-Native Hive Tables (Hbase, KAFKA, JDBC, Druid, etc..)" in "Features") for more information.

Use this to migrate non-native hive tables like: Kafka, HBase, JDBC Federated, etc.. This option excludes native tables from processing, so you should run it separately.

Note that many of these tables REQUIRE that the supporting references EXIST first, or their creation will FAIL. For instance, migrating a table with the HBase Storage handler requires that the HBase table is present AND visible from Hive.

[-v|--views-only]

See Views ("VIEWS" in "Features") for more information.

Like -mnno the -v option excludes other tables from processing, so it's designed to run along. And like the -mnno option, it too requires the supporting tables to exist, otherwise the view create statements will fail.

-asm|--avro-schema-migration

AVRO tables have a TBLPROPERTIES key avro.schema.url that identifies the supporting avro schemas location on the current cluster. When migrating AVRO tables, use this option to check for this setting. As a result, hms-mirror will parse out the location and replace it with the same relative location.

hms-mirror will use the configured environments hdfs|core-site.xml files to launch an hdfs client that will copy the supporting avro schema between clusters. For this to work, the clusters MUST be visible to each other (Linking Cluster Storage Layers).

The DUMP strategy is like SCHEMA_ONLY; it just doesn't require the RIGHT cluster to be connected. Although, it does need the following settings for the RIGHT cluster to make the proper adjustments:

legacyHive
hcfsNamespace
hiveServer2 -> partitionDiscovery

When the option -ma (migrate acid) is specified, the ACID schema's will be migrated/dumped. It is essential to know that the data for ACID tables can NOT simply be copied from one clusters hive instance to another. The data needs to be extracted to a none ACID table, then use an external table definition to read and INSERT the data into the new ACID table on the RIGHT cluster. For those that insist on trying to simply copy the data.... you've been warned ;).

With the DUMP strategy, you'll have a 'translated' (for legacy hive) table DDL that can be run on the new cluster independently.

-sp,--sql-partition-count <limit>

Sets the limit for the number of partitions that the SQL (SQL) strategy will process. If the value is exceeded, the process will NOT migrate the table. The default is 500 .

To persist a higher value without specifying the -sp option, add the following to the config (Default Configuration Template).

hybrid:
  sqlPartitionLimit: <limit>

-ma|--migrate-acid or -mao|--migrate-acid-only

Include ACID tables in processing.

-da|--downgrade-acid

Applicable only when -ma|o is used. This will take the ACID tables and downgrade them to EXTERNAL/PURGE tables. Buckets adjustments ("ACID Tables" in "Features") are applicable.

-dc|--distcp

For EXTERNAL tables, this will build a distcp plan that can be used to transfer the database tables.

When the option -ma|o is used, ACID tables will be migrated. ACID tables will be converted to EXTERNAL transfer tables for the distcp operations. On the other cluster, the tables will be created initially as EXTERNAL, then transferred back over to ACID tables, unless -da is used.

If the -da is used the tables will remain EXTERNAL on the new cluster.

When -is is used, there will be distcp operations required on both clusters to handle data movement to/from the intermediate-storage area.

-is|--intermediate-storage or -cs|--common-storage

When the clusters aren't linked (Linking Cluster Storage Layers) these two options provide a way to transfer data through an intermediate/common storage location. Each cluster must have access to these locations. These values are mutually exclusive.

--intermediate-storage requires an addition transfer of data. The LEFT transfers data to the -is location and the RIGHT cluster uses that data and initiates a transfer from the location to the final hcfsNamespace value set for the cluster. This is a two copy migration.

--common-storage assumes the location is also the final resting place for the data. Some optimizations are possible (ACID downgrades) that reduce that times data need to move. This is a single copy migration.

-wd|--warehouse-directory and -ewd|--external-warehouse-directory

Are used to set the databases default locations for managed and external tables. This overrides the systems hive metastore properties.

-rdl|--reset-to-default-location

Regardless of where the source data relative location was on the filesystem, this will reset it to the default location on the new cluster.

If -dc|--distcp is used, then the warehouse options are required in order for hms-mirror to build the distcp workplan.

-ep|--export-partition-count <limit>

Sets the limit for the number of partitions to use EXPORT_IMPORT. The default is 100. When a table has a partition count that exceeds this value, the SQL (SQL) strategy will be used.

To persist a higher value without specifying the -ep option, add the following to the config (Default Configuration Template).

hybrid:
  exportImportPartitionLimit: <limit>

Interesting Options

When the cluster don't have direct line of sight to each other and can NOT be linked (Linking Cluster Storage Layers), you can use options like -cs or -is to bridge the gap.

Important Properties

The following screen shot shows the properties that are important for the Storage Migration feature.

Target Namespace

Identifies the 'namespace' where all the translations will be made to.

Location Translation Strategy

RELATIVE or ALIGNED. In this case, we DO want to ALIGN the locations with the new 'namespace' and locations defined in the Warehouse Plan for the database.

Movement Strategy

DISTCP or SQL. This will direct hms-mirror to build a distcp plan/scripts or construct SQL to move the data.

Skip Database Location Adjustments

Defaults to 'false'. If you want to skip the location adjustments, set this to 'true'. This is useful if you're trying to archive tables in a database to another storage system, but want to leave the database location as is for new tables in the database.

Demo

Here's how the Warehouse Plans would look for the above example:

Here's one of the report details for the 'finance.db1' database. See how the 'external' and 'managed' locations are have been altered to reflect the new 'namespace' structure in Ozone we requested in the Warehouse Plan.

Requirements

• Requires Hive with Iceberg Support.

  • CDP Private Cloud Base 7.1.9 Hive does NOT support Iceberg.

  • CDP Private Cloud Base 7.3.1 Hive does support Iceberg.

  • CDP Private Cloud CDW 1.5.1 Hive does support Iceberg. You need CDW Data Services 1.5.1 or higher.

  • CDP Public Cloud Hive does support Iceberg as of August 2023 (Datahub and CDW)

Caution

Make sure you know the component limitations with Iceberg Tables here (https://docs.cloudera.com/cdp-public-cloud/cloud/cdp-iceberg/topics/iceberg-in-cdp.html) so you aren't caught by surprise.

Copy Avro Schema Urls

CLI

-asm,--avro-schema-migration Migrate AVRO Schema Files referenced in TBLPROPERTIES by 'avro.schema.url'. Without migration it is expected that the file will exist on the other cluster and match the 'url' defined in the schema DDL. If it's not present, schema creation will FAIL. Specifying this option REQUIRES the LEFT and RIGHT cluster to be LINKED. See docs: https://github.com/cloudera-labs/hms-mirror#linking-clusters-storage-layers

Web UI

Config File

copyAvroSchemaUrls is a boolean value that determines if the Avro schema URLs should be copied from the source to the target. This is useful if you're using Avro schemas in your source and target environments and want to maintain the same schema URLs in both environments. The default value is false.

copyAvroSchemaUrls: true|false

Data Strategy

Data Strategy controls how schemas and data are migrated. The following values are supported:

• SCHEMA_ONLY

• SQL

• EXPORT_IMPORT

• HYBRID

• DUMP

• STORAGE_MIGRATION

• COMMON

• LINKED

CLI

-d|--data-strategy <strategy> 'strategy' is one of the strategies listed above.

Web UI

The Web UI approaches this setting a little differently. When you 'initialized' a migration session, you'll be asked to select a 'Data Strategy' from a drop-down list. Saved sessions will have the 'data-strategy' embedded in them and is NOT editable. You 'can' create a new session based on an existing session and maintain most of the properties of the original session.

Config File

dataStrategy: "SCHEMA_ONLY|SQL|EXPORT_IMPORT|HYBRID|DUMP|STORAGE_MIGRATION|COMMON|LINKED"

Database Only

When set we'll only migrate the database objects and not the tables.

CLI

-dbo|--database-only will only migrate the database objects and not the tables.

Web UI

Config File

databaseOnly: true|false

Dump Test Data

Dumping test data uses the running configuration to create a processable artifact that can be used for testing offline.

dumpTestData: true|false

Load Test Data File

Run the process with a test data file that was created with the --dump-test-data option.

loadTestDataFile: "<path_to_test_data_file>"

Skip Link Check

When there are issues connecting to the source or target cluster storage systems, you can skip the link check. Doing so may mean that certain features will not work as expected. EG: Limits used to determine processing that need to review the storage layer or the ability to copy Avro schema files or optimizations made based on file

CLI

-slc|--skip-link-check will skip the link check.

Web UI

Config File

Default is false.

skipLinkCheck: true|false

Database Prefix

dbPrefix is a value to pre-pend to the database name when creating the database in the target cluster. This is way to avoid conflicts with existing databases in the target cluster. The default value is an empty string.

dbPrefix: "<prefix>"

Database Rename

dbRename is a string value that identifies the new name of the database in the target cluster. This is useful for testing a single database migration to an alternate database in the target cluster. This is only valid for a single database migration.

dbRename: "<new_db_name>"

Execute

execute is a boolean value that determines if the migration should be executed. The default value is false, which is the dry-run mode. In the 'dry-run' mode, all the reports are generated and none of the actual migration is done.

You should ALWAYS run a 'dry-run' before executing a migration. This will give you a good idea of what will be done and provide you with the reports to review.

execute: true|false

Output Directory

Output directory is a string value that identifies the directory where the reports will be written. The default is $HOME/.hms-mirror/reports. When this value is defined, the reports will be written to the specified output directory with the timestamp as the 'name' of the report.

outputDirectory: "<path_to_output_directory>"

Encrypted Passwords

encryptedPasswords is a boolean value that determines if the passwords in the configuration file are encrypted.

encryptedPasswords: true|false

Read-Only

Read-Only is a boolean value that determines if the migration should be read-only. The default value is false. When this value is set to true, table schema's created will not have a 'purge' flag set to ensure they can't drop data. This is useful for testing migrations and for DR scenarios where you want to limit the exposure of potential changes on the target cluster.

This does NOT prevent data from being manipulated on the target cluster after the migration.

readOnly: true|false

Save Working Tables

saveWorkingTables is a boolean value that determines if the working tables should be saved. The default value is false.

CLI

-swt|--save-working-tables

Web UI

Config File

saveWorkingTables: true|false

Skip Features

Skip Features is a boolean value that is false by default so feature check will be made. Features are a framework of checks that examine a table definition and make corrections to it to ensure it's compatible with the target cluster. We've found several circumstances where definitions extracted from the source cluster can NOT be replayed on the target cluster for some reason. These features attempt to correct those issues during the migration.

skipFeatures: true|false

Transfer Ownership

You can choose to transfer ownership of objects from the source to target clusters. Ownership is defined as the owner of the database or table in the metastore. This is useful if you are using Ranger policies that rely on the owner of the object.

The user running the migration commands will need permissions to make these settings to the table. This would include the ability to `ALTER` the target table. If you're using Ranger, please run some test with the user in beeline to ensure there isn't an issue when running the process.

CLI

-to|--transfer-ownership will transfer ownership of databases and tables from the source to the target cluster.

-todb|--transfer-ownership-database will transfer ownership of databases from the source to the target cluster.

-totbl|--transfer-ownership-table will transfer ownership of tables from the source to the target cluster.

Web UI

Config File

ownershipTransfer:
  database: true|false
  table: true|false

ACID Tables

ACID tables require special handling. By default, ACID tables are NOT migrated unless the following options are set.

CLI

-ma|--migrate-acid will set the migration to include ACID tables. -mao|--migrate-acid-only will set the migration to include ONLY ACID tables.

Web UI

Config File

migrateACID:
  "on": true
  only: false

Non-Native Tables

By default, non-native tables are NOT migrated. Usually because there are requirements that the underlying technology be in place before the migration can happen.

If those prerequisites are met, you can set the migrateNonNative option to true in the configuration file.

This is typical of tables in Hive that rely on other technologies to store the data. EG: HBase, Kafka, JDBC Federation, etc.

CLI

-mnn|--migrate-non-native will set the migration to include non-native tables. -mnno|--migrate-non-native-only will set the migration to include ONLY non-native tables.

Web UI

Config File

migrateNonNative: true

Views

Views are NOT migrated by default. Views require the underlying tables to be in place before they can be created. If you have views, migrate the tables first and then rerun the migration for the views.

Do not rename databases or tables that view depend on. The migration will fail.

CLI

-v|--views-only will set the migration to include ONLY views.

Web UI

Config File

migrateVIEW:
  "on": false

Databases

There are several ways to specify which databases to process. The most simple method is to provide a list of database(s) to process.

The first is to set the database(s) to process explicitly.

CLI

-db <databases>|--database <databases> 'databases' is a comma separated list of target databases to process.

Web UI

Config File

databases:
- db_name
- db_name2

The second method is to use a Regular Expression to match the database(s) to process.

CLI

-dbRegEx <regex>|--database-regex <regex> 'regex' is a valid "Regular Expression" used to filter which databases to process from the target cluster.

Web UI

Config File

Using the -dbRegEx on the CLI will populate the databases section of the config file with the regular expression matches.

databases:
- db_name
- db_name2

The third method is the most flexible and complete option. Use Warehouse Plans (Warehouse Plans) to define the which databases to process. This method also allows you to define individual database location values.

CLI

-wps,--warehouse-plans <db=ext-dir:mngd-dir[,db=ext-dir:mngd-dir]...>

Web UI

Config File

translator:
  warehousePlans:
    db1:
      source: "PLAN"
      externalDirectory: "/finance/external"
      managedDirectory: "/finance/managed"
    db2:
        source: "PLAN"
        externalDirectory: "/marketing/external"
        managedDirectory: "/marketing/managed"

Database Skip Properties (via RegEx)

A user managed list of properties that will be filter OUT from the migration. For example: If you don't want to migrate a DBPROPERTY like repl.incl.test=hello_world, then add repl\.incl.* to this list.

This is a RegEx, so ensure you follow that syntax.

CLI

-dbsp,--database-skip-properties <properties> Comma separated list of database properties (regex) to skip during the migration process. This will prevent the property from being set on the target cluster.

Web UI

Config File

filter:
  dbPropertySkipList:
    - "repl\\.inc.*"

Tables

When nothing is specified, all tables in the processed databases are included. To limit the tables processed, you can use the following options.

By Regular Expression

CLI

-tf <regex> 'regex' used to match tables in the database to process. process.

-tef <regex> 'regex' used to match tables that would be 'EXCLUDED' in the database to process. process.

Web UI

Config File

filter:
  tblRegEx: "test.*"

filter:
  tblExcludeRegEx: "tmp.*"

By Limits

CLI

-tfp <partition-count> 'partition-count' would be a limit of the number of partitions in a table that would be processed. Tables with more partitions than the limit would be excluded. A value of -1 would include all tables.

-tfs <size MB> 'size MB' would be a limit in size for tables that would be processed. Tables greater in size would be excluded. A value of -1 would include all tables.

Web UI

A value of -1 would include all tables.

Config File

A value of -1 would include all tables.

filter:
  tblSizeLimit: -1
  tblPartitionLimit: -1

Target Namespace

transfer:
  storageMigration:

Intermediate Storage

transfer:
  storageMigration:
    intermediateStorage: ...

Location Translation Strategy

transfer:
  storageMigration:
    translationType: "ALIGNED|RELATIVE"

Data Movement Strategy

transfer:
  storageMigration:
    dataMovementStrategy: "SQL|DISTCP"

Skip Database Location Adjustments

When true, the database location will NOT be adjusted to match the table locations that are migrated. " + "This is useful in the case of an 'archive' strategy where you only want to migrate the table(s) but are not yet " + "ready to migration or stage future tables at the new location.

transfer:
  storageMigration:
    skipDatabaseLocationAdjustments: true|false

Create Archive

When true (default is false), the tables being migrated will be archived vs. simply changing the location details " + "of the tables metadata. This is relevant only for STORAGE_MIGRATION when using the 'DISTCP' data movement " + "strategy. When the data movement strategy is 'SQL' for STORAGE_MIGRATION, this flag is ignored because the default " + "behavior is to create an archive table anyhow.

transfer:
  storageMigration:
    createArchive: true|false

Consolidate Source Tables

Used to help define how a distcp plan will be built when asked for. The default is false which means that will NOT be consolidating table locations.

If this is set to true, the distcp plan will remove the table location from the source (which is generally the database location) and use it for all transfers. This looks more simple, but could lead to copying more data than you expect since there's no guarantee that there isn't a lot of other 'extra' data in the source location.

transfer:
  storageMigration:
    consolidateTablesForDistcp: true|false

Strict Mode

When true and using distcp, the migration will fail if the table partition locations aren't standard. The default is false. The default mode will allow you to move forward with warnings about the non-standard partition locations. It's highly recommended that you validate the location mappings in the distcp plan before running the migration.

transfer:
  storageMigration:
    strict: true|false

Application doesn't seem to be making progress

All the counters for table processing aren't moving (review the hms-mirror.log) or (1.6.1.0+) the on screen logging of what tables are being added and metadata collected for has stopped.

Solution

The application creates a pool of connection to the HiveServer2 instances on the LEFT and RIGHT to be used for processing. When the HiveServer2 doesn't support or doesn't have available the number of connections being requested from hms-mirror, the application will 'wait' forever on getting the connections requested.

Stop the application and lower the concurrency value to a value that can be supported.

transfer:
  concurrency: 4

Or, you could modify the HiveServer2 instance to handle the number of connections being requested.

Application won't start NoClassDefFoundError

Error

Exception in thread "main" java.lang.NoClassDefFoundError:
java/sql/Driver at java.base/java.lang.ClassLoader.defineClass1

hms-mirror uses a classloader to separate the various jdbc classes (and versions) used to manage migrations between two different clusters. The application also has a requirement to run on older platforms, so the most common denominator is Java 8. Our method of loading and separating these libraries doesn't work in Java 9+.

Solution

Please use Java 8 to run hms-mirror.

CDP Hive Standalone Driver for CDP 7.1.8 CHF x (Cummulative Hot Fix) won't connect

If you are attempting to connect to a CDP 7.1.8 clusters Hive Server 2 with the CDP Hive Standalone Driver identified in the clusters jarFile property, you may not be able to connect. A security item addressed in these drivers changed the required classes.

If you see:

java.lang.RuntimeException: java.lang.RuntimeException: java.lang.NoClassDefFoundError: org/apache/log4j/Level

You will need to include additional jars in the jarFile property. The following jars are required:

log4j-1.2-api-2.18.0.ja
log4j-api-2.18.0.jar
log4j-core-2.18.0.jar

The feature enhancement that allows multiple jars to be specified in the jarFile property is available in hms-mirror 1.6.0.0 and later. See Issue #47 (https://github.com/cloudera-labs/hms-mirror/issues/67)

Solution

Using hms-mirror v1.6.0.0 or later, specify the additional jars in the jarFile property. For example: jarFile: "<absolute_path_to>/hive-jdbc-3.1.3000.7.1.8.28-1-standalone.jar:<absolute_path_to>/log4j-1.2-api-2.18.0.jar:<absolute_path_to>/log4j-api-2.18.0.jar:<absolute_path_to>/log4j-core-2.18.0.jar"

These jar files can be found on the CDP edge node in /opt/cloudera/parcels/CDH/jars/.

Ensure that the standalone driver is list 'FIRST' in the jarFile property.

Failed AVRO Table Creation

Error while compiling statement: FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.ddl.DDLTask. java.lang.RuntimeException: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException Encountered AvroSerdeException determining schema. Returning signal schema to indicate problem: Unable to read schema from given path: /user/dstreev/test.avsc)

Solution

Validate that the 'schema' file has been copied over to the new cluster. If that has been done, check the permissions. In a non-impersonation environment (doas=false), the hive user must have access to the file.

Table processing completed with ERROR.

We make various checks as we perform the migrations, and when those checks don't pass, the result is an error.

Solution

In tips (Tips) we suggest running with dry-run first (default). This will catch the potential issues first, without taking a whole lot of time. Use this to remediate issues before executing.

If the scenario that causes the ERROR is known, a remediation summary will be in the output report under Issues for that table. Follow those instructions, then rerun the process with --retry. NOTE: --retry is currently tech preview and not thoroughly tested.

Connecting to HS2 via Kerberos

Connecting to an HDP cluster running 2.6.5 with Binary protocol and Kerberos triggers an incompatibility issue: Unrecognized Hadoop major version number: 3.1.1.7.1....

Solution

The application is built with CDP libraries (excluding the Hive Standalone Driver). When Kerberos is the auth` protocol to connect to Hive 1, it will get the application libs which will NOT be compatible with the older cluster.

Kerberos connections are only supported to the CDP cluster.

When connecting via Kerberos, you will need to include the --hadoop-classpath when launching hms-mirror`.

Auto Partition Discovery not working

I've set the partitionDiscovery:auto to true, but the partitions aren't getting discovered.

Solution

In CDP Base/PVC versions < 7.1.6 have not set the housekeeping thread that runs to activate this feature.

In the Hive metastore configuration in Cloudera Manager, set metastore.housekeeping.threads.on=true in the Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml

Hive SQL Exception / HDFS Permissions Issues

Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAccessControlException:Permission denied: user [dstreev] does not have [ALL] privilege on [hdfs://HDP50/apps/hive/warehouse/tpcds_bin_partitioned_orc_10.db/web_site]

This error is a permission error to HDFS. For HYBRID, EXPORT_IMPORT, SQL, and SCHEMA_ONLY (with -ams enabled), this could be an issue with cross-cluster HDFS access.

Review the output report for details of where this error occurred (LEFT or RIGHT cluster).

When dealing with CREATE DDL statements submitted through HS2 with a LOCATION element in them, the submitting user AND the HS2 service account must have permissions to the directory. Remember, with cross-cluster access, the user identity will originate on the RIGHT cluster and will be EVALUATED on the LEFT clusters storage layer.

For migrations, the hms-mirror running user (JDBC) and keytab user (HDFS) should be privileged users.

YARN Submission stuck in ACCEPTED phase

The process uses a connection pool to hive. If the concurrency value for the cluster is too high, you may have reached the maximum ratio of AM (Application Masters) for the YARN queue.

Review the ACCEPTED jobs and review the jobs Diagnostics status for details on why the jobs is stuck.

Solution

Either of:

1. Reduce the concurrency in the configuration file for hms-mirror

2. Increase the AM ratio or Queue size to allow the jobs to be submitted. This can be done while the process is running.

Spark DFS Access

If you have problems accessing HDFS from spark-shell or spark-submit try adding the following configuration to spark:

--conf spark.yarn.access.hadoopFileSystems=hdfs://<NEW_NAMESPACE>,hdfs://<OLD_NAMESPACE>

Permission Issues

HiveAccessControlException Permission denied user: [xxxx] does not have [ALL] privileges on ['location'] [state=42000,code=40000]

and possibly

In HS2 Logs: Unauthorized connection for super-user

Solution

Caused by the following:

• The 'user' doesn't have access to the location as indicated in the message. Verify through 'hdfs' that this is true or not. If the user does NOT have access, grant them access and try again.

• The 'hive' service account running HS2 does NOT have access to the location. The message will mask this and present it as a 'user' issue, when it is in fact an issue with the 'hive' service account. Grant the account the appropriate access.

• The 'hive' service does NOT have proxy permissions to the storage layer.

  • Check the hadoop.proxyuser.hive.hosts|groups setting in core-site.xml. If you are running into this super-user error on the RIGHT cluster, while trying to access a storage location on the LEFT cluster, ensure the proxy settings include the rights values in the RIGHT clusters core-site.xml, since that is where HS2 will pick it up from.

Must use HiveInputFormat to read ACID tables

We've seen this while attempting to migrate ACID tables from older clusters (HDP 2.6). The error occurs when we try to extract the ACID table data to a 'transfer' external table on the LEFT cluster, which is 'legacy'.

Solution

HDP 2.6.5, the lowest supported cluster version intended for this process, should be using the 'tez' execution engine set hive.execution.engine=tez. If the cluster has been upgraded from an older HDP version OR they've simply decided NOT to use the tez execution engine', you may get this error.

In hms-mirror releases 1.3.0.5 and above, we will explicitly run set hive.execution.engine=tez on the LEFT cluster when identified as a 'legacy' cluster. For version 1.3.0.4 (the first version to support ACID transfers), you'll need to set the hive environment for the HS2 instance you're connecting to use tez as the execution engine.

ACL issues across cross while using LOWER clusters storage

Are you seeing something like this?

org.apache.hadoop.hive.ql.ddl.DDLTask. MetaException(message:Got exception: org.apache.hadoop.security.AccessControlException Permission denied: user=hive, access=WRITE, inode="/apps/hive/warehouse/merge_files.db/merge_files_part_a_small_replacement":dstreev:hadoop:drwxr-xr-x at

This is caused when trying to CREATE a table on the RIGHT cluster that references data on the LEFT cluster. When the LEFT cluster is setup differently with regard to impersonation (doas) than the RIGHT, transfer tables are created with POSIX permissions that may not allow the RIGHT cluster/user to access that location.

Solution

Using Ranger on the LEFT cluster, open up the permissions to allow the requesting user access as identified.

Environments

HDP 2.6 -> CDP CDH 5.x -> CDP CDH 6.x -> CDP

LEFT clusters are the LEGACY clusters. RIGHT clusters are the NON-LEGACY clusters (hive3).

Assumptions

• LEFT HS2 is NOT Kerberized. Since this is a Legacy cluster (which is kerberized) we need to establish a 'non' kerberized HS2 endpoint. Use KNOX or setup an additional HS2 in the management console that isn't Kerberized. hms-mirror is built (default) with Hadoop 3, of which the libraries are NOT backwardly compatible.

• Standalone JDBC jar files for the LEGACY and NON-LEGACY clusters are available to the running host as specified in the 'configuration'.

• hms-mirror is run from an EdgeNode on CDP

  • The edgenode has network access to the Legacy HS2 endpoint

• No ACID tables (HDP)

• No VIEWs

• No Non-Native tables (Hive tables backed by HBase, JDBC, Kafka)

• The HiveServer2's on each cluster have enough concurrency to support the configured connections transfer->concurrency. If not specified, the default is 4.

NOTES

hms-mirror runs in DRY-RUN mode by default. Add -e|--execute to your command to actually run the process on the clusters. Use --accept to avoid the verification questions (but don't deny their meaning).

All actions performed by hms-mirror are recorded in the *_execute.sql files. Review them to understand the orchestration and process.

Review the report markdown files (html version also available) for details about the job. Explanations regarding steps, issues, and failure reasons can be found there.

One-time migration of SCHEMA's from LEFT to RIGHT.

This is done with the basic SCHEMA_ONLY data strategy (default) and will extract the schema's from the LEFT and replay them on the RIGHT cluster. In this mode, NO DATA is moved.

hms-mirror -db tpcds_bin_partitioned_orc_10 -o temp

Examine the reports place in the relative temp directory specified with the -o option. A report set is generated to each database.

Since no data is transferred in this scenario, the expectation is that the data is managed by another process, like distcp. The output of hms-mirror will create some template distcp calls with the appropriate location details. You can use this as a starting point to managed this transfer.

DUMP of clusters SCHEMA

We want a simple extraction of a database schema. An optional -ds LEFT|RIGHT option can be specified to target which configured cluster to use. The default is LEFT. Note: When you specify RIGHT, the DUMP output with still show up in the LEFT output report.

No translations are done in this scenario. So if the DUMP is taken from a 'legacy' cluster, beware of the implications of 'replaying' it on a non-legacy cluster.

This will use the LEFT cluster configuration for the schema DUMP of tpcds_bin_partititoned_orc_10. hms-mirror -db tpcds_bin_partitioned_orc_10 -o temp -d DUMP

This will use the RIGHT cluster configuration for the schema DUMP of tpcds_bin_partititoned_orc_10. hms-mirror -db tpcds_bin_partitioned_orc_10 -o temp -d DUMP -ds RIGHT

Data Migration for Non-ACID tables using SQL

This approach assumes the clusters are linked (Linking Cluster Storage Layers). The SQL data strategy uses the RIGHT's clusters view into the HDFS filesystem of the LEFT cluster to facilitate data movement.

hms-mirror -d SQL -db tpcds_bin_partitioned_orc_10 -o temp

Data Migration for ACID tables using SQL or HYBRID

This approach assumes the clusters are linked (Linking Cluster Storage Layers). The SQL data strategy uses the RIGHT's clusters view into the HDFS filesystem of the LEFT cluster to facilitate data movement.

hms-mirror -d SQL -db tpcds_bin_partitioned_orc_10 -o temp -ma

Use -mao to migrate ONLY ACID tables. -ma will migrate ACID and Non-ACID tables.

Schema Migration of specific tables using RegEx

Using a RegEx pattern, filter the tables in the db to migrate.

hms-mirror -db tpcds_bin_partitioned_orc_10 -tf call_*. -o temp

-tf|--table-filter followed by a RegEx to filter tables.

Migrate VIEWS for a specific database

View migration requires the underlying referenced tables exist BEFORE the 'view' can be created. This isn't a requirement of hms-mirror rather a Hive requirement. Therefore, you should migrate the tables first as shown above and in a followup process run the following.

hms-mirror -db tpcds_bin_partitioned_orc_10 -v

Create schema in RIGHT cluster using the LEFT clusters data

This is a helpful scenario for 'testing' workflows on the RIGHT cluster. The tables on the right cluster will NOT be configured with 'PURGE' to avoid the deletion of data on the LEFT cluster. These tables should be considered READ-ONLY. Test this against a sample dataset to THOROUGHLY understand the relationships here. This is NOT intended for 'production' use and should be used only as a validation mechanism for the RIGHT cluster.

The clusters must be linked (Linking Cluster Storage Layers). Only legacy managed tables, external tables, and views can be linked. ACID tables can NOT be linked.

hms-mirror -d LINKED -db tpcds_bin_partitioned_orc_10 -o temp

The tables created on the RIGHT cluster will use the data located on the LEFT cluster. The 'database' will be created to match the database in the LEFT cluster.

WARNING: If the LOCATION element is specified in the database definition AND you use DROP DATABASE ... CASCADE from the RIGHT cluster, YOU WILL DROP THE DATA ON THE LEFT CLUSTER even though the tables are NOT purgeable. This is the DEFAULT behavior of hive 'DROP DATABASE'. So BE CAREFUL!!!!

Migrate SCHEMA's and Data using SQL

The clusters must be linked (Linking Cluster Storage Layers). In this scenario, we'll use the connected clusters and SQL to migrate data from the LEFT cluster to the RIGHT.

There are limits regarding partitioned tables. For SQL migrations, the default is 500. Meaning that tables with more than 500 partitions will NOT attempt this transfer. This can be changed in the 'config' file by adding/changing hybrid->sqlPartitionLimit. This was put in place as a general safeguard against attempts at tables with a partition count that may fail. It doesn't mean they'll always fail, it's just a place holder.

hms-mirror -d SQL -db tpcds_bin_partitioned_orc_10 -o temp

To transfer ACID tables, add -ma|-mao.

This is a one time transfer. Incremental updates aren't supported with this configuration. For incremental schema updates, see:

Migrate SCHEMA's and Data using EXPORT_IMPORT

EXPORT/IMPORT is a basic Hive process used to package table schemas and data into a transferable unit that can be replayed on the new cluster. For hms-mirror there is a defined prefix for a transfer directory in the configuration transfer->exportBaseDirPrefix. If this isn't defined, the default is /apps/hive/warehouse/export_.

There are performance implications to using EXPORT_IMPORT with partitioned tables. The IMPORT process is quite slow at loading partitions. We've defined limits in the config (which can be changed) hybrid->exportImportPartitionLimit. The default is 100. If the number of partitions exceeds this value, we will NOT attempt the transfer and will note this in the output report.

The clusters must be linked (Linking Cluster Storage Layers). The before mentioned prefix directory on the LEFT cluster is accessed by the IMPORT process that runs on the RIGHT cluster. If the namespace (and permissions) aren't correct, the IMPORT process will fail.

hms-mirror -d EXPORT_IMPORT -db tpcds_bin_partitioned_orc_10 -o temp

EXPORT_IMPORT will NOT work for ACIDv1 -> ACIDv2 (Hive 1/2 to 3) conversions. Use SQL or HYBRID instead.

Migrate SCHEMA's and Data using HYBRID

The HYBRID data strategy is a combination of the SQL and EXPORT_IMPORT data strategies. It uses basic rules to choose the more appropriate method for the table in question.

The clusters must be linked (Linking Cluster Storage Layers).

The process will first consider using EXPORT_IMPORT unless:

• The table is ACIDv1 and you're migrating to ACIDv2 (Legacy to Non-Legacy Clusters)

• The number of partitions exceed the value set by: hybrid->exportImportPartitionLimit. The default is 100. When exceeded, the SQL method will be used. The SQL method will fail is the partition count exceeds the value of hybrid->sqlPartitionLimit. The default is 500.

hms-mirror -d HYBRID -db tpcds_bin_partitioned_orc_10 -o temp

Disaster Recovery (RIGHT Cluster is DR and READ-ONLY)

The DR scenario will transfer schemas and subsequent runs will update the 'schema' if changes are made. This process does NOT move data. The process will generate a distcp work plan for you to get started. You should modify that to use 'snapshot diffs' and managed the data migration through distcp.

You should first run the process in DRY-RUN mode to get the distcp plan. The data must be transferred first! This ensures that the database and table/partition directories are created BEFORE the schemas are replayed. If the schemas are applied before the distcp with SNAPSHOT diffs, then hive will own the directories and the distcp with snapshots will fail.

WARNING: Do not attempt to DROP DATABASE ... CASCADE on the RIGHT cluster, this will modify the filesystem and cause the incremental distcp with snapshots to fail.

hms-mirror -d SCHEMA_ONLY -db tpcds_bin_partitioned_orc_10 -ro -sync

This process will review the tables on the LEFT cluster with the RIGHT and either update the schema when it's changed (by dropping and recreating), add missing tables, or drop tables that don't exist anymore.

Tables that are migrated this way will NOT have the PURGE flag set on the RIGHT cluster. This allows us to DROP a table without affecting the data for the -sync process.

Downgrade and Replace ACID tables

In this scenario, you're choosing to downgrade ACID tables that are migrated, as well as the current tables on the source cluster.

hms-mirror -db tpcds_bin_partitioned_orc_10 -mao -da -r

This will migrate ACID tables only (-mao vs. -ma), downgrade them to EXTERNAL/PURGE tables, and 'replace' the current ACID table with a MANAGED Non-Transactional table in legacy environments OR a EXTERNAL/PURGE table in Hive3+ environments.

Using the DRY-RUN mode, experiment with options -is and -cs for various implementations of this conversion.

Requirements

• The source and target clusters on AWS are running CDP Cloud with an available HS2 endpoint.

• Provide a mechanism to migrate Hive metadata from the source cluster to the target cluster, to include making adjustments to the metadata to account for differences in the clusters storage locations.

• Establish the RPO and RTO for the DR scenario and ensure the migration process can meet those requirements.

• We'll only target external tables for this scenario. Managed tables will require additional considerations, since the data and metadata are intermingled and can't be supported through a simple copy operation.

Assumptions

• The data on S3 is already replicated to the target cluster through some other mechanism (e.g. S3 replication, etc.).

• The S3 replication will meet the RPO and RTO requirements.

• The data replication is in place before DR is invoked and the scripts are run to build the schemas on the target cluster.

• There are no managed tables being migrated. I would recommend setting the database property ‘EXTERNAL_TABLES_ONLY’=’TRUE’ with: ALTER DATABASE <db_name> SET TBLPROPERTIES ('EXTERNAL_TABLES_ONLY'='TRUE'); to ensure only external tables can be created.

• Partitions follow standard naming conventions regarding directory names/structures. Tables with non-standard partitioning will require additional considerations. hms-mirror doesn't translate partition details and relies on MSCK REPAIR <table> SYNC PARTITIONS to discover / rebuild a tables partitions. If the partitions are not in a standard format, the MSCK REPAIR will not work and the partitions will need to be manually created.

• We don't support schema evolution. All tables will be created in there current state.

• The "LEFT" clusters hcfsNamespace can only address a single namespace at a time. If you have multiple namespaces, you'll need to run hms-mirror multiple times, once for each namespace. The "RIGHT" cluster can address multiple namespaces through the hcfsNamespace element. This element is used to match and adjust the storage location of the tables on the target cluster.

The Process

The process is fairly straight forward. We'll use hms-mirror to migrate the Hive metadata from the source cluster to the target cluster. We'll use the --common-storage or set the hcfsNamespace element for the RIGHT cluster to ensure the schemas are built with the DR bucket adjustments.

You have a few options regarding the transfer:

• If the target is truly a DR cluster, you can run hms-mirror on the source cluster and generate the metadata files locally. Then copy the metadata files to the target cluster and build out the schemas there. This doesn't need to be done until the DR is invoked.

• If you want/need to keep the metadata in-sync between the clusters, you can run hms-mirror with the -ro and -sync flags (and eventually with -e) to keep the metadata in-sync between the clusters. Tables created on the source cluster will require the data to be replicated to the target cluster before the table can be created in DR. While we're only migrating external tables, they may have set external.table.purge to true on the source cluster. In this case, these tables will be set to NON purge on the target cluster. This is to prevent the table data (being managed through S3 replication) from being dropped by subsequent sync runs where the tables might have changed.

Running hms-mirror

clusters:
  LEFT:
    environment: "LEFT"
    legacyHive: false
    hcfsNamespace: "s3a://<my_source_s3_bucket>"
    hiveServer2:
      # Recommend using a KNOX endpoint to remove need for Kerberos Authentication
      uri: "jdbc:hive2://<my_source_hs2_endpoint>"
      connectionProperties:
        user: "<user>"
        maxWaitMillis: "5000"
        password: "*****"
        maxTotal: "-1"
      jarFile: "<local_location_of_hive-jdbc_driver>"
  RIGHT:
    environment: "RIGHT"
    legacyHive: false
    hcfsNamespace: "s3a://<my_target_s3_bucket>"
    hiveServer2:
      uri: "jdbc:hive2://<my_target_hs2_endpoint>"
      connectionProperties:
        user: "<user>"
        maxWaitMillis: "5000"
        password: "*****"
        maxTotal: "-1"
      jarFile: "<local_location_of_hive-jdbc_driver>"
    partitionDiscovery:
      # Optional, but recommended it the cluster isn't overburdened.
      auto: true
      # Required if auto is false and/or you want to ensure the partitions are in sync after the 
      # transfer is made.
      initMSCK: true

On-Prem to Cloud (Direct)

On-Prem to Cloud (In-Direct)

Cloud to On-Prem (In-Direct)