Data maintenance #

From the Starburst Galaxy navigation menu, select Data > Data maintenance.

Data maintenance jobs run tasks that improve performance and reduce storage in Apache Iceberg tables. Supported tasks include data file compaction, statistics collection, and deletion of outdated snapshots and orphaned files.

To perform data maintenance operations on live tables, see data maintenance for Kafka streaming ingestion and data maintenance for file ingestion

The Data maintenance pane has the following levels:

• Top: Shows a list of catalogs that include at least one maintenance task.
• Catalog: For the current catalog, shows a list of its schemas with maintenance tasks.
• Schema: For the current schema, shows a list of its tables with maintenance tasks. You can create, edit, or delete tasks that apply to the entire schema from this level.
• Table: Shows the current table’s defined maintenance tasks and run history. From this level, you can run, edit, or delete the current data maintenance job.

Create data maintenance job #

To create a data maintenance job, click Create maintenance task in the top, catalog, or schema details levels.

Provide the following information in the Configure data maintenance dialog:

• In the Maintenance target section:
  • Specify a catalog and schema from the respective drop-down menus. If you opened this dialog from the catalog or schema details levels, those fields are pre-selected.
  • Select the All tables radio button to include all tables. This selection automatically applies maintenance tasks to all future Iceberg tables created as part of this schema. This selection also automatically stops maintenance jobs for tables that are removed. To include tables with separate schedules, see Edit data maintenance jobs.
  • Select the Select tables radio button to invoke the table drop-down menu. Expand the menu, and select one or multiple tables.
    
    
• In the Maintenance tasks section, select at least one maintenance task:

• In the Execution details section, select an executing role and a cluster from the Select cluster the respective drop-down menus.

• In the Job schedule section:

  • Select a Time zone from the drop-down menu.
  • Choose the Select frequency or Enter cron expression recurring interval format.
    
    

  For Select frequency: Choose an hourly, daily, weekly, monthly, or annual schedule from the drop-down menu. The corresponding values depend on the schedule:

  • Hourly: Enter a value between 0 minutes and 59 minutes.
  • Daily: Enter a time in the format hh:mm, then specify AM or PM.
  • Weekly: Enter a time in the format hh:mm, specify AM or PM, then select one or more days of the week.
  • Monthly: Enter a time in the format hh:mm, specify AM or PM, then select a date.
  • Annually: Enter a month, day, hour, and minutes in the format MM/DD hh:mm. Specify AM or PM.
    
    

  For Enter cron expression: Enter the desired schedule in the form of a UNIX cron expression. For example, a cycle scheduled to run weekly at 9:30 AM on Monday, Wednesday, and Friday:

30 9 * * 1,3,5

• Click Save.

  
  

  ×

  

Data maintenance job details #

All scheduled data maintenance jobs are listed in the Data maintenance pane beginning at the top details level.

As with other panes in Starburst Galaxy, the top row of this pane provides catalog > schema > table breadcrumbs to show which detail level you are on. Click the names in the breadcrumb list to navigate among the levels.

Header sections #

The header for the catalog and schema details levels include the symbol key, which explains the task symbols:

• compress Compaction
• search_insights Profile and statistics
• deployed_code_history Snapshot expiration
• vacuum Delete orphan files

The Search field at the top, catalog, and schema details levels let you restrict the list to matching values.

The Last run status drop-down menu at the schema details level lets you restrict the list to jobs that are scheduled, running, completed, or failed.

The Maintenance task drop-down menu at the catalog and schema details levels lets you restrict the list to a single task type.

Top level details #

The header for the top details level shows the total number of catalogs with tasks, and provides a search bar and drop-down menus that let you customize which details appear in the list of catalogs.

To view data maintenance jobs of a certain status, use the Last run status drop-down menu. Use the search bar to find catalogs.

The list of catalogs has the following columns:

• Catalog: The name of the catalog with defined maintenance tasks.
• Schemas with maintenance: The total number of schema-level jobs.

• Tables with maintenance: The total number of table-level jobs.

  
  

  ×

  

Catalog level details #

To view catalog level details, click the name of a catalog from the top details level.

Catalog level details are organized in the following columns:

• Location: The specified schema.
• Tasks: The symbols representing the tasks included in the data maintenance job.
• Last run: The date and time the data maintenance job was last run.

• Schedule: The next scheduled run time.

  
  

  ×

  

Schema level details #

To view schema level details, click the name of a schema from the catalog details level. The schema level details list can include individual tables or maintenance tasks set up to run for all tables in a schema.

Schema level details have the following columns:

• Location: The tables with defined maintenance tasks.
• Status: An icon showing status of the data maintenance job:
  • hourglass_topQueued
  • check_circle Completed
  • error Failed
  • schedule Scheduled to run
  • sync Currently running
• Tasks: The symbols representing the tasks included in the data maintenance job.
• Last run: The date and time the data maintenance job was last run.
• Schedule: The next scheduled run time.
• The more_vertoptions menu.

The Status and Tasks columns remain blank until a task has been executed at least once.

Use the options menu to edit the task or to run it now without waiting for its start time.

×

Table level details #

For more information on individual data maintenance jobs, click a table name from the schema details level.

The title of the table details level task pane is the name of the table. The top portion at the pane provides a summary of the selected data maintenance job, a Run now button, and an options menu that allows you to edit the task.

The Task history section is organized in the following columns:

• Query ID: The unique identifier for the statement. Click the Query ID to view Query details.
• Task: The selected data maintenance task.
• Run ID: The unique identifier for the task run.
• Status: The status of the data maintenance job.
  • check Completed.
  • error Failed; the Debug link opens a dialog with information about the failed task.
• Started: When the data maintenance job started.

• Elapsed time: The duration of data maintenance job.

  
  

  ×

  

Manage data maintenance jobs #

All editing is performed at the schema and table details levels. The schema details level allows for bulk and individual job edits, while the table details level allows for individual job edits only.

Edit data maintenance jobs #

To make bulk edits, go to the schema details level, and follow these steps:

• Click themore_vertoptions menu in the header.
• Click Edit jobs.
• Click the top checkbox to select all tables or select individual tables by clicking their checkboxes.
• Click Bulk edit to invoke the Edit data maintenance dialog.
• Make changes and resolve any discrepancies in tasks and schedules in the selected set of tables.
• Click Save.

To make individual edits:

• At the schema details level:
  • Click themore_vertoptions menu in the row, then select Edit.
  • Make changes, then click Save.
    
    
• At the table details level:
  • Click themore_vertoptions menu in the header, then select Edit.
  • Make changes, then click Save.

Existing data maintenance jobs exclude tables that have separate maintenance schedules. To include these tables, delete the data maintenance job associated with them. After you delete the job, the previously excluded tables are included in the data maintenance job.

Delete data maintenance jobs #

To delete jobs in bulk, go to the schema details level, and follow these steps:

• Click themore_vertoptions menu in the header.
• Click Delete jobs.
• Click the top checkbox to select all tables or select individual tables by clicking their checkboxes.
• Select Bulk delete.
• In the dialog, Yes, delete.

To make individual deletes:

• At the schema details level:

  • Click themore_vertoptions menu in the row of the table of interest.
  • Select Delete job.
  • In the dialog, Yes, delete.
    
    

• At the table details level:

  • Click themore_vertoptions menu in the header.
  • Select Delete job.
  • In the dialog, Yes, delete.