Spark SQL Formatter

Spark SQL is usually written inside something else — a triple-quoted Python string in a notebook, a Scala interpolation, a Databricks cell, an Airflow operator argument. Being embedded is what makes it get squashed onto one line, and being distributed is what makes it worth reading carefully: the difference between a partition-pruned scan and a full table read is a single predicate.

With the Spark dialect selected, the formatter parses the Hive-inherited syntax that generic parsers do not know: LATERAL VIEW EXPLODE, DISTRIBUTE BY, SORT BY, CLUSTER BY, INSERT OVERWRITE … PARTITION, and the Delta Lake MERGE INTO with its wildcard update and insert forms.

Hive lineage, and the clauses that come with it

Spark SQL began as a Hive-compatible layer and kept the syntax. That inheritance is why these clauses exist and why a generic SQL parser has never heard of them:

• LATERAL VIEW EXPLODE. The Hive way to expand an array column into rows, given its own top-level line with the generated table and column aliases attached. Spark also supports explode() directly in the select list, but LATERAL VIEW remains everywhere in inherited code.
• DISTRIBUTE BY and SORT BY. Each gets its own clause line. DISTRIBUTE BY controls which partition a row lands in; SORT BY orders rows within each partition, unlike ORDER BY, which forces a global sort through a single reducer. Getting those two confused is a classic cause of a job that either runs slowly or produces almost-sorted output.
• CLUSTER BY. Shorthand for DISTRIBUTE BY x SORT BY x on the same expression. Formatted as its own clause so it is not mistaken for the Databricks table property of the same name.
• INSERT OVERWRITE TABLE … PARTITION (…). The partition specification stays with the target table, above the SELECT that supplies the rows — which is the layout that makes it obvious which partition you are about to replace.

Delta Lake MERGE INTO

Delta MERGE INTO is the workhorse of Lakehouse upserts, and it accepts path-style targets such as delta.`/mnt/silver/users` as well as catalog table names. The backtick-quoted path is preserved as a single identifier, slashes and all.

Be aware of one honest limitation: the Spark dialect does not break a MERGE INTO into one line per WHEN branch the way the T-SQL dialect does. A merge with several conditional branches stays largely on one line. The statement is emitted correctly and unchanged, but if you want branch-per-line layout you will still add the line breaks by hand. The SELECT statements around it are formatted fully.

Databricks SQL, and what the tool does not know

Databricks SQL is Spark SQL plus extensions, so this dialect covers the great majority of it. Unity Catalog three-part names such as catalog.schema.table format as ordinary qualified references. Statements built out of Databricks-specific keywords the parser does not recognise are still emitted, just without dedicated line breaks.

What the tool cannot do is anything runtime. It has no cluster, no catalog and no statistics: it will not tell you that a join is going to broadcast, that your partitions are skewed, or that a table has too many small files. It is a layout tool, plus a light check for unbalanced quotes and parentheses, and it never leaves your browser.