fabric-cli

Spark pool examples

Type: .SparkPool

Supported commands: fab desc .SparkPool

• Navigating to a Spark pool
• Checking if a Spark pool exists
• Getting and querying Spark pool details
• Listing Spark pools
• Creating a Spark pool
• Removing a Spark pool
• Setting and updating Spark pool properties

Navigating to a Spark pool

Navigate to a Spark pool using cd command. Spark pools are included in .sparkpools within a workspace. You can use relative and absolute paths based on current context.

# use absolute path
fab:/ws1.Workspace$ cd /ws1.Workspace/.sparkpools
* Switched to '.sparkpools'

# use relative path
fab:/ws1.Workspace/.sparkpools$ cd pool1.SparkPool
* Switched to 'pool1.SparkPool'

Checking if a Spark pool exists

Verify if a Spark pool exists. Using exists on .sparkpools has no effect.

fab:/ws1.Workspace/.sparkpools$ exists pool1.SparkPool
* true

fab:/ws1.Workspace/.sparkpools$ exists pool1_.SparkPool
* false

Getting and querying Spark pool details

Retrieve and query Spark pool details with -q (JMESPath query). Output the result to a local path or Lakehouse/Files with -o.

# get JSON propertie
fab:/ws1.Workspace/.sparkpools$ get pool1.SparkPool

# export JMESPath query result
fab:/ws1.Workspace/.sparkpools$ get pool1.SparkPool -q . -o /tmp

Listing Spark pools

List existing Spark pools and their details using -l.

# get JSON properties
fab:/ws1.Workspace/.sparkpools$ ls

# query using JMESPath
fab:/ws1.Workspace/.sparkpools$ ls -l

Creating a Spark pool

Create a Spark pool. Use optional parameters with -P as needed; defaults apply if unspecified.

• Optional params: nodeSize (default: Small), autoScale.maxNodeCount (default: 1), autoScale.minNodeCount (default: 1). Parameters are case insensitive.
• nodeSize: Small, Medium, Large, XLarge, and XXLarge
• maxNodeCount >= minNodeCount

# Small single-node by default
fab:/ws1.Workspace/.sparkpools$ create pool2.SparkPool

# with parameters
fab:/ws1.Workspace/.sparkpools$ create pool3.SparkPool -P nodesize=Medium,autoScale.minnodecount=1,autoScale.maxnodecount=3

Removing a Spark pool

Remove a Spark pool. Use -f to force.

fab:/ws1.Workspace/.sparkpools$ rm pool2.SparkPool
fab:/ws1.Workspace/.sparkpools$ rm pool3.SparkPool -f

Setting and updating Spark pool properties

Set Spark pool properties. Use -f to force. Properties are case sensitive.

• Supported properties: name, nodeSize, autoScale.enabled, autoScale.minNodeCount and autoScale.maxNodeCount.

# rename pool
fab:/ws1.Workspace/.sparkpools$ set pool1.SparkPool -q name -i pool1ren
! Modifying properties may lead to unintended consequences
? Are you sure? Yes
Setting new property for 'pool1.SparkPool'...
* Spark Pool updated

# change node size
fab:/ws1.Workspace/.sparkpools$ set pool1ren.SparkPool -q nodeSize -i Medium -f

# enable auto-scale
fab:/ws1.Workspace/.sparkpools$ set pool1ren.SparkPool -q autoScale.enabled -i True -f

# change min/max nodes
fab:/ws1.Workspace/.sparkpools$ set pool1ren.SparkPool -q autoScale.maxNodeCount -i 3 -f
fab:/ws1.Workspace/.sparkpools$ set pool1ren.SparkPool -q autoScale.minNodeCount -i 2 -f

See all examples