Agent Native

databricks-spark-structured-streaming

This skill provides patterns and best practices for building production Spark Structured Streaming pipelines on Databricks.

by DatabricksOfficial publisherNot reviewed

Identity proven by GitHub's verified organisation badge (databricks)

Agents
Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, OpenCode
License
LicenseRef-Databricks (restricted)
Pin
e3df77bpinned Oct 9, 2026
Context cost
about 810 tokenswhen the skill loads
GitHub stars
345on databricks/databricks-agent-skills
Scripts
None found

What it does

“Comprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks, or tuning streaming cost and performance.”
The author's description, from SKILL.md

Generated summary from the skill's own text

This skill provides patterns and best practices for building production Spark Structured Streaming pipelines on Databricks. It covers Kafka ingestion, Real-Time Mode, stream joins, stateful operations, checkpoints, and multi-sink writes with detailed reference guides and configuration options.

When to use it

  • Building Kafka to Delta or Kafka to Kafka streaming pipelines
  • Implementing stateful operations with watermarks and state stores
  • Configuring stream-stream or stream-static joins for real-time processing
  • Optimizing checkpoint management and trigger selection for cost and performance

When not to

  • When batch processing is sufficient for your latency requirements
  • Without access to Databricks CLI version 1.0.0 or higher

Install

Restricted license: install from the publisher's repository and read its terms.

These commands install from the publisher's repository and need no account.

skills CLI

npx skills add databricks/databricks-agent-skills --skill databricks-spark-structured-streaming

Installs from the repository's default branch, which may have changed since the commit we checked.

GitHub CLI 2.90 or later

gh skill install databricks/databricks-agent-skills databricks-spark-structured-streaming@e3df77b3f0eb5d70392eb35c6761ca0f2929a4f6

Installs the commit we checked (e3df77b).

Copy by hand

Download the folder at e3df77b and put it in .claude/skills/databricks-spark-structured-streaming for Claude Code and in .agents/skills/databricks-spark-structured-streaming for Codex, Cursor, Gemini CLI, GitHub Copilot and OpenCode.

Skill folder on GitHub at e3df77b

Claude Code plugin

/plugin install databricks@claude-plugins-official

The publisher's plugin in Claude's official marketplace. It may hold more than this skill, and we have not pinned it.

Copy for your agent

A prompt for any coding agent. It installs this skill from GitHub at the pinned commit, copies the files as they are, and runs nothing unless you ask.

Prompt

Install the agent skill "databricks-spark-structured-streaming" by Databricks.

Source, pinned to commit e3df77b3f0eb5d70392eb35c6761ca0f2929a4f6:
https://github.com/databricks/databricks-agent-skills/tree/e3df77b3f0eb5d70392eb35c6761ca0f2929a4f6/plugins/databricks/claude/skills/databricks-spark-structured-streaming

1. Download that folder at exactly this commit. Do not use the default branch.
2. Copy every file unchanged, keeping the folder structure, into .claude/skills/databricks-spark-structured-streaming/ for Claude Code and .agents/skills/databricks-spark-structured-streaming/ for Codex, Cursor, Gemini CLI, GitHub Copilot and OpenCode. Create the folders if they do not exist.
3. Do not edit the files, and do not run any script in the folder unless I ask you to.
4. When you are done, list the files you copied and where you put them.

Its license is restricted or not stated. Show me the license terms from the repository before you copy anything.

Listing and checks: https://www.agentnative.dev/skills/databricks/databricks-spark-structured-streaming

SKILL.md at the pin

The license of this skill does not let us show its files here.

View SKILL.md on GitHub at e3df77b

Files

15 files, 176 KB in total. Each file has a sha256 in the pin's manifest; the first 12 characters are shown.

Skill folder on GitHub at e3df77b

Also in

The same SKILL.md content appears in these places too. We list it once and credit the oldest public source.

All 37 skills from Databricks

More from Databricks

  • databricks-agent-bricks

    This skill enables agents to create and manage Knowledge Assistants for document question-answering via RAG on Volumes, and Supervisor Agents that orchestrate…

    Claude Code, Codex, Cursor +3 · about 1.7K tokens · pinned 6 days ago

  • databricks-ai-functions

    This skill provides built-in Databricks AI Functions that add AI capabilities directly to SQL and PySpark pipelines, including classification, extraction…

    Claude Code, Codex, Cursor +3 · about 3.8K tokens · pinned 6 days ago

  • databricks-ai-runtime

    This skill enables agents to submit and manage GPU training workloads on Databricks serverless compute using the `databricks air` CLI.

    Claude Code, Codex, Cursor +4 · about 2.9K tokens · pinned 6 days ago

  • databricks-aibi-dashboards

    Creates Databricks AI/BI dashboards using JSON specifications with support for multiple visualization types.

    Claude Code, Codex, Cursor +3 · about 9.5K tokens · pinned 6 days ago

  • databricks-app-design

    This skill helps design the user experience of custom-code Databricks Apps built with AppKit and React that display data.

    Claude Code, Codex, Cursor +3 · about 1.6K tokens · pinned 6 days ago

  • databricks-apps

    Build custom interactive data apps on Databricks using AppKit (TypeScript/React framework) or other frameworks like Streamlit or FastAPI.

    Claude Code, Codex, Cursor +3 · about 5.5K tokens · pinned 6 days ago

All Data and databases skills

Related in Data and databases

  • databricks-ml-training

    Train machine learning models on Databricks using MLflow, XGBoost, scikit-learn, LightGBM, or PyTorch with Optuna hyperparameter tuning.

    by DatabricksOfficial publisher

    Claude Code, Codex, Cursor +3 · about 4.4K tokens · pinned 6 days ago

  • airflow-state-store

    Helps agents work with Airflow 3.3's task and asset state stores for persisting coordination state across task retries, implementing crash-safe external job…

    by AstronomerOfficial publisher

    Claude Code, Codex, Cursor +4 · about 5.9K tokens · pinned 6 days ago

  • databricks-core

    This skill enables the agent to authenticate with Databricks, manage CLI profiles, explore data catalogs and tables, and route to specialized product skills…

    by DatabricksOfficial publisher

    Contains scripts

    Claude Code, Codex, Cursor +3 · about 1.8K tokens · pinned 6 days ago

  • databricks-data-discovery

    This skill routes data discovery requests to Databricks Genie, enabling the agent to find tables and schemas, answer questions about data, and generate SQL…

    by DatabricksOfficial publisher

    Claude Code, Codex, Cursor +3 · about 2K tokens · pinned 6 days ago

  • databricks-dbsql

    This skill provides access to Databricks SQL advanced features including SQL scripting, stored procedures, materialized views, geospatial functions…

    by DatabricksOfficial publisher

    Claude Code, Codex, Cursor +3 · about 2.5K tokens · pinned 6 days ago

  • databricks-docs

    This skill provides reference access to Databricks documentation via llms.txt index to supplement other skills.

    by DatabricksOfficial publisher

    Claude Code, Codex, Cursor +3 · about 630 tokens · pinned 6 days ago

Open roles at Databricks

5 of 136 open roles on the jobs board

See all 136 roles at Databricks on the jobs board

The skills listed here are third-party software, collected from public GitHub repositories as a catalog of recent skills and provided as is. Agent Native does not write or control them and is not responsible for any harm, loss or damage caused by installing or using them. Labels and checks describe what we saw at one commit and are not a guarantee. Read a skill's files before you run it. Terms of use