Agent Native

spark-python-data-source

Build custom Python data sources for Apache Spark 4.0+ to connect with external systems like databases, APIs, and message queues.

by DatabricksOfficial publisherNot reviewed

Identity proven by GitHub's verified organisation badge (databricks)

Agents
Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, OpenCode, claude.ai and API
License
LicenseRef-Databricks (restricted)
Pin
e3df77bpinned Oct 9, 2026
Context cost
about 1,900 tokenswhen the skill loads
GitHub stars
345on databricks/databricks-agent-skills
Scripts
None found

What it does

“Build custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers/writers for external systems. Use this skill whenever someone wants to connect Spark to an external system (database, API, message queue, custom protocol), build a Spark connector or plugin in Python, implement a DataSourceReader or DataSourceWriter, pull data from or push data to a system via Spark, or work with the PySpark DataSource API in any way. Even if they just say "read from X in Spark" or "write DataFrame to Y" and there's no native connector, this skill applies.”
The author's description, from SKILL.md

Generated summary from the skill's own text

Build custom Python data sources for Apache Spark 4.0+ to connect with external systems like databases, APIs, and message queues. The skill provides patterns for batch and streaming readers/writers, partitioning strategies, authentication, type conversion, error handling, and testing using the PySpark DataSource API.

When to use it

  • Connecting Spark to a system without a native connector
  • Implementing batch or streaming reads/writes for external data sources
  • Building a Spark plugin or connector in Python
  • Migrating data between Spark and external systems via custom sources

When not to

  • Using existing official Spark connectors or dedicated libraries
  • Working with systems that already have native Spark support

Install

Restricted license: install from the publisher's repository and read its terms.

These commands install from the publisher's repository and need no account.

skills CLI

npx skills add databricks/databricks-agent-skills --skill spark-python-data-source

Installs from the repository's default branch, which may have changed since the commit we checked.

GitHub CLI 2.90 or later

gh skill install databricks/databricks-agent-skills spark-python-data-source@e3df77b3f0eb5d70392eb35c6761ca0f2929a4f6

Installs the commit we checked (e3df77b).

Copy by hand

Download the folder at e3df77b and put it in .claude/skills/spark-python-data-source for Claude Code and in .agents/skills/spark-python-data-source for Codex, Cursor, Gemini CLI, GitHub Copilot and OpenCode.

Skill folder on GitHub at e3df77b

Claude Code plugin

/plugin install databricks@claude-plugins-official

The publisher's plugin in Claude's official marketplace. It may hold more than this skill, and we have not pinned it.

Copy for your agent

A prompt for any coding agent. It installs this skill from GitHub at the pinned commit, copies the files as they are, and runs nothing unless you ask.

Prompt

Install the agent skill "spark-python-data-source" by Databricks.

Source, pinned to commit e3df77b3f0eb5d70392eb35c6761ca0f2929a4f6:
https://github.com/databricks/databricks-agent-skills/tree/e3df77b3f0eb5d70392eb35c6761ca0f2929a4f6/experimental/spark-python-data-source

1. Download that folder at exactly this commit. Do not use the default branch.
2. Copy every file unchanged, keeping the folder structure, into .claude/skills/spark-python-data-source/ for Claude Code and .agents/skills/spark-python-data-source/ for Codex, Cursor, Gemini CLI, GitHub Copilot and OpenCode. Create the folders if they do not exist.
3. Do not edit the files, and do not run any script in the folder unless I ask you to.
4. When you are done, list the files you copied and where you put them.

Its license is restricted or not stated. Show me the license terms from the repository before you copy anything.

Listing and checks: https://www.agentnative.dev/skills/databricks/spark-python-data-source

SKILL.md at the pin

The license of this skill does not let us show its files here.

View SKILL.md on GitHub at e3df77b

Files

12 files, 111 KB in total. Each file has a sha256 in the pin's manifest; the first 12 characters are shown.

Skill folder on GitHub at e3df77b
All 37 skills from Databricks

More from Databricks

  • databricks-agent-bricks

    This skill enables agents to create and manage Knowledge Assistants for document question-answering via RAG on Volumes, and Supervisor Agents that orchestrate…

    Claude Code, Codex, Cursor +3 · about 1.7K tokens · pinned 6 days ago

  • databricks-ai-functions

    This skill provides built-in Databricks AI Functions that add AI capabilities directly to SQL and PySpark pipelines, including classification, extraction…

    Claude Code, Codex, Cursor +3 · about 3.8K tokens · pinned 6 days ago

  • databricks-ai-runtime

    This skill enables agents to submit and manage GPU training workloads on Databricks serverless compute using the `databricks air` CLI.

    Claude Code, Codex, Cursor +4 · about 2.9K tokens · pinned 6 days ago

  • databricks-aibi-dashboards

    Creates Databricks AI/BI dashboards using JSON specifications with support for multiple visualization types.

    Claude Code, Codex, Cursor +3 · about 9.5K tokens · pinned 6 days ago

  • databricks-app-design

    This skill helps design the user experience of custom-code Databricks Apps built with AppKit and React that display data.

    Claude Code, Codex, Cursor +3 · about 1.6K tokens · pinned 6 days ago

  • databricks-apps

    Build custom interactive data apps on Databricks using AppKit (TypeScript/React framework) or other frameworks like Streamlit or FastAPI.

    Claude Code, Codex, Cursor +3 · about 5.5K tokens · pinned 6 days ago

All Data and databases skills

Related in Data and databases

  • gcp-spark

    This skill develops, optimizes, and executes PySpark and Spark SQL code on Google Cloud's Managed Spark (Dataproc Serverless and Clusters), using BigLake…

    by GoogleOfficial publisher

    Claude Code, Codex, Cursor +4 · about 2.9K tokens · pinned 6 days ago

  • alphagenome-single-variant-analysis

    Analyzes how genetic variants affect gene expression, chromatin accessibility, and splicing using the AlphaGenome API.

    by GoogleOfficial publisher

    Contains scripts

    Claude Code, Codex, Cursor +3 · about 2.8K tokens · pinned 6 days ago

  • alphagenome-variant-impact-score

    Score genetic variants using AlphaGenome Variant Impact models to assess functional impact through 18 biological feature attributions.

    by GoogleOfficial publisher

    Contains scripts

    Claude Code, Codex, Cursor +3 · about 4.1K tokens · pinned 6 days ago

  • chdb-datastore

    This skill enables working with tabular data using a pandas-compatible API backed by ClickHouse for faster execution.

    by ClickHouseOfficial publisher

    Contains scripts

    Claude Code, Codex, Cursor +3 · about 1.2K tokens · pinned 6 days ago

  • chdb-sql

    The skill enables running ClickHouse SQL queries on local files (parquet, csv, json), remote databases (Postgres, MySQL, MongoDB, ClickHouse Cloud), and cloud…

    by ClickHouseOfficial publisher

    Contains scripts

    Claude Code, Codex, Cursor +3 · about 960 tokens · pinned 6 days ago

  • blueprint

    Blueprint enables composing Apache Airflow DAGs from YAML templates using reusable Python task group classes with Pydantic validation.

    by AstronomerOfficial publisher

    Claude Code, Codex, Cursor +4 · about 8.3K tokens · pinned 6 days ago

Open roles at Databricks

5 of 136 open roles on the jobs board

See all 136 roles at Databricks on the jobs board

The skills listed here are third-party software, collected from public GitHub repositories as a catalog of recent skills and provided as is. Agent Native does not write or control them and is not responsible for any harm, loss or damage caused by installing or using them. Labels and checks describe what we saw at one commit and are not a guarantee. Read a skill's files before you run it. Terms of use