Beyond prompts

Programming with LLMs for Data Practitioners

Julia Silge | Posit PBC

Hello!

@juliasilge

@juliasilge.com

juliasilge.com

Day 1: Build on AI

  1. Talking with LLMs via code
  2. Programming with LLMs
  3. Prompt engineering and RAG
  4. Beyond prompts (this session!)

Tool calling

Letting the model use your functions

  • The model decides when to call but you decide what is callable
  • The shape:
    • Function name, description, JSON schema for arguments
  • You allow the model to execute the function on your behalf and send the result back

Demo

Human-in-the-loop: the model writes its tool request as text, and a person runs it and reports back

# demo/04-manual-tools/04-manual-tools-app.R
shiny::runApp()

Letting the model use your functions

  • The model decides when to call but you decide what is callable
  • The shape:
    • Function name, description, JSON schema for arguments
  • You allow the model to execute the function on your behalf and send the result back

Demo

The automatic tool-calling loop: the library handles the round trips for you

# demo/04-weather-tool/04-weather-tool.qmd
chat.register_tool(get_weather)
chat.chat("What should I wear to SIAS 2026 in Provo, UT?")

Your turn

Activity

Open 13-quiz-game-tools and wire tool calling into the quiz show app:

  • Provide a function that plays a sound (R: beepr, Python: playsound)
  • Document it well and register it as a tool in the quiz show app
  • Watch the model learn when to play which sound
20:00

Tool calling UI for Shiny

Showing tool activity to users

  • Tool annotations can give each tool an icon and title
  • ContentToolResult can return a custom title and icon
  • Tool calls can update reactive state in your Shiny app

Your turn

Activity

Open 14-quiz-game-ui and progressively upgrade the quiz show app:

  • Add tool annotations: give each tool an icon and title
  • Return a ContentToolResult with a custom title and icon
  • Track answers and score in reactive value boxes that update from tool calls
  • (Optional) Add a “finish round” tool that displays the final score
15:00

querychat

Chatting with your data

  • A drop-in Shiny module that turns a data frame into a chat-driven explorer
  • The model writes the SQL or dplyr, and the app runs it!
  • Explore querychat on your own

MCP, agents, and more

A preview of Day 2

  • Model Context Protocol: a common standard for exposing tools and resources to LLM clients
    • Write a server once, use it from many clients (Positron, VS Code, Claude Desktop, …)
  • Agent: an LLM with a read tool and a write tool
    • Autonomous, long-context, minimal intervention
  • We pick both of these up tomorrow, wiring MCP servers into an AI coding assistant and understanding agents

Where are we headed?

Sitting with the tension

Hopes

  • Lower the barrier to programming and reproducibility for many more people
  • Translation opens the ecosystem beyond English speakers
  • Voice input can remove the typing-fluency gate
  • “Play in everyone’s backyard” like Tukey! Shallow expertise in basically any new domain
  • Productivity, perhaps 2-5x faster, with careful vetting to keep quality

Fears

  • Environmental cost: data-center energy and water
  • Copyright: industrial-scale training on others’ work
  • Wealth concentration toward a few companies
  • “Never really learn”: shallow engagement where skills don’t grow
  • Access: premium tools need US dollars, unlike open source tools

Your turn

Activity

  • What is your biggest hope and biggest fear about working with AI?
  • Discuss with a neighbor
05:00

What we pick up on Day 2

  • What do these tools change about your day-to-day work? How can you build on them?
  • Watch out for: cost, lock-in, evaluation, trust
  • Build with AI: AI coding assistants, from your editor to autonomous agents

Thanks!

@juliasilge

@juliasilge.com

juliasilge.com

Illustrations from unDraw