Help & Documentation
Everything you need to get started and get the most out of Data Shepherd.
Getting Started
Your first transformation
- Go to Transformations → New Transformation
- Upload your source file — CSV, JSON, EDI, TXT, or fixed-width. If you have a spec document (PDF, Word) that describes the file format, upload it as a Data Dictionary for better results.
- Optionally upload a target file if you need to map one format to another.
- Choose Quick Mode for a single prompt, or Chat Mode to refine the transformation conversationally.
- Preview the output on your sample data. If something is off, tell the agent what to fix.
- Save the transformation. It's now ready to run on any file with the same structure.
Running a saved transformation
Open any saved transformation and click Run. Upload the file you want to transform, and Data Shepherd will execute the scripts and return the result. You can download the output or view it in the browser.
Same transformation, same output, every time — the scripts are deterministic, so you get consistent results regardless of when you run them.
Supported File Formats
You can also upload PDF, Word, and Excel files as data dictionaries or supplemental context — these help the AI understand your file format and produce more accurate results.
Using the Agent Chat
Chat Mode lets you build transformations conversationally. Describe what you need in plain English, and the agent will analyze your file, propose a transformation plan, and generate the scripts.
Tips for better results
- Be specific about field names and output format — "combine first_name and last_name into full_name" beats "merge the name columns"
- Upload a data dictionary if you have one. Spec documents, PDFs, and Word files all work.
- If the preview isn't right, tell the agent what's wrong. It will revise and show you a new preview.
- You can upload additional files mid-conversation if the agent needs more context.
- For complex mappings, upload both a source and target sample so the agent can see exactly what you need.
What the agent will ask you to approve
- Data dictionaries — the agent extracts field definitions from your documents and asks you to confirm before proceeding.
- Transformation plans — before generating scripts, the agent shows you what it intends to do. Approve or request changes.
How Your Data Stays Private
Data Shepherd is designed so that AI never processes your actual data records.
- AI sees structure, not content. The AI analyzes a small sample to understand field names, formats, and types. Your full dataset is never sent to the AI model.
- Scripts are deterministic. Once generated, the scripts run server-side without AI involvement. Same input, same output, every time.
- You approve before anything runs. Preview the output on sample data first. Nothing touches your full dataset until you say so.
- Data stays on the platform. Files are processed in isolated server environments and are never shared with third parties.
Connectors & Schedules
Connectors
Connectors link Data Shepherd to your cloud storage so transformations can run automatically. Supported types:
- SFTP — authenticate with password or SSH private key
- FTPS — FTP over SSL/TLS with password authentication
- Amazon S3 — authenticate with access key and secret
- Azure Blob Storage — authenticate with connection string or SAS token
You can test connectivity before saving. Credentials are stored securely and never exposed in the UI after creation.
Schedules
Schedules run transformations automatically on a recurring basis. Connect a source (where files arrive), a transformation (what to do with them), and optionally a destination (where to send the output).
- Set a file pattern (e.g.,
*.csvorreport_*.json) to pick up specific files. - Choose what happens after pickup — archive, move, delete, or leave in place.
- Skip duplicates to avoid reprocessing files you've already handled.
- Pro and Max plans support encryption (PGP/GPG or AES-256) for both source decryption and destination encryption.
Auto-Heal
Source data changes — columns get renamed, formats shift, new fields appear. Auto-heal detects when a saved transformation fails on new data and automatically generates revised scripts to handle the changes.
- Enable or disable auto-heal per transformation from its detail page.
- When a job is auto-healed, you'll see a summary of what changed and can choose to accept the updated version.
- Available on Pro and Max plans.
API & Webhooks
Integrate Data Shepherd into your own systems with the REST API. Run transformations, check job results, and monitor webhooks programmatically.
- Create API keys in Settings → API Keys & Webhooks with scoped permissions (read, run).
- API keys are for running transformations and monitoring results. Creating and managing transformations, schedules, and connectors is done on the site.
- Set up webhooks on the site to get notified when jobs complete or fail. Monitor webhook deliveries via the API.
- API documentation is available in the API Keys & Webhooks settings tab.
- Available on Growth, Pro, and Max plans.
Account & Billing
- Upgrade or downgrade anytime from Settings → Subscription. Changes are prorated.
- Manage billing (update card, download invoices) through the Stripe billing portal, accessible from Settings.
- View usage on your Dashboard — saved transformations, jobs this period, and generations remaining.
- Email notifications can be configured in Settings → Notifications for job completions, failures, and weekly summaries.
- Delete your account from Settings → Account. This permanently removes all transformations, job history, and subscription data.
Plan Limits
| Free | Growth | Pro | Max | |
|---|---|---|---|---|
| Price | $0 | $19/mo | $59/mo | $249/mo |
| Transformations | 3 | 15 | 50 | 200 |
| Runs / month | 25 | 75 | 150 | Unlimited |
| Max file size | 5 MB | 25 MB | 50 MB | 75 MB |
| API access | — | Yes | Yes | Yes |
| Connectors | — | 1 | 5 | Unlimited |
| Schedules | — | 1 | 10 | Unlimited |
| Auto-heal | — | — | Yes | Yes |
| Encryption | — | — | Yes | Yes |
Annual billing saves approximately 17%. See Pricing for full details.
Need Help?
If you run into something that isn't covered here, reach out at support@datashepherd.ai. We typically respond within one business day.