itziarZG/CasesBuides
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: add autoskills

Browse Source
This commit is contained in:
itziarZG
2026-07-21 19:35:31 +02:00
parent ffc170d054
commit 1f45866dc5
27 changed files with 5983 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/skills/accessibility/SKILL.md
+440
View File
@@ -0,0 +1,440 @@
---
name: accessibility
description: Audit and improve web accessibility following WCAG 2.2 guidelines. Use when asked to "improve accessibility", "a11y audit", "WCAG compliance", "screen reader support", "keyboard navigation", or "make accessible".
license: MIT
metadata:
author: web-quality-skills
version: "1.1"
---
# Accessibility (a11y)
Comprehensive accessibility guidelines based on WCAG 2.2 and Lighthouse accessibility audits. Goal: make content usable by everyone, including people with disabilities.
## WCAG Principles: POUR
| Principle | Description |
|-----------|-------------|
| **P**erceivable | Content can be perceived through different senses |
| **O**perable | Interface can be operated by all users |
| **U**nderstandable | Content and interface are understandable |
| **R**obust | Content works with assistive technologies |
## Conformance levels
| Level | Requirement | Target |
|-------|-------------|--------|
| **A** | Minimum accessibility | Must pass |
| **AA** | Standard compliance | Should pass (legal requirement in many jurisdictions) |
| **AAA** | Enhanced accessibility | Nice to have |
---
## Perceivable
### Text alternatives (1.1)
**Images require alt text:**
```html
<!-- ❌ Missing alt -->
<img src="chart.png">
<!-- ✅ Descriptive alt -->
<img src="chart.png" alt="Bar chart showing 40% increase in Q3 sales">
<!-- ✅ Decorative image (empty alt) -->
<img src="decorative-border.png" alt="" role="presentation">
<!-- ✅ Complex image with longer description -->
<figure>
<img src="infographic.png" alt="2024 market trends infographic"
aria-describedby="infographic-desc">
<figcaption id="infographic-desc">
<!-- Detailed description -->
</figcaption>
</figure>
```
**Icon buttons need accessible names:**
```html
<!-- ❌ No accessible name -->
<button><svg><!-- menu icon --></svg></button>
<!-- ✅ Using aria-label -->
<button aria-label="Open menu">
<svg aria-hidden="true"><!-- menu icon --></svg>
</button>
<!-- ✅ Using visually hidden text -->
<button>
<svg aria-hidden="true"><!-- menu icon --></svg>
<span class="visually-hidden">Open menu</span>
</button>
```
**Visually hidden class:**
```css
.visually-hidden {
position: absolute;
width: 1px;
height: 1px;
padding: 0;
margin: -1px;
overflow: hidden;
clip: rect(0, 0, 0, 0);
white-space: nowrap;
border: 0;
}
```
### Color contrast (1.4.3, 1.4.6)
| Text Size | AA minimum | AAA enhanced |
|-----------|------------|--------------|
| Normal text (< 18px / < 14px bold) | 4.5:1 | 7:1 |
| Large text (≥ 18px / ≥ 14px bold) | 3:1 | 4.5:1 |
| UI components & graphics | 3:1 | 3:1 |
```css
/* ❌ Low contrast (2.5:1) */
.low-contrast {
color: #999;
background: #fff;
}
/* ✅ Sufficient contrast (7:1) */
.high-contrast {
color: #333;
background: #fff;
}
/* ✅ Focus states need contrast too */
:focus-visible {
outline: 2px solid #005fcc;
outline-offset: 2px;
}
```
**Don't rely on color alone:**
```html
<!-- ❌ Only color indicates error -->
<input class="error-border">
<style>.error-border { border-color: red; }</style>
<!-- ✅ Color + icon + text -->
<div class="field-error">
<input aria-invalid="true" aria-describedby="email-error">
<span id="email-error" class="error-message">
<svg aria-hidden="true"><!-- error icon --></svg>
Please enter a valid email address
</span>
</div>
```
### Media alternatives (1.2)
```html
<!-- Video with captions -->
<video controls>
<source src="video.mp4" type="video/mp4">
<track kind="captions" src="captions.vtt" srclang="en" label="English" default>
<track kind="descriptions" src="descriptions.vtt" srclang="en" label="Descriptions">
</video>
<!-- Audio with transcript -->
<audio controls>
<source src="podcast.mp3" type="audio/mp3">
</audio>
<details>
<summary>Transcript</summary>
<p>Full transcript text...</p>
</details>
```
---
## Operable
### Keyboard accessible (2.1)
**All functionality must be keyboard accessible:**
```javascript
// ❌ Only handles click
element.addEventListener('click', handleAction);
// ✅ Handles both click and keyboard
element.addEventListener('click', handleAction);
element.addEventListener('keydown', (e) => {
if (e.key === 'Enter' || e.key === ' ') {
e.preventDefault();
handleAction();
}
});
```
**No keyboard traps.** Users must be able to Tab into and out of every component. Use the [modal focus trap pattern](references/A11Y-PATTERNS.md#modal-focus-trap) for dialogs—the native `<dialog>` element handles this automatically.
### Focus visible (2.4.7)
```css
/* ❌ Never remove focus outlines */
*:focus { outline: none; }
/* ✅ Use :focus-visible for keyboard-only focus */
:focus {
outline: none;
}
:focus-visible {
outline: 2px solid #005fcc;
outline-offset: 2px;
}
/* ✅ Or custom focus styles */
button:focus-visible {
box-shadow: 0 0 0 3px rgba(0, 95, 204, 0.5);
}
```
### Focus not obscured (2.4.11) — new in 2.2
When an element receives keyboard focus, it must not be entirely hidden by other author-created content such as sticky headers, footers, or overlapping panels. At Level AAA (2.4.12), no part of the focused element may be hidden.
```css
/* ✅ Account for sticky headers when scrolling to focused elements */
:target {
scroll-margin-top: 80px;
}
/* ✅ Ensure focused items clear fixed/sticky bars */
:focus {
scroll-margin-top: 80px;
scroll-margin-bottom: 60px;
}
```
### Skip links (2.4.1)
Provide a skip link so keyboard users can bypass repetitive navigation. See the [skip link pattern](references/A11Y-PATTERNS.md#skip-link) for full markup and styles.
### Target size (2.5.8) — new in 2.2
Interactive targets must be at least **24 × 24 CSS pixels** (AA). Exceptions: inline text links, elements where the browser controls the size, and targets where a 24px circle centered on the bounding box does not overlap another target.
```css
/* ✅ Minimum target size */
button,
[role="button"],
input[type="checkbox"] + label,
input[type="radio"] + label {
min-width: 24px;
min-height: 24px;
}
/* ✅ Comfortable target size (recommended 44×44) */
.touch-target {
min-width: 44px;
min-height: 44px;
display: inline-flex;
align-items: center;
justify-content: center;
}
```
### Dragging movements (2.5.7) — new in 2.2
Any action that requires dragging must have a single-pointer alternative (e.g., buttons, inputs). See the [dragging movements pattern](references/A11Y-PATTERNS.md#dragging-movements) for a sortable-list example.
### Timing (2.2)
```javascript
// Allow users to extend time limits
function showSessionWarning() {
const modal = createModal({
title: 'Session Expiring',
content: 'Your session will expire in 2 minutes.',
actions: [
{ label: 'Extend session', action: extendSession },
{ label: 'Log out', action: logout }
],
timeout: 120000
});
}
```
### Motion (2.3)
```css
/* Respect reduced motion preference */
@media (prefers-reduced-motion: reduce) {
*,
*::before,
*::after {
animation-duration: 0.01ms !important;
animation-iteration-count: 1 !important;
transition-duration: 0.01ms !important;
scroll-behavior: auto !important;
}
}
```
---
## Understandable
### Page language (3.1.1)
```html
<!-- ❌ No language specified -->
<html>
<!-- ✅ Language specified -->
<html lang="en">
<!-- ✅ Language changes within page -->
<p>The French word for hello is <span lang="fr">bonjour</span>.</p>
```
### Consistent navigation (3.2.3)
```html
<!-- Navigation should be consistent across pages -->
<nav aria-label="Main">
<ul>
<li><a href="/" aria-current="page">Home</a></li>
<li><a href="/products">Products</a></li>
<li><a href="/about">About</a></li>
</ul>
</nav>
```
### Consistent help (3.2.6) — new in 2.2
If a help mechanism (contact info, chat widget, FAQ link, self-help option) is repeated across multiple pages, it must appear in the **same relative order** each time. Users who rely on consistent placement shouldn't have to hunt for help on every page.
### Form labels (3.3.2)
Every input needs a programmatically associated label. See the [form labels pattern](references/A11Y-PATTERNS.md#form-labels) for explicit, implicit, and instructional examples.
### Error handling (3.3.1, 3.3.3)
Announce errors to screen readers with `role="alert"` or `aria-live`, set `aria-invalid="true"` on invalid fields, and focus the first error on submit. See the [error handling pattern](references/A11Y-PATTERNS.md#error-handling) for full markup and JS.
### Redundant entry (3.3.7) — new in 2.2
Don't force users to re-enter information they already provided in the same session. Auto-populate from earlier steps, or let users select from previously entered values. Exceptions: security re-confirmation and content that has expired.
```html
<!-- ✅ Auto-fill shipping address from billing -->
<fieldset>
<legend>Shipping address</legend>
<label>
<input type="checkbox" id="same-as-billing" checked>
Same as billing address
</label>
<!-- Fields auto-populated when checked -->
</fieldset>
```
### Accessible authentication (3.3.8) — new in 2.2
Login flows must not rely on cognitive function tests (e.g., remembering a password, solving a puzzle) unless at least one of:
- A copy-paste or autofill mechanism is available
- An alternative method exists (e.g., passkey, SSO, email link)
- The test uses object recognition or personal content (AA only; AAA removes this exception)
```html
<!-- ✅ Allow paste in password fields -->
<input type="password" id="password" autocomplete="current-password">
<!-- ✅ Offer passwordless alternatives -->
<button type="button">Sign in with passkey</button>
<button type="button">Email me a login link</button>
```
---
## Robust
### ARIA usage (4.1.2)
**Prefer native elements:**
```html
<!-- ❌ ARIA role on div -->
<div role="button" tabindex="0">Click me</div>
<!-- ✅ Native button -->
<button>Click me</button>
<!-- ❌ ARIA checkbox -->
<div role="checkbox" aria-checked="false">Option</div>
<!-- ✅ Native checkbox -->
<label><input type="checkbox"> Option</label>
```
**When ARIA is needed,** use the correct roles and states. See the [ARIA tabs pattern](references/A11Y-PATTERNS.md#aria-tabs) for a complete tablist example.
### Live regions (4.1.3)
Use `aria-live` regions to announce dynamic content changes without moving focus. See the [live regions pattern](references/A11Y-PATTERNS.md#live-regions-and-notifications) for markup and a `showNotification()` helper.
---
## Testing checklist
### Automated testing
```bash
# Lighthouse accessibility audit
npx lighthouse https://example.com --only-categories=accessibility
# axe-core
npm install @axe-core/cli -g
axe https://example.com
```
### Manual testing
- [ ] **Keyboard navigation:** Tab through entire page, use Enter/Space to activate
- [ ] **Screen reader:** Test with VoiceOver (Mac), NVDA (Windows), or TalkBack (Android)
- [ ] **Zoom:** Content usable at 200% zoom
- [ ] **High contrast:** Test with Windows High Contrast Mode
- [ ] **Reduced motion:** Test with `prefers-reduced-motion: reduce`
- [ ] **Focus order:** Logical and follows visual order
- [ ] **Target size:** Interactive elements meet 24×24px minimum
See the [screen reader commands reference](references/A11Y-PATTERNS.md#screen-reader-commands) for VoiceOver and NVDA shortcuts.
---
## Common issues by impact
### Critical (fix immediately)
1. Missing form labels
2. Missing image alt text
3. Insufficient color contrast
4. Keyboard traps
5. No focus indicators
### Serious (fix before launch)
1. Missing page language
2. Missing heading structure
3. Non-descriptive link text
4. Auto-playing media
5. Missing skip links
### Moderate (fix soon)
1. Missing ARIA labels on icons
2. Inconsistent navigation
3. Missing error identification
4. Timing without controls
5. Missing landmark regions
## References
- [WCAG 2.2 Quick Reference](https://www.w3.org/WAI/WCAG22/quickref/)
- [WAI-ARIA Authoring Practices](https://www.w3.org/WAI/ARIA/apg/)
- [Deque axe Rules](https://dequeuniversity.com/rules/axe/)
- [Web Quality Audit](../web-quality-audit/SKILL.md)
- [WCAG criteria reference](references/WCAG.md)
- [Accessibility code patterns](references/A11Y-PATTERNS.md)
.agents/skills/accessibility/references/A11Y-PATTERNS.md
+233
View File
@@ -0,0 +1,233 @@
# Accessibility Code Patterns
Practical, copy-paste-ready patterns for common accessibility requirements. Each pattern is self-contained and linked from the main [SKILL.md](../SKILL.md).
---
## Modal focus trap
Trap keyboard focus inside a modal dialog so Tab/Shift+Tab cycle through its focusable elements and Escape closes it.
```javascript
function openModal(modal) {
const focusableElements = modal.querySelectorAll(
'button, [href], input, select, textarea, [tabindex]:not([tabindex="-1"])'
);
const firstElement = focusableElements[0];
const lastElement = focusableElements[focusableElements.length - 1];
modal.addEventListener('keydown', (e) => {
if (e.key === 'Tab') {
if (e.shiftKey && document.activeElement === firstElement) {
e.preventDefault();
lastElement.focus();
} else if (!e.shiftKey && document.activeElement === lastElement) {
e.preventDefault();
firstElement.focus();
}
}
if (e.key === 'Escape') {
closeModal();
}
});
firstElement.focus();
}
```
The native `<dialog>` element handles focus trapping automatically—prefer it when browser support allows.
---
## Skip link
Allows keyboard users to bypass repetitive navigation and jump straight to main content.
```html
<body>
<a href="#main-content" class="skip-link">Skip to main content</a>
<header><!-- navigation --></header>
<main id="main-content" tabindex="-1">
<!-- main content -->
</main>
</body>
```
```css
.skip-link {
position: absolute;
top: -40px;
left: 0;
background: #000;
color: #fff;
padding: 8px 16px;
z-index: 100;
}
.skip-link:focus {
top: 0;
}
```
---
## Error handling
Announce errors to screen readers and focus the first invalid field on submit.
```html
<form novalidate>
<div class="field" aria-live="polite">
<label for="email">Email</label>
<input type="email" id="email"
aria-invalid="true"
aria-describedby="email-error">
<p id="email-error" class="error" role="alert">
Please enter a valid email address (e.g., name@example.com)
</p>
</div>
</form>
```
```javascript
form.addEventListener('submit', (e) => {
const firstError = form.querySelector('[aria-invalid="true"]');
if (firstError) {
e.preventDefault();
firstError.focus();
const errorSummary = document.getElementById('error-summary');
errorSummary.textContent =
`${errors.length} errors found. Please fix them and try again.`;
errorSummary.focus();
}
});
```
---
## Form labels
Every input needs an associated label—either explicit (`for`/`id`) or implicit (wrapping `<label>`).
```html
<!-- ❌ No label association -->
<input type="email" placeholder="Email">
<!-- ✅ Explicit label -->
<label for="email">Email address</label>
<input type="email" id="email" name="email"
autocomplete="email" required>
<!-- ✅ Implicit label -->
<label>
Email address
<input type="email" name="email" autocomplete="email" required>
</label>
<!-- ✅ With instructions -->
<label for="password">Password</label>
<input type="password" id="password"
aria-describedby="password-requirements">
<p id="password-requirements">
Must be at least 8 characters with one number.
</p>
```
---
## Dragging movements
Any action triggered by dragging must offer a single-pointer alternative (WCAG 2.5.7).
```html
<!-- ❌ Drag-only reorder -->
<ul class="sortable-list" draggable="true">
<li>Item 1</li>
<li>Item 2</li>
</ul>
<!-- ✅ Drag + button alternatives -->
<ul class="sortable-list">
<li>
<span>Item 1</span>
<button aria-label="Move Item 1 up">↑</button>
<button aria-label="Move Item 1 down">↓</button>
</li>
<li>
<span>Item 2</span>
<button aria-label="Move Item 2 up">↑</button>
<button aria-label="Move Item 2 down">↓</button>
</li>
</ul>
```
Also applies to sliders, map panning, colour pickers, and similar drag-based widgets—always provide an equivalent click/tap or keyboard path.
---
## ARIA tabs
Tabs require `role="tablist"`, `role="tab"`, and `role="tabpanel"` with proper `aria-selected`, `aria-controls`, and keyboard support.
```html
<div role="tablist" aria-label="Product information">
<button role="tab" id="tab-1" aria-selected="true"
aria-controls="panel-1">Description</button>
<button role="tab" id="tab-2" aria-selected="false"
aria-controls="panel-2" tabindex="-1">Reviews</button>
</div>
<div role="tabpanel" id="panel-1" aria-labelledby="tab-1">
<!-- Panel content -->
</div>
<div role="tabpanel" id="panel-2" aria-labelledby="tab-2" hidden>
<!-- Panel content -->
</div>
```
Arrow keys should move focus between tabs; the active tab receives `tabindex="0"` while inactive tabs use `tabindex="-1"`.
---
## Live regions and notifications
Use `aria-live` to announce dynamic content changes to screen readers without moving focus.
```html
<!-- Status updates (polite — waits for pause in speech) -->
<div aria-live="polite" aria-atomic="true" class="status">
<!-- Content updates announced to screen readers -->
</div>
<!-- Urgent alerts (assertive — interrupts) -->
<div role="alert" aria-live="assertive">
<!-- Interrupts current announcement -->
</div>
```
```javascript
function showNotification(message, type = 'polite') {
const container = document.getElementById(`${type}-announcer`);
container.textContent = '';
requestAnimationFrame(() => {
container.textContent = message;
});
}
```
Clear the container before writing to ensure the same message triggers a new announcement.
---
## Screen reader commands
Quick reference for the most common screen reader shortcuts.
| Action | VoiceOver (Mac) | NVDA (Windows) |
|--------|-----------------|----------------|
| Start/Stop | ⌘ + F5 | Ctrl + Alt + N |
| Next item | VO + → | ↓ |
| Previous item | VO + ← | ↑ |
| Activate | VO + Space | Enter |
| Headings list | VO + U, then arrows | H / Shift + H |
| Links list | VO + U | K / Shift + K |
.agents/skills/accessibility/references/WCAG.md
+191
View File
@@ -0,0 +1,191 @@
# WCAG 2.2 Quick Reference
## Success criteria by level
### Level A (minimum)
| Criterion | Description |
|-----------|-------------|
| **1.1.1** Non-text Content | All images, icons have text alternatives |
| **1.2.1** Audio-only/Video-only | Provide transcript or audio description |
| **1.2.2** Captions | Video with audio has captions |
| **1.2.3** Audio Description | Video has audio description |
| **1.3.1** Info and Relationships | Information conveyed through presentation is available programmatically |
| **1.3.2** Meaningful Sequence | Reading order is logical |
| **1.3.3** Sensory Characteristics | Instructions don't rely solely on shape, color, size, location, orientation, or sound |
| **1.4.1** Use of Color | Color is not the only visual means of conveying information |
| **1.4.2** Audio Control | Audio playing automatically can be paused/stopped |
| **2.1.1** Keyboard | All functionality available via keyboard |
| **2.1.2** No Keyboard Trap | Keyboard focus can be moved away from any component |
| **2.1.4** Character Key Shortcuts | Single-key shortcuts can be turned off or remapped |
| **2.2.1** Timing Adjustable | Time limits can be extended |
| **2.2.2** Pause, Stop, Hide | Moving/blinking content can be paused |
| **2.3.1** Three Flashes | Nothing flashes more than 3 times per second |
| **2.4.1** Bypass Blocks | Skip link or landmark navigation available |
| **2.4.2** Page Titled | Pages have descriptive titles |
| **2.4.3** Focus Order | Focus order preserves meaning |
| **2.4.4** Link Purpose | Link purpose clear from link text or context |
| **2.5.1** Pointer Gestures | Multi-point gestures have single-pointer alternatives |
| **2.5.2** Pointer Cancellation | Down-event doesn't trigger action (use up-event or click) |
| **2.5.3** Label in Name | Accessible name contains visible label text |
| **2.5.4** Motion Actuation | Motion-triggered functions have alternatives |
| **3.1.1** Language of Page | Default language specified in HTML |
| **3.2.1** On Focus | Focus doesn't trigger unexpected changes |
| **3.2.2** On Input | Input doesn't trigger unexpected changes |
| **3.2.6** Consistent Help | Help mechanisms appear in the same relative order across pages |
| **3.3.1** Error Identification | Input errors clearly described |
| **3.3.2** Labels or Instructions | Form inputs have labels or instructions |
| **3.3.7** Redundant Entry | Information previously entered is auto-populated or available to select |
| **4.1.2** Name, Role, Value | UI components have accessible names and correct roles |
### Level AA (standard)
| Criterion | Description |
|-----------|-------------|
| **1.2.4** Captions (Live) | Live audio has captions |
| **1.2.5** Audio Description | Pre-recorded video has audio description |
| **1.3.4** Orientation | Content doesn't restrict orientation |
| **1.3.5** Identify Input Purpose | Input purpose can be programmatically determined |
| **1.4.3** Contrast (Minimum) | 4.5:1 for normal text, 3:1 for large text |
| **1.4.4** Resize Text | Text can be resized to 200% without loss of functionality |
| **1.4.5** Images of Text | Text used instead of images of text |
| **1.4.10** Reflow | Content reflows at 320px width without horizontal scroll |
| **1.4.11** Non-text Contrast | UI components have 3:1 contrast |
| **1.4.12** Text Spacing | Content adapts to text spacing changes |
| **1.4.13** Content on Hover/Focus | Additional content is dismissible, hoverable, persistent |
| **2.4.5** Multiple Ways | Multiple ways to find pages |
| **2.4.6** Headings and Labels | Headings and labels are descriptive |
| **2.4.7** Focus Visible | Focus indicator is visible |
| **2.4.11** Focus Not Obscured (Minimum) | Focused element is not entirely hidden by author-created content |
| **2.5.7** Dragging Movements | Dragging actions have single-pointer alternatives |
| **2.5.8** Target Size (Minimum) | Interactive targets are at least 24×24 CSS pixels (with exceptions) |
| **3.1.2** Language of Parts | Language changes are marked |
| **3.2.3** Consistent Navigation | Navigation is consistent across pages |
| **3.2.4** Consistent Identification | Same functionality uses same labels |
| **3.3.3** Error Suggestion | Error corrections suggested when known |
| **3.3.4** Error Prevention (Legal) | Actions can be reversed or confirmed |
| **3.3.8** Accessible Authentication (Minimum) | No cognitive function test for login unless an alternative or assistance is provided |
| **4.1.3** Status Messages | Status messages announced to screen readers |
### Level AAA (enhanced)
| Criterion | Description |
|-----------|-------------|
| **1.4.6** Contrast (Enhanced) | 7:1 for normal text, 4.5:1 for large text |
| **1.4.8** Visual Presentation | Foreground/background colors can be selected |
| **1.4.9** Images of Text (No Exception) | No images of text |
| **2.1.3** Keyboard (No Exception) | All functionality keyboard accessible |
| **2.2.3** No Timing | No time limits |
| **2.2.4** Interruptions | Interruptions can be postponed |
| **2.2.5** Re-authenticating | Data preserved on re-authentication |
| **2.2.6** Timeouts | Users warned about data loss from inactivity |
| **2.3.2** Three Flashes | No content flashes more than 3 times |
| **2.3.3** Animation from Interactions | Motion animation can be disabled |
| **2.4.8** Location | User location within site is available |
| **2.4.9** Link Purpose (Link Only) | Link purpose clear from link text alone |
| **2.4.10** Section Headings | Sections have headings |
| **2.4.12** Focus Not Obscured (Enhanced) | No part of the focused element is hidden by author-created content |
| **2.4.13** Focus Appearance | Focus indicator has sufficient area, contrast, and is not obscured |
| **3.1.3** Unusual Words | Definitions available for unusual words |
| **3.1.4** Abbreviations | Abbreviations expanded |
| **3.1.5** Reading Level | Alternative content for complex text |
| **3.1.6** Pronunciation | Pronunciation available where needed |
| **3.2.5** Change on Request | Changes initiated only by user |
| **3.3.5** Help | Context-sensitive help available |
| **3.3.6** Error Prevention (All) | All form submissions can be reviewed |
| **3.3.9** Accessible Authentication (Enhanced) | No cognitive function test for login (no object or personal content recognition exceptions) |
## Common ARIA patterns
### Buttons
```html
<button>Label</button>
<!-- or -->
<button aria-label="Close dialog">×</button>
```
### Links
```html
<a href="/page">Descriptive link text</a>
<!-- External links -->
<a href="https://external.com" target="_blank" rel="noopener">
External site
<span class="visually-hidden">(opens in new tab)</span>
</a>
```
### Form fields
```html
<label for="email">Email address</label>
<input type="email" id="email" aria-describedby="email-hint">
<p id="email-hint">We'll never share your email.</p>
```
### Error states
```html
<label for="email">Email</label>
<input type="email" id="email" aria-invalid="true" aria-describedby="email-error">
<p id="email-error" role="alert">Please enter a valid email address.</p>
```
### Navigation
```html
<nav aria-label="Main">
<ul>
<li><a href="/" aria-current="page">Home</a></li>
<li><a href="/about">About</a></li>
</ul>
</nav>
```
### Modals
```html
<div role="dialog" aria-modal="true" aria-labelledby="dialog-title">
<h2 id="dialog-title">Confirm Action</h2>
<!-- content -->
</div>
```
### Live regions
```html
<!-- Polite (waits for pause in speech) -->
<div aria-live="polite">Status update here</div>
<!-- Assertive (interrupts immediately) -->
<div aria-live="assertive" role="alert">Error message here</div>
<!-- Status (polite, implicit) -->
<div role="status">Loading complete</div>
```
## What changed from 2.1 to 2.2
| Change | Criterion | Level |
|--------|-----------|-------|
| **Removed** | 4.1.1 Parsing | A |
| **Added** | 2.4.11 Focus Not Obscured (Minimum) | AA |
| **Added** | 2.4.12 Focus Not Obscured (Enhanced) | AAA |
| **Added** | 2.4.13 Focus Appearance | AAA |
| **Added** | 2.5.7 Dragging Movements | AA |
| **Added** | 2.5.8 Target Size (Minimum) | AA |
| **Added** | 3.2.6 Consistent Help | A |
| **Added** | 3.3.7 Redundant Entry | A |
| **Added** | 3.3.8 Accessible Authentication (Minimum) | AA |
| **Added** | 3.3.9 Accessible Authentication (Enhanced) | AAA |
## Testing tools
| Tool | Type | URL |
|------|------|-----|
| axe DevTools | Browser extension | [deque.com/axe](https://www.deque.com/axe/) |
| WAVE | Browser extension | [wave.webaim.org](https://wave.webaim.org/) |
| Lighthouse | Built into Chrome | DevTools → Lighthouse |
| NVDA | Screen reader (Windows) | [nvaccess.org](https://www.nvaccess.org/) |
| VoiceOver | Screen reader (Mac) | Built into macOS |
| Colour Contrast Analyser | Desktop app | [tpgi.com](https://www.tpgi.com/color-contrast-checker/) |
## Sources
- [WCAG 2.2 W3C Recommendation](https://www.w3.org/TR/WCAG22/)
- [WCAG 2.2 Quick Reference](https://www.w3.org/WAI/WCAG22/quickref/)
- [What's New in WCAG 2.2](https://www.w3.org/WAI/standards-guidelines/wcag/new-in-22/)
.agents/skills/frontend-design/LICENSE.txt
+177
View File
@@ -0,0 +1,177 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
.agents/skills/frontend-design/SKILL.md
+42
View File
@@ -0,0 +1,42 @@
---
name: frontend-design
description: Create distinctive, production-grade frontend interfaces with high design quality. Use this skill when the user asks to build web components, pages, artifacts, posters, or applications (examples include websites, landing pages, dashboards, React components, HTML/CSS layouts, or when styling/beautifying any web UI). Generates creative, polished code and UI design that avoids generic AI aesthetics.
license: Complete terms in LICENSE.txt
---
This skill guides creation of distinctive, production-grade frontend interfaces that avoid generic "AI slop" aesthetics. Implement real working code with exceptional attention to aesthetic details and creative choices.
The user provides frontend requirements: a component, page, application, or interface to build. They may include context about the purpose, audience, or technical constraints.
## Design Thinking
Before coding, understand the context and commit to a BOLD aesthetic direction:
- **Purpose**: What problem does this interface solve? Who uses it?
- **Tone**: Pick an extreme: brutally minimal, maximalist chaos, retro-futuristic, organic/natural, luxury/refined, playful/toy-like, editorial/magazine, brutalist/raw, art deco/geometric, soft/pastel, industrial/utilitarian, etc. There are so many flavors to choose from. Use these for inspiration but design one that is true to the aesthetic direction.
- **Constraints**: Technical requirements (framework, performance, accessibility).
- **Differentiation**: What makes this UNFORGETTABLE? What's the one thing someone will remember?
**CRITICAL**: Choose a clear conceptual direction and execute it with precision. Bold maximalism and refined minimalism both work - the key is intentionality, not intensity.
Then implement working code (HTML/CSS/JS, React, Vue, etc.) that is:
- Production-grade and functional
- Visually striking and memorable
- Cohesive with a clear aesthetic point-of-view
- Meticulously refined in every detail
## Frontend Aesthetics Guidelines
Focus on:
- **Typography**: Choose fonts that are beautiful, unique, and interesting. Avoid generic fonts like Arial and Inter; opt instead for distinctive choices that elevate the frontend's aesthetics; unexpected, characterful font choices. Pair a distinctive display font with a refined body font.
- **Color & Theme**: Commit to a cohesive aesthetic. Use CSS variables for consistency. Dominant colors with sharp accents outperform timid, evenly-distributed palettes.
- **Motion**: Use animations for effects and micro-interactions. Prioritize CSS-only solutions for HTML. Use Motion library for React when available. Focus on high-impact moments: one well-orchestrated page load with staggered reveals (animation-delay) creates more delight than scattered micro-interactions. Use scroll-triggering and hover states that surprise.
- **Spatial Composition**: Unexpected layouts. Asymmetry. Overlap. Diagonal flow. Grid-breaking elements. Generous negative space OR controlled density.
- **Backgrounds & Visual Details**: Create atmosphere and depth rather than defaulting to solid colors. Add contextual effects and textures that match the overall aesthetic. Apply creative forms like gradient meshes, noise textures, geometric patterns, layered transparencies, dramatic shadows, decorative borders, custom cursors, and grain overlays.
NEVER use generic AI-generated aesthetics like overused font families (Inter, Roboto, Arial, system fonts), cliched color schemes (particularly purple gradients on white backgrounds), predictable layouts and component patterns, and cookie-cutter design that lacks context-specific character.
Interpret creatively and make unexpected choices that feel genuinely designed for the context. No design should be the same. Vary between light and dark themes, different fonts, different aesthetics. NEVER converge on common choices (Space Grotesk, for example) across generations.
**IMPORTANT**: Match implementation complexity to the aesthetic vision. Maximalist designs need elaborate code with extensive animations and effects. Minimalist or refined designs need restraint, precision, and careful attention to spacing, typography, and subtle details. Elegance comes from executing the vision well.
Remember: Claude is capable of extraordinary creative work. Don't hold back, show what can truly be created when thinking outside the box and committing fully to a distinctive vision.
.agents/skills/pandas-data-analysis/SKILL.md
+279
View File
@@ -0,0 +1,279 @@
---
name: Pandas Data Analysis
description: Master data manipulation, analysis, and visualization with Pandas, NumPy, and Matplotlib
version: "2.1.0"
sasmp_version: "1.3.0"
bonded_agent: 03-data-science
bond_type: PRIMARY_BOND
# Skill Configuration
retry_strategy: exponential_backoff
observability:
logging: true
metrics: data_processing_time
---
# Pandas Data Analysis
## Overview
Master data analysis with Pandas, the powerful Python library for data manipulation and analysis. Learn to clean, transform, analyze, and visualize data effectively.
## Learning Objectives
- Load and manipulate data from various sources (CSV, Excel, SQL, APIs)
- Clean and transform messy datasets
- Perform exploratory data analysis (EDA)
- Aggregate and group data for insights
- Create compelling visualizations
- Optimize performance for large datasets
## Core Topics
### 1. Pandas DataFrames & Series
- Creating DataFrames from various sources
- Indexing and selecting data (loc, iloc, at, iat)
- Filtering and boolean indexing
- Adding/removing columns and rows
- Data types and conversions
**Code Example:**
```python
import pandas as pd
import numpy as np
# Create DataFrame
data = {
'name': ['Alice', 'Bob', 'Charlie', 'David'],
'age': [25, 30, 35, 28],
'salary': [50000, 60000, 75000, 55000],
'department': ['IT', 'HR', 'IT', 'Sales']
}
df = pd.DataFrame(data)
# Indexing and filtering
it_employees = df[df['department'] == 'IT']
high_earners = df.loc[df['salary'] > 55000, ['name', 'salary']]
# Adding calculated columns
df['annual_bonus'] = df['salary'] * 0.10
df['age_group'] = pd.cut(df['age'], bins=[0, 30, 40, 100], labels=['Young', 'Mid', 'Senior'])
print(df)
```
### 2. Data Cleaning & Transformation
- Handling missing data (dropna, fillna, interpolate)
- Removing duplicates
- String operations and text cleaning
- Date/time parsing and manipulation
- Type conversions and casting
- Applying custom functions (apply, map, applymap)
**Code Example:**
```python
import pandas as pd
# Load data with missing values
df = pd.read_csv('sales_data.csv')
# Handle missing values
df['price'].fillna(df['price'].median(), inplace=True)
df['category'].fillna('Unknown', inplace=True)
df.dropna(subset=['customer_id'], inplace=True)
# Clean text data
df['product_name'] = df['product_name'].str.strip().str.lower()
df['product_name'] = df['product_name'].str.replace('[^a-zA-Z0-9 ]', '', regex=True)
# Convert dates
df['order_date'] = pd.to_datetime(df['order_date'])
df['year'] = df['order_date'].dt.year
df['month'] = df['order_date'].dt.month
# Remove duplicates
df.drop_duplicates(subset=['order_id'], keep='first', inplace=True)
# Apply custom function
def categorize_price(price):
if price < 50:
return 'Low'
elif price < 100:
return 'Medium'
else:
return 'High'
df['price_category'] = df['price'].apply(categorize_price)
```
### 3. Aggregation & Grouping
- GroupBy operations
- Aggregation functions (sum, mean, count, etc.)
- Pivot tables and cross-tabulation
- Multi-level indexing
- Window functions (rolling, expanding)
**Code Example:**
```python
import pandas as pd
# Sample sales data
df = pd.read_csv('sales.csv')
# GroupBy aggregation
dept_stats = df.groupby('department').agg({
'salary': ['mean', 'min', 'max'],
'employee_id': 'count'
})
# Multiple groupby
sales_by_region_product = df.groupby(['region', 'product_category'])['sales'].sum()
# Pivot table
pivot = df.pivot_table(
values='sales',
index='product_category',
columns='quarter',
aggfunc='sum',
fill_value=0
)
# Rolling window (moving average)
df['sales_ma_7d'] = df.groupby('product_id')['sales'].transform(
lambda x: x.rolling(window=7, min_periods=1).mean()
)
# Cumulative sum
df['cumulative_sales'] = df.groupby('product_id')['sales'].cumsum()
```
### 4. Data Visualization
- Matplotlib basics
- Seaborn for statistical plots
- Pandas built-in plotting
- Customizing plots
- Creating dashboards
**Code Example:**
```python
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
# Set style
sns.set_style('whitegrid')
# Load data
df = pd.read_csv('sales_data.csv')
# 1. Line plot - Sales trend over time
df.groupby('month')['sales'].sum().plot(kind='line', figsize=(10, 6))
plt.title('Monthly Sales Trend')
plt.xlabel('Month')
plt.ylabel('Total Sales ($)')
plt.show()
# 2. Bar plot - Sales by category
category_sales = df.groupby('category')['sales'].sum().sort_values(ascending=False)
category_sales.plot(kind='bar', figsize=(10, 6))
plt.title('Sales by Category')
plt.xlabel('Category')
plt.ylabel('Total Sales ($)')
plt.xticks(rotation=45)
plt.show()
# 3. Histogram - Price distribution
df['price'].hist(bins=30, figsize=(10, 6))
plt.title('Price Distribution')
plt.xlabel('Price ($)')
plt.ylabel('Frequency')
plt.show()
# 4. Box plot - Salary by department
df.boxplot(column='salary', by='department', figsize=(10, 6))
plt.title('Salary Distribution by Department')
plt.suptitle('')
plt.show()
# 5. Heatmap - Correlation matrix
corr = df[['age', 'salary', 'years_experience']].corr()
sns.heatmap(corr, annot=True, cmap='coolwarm', center=0)
plt.title('Correlation Matrix')
plt.show()
```
## Hands-On Practice
### Project 1: Customer Analytics
Analyze customer purchase behavior and segmentation.
**Requirements:**
- Load customer transaction data
- Clean and prepare dataset
- Calculate RFM (Recency, Frequency, Monetary) metrics
- Customer segmentation
- Visualize insights
- Generate executive summary
**Key Skills:** Data cleaning, aggregation, visualization
### Project 2: Time Series Analysis
Analyze sales trends and forecast future performance.
**Requirements:**
- Load time series data
- Handle missing dates
- Calculate moving averages
- Identify trends and seasonality
- Detect anomalies
- Create interactive visualizations
**Key Skills:** Time series operations, rolling windows, plotting
### Project 3: Data Quality Report
Build automated data quality assessment tool.
**Requirements:**
- Check for missing values
- Identify duplicates
- Detect outliers
- Validate data types
- Generate quality metrics
- Export HTML report
**Key Skills:** Data validation, statistical analysis, reporting
## Assessment Criteria
- [ ] Load and clean real-world datasets efficiently
- [ ] Perform complex data transformations
- [ ] Use GroupBy for aggregations
- [ ] Create insightful visualizations
- [ ] Handle missing and inconsistent data
- [ ] Optimize performance for large datasets
- [ ] Document analysis with clear explanations
## Resources
### Official Documentation
- [Pandas Docs](https://pandas.pydata.org/docs/) - Official documentation
- [NumPy Docs](https://numpy.org/doc/) - NumPy documentation
- [Matplotlib Docs](https://matplotlib.org/) - Plotting library
### Learning Platforms
- [Kaggle](https://www.kaggle.com/learn/pandas) - Free Pandas course
- [DataCamp](https://www.datacamp.com/courses/pandas-foundations) - Interactive courses
- [Python for Data Analysis](https://wesmckinney.com/book/) - Wes McKinney's book
### Tools
- [Jupyter Notebook](https://jupyter.org/) - Interactive development
- [Google Colab](https://colab.research.google.com/) - Cloud notebooks
- [Anaconda](https://www.anaconda.com/) - Data science distribution
## Next Steps
After mastering Pandas, explore:
- **Scikit-learn** - Machine learning
- **SQL** - Database querying
- **Apache Spark** - Big data processing
- **Tableau/Power BI** - Business intelligence tools
.agents/skills/pandas-data-analysis/assets/config.yaml
+1
View File
@@ -0,0 +1 @@
python_skill: pandas-data-analysis
.agents/skills/pandas-data-analysis/references/GUIDE.md
+1
View File
@@ -0,0 +1 @@
# pandas-data-analysis Guide
.agents/skills/pandas-data-analysis/scripts/helper.py
+3
View File
@@ -0,0 +1,3 @@
#!/usr/bin/env python3
import json
print(json.dumps({"skill": "pandas-data-analysis"}, indent=2))
.agents/skills/pandas-pro/SKILL.md
+180
View File
@@ -0,0 +1,180 @@
---
name: pandas-pro
description: Performs pandas DataFrame operations for data analysis, manipulation, and transformation. Use when working with pandas DataFrames, data cleaning, aggregation, merging, or time series analysis. Invoke for data manipulation tasks such as joining DataFrames on multiple keys, pivoting tables, resampling time series, handling NaN values with interpolation or forward-fill, groupby aggregations, type conversion, or performance optimization of large datasets.
license: MIT
metadata:
author: https://github.com/Jeffallan
version: "1.1.0"
domain: data-ml
triggers: pandas, DataFrame, data manipulation, data cleaning, aggregation, groupby, merge, join, time series, data wrangling, pivot table, data transformation
role: expert
scope: implementation
output-format: code
related-skills: python-pro
---
# Pandas Pro
Expert pandas developer specializing in efficient data manipulation, analysis, and transformation workflows with production-grade performance patterns.
## Core Workflow
1. **Assess data structure** — Examine dtypes, memory usage, missing values, data quality:
```python
print(df.dtypes)
print(df.memory_usage(deep=True).sum() / 1e6, "MB")
print(df.isna().sum())
print(df.describe(include="all"))
```
2. **Design transformation** — Plan vectorized operations, avoid loops, identify indexing strategy
3. **Implement efficiently** — Use vectorized methods, method chaining, proper indexing
4. **Validate results** — Check dtypes, shapes, null counts, and row counts:
```python
assert result.shape[0] == expected_rows, f"Row count mismatch: {result.shape[0]}"
assert result.isna().sum().sum() == 0, "Unexpected nulls after transform"
assert set(result.columns) == expected_cols
```
5. **Optimize** — Profile memory, apply categorical types, use chunking if needed
## Reference Guide
Load detailed guidance based on context:
| Topic | Reference | Load When |
|-------|-----------|-----------|
| DataFrame Operations | `references/dataframe-operations.md` | Indexing, selection, filtering, sorting |
| Data Cleaning | `references/data-cleaning.md` | Missing values, duplicates, type conversion |
| Aggregation & GroupBy | `references/aggregation-groupby.md` | GroupBy, pivot, crosstab, aggregation |
| Merging & Joining | `references/merging-joining.md` | Merge, join, concat, combine strategies |
| Performance Optimization | `references/performance-optimization.md` | Memory usage, vectorization, chunking |
## Code Patterns
### Vectorized Operations (before/after)
```python
# ❌ AVOID: row-by-row iteration
for i, row in df.iterrows():
df.at[i, 'tax'] = row['price'] * 0.2
# ✅ USE: vectorized assignment
df['tax'] = df['price'] * 0.2
```
### Safe Subsetting with `.copy()`
```python
# ❌ AVOID: chained indexing triggers SettingWithCopyWarning
df['A']['B'] = 1
# ✅ USE: .loc[] with explicit copy when mutating a subset
subset = df.loc[df['status'] == 'active', :].copy()
subset['score'] = subset['score'].fillna(0)
```
### GroupBy Aggregation
```python
summary = (
df.groupby(['region', 'category'], observed=True)
.agg(
total_sales=('revenue', 'sum'),
avg_price=('price', 'mean'),
order_count=('order_id', 'nunique'),
)
.reset_index()
)
```
### Merge with Validation
```python
merged = pd.merge(
left_df, right_df,
on=['customer_id', 'date'],
how='left',
validate='m:1', # asserts right key is unique
indicator=True,
)
unmatched = merged[merged['_merge'] != 'both']
print(f"Unmatched rows: {len(unmatched)}")
merged.drop(columns=['_merge'], inplace=True)
```
### Missing Value Handling
```python
# Forward-fill then interpolate numeric gaps
df['price'] = df['price'].ffill().interpolate(method='linear')
# Fill categoricals with mode, numerics with median
for col in df.select_dtypes(include='object'):
df[col] = df[col].fillna(df[col].mode()[0])
for col in df.select_dtypes(include='number'):
df[col] = df[col].fillna(df[col].median())
```
### Time Series Resampling
```python
daily = (
df.set_index('timestamp')
.resample('D')
.agg({'revenue': 'sum', 'sessions': 'count'})
.fillna(0)
)
```
### Pivot Table
```python
pivot = df.pivot_table(
values='revenue',
index='region',
columns='product_line',
aggfunc='sum',
fill_value=0,
margins=True,
)
```
### Memory Optimization
```python
# Downcast numerics and convert low-cardinality strings to categorical
df['category'] = df['category'].astype('category')
df['count'] = pd.to_numeric(df['count'], downcast='integer')
df['score'] = pd.to_numeric(df['score'], downcast='float')
print(df.memory_usage(deep=True).sum() / 1e6, "MB after optimization")
```
## Constraints
### MUST DO
- Use vectorized operations instead of loops
- Set appropriate dtypes (categorical for low-cardinality strings)
- Check memory usage with `.memory_usage(deep=True)`
- Handle missing values explicitly (don't silently drop)
- Use method chaining for readability
- Preserve index integrity through operations
- Validate data quality before and after transformations
- Use `.copy()` when modifying subsets to avoid SettingWithCopyWarning
### MUST NOT DO
- Iterate over DataFrame rows with `.iterrows()` unless absolutely necessary
- Use chained indexing (`df['A']['B']`) — use `.loc[]` or `.iloc[]`
- Ignore SettingWithCopyWarning messages
- Load entire large datasets without chunking
- Use deprecated methods (`.ix`, `.append()` — use `pd.concat()`)
- Convert to Python lists for operations possible in pandas
- Assume data is clean without validation
## Output Templates
When implementing pandas solutions, provide:
1. Code with vectorized operations and proper indexing
2. Comments explaining complex transformations
3. Memory/performance considerations if dataset is large
4. Data validation checks (dtypes, nulls, shapes)
[Documentation](https://jeffallan.github.io/claude-skills/skills/data-ml/pandas-pro/)
.agents/skills/pandas-pro/references/aggregation-groupby.md
+545
View File
@@ -0,0 +1,545 @@
# Aggregation and GroupBy
---
## Overview
Aggregation transforms data from individual records to summary statistics. This reference covers GroupBy, pivot tables, crosstab, and advanced aggregation patterns with pandas 2.0+.
---
## GroupBy Fundamentals
### Basic GroupBy
```python
import pandas as pd
import numpy as np
df = pd.DataFrame({
'department': ['Eng', 'Eng', 'Sales', 'Sales', 'Eng', 'HR'],
'team': ['Backend', 'Frontend', 'East', 'West', 'Backend', 'Recruit'],
'employee': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve', 'Frank'],
'salary': [80000, 75000, 65000, 70000, 85000, 60000],
'years': [5, 3, 7, 4, 6, 2]
})
# Single column groupby with single aggregation
avg_salary = df.groupby('department')['salary'].mean()
# Multiple aggregations
stats = df.groupby('department')['salary'].agg(['mean', 'min', 'max', 'count'])
# GroupBy multiple columns
grouped = df.groupby(['department', 'team'])['salary'].mean()
# Reset index to get DataFrame instead of Series
grouped = df.groupby('department')['salary'].mean().reset_index()
```
### Multiple Columns, Multiple Aggregations
```python
# Named aggregation (pandas 2.0+ preferred)
result = df.groupby('department').agg(
avg_salary=('salary', 'mean'),
max_salary=('salary', 'max'),
total_years=('years', 'sum'),
headcount=('employee', 'count'),
)
# Dictionary syntax (traditional)
result = df.groupby('department').agg({
'salary': ['mean', 'max', 'std'],
'years': ['sum', 'mean'],
})
# Flatten multi-level column names
result.columns = ['_'.join(col).strip() for col in result.columns.values]
```
### Custom Aggregation Functions
```python
# Lambda functions
result = df.groupby('department').agg({
'salary': lambda x: x.max() - x.min(), # Range
'years': lambda x: x.quantile(0.75), # 75th percentile
})
# Named functions for clarity
def salary_range(x):
return x.max() - x.min()
def coefficient_of_variation(x):
return x.std() / x.mean() if x.mean() != 0 else 0
result = df.groupby('department').agg(
salary_range=('salary', salary_range),
salary_cv=('salary', coefficient_of_variation),
)
# Multiple custom functions
result = df.groupby('department')['salary'].agg([
('range', lambda x: x.max() - x.min()),
('iqr', lambda x: x.quantile(0.75) - x.quantile(0.25)),
('median', 'median'),
])
```
---
## Transform and Apply
### Transform - Returns Same Shape
```python
# Transform returns Series with same index as original
# Useful for adding aggregated values back to original DataFrame
# Add group mean as new column
df['dept_avg_salary'] = df.groupby('department')['salary'].transform('mean')
# Normalize within group
df['salary_zscore'] = df.groupby('department')['salary'].transform(
lambda x: (x - x.mean()) / x.std()
)
# Rank within group
df['salary_rank'] = df.groupby('department')['salary'].transform('rank', ascending=False)
# Percentage of group total
df['salary_pct'] = df.groupby('department')['salary'].transform(
lambda x: x / x.sum() * 100
)
# Fill missing with group mean
df['salary'] = df.groupby('department')['salary'].transform(
lambda x: x.fillna(x.mean())
)
```
### Apply - Flexible Operations
```python
# Apply runs function on each group DataFrame
def top_n_by_salary(group, n=2):
return group.nlargest(n, 'salary')
top_earners = df.groupby('department').apply(top_n_by_salary, n=2)
# Reset index after apply
top_earners = df.groupby('department', group_keys=False).apply(
top_n_by_salary, n=2
).reset_index(drop=True)
# Complex group operations
def group_summary(group):
return pd.Series({
'headcount': len(group),
'avg_salary': group['salary'].mean(),
'top_earner': group.loc[group['salary'].idxmax(), 'employee'],
'avg_tenure': group['years'].mean(),
})
summary = df.groupby('department').apply(group_summary)
```
### Filter - Keep/Remove Groups
```python
# Keep only groups meeting a condition
# Groups with average salary > 70000
filtered = df.groupby('department').filter(lambda x: x['salary'].mean() > 70000)
# Groups with more than 2 members
filtered = df.groupby('department').filter(lambda x: len(x) > 2)
# Combined conditions
filtered = df.groupby('department').filter(
lambda x: (len(x) >= 2) and (x['salary'].mean() > 65000)
)
```
---
## Pivot Tables
### Basic Pivot Table
```python
df = pd.DataFrame({
'date': pd.date_range('2024-01-01', periods=6),
'product': ['A', 'B', 'A', 'B', 'A', 'B'],
'region': ['East', 'East', 'West', 'West', 'East', 'West'],
'sales': [100, 150, 120, 180, 90, 200],
'quantity': [10, 15, 12, 18, 9, 20],
})
# Simple pivot
pivot = df.pivot_table(
values='sales',
index='product',
columns='region',
aggfunc='sum'
)
# Multiple values
pivot = df.pivot_table(
values=['sales', 'quantity'],
index='product',
columns='region',
aggfunc='sum'
)
# Multiple aggregation functions
pivot = df.pivot_table(
values='sales',
index='product',
columns='region',
aggfunc=['sum', 'mean', 'count']
)
```
### Advanced Pivot Table Options
```python
# Fill missing values
pivot = df.pivot_table(
values='sales',
index='product',
columns='region',
aggfunc='sum',
fill_value=0
)
# Add margins (totals)
pivot = df.pivot_table(
values='sales',
index='product',
columns='region',
aggfunc='sum',
margins=True,
margins_name='Total'
)
# Multiple index levels
pivot = df.pivot_table(
values='sales',
index=['product', df['date'].dt.month],
columns='region',
aggfunc='sum'
)
# Observed categories only (for categorical data)
pivot = df.pivot_table(
values='sales',
index='product',
columns='region',
aggfunc='sum',
observed=True # pandas 2.0+ default changed
)
```
### Unpivoting (Melt)
```python
# Wide to long format
wide_df = pd.DataFrame({
'product': ['A', 'B'],
'Q1_sales': [100, 150],
'Q2_sales': [120, 180],
'Q3_sales': [90, 200],
})
# Melt to long format
long_df = pd.melt(
wide_df,
id_vars=['product'],
value_vars=['Q1_sales', 'Q2_sales', 'Q3_sales'],
var_name='quarter',
value_name='sales'
)
# Clean quarter column
long_df['quarter'] = long_df['quarter'].str.replace('_sales', '')
```
---
## Crosstab
### Basic Crosstab
```python
df = pd.DataFrame({
'gender': ['M', 'F', 'M', 'F', 'M', 'F', 'M', 'M'],
'department': ['Eng', 'Eng', 'Sales', 'Sales', 'Eng', 'HR', 'HR', 'Eng'],
'level': ['Senior', 'Junior', 'Senior', 'Senior', 'Junior', 'Junior', 'Senior', 'Junior'],
})
# Simple crosstab (counts)
ct = pd.crosstab(df['gender'], df['department'])
# Normalized crosstab
ct_pct = pd.crosstab(df['gender'], df['department'], normalize='all') # Total
ct_pct = pd.crosstab(df['gender'], df['department'], normalize='index') # Row
ct_pct = pd.crosstab(df['gender'], df['department'], normalize='columns') # Column
# With margins
ct = pd.crosstab(df['gender'], df['department'], margins=True)
# Multiple levels
ct = pd.crosstab(
[df['gender'], df['level']],
df['department']
)
```
### Crosstab with Aggregation
```python
df['salary'] = [80000, 75000, 65000, 70000, 85000, 60000, 72000, 78000]
# Crosstab with values and aggregation
ct = pd.crosstab(
df['gender'],
df['department'],
values=df['salary'],
aggfunc='mean'
)
# Multiple aggregations
ct = pd.crosstab(
df['gender'],
df['department'],
values=df['salary'],
aggfunc=['mean', 'sum', 'count']
)
```
---
## Window Functions with GroupBy
### Rolling Aggregations
```python
df = pd.DataFrame({
'date': pd.date_range('2024-01-01', periods=10),
'product': ['A', 'B'] * 5,
'sales': [100, 150, 110, 160, 120, 170, 130, 180, 140, 190],
})
# Rolling mean within groups
df['rolling_avg'] = df.groupby('product')['sales'].transform(
lambda x: x.rolling(window=3, min_periods=1).mean()
)
# Expanding aggregations
df['cumulative_sales'] = df.groupby('product')['sales'].transform('cumsum')
df['expanding_avg'] = df.groupby('product')['sales'].transform(
lambda x: x.expanding().mean()
)
# Rank within groups
df['sales_rank'] = df.groupby('product')['sales'].rank(method='dense')
```
### Shift and Diff
```python
# Previous value within group
df['prev_sales'] = df.groupby('product')['sales'].shift(1)
# Next value
df['next_sales'] = df.groupby('product')['sales'].shift(-1)
# Period-over-period change
df['sales_change'] = df.groupby('product')['sales'].diff()
# Percentage change
df['sales_pct_change'] = df.groupby('product')['sales'].pct_change()
```
---
## Common Aggregation Patterns
### Summary Statistics
```python
# Comprehensive summary by group
def full_summary(group):
return pd.Series({
'count': len(group),
'mean': group['salary'].mean(),
'std': group['salary'].std(),
'min': group['salary'].min(),
'q25': group['salary'].quantile(0.25),
'median': group['salary'].median(),
'q75': group['salary'].quantile(0.75),
'max': group['salary'].max(),
'sum': group['salary'].sum(),
})
summary = df.groupby('department').apply(full_summary)
```
### Top N Per Group
```python
# Top 2 salaries per department
top_2 = df.groupby('department', group_keys=False).apply(
lambda x: x.nlargest(2, 'salary')
)
# Using head after sorting
top_2 = df.sort_values('salary', ascending=False).groupby(
'department', group_keys=False
).head(2)
# Bottom N
bottom_2 = df.groupby('department', group_keys=False).apply(
lambda x: x.nsmallest(2, 'salary')
)
```
### First/Last Per Group
```python
# First row per group
first = df.groupby('department').first()
# Last row per group
last = df.groupby('department').last()
# First row after sorting
first_by_salary = df.sort_values('salary', ascending=False).groupby(
'department'
).first()
# Nth row
nth = df.groupby('department').nth(1) # Second row (0-indexed)
```
### Cumulative Operations
```python
# Cumulative sum
df['cum_sales'] = df.groupby('department')['salary'].cumsum()
# Cumulative max/min
df['cum_max'] = df.groupby('department')['salary'].cummax()
df['cum_min'] = df.groupby('department')['salary'].cummin()
# Cumulative count
df['cum_count'] = df.groupby('department').cumcount() + 1
# Running percentage of total
df['running_pct'] = df.groupby('department')['salary'].transform(
lambda x: x.cumsum() / x.sum() * 100
)
```
---
## Performance Tips for GroupBy
### Efficient GroupBy Operations
```python
# Pre-sort for faster groupby operations
df = df.sort_values('department')
grouped = df.groupby('department', sort=False) # Already sorted
# Use observed=True for categorical columns (pandas 2.0+ default)
df['department'] = df['department'].astype('category')
grouped = df.groupby('department', observed=True)['salary'].mean()
# Avoid apply when possible - use built-in aggregations
# SLOWER:
result = df.groupby('department')['salary'].apply(lambda x: x.sum())
# FASTER:
result = df.groupby('department')['salary'].sum()
# Use numba for custom aggregations (if available)
@numba.jit(nopython=True)
def custom_agg(values):
return values.sum() / len(values)
```
### Memory-Efficient Aggregation
```python
# For large DataFrames, compute aggregations separately
groups = df.groupby('department')
means = groups['salary'].mean()
sums = groups['salary'].sum()
counts = groups.size()
result = pd.DataFrame({
'mean': means,
'sum': sums,
'count': counts
})
# Avoid creating intermediate large DataFrames
# BAD: Creates full transformed DataFrame
df['z_score'] = (df['salary'] - df.groupby('department')['salary'].transform('mean')) / df.groupby('department')['salary'].transform('std')
# BETTER: Compute once
group_stats = df.groupby('department')['salary'].agg(['mean', 'std'])
df = df.merge(group_stats, on='department')
df['z_score'] = (df['salary'] - df['mean']) / df['std']
```
---
## Best Practices Summary
1. **Use named aggregation** - Clearer than dictionary syntax
2. **Choose transform vs apply wisely** - Transform for same-shape, apply for flexible
3. **Pre-sort for performance** - Use `sort=False` after sorting
4. **Prefer built-in aggregations** - Faster than lambda/apply
5. **Use observed=True** - Especially for categorical data
6. **Reset index when needed** - Keep DataFrames easier to work with
7. **Validate group counts** - Check for unexpected groups
---
## Anti-Patterns to Avoid
```python
# BAD: Iterating over groups manually
for name, group in df.groupby('department'):
# process group
pass
# GOOD: Use vectorized operations
df.groupby('department').agg(...)
# BAD: Multiple groupby calls
df.groupby('dept')['salary'].mean()
df.groupby('dept')['salary'].sum()
df.groupby('dept')['salary'].count()
# GOOD: Single groupby, multiple aggs
df.groupby('dept')['salary'].agg(['mean', 'sum', 'count'])
# BAD: Apply for simple aggregations
df.groupby('dept')['salary'].apply(np.mean)
# GOOD: Built-in method
df.groupby('dept')['salary'].mean()
```
---
## Related References
- `dataframe-operations.md` - Filtering before aggregation
- `merging-joining.md` - Join aggregated results back
- `performance-optimization.md` - Optimize large-scale aggregations
.agents/skills/pandas-pro/references/data-cleaning.md
+500
View File
@@ -0,0 +1,500 @@
# Data Cleaning
---
## Overview
Data cleaning is critical for reliable analysis. This reference covers handling missing values, duplicates, type conversion, and data validation with pandas 2.0+ patterns.
---
## Missing Values
### Detecting Missing Values
```python
import pandas as pd
import numpy as np
df = pd.DataFrame({
'name': ['Alice', 'Bob', None, 'Diana'],
'age': [25, np.nan, 35, 28],
'salary': [50000, 60000, np.nan, np.nan],
'department': ['Eng', '', 'Eng', 'Sales']
})
# Check for any missing values
df.isna().any() # Per column
df.isna().any().any() # Entire DataFrame
# Count missing values
df.isna().sum() # Per column
df.isna().sum().sum() # Total
# Percentage of missing values
(df.isna().sum() / len(df) * 100).round(2)
# Rows with any missing values
df[df.isna().any(axis=1)]
# Rows with all values present
df[df.notna().all(axis=1)]
# Missing value heatmap info
missing_info = pd.DataFrame({
'missing': df.isna().sum(),
'percent': (df.isna().sum() / len(df) * 100).round(2),
'dtype': df.dtypes
})
```
### Handling Missing Values - Dropping
```python
# Drop rows with any missing value
df_clean = df.dropna()
# Drop rows where specific columns have missing values
df_clean = df.dropna(subset=['name', 'age'])
# Drop rows where ALL values are missing
df_clean = df.dropna(how='all')
# Drop rows with minimum non-null values
df_clean = df.dropna(thresh=3) # Keep rows with at least 3 non-null
# Drop columns with missing values
df_clean = df.dropna(axis=1)
# Drop columns with more than 50% missing
threshold = len(df) * 0.5
df_clean = df.dropna(axis=1, thresh=threshold)
```
### Handling Missing Values - Filling
```python
# Fill with constant value
df['age'] = df['age'].fillna(0)
# Fill with column mean/median/mode
df['age'] = df['age'].fillna(df['age'].mean())
df['salary'] = df['salary'].fillna(df['salary'].median())
df['department'] = df['department'].fillna(df['department'].mode()[0])
# Forward fill (use previous value)
df['salary'] = df['salary'].ffill()
# Backward fill (use next value)
df['salary'] = df['salary'].bfill()
# Fill with different values per column
fill_values = {'age': 0, 'salary': df['salary'].median(), 'name': 'Unknown'}
df = df.fillna(fill_values)
# Fill with interpolation (numeric data)
df['salary'] = df['salary'].interpolate(method='linear')
# Group-specific fill (fill with group mean)
df['salary'] = df.groupby('department')['salary'].transform(
lambda x: x.fillna(x.mean())
)
```
### Handling Empty Strings vs NaN
```python
# Empty strings are NOT detected as NaN
df['department'].isna().sum() # Won't count ''
# Replace empty strings with NaN
df['department'] = df['department'].replace('', np.nan)
# Or
df['department'] = df['department'].replace(r'^\s*$', np.nan, regex=True)
# Replace multiple values with NaN
df = df.replace(['', 'N/A', 'null', 'None', '-'], np.nan)
# Using na_values when reading files
df = pd.read_csv('file.csv', na_values=['', 'N/A', 'null', 'None', '-'])
```
---
## Handling Duplicates
### Detecting Duplicates
```python
df = pd.DataFrame({
'id': [1, 2, 2, 3, 4, 4],
'name': ['Alice', 'Bob', 'Bob', 'Charlie', 'Diana', 'Diana'],
'email': ['a@x.com', 'b@x.com', 'b@x.com', 'c@x.com', 'd@x.com', 'd2@x.com']
})
# Check for duplicate rows (all columns)
df.duplicated().sum()
# Check specific columns
df.duplicated(subset=['id']).sum()
df.duplicated(subset=['name', 'email']).sum()
# View duplicate rows
df[df.duplicated(keep=False)] # All duplicates
df[df.duplicated(keep='first')] # Duplicates except first occurrence
df[df.duplicated(keep='last')] # Duplicates except last occurrence
# Count duplicates per key
df.groupby('id').size().loc[lambda x: x > 1]
```
### Removing Duplicates
```python
# Remove duplicate rows (keep first)
df_clean = df.drop_duplicates()
# Keep last occurrence
df_clean = df.drop_duplicates(keep='last')
# Remove all duplicates (keep none)
df_clean = df.drop_duplicates(keep=False)
# Based on specific columns
df_clean = df.drop_duplicates(subset=['id'])
df_clean = df.drop_duplicates(subset=['name', 'email'], keep='last')
# In-place modification
df.drop_duplicates(inplace=True)
```
### Handling Duplicates with Aggregation
```python
# Instead of dropping, aggregate duplicates
df_agg = df.groupby('id').agg({
'name': 'first',
'email': lambda x: ', '.join(x.unique())
}).reset_index()
# Keep row with max/min value
df_best = df.loc[df.groupby('id')['score'].idxmax()]
# Rank duplicates
df['rank'] = df.groupby('id').cumcount() + 1
```
---
## Type Conversion
### Checking and Converting Types
```python
# Check current types
df.dtypes
df.info()
# Convert to specific type
df['age'] = df['age'].astype(int)
df['salary'] = df['salary'].astype(float)
df['name'] = df['name'].astype(str)
# Safe conversion with errors handling
df['age'] = pd.to_numeric(df['age'], errors='coerce') # Invalid -> NaN
df['age'] = pd.to_numeric(df['age'], errors='ignore') # Keep original if invalid
# Convert multiple columns
df = df.astype({'age': 'int64', 'salary': 'float64'})
# Convert object to string (pandas 2.0+ StringDtype)
df['name'] = df['name'].astype('string') # Nullable string type
```
### Datetime Conversion
```python
df = pd.DataFrame({
'date_str': ['2024-01-15', '2024-02-20', 'invalid', '2024-03-10'],
'timestamp': [1705276800, 1708387200, 1710028800, 1710028800]
})
# String to datetime
df['date'] = pd.to_datetime(df['date_str'], errors='coerce')
# Specify format for faster parsing
df['date'] = pd.to_datetime(df['date_str'], format='%Y-%m-%d', errors='coerce')
# Unix timestamp to datetime
df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')
# Extract components
df['year'] = df['date'].dt.year
df['month'] = df['date'].dt.month
df['day_of_week'] = df['date'].dt.day_name()
# Handle mixed formats
df['date'] = pd.to_datetime(df['date_str'], format='mixed', dayfirst=False)
```
### Categorical Conversion
```python
# Convert to categorical (memory efficient for low cardinality)
df['department'] = df['department'].astype('category')
# Ordered categorical
df['size'] = pd.Categorical(
df['size'],
categories=['Small', 'Medium', 'Large'],
ordered=True
)
# Check memory savings
print(f"Object: {df['department'].nbytes}")
df['department'] = df['department'].astype('category')
print(f"Category: {df['department'].nbytes}")
```
### Nullable Integer Types (pandas 2.0+)
```python
# Standard int doesn't support NaN
# Use nullable integer types
df['age'] = df['age'].astype('Int64') # Note capital I
# All nullable types
df = df.astype({
'count': 'Int64', # Nullable integer
'price': 'Float64', # Nullable float
'flag': 'boolean', # Nullable boolean
'name': 'string', # Nullable string
})
# Convert with NA handling
df['age'] = pd.array([1, 2, None, 4], dtype='Int64')
```
---
## String Cleaning
### Common String Operations
```python
df = pd.DataFrame({
'name': [' Alice ', 'BOB', 'charlie', None, 'Diana Smith'],
'email': ['ALICE@EXAMPLE.COM', 'bob@test', 'invalid', None, 'diana@example.com']
})
# Strip whitespace
df['name'] = df['name'].str.strip()
# Case normalization
df['name'] = df['name'].str.lower()
df['name'] = df['name'].str.upper()
df['name'] = df['name'].str.title() # Title Case
# Replace patterns
df['name'] = df['name'].str.replace(r'\s+', ' ', regex=True) # Multiple spaces to one
df['phone'] = df['phone'].str.replace(r'[^0-9]', '', regex=True) # Keep only digits
# Extract with regex
df['domain'] = df['email'].str.extract(r'@(.+)$')
df['first_name'] = df['name'].str.extract(r'^(\w+)')
# Split strings
df[['first', 'last']] = df['name'].str.split(' ', n=1, expand=True)
```
### String Validation
```python
# Check patterns
df['valid_email'] = df['email'].str.match(r'^[\w.]+@[\w.]+\.\w+$', na=False)
# String length
df['name_length'] = df['name'].str.len()
df['valid_length'] = df['name'].str.len().between(2, 50)
# Contains check
df['has_domain'] = df['email'].str.contains('@', na=False)
```
---
## Data Validation
### Validation Functions
```python
def validate_dataframe(df: pd.DataFrame) -> dict:
"""Comprehensive DataFrame validation."""
report = {
'rows': len(df),
'columns': len(df.columns),
'duplicates': df.duplicated().sum(),
'missing_by_column': df.isna().sum().to_dict(),
'dtypes': df.dtypes.astype(str).to_dict(),
}
return report
# Range validation
def validate_range(series: pd.Series, min_val, max_val) -> pd.Series:
"""Return boolean mask for values in range."""
return series.between(min_val, max_val)
df['valid_age'] = validate_range(df['age'], 0, 120)
# Custom validation
def validate_email(series: pd.Series) -> pd.Series:
"""Validate email format."""
pattern = r'^[\w.+-]+@[\w-]+\.[\w.-]+$'
return series.str.match(pattern, na=False)
df['valid_email'] = validate_email(df['email'])
```
### Schema Validation with pandera
```python
# Using pandera for schema validation (recommended for production)
import pandera as pa
from pandera import Column, Check
schema = pa.DataFrameSchema({
'name': Column(str, Check.str_length(min_value=1, max_value=100)),
'age': Column(int, Check.in_range(0, 120)),
'email': Column(str, Check.str_matches(r'^[\w.+-]+@[\w-]+\.[\w.-]+$')),
'salary': Column(float, Check.greater_than(0), nullable=True),
})
# Validate DataFrame
try:
schema.validate(df)
except pa.errors.SchemaError as e:
print(f"Validation failed: {e}")
```
---
## Data Cleaning Pipeline
### Method Chaining Pattern
```python
def clean_dataframe(df: pd.DataFrame) -> pd.DataFrame:
"""Complete data cleaning pipeline using method chaining."""
return (
df
# Make a copy
.copy()
# Standardize column names
.rename(columns=lambda x: x.lower().strip().replace(' ', '_'))
# Drop fully empty rows
.dropna(how='all')
# Clean string columns
.assign(
name=lambda x: x['name'].str.strip().str.title(),
email=lambda x: x['email'].str.lower().str.strip(),
)
# Handle missing values
.fillna({'department': 'Unknown'})
# Convert types
.astype({'age': 'Int64', 'department': 'category'})
# Remove duplicates
.drop_duplicates(subset=['email'])
# Reset index
.reset_index(drop=True)
)
df_clean = clean_dataframe(df)
```
### Pipeline with Validation
```python
def clean_and_validate(
df: pd.DataFrame,
required_columns: list[str],
unique_columns: list[str] | None = None,
) -> tuple[pd.DataFrame, dict]:
"""Clean DataFrame and return validation report."""
# Validate required columns exist
missing_cols = set(required_columns) - set(df.columns)
if missing_cols:
raise ValueError(f"Missing required columns: {missing_cols}")
# Track cleaning stats
stats = {
'initial_rows': len(df),
'dropped_empty': 0,
'dropped_duplicates': 0,
'filled_missing': {},
}
# Clean
df = df.copy()
# Drop empty rows
before = len(df)
df = df.dropna(how='all')
stats['dropped_empty'] = before - len(df)
# Handle duplicates
if unique_columns:
before = len(df)
df = df.drop_duplicates(subset=unique_columns)
stats['dropped_duplicates'] = before - len(df)
stats['final_rows'] = len(df)
return df, stats
```
---
## Best Practices Summary
1. **Always check data quality first** - Use `.info()`, `.describe()`, and missing value analysis
2. **Document cleaning decisions** - Track what was dropped/filled and why
3. **Use nullable types** - `Int64`, `string`, `boolean` for proper NA handling
4. **Validate after cleaning** - Ensure data meets expectations
5. **Use method chaining** - Readable, maintainable cleaning pipelines
6. **Copy before modifying** - Avoid SettingWithCopyWarning
7. **Handle edge cases** - Empty strings, whitespace, invalid formats
---
## Anti-Patterns to Avoid
```python
# BAD: Dropping NaN without understanding impact
df = df.dropna() # May lose significant data
# GOOD: Investigate first, then decide
print(f"Missing values: {df.isna().sum()}")
print(f"Rows affected: {df.isna().any(axis=1).sum()}")
# Then make informed decision
# BAD: Filling without domain knowledge
df['age'] = df['age'].fillna(0) # Age 0 is not valid
# GOOD: Use appropriate fill strategy
df['age'] = df['age'].fillna(df['age'].median())
# BAD: Type conversion without error handling
df['id'] = df['id'].astype(int) # Will fail on NaN or invalid
# GOOD: Safe conversion
df['id'] = pd.to_numeric(df['id'], errors='coerce').astype('Int64')
```
---
## Related References
- `dataframe-operations.md` - Selection and filtering for targeted cleaning
- `aggregation-groupby.md` - Aggregate duplicates instead of dropping
- `performance-optimization.md` - Efficient cleaning of large datasets
.agents/skills/pandas-pro/references/dataframe-operations.md
+420
View File
@@ -0,0 +1,420 @@
# DataFrame Operations
---
## Overview
DataFrame operations form the foundation of pandas work. This reference covers indexing, selection, filtering, and sorting with pandas 2.0+ best practices.
---
## Indexing and Selection
### Label-Based Selection with `.loc[]`
Use `.loc[]` for label-based indexing. Always preferred over chained indexing.
```python
import pandas as pd
import numpy as np
# Sample DataFrame
df = pd.DataFrame({
'name': ['Alice', 'Bob', 'Charlie', 'Diana'],
'age': [25, 30, 35, 28],
'salary': [50000, 60000, 70000, 55000],
'department': ['Engineering', 'Sales', 'Engineering', 'Marketing']
}, index=['a', 'b', 'c', 'd'])
# Single value
value = df.loc['a', 'name'] # 'Alice'
# Single row (returns Series)
row = df.loc['a']
# Multiple rows
rows = df.loc[['a', 'c']]
# Row and column slices (inclusive on both ends)
subset = df.loc['a':'c', 'name':'salary']
# Boolean indexing with .loc
adults = df.loc[df['age'] >= 30]
# Boolean indexing with column selection
adults_names = df.loc[df['age'] >= 30, 'name']
# Multiple conditions
engineering_seniors = df.loc[
(df['department'] == 'Engineering') & (df['age'] >= 30),
['name', 'salary']
]
```
### Position-Based Selection with `.iloc[]`
Use `.iloc[]` for integer position-based indexing.
```python
# Single value by position
value = df.iloc[0, 0] # First row, first column
# Single row by position
first_row = df.iloc[0]
# Slice rows (exclusive end, like Python)
first_three = df.iloc[:3]
# Specific rows and columns by position
subset = df.iloc[[0, 2], [0, 2]] # Rows 0,2 and columns 0,2
# Range selection
block = df.iloc[1:3, 0:2] # Rows 1-2, columns 0-1
```
### When to Use `.loc[]` vs `.iloc[]`
| Scenario | Use | Example |
|----------|-----|---------|
| Known column names | `.loc[]` | `df.loc[:, 'name']` |
| Filter by condition | `.loc[]` | `df.loc[df['age'] > 25]` |
| First/last N rows | `.iloc[]` | `df.iloc[:5]` or `df.iloc[-5:]` |
| Specific row positions | `.iloc[]` | `df.iloc[[0, 5, 10]]` |
| Unknown column order | `.iloc[]` | `df.iloc[:, 0]` |
---
## Filtering DataFrames
### Boolean Masks
```python
# Single condition
mask = df['age'] > 25
filtered = df[mask]
# Multiple conditions (use parentheses!)
mask = (df['age'] > 25) & (df['salary'] < 65000)
filtered = df[mask]
# OR conditions
mask = (df['department'] == 'Engineering') | (df['department'] == 'Sales')
filtered = df[mask]
# NOT condition
mask = ~(df['department'] == 'Marketing')
filtered = df[mask]
```
### Using `.query()` for Readable Filters
```python
# Simple query - more readable for complex conditions
result = df.query('age > 25 and salary < 65000')
# Using variables with @
min_age = 25
result = df.query('age > @min_age')
# String comparisons
result = df.query('department == "Engineering"')
# In-list filtering
depts = ['Engineering', 'Sales']
result = df.query('department in @depts')
# Complex expressions
result = df.query('(age > 25) and (department != "Marketing")')
```
### Using `.isin()` for Multiple Values
```python
# Filter by multiple values
departments = ['Engineering', 'Sales']
filtered = df[df['department'].isin(departments)]
# Negation
filtered = df[~df['department'].isin(departments)]
# Multiple columns
conditions = {
'department': ['Engineering', 'Sales'],
'age': [25, 30, 35]
}
# Filter where department is in list AND age is in list
mask = df['department'].isin(conditions['department']) & df['age'].isin(conditions['age'])
```
### String Filtering with `.str` Accessor
```python
df = pd.DataFrame({
'email': ['alice@example.com', 'bob@test.org', 'charlie@example.com'],
'name': ['Alice Smith', 'Bob Jones', 'Charlie Brown']
})
# Contains
mask = df['email'].str.contains('example')
# Starts/ends with
mask = df['email'].str.endswith('.com')
mask = df['name'].str.startswith('A')
# Regex matching
mask = df['email'].str.match(r'^[a-z]+@example\.com$')
# Case-insensitive
mask = df['name'].str.lower().str.contains('alice')
# Or with case parameter
mask = df['name'].str.contains('alice', case=False)
# Handle NaN in string columns
mask = df['email'].str.contains('example', na=False)
```
---
## Sorting
### Basic Sorting
```python
# Sort by single column (ascending)
sorted_df = df.sort_values('age')
# Sort descending
sorted_df = df.sort_values('age', ascending=False)
# Sort by multiple columns
sorted_df = df.sort_values(['department', 'salary'], ascending=[True, False])
# Sort by index
sorted_df = df.sort_index()
sorted_df = df.sort_index(ascending=False)
```
### Advanced Sorting
```python
# Sort with NaN handling
df_with_nan = pd.DataFrame({
'name': ['Alice', 'Bob', 'Charlie'],
'score': [85.0, np.nan, 90.0]
})
# NaN at end (default)
sorted_df = df_with_nan.sort_values('score', na_position='last')
# NaN at beginning
sorted_df = df_with_nan.sort_values('score', na_position='first')
# Custom sort order using Categorical
order = ['Marketing', 'Sales', 'Engineering']
df['department'] = pd.Categorical(df['department'], categories=order, ordered=True)
sorted_df = df.sort_values('department')
# Sort by computed values without adding column
sorted_df = df.iloc[df['name'].str.len().argsort()]
```
### In-Place Sorting
```python
# Modify DataFrame in place
df.sort_values('age', inplace=True)
# Reset index after sorting
df.sort_values('age', inplace=True)
df.reset_index(drop=True, inplace=True)
# Or chain
df = df.sort_values('age').reset_index(drop=True)
```
---
## Column Operations
### Adding and Modifying Columns
```python
# Add new column
df['bonus'] = df['salary'] * 0.1
# Conditional column with np.where
df['seniority'] = np.where(df['age'] >= 30, 'Senior', 'Junior')
# Multiple conditions with np.select
conditions = [
df['age'] < 25,
df['age'] < 35,
df['age'] >= 35
]
choices = ['Junior', 'Mid', 'Senior']
df['level'] = np.select(conditions, choices, default='Unknown')
# Using .assign() for method chaining (returns new DataFrame)
df_new = df.assign(
bonus=lambda x: x['salary'] * 0.1,
total_comp=lambda x: x['salary'] + x['salary'] * 0.1
)
```
### Renaming Columns
```python
# Rename specific columns
df = df.rename(columns={'name': 'full_name', 'age': 'years'})
# Rename all columns with function
df.columns = df.columns.str.lower().str.replace(' ', '_')
# Using rename with function
df = df.rename(columns=str.upper)
```
### Dropping Columns
```python
# Drop single column
df = df.drop('bonus', axis=1)
# Or
df = df.drop(columns=['bonus'])
# Drop multiple columns
df = df.drop(columns=['bonus', 'level'])
# Drop columns by condition
cols_to_drop = [col for col in df.columns if col.startswith('temp_')]
df = df.drop(columns=cols_to_drop)
```
### Reordering Columns
```python
# Explicit order
new_order = ['name', 'department', 'age', 'salary']
df = df[new_order]
# Move specific column to front
cols = ['salary'] + [c for c in df.columns if c != 'salary']
df = df[cols]
# Using .reindex()
df = df.reindex(columns=['name', 'age', 'salary', 'department'])
```
---
## Index Operations
### Setting and Resetting Index
```python
# Set column as index
df = df.set_index('name')
# Reset index back to column
df = df.reset_index()
# Drop index completely
df = df.reset_index(drop=True)
# Set multiple columns as index (MultiIndex)
df = df.set_index(['department', 'name'])
```
### Working with MultiIndex
```python
# Create MultiIndex DataFrame
df = pd.DataFrame({
'department': ['Eng', 'Eng', 'Sales', 'Sales'],
'team': ['Backend', 'Frontend', 'East', 'West'],
'headcount': [10, 8, 15, 12]
}).set_index(['department', 'team'])
# Select from MultiIndex
df.loc['Eng'] # All Eng rows
df.loc[('Eng', 'Backend')] # Specific row
# Cross-section with .xs()
df.xs('Backend', level='team') # All Backend teams
# Reset specific level
df.reset_index(level='team')
```
---
## Copying DataFrames
### When to Use `.copy()`
```python
# ALWAYS copy when modifying a subset
subset = df[df['age'] > 25].copy()
subset['new_col'] = 100 # Safe, no SettingWithCopyWarning
# Without copy - may raise warning or fail silently
# BAD:
# subset = df[df['age'] > 25]
# subset['new_col'] = 100 # SettingWithCopyWarning!
# Deep copy (default) - copies data
df_copy = df.copy() # or df.copy(deep=True)
# Shallow copy - shares data, only copies structure
df_shallow = df.copy(deep=False)
```
---
## Best Practices Summary
1. **Use `.loc[]` and `.iloc[]`** - Never use chained indexing
2. **Parenthesize conditions** - `(cond1) & (cond2)` not `cond1 & cond2`
3. **Use `.query()` for readability** - Especially with complex filters
4. **Copy before modifying subsets** - Always use `.copy()`
5. **Use vectorized operations** - Avoid row iteration for filtering
6. **Handle NaN explicitly** - Use `na=False` in string operations
7. **Prefer method chaining** - Use `.assign()` for column creation
---
## Anti-Patterns to Avoid
```python
# BAD: Chained indexing
df['A']['B'] = value # May not work, raises warning
# GOOD: Use .loc
df.loc[:, ('A', 'B')] = value
# Or for row selection then assignment:
df.loc[df['A'] > 0, 'B'] = value
# BAD: Iterating for filtering
result = []
for idx, row in df.iterrows():
if row['age'] > 25:
result.append(row)
# GOOD: Boolean indexing
result = df[df['age'] > 25]
# BAD: Multiple separate assignments
df = df[df['age'] > 25]
df = df[df['salary'] > 50000]
# GOOD: Combined filter
df = df[(df['age'] > 25) & (df['salary'] > 50000)]
```
---
## Related References
- `data-cleaning.md` - After selection, clean the data
- `aggregation-groupby.md` - Group and aggregate filtered data
- `performance-optimization.md` - Optimize filtering on large datasets
.agents/skills/pandas-pro/references/merging-joining.md
+596
View File
@@ -0,0 +1,596 @@
# Merging and Joining
---
## Overview
Combining DataFrames is essential for working with relational data. This reference covers merge, join, concat, and advanced combination strategies with pandas 2.0+.
---
## Merge (SQL-Style Joins)
### Basic Merge
```python
import pandas as pd
import numpy as np
# Sample DataFrames
employees = pd.DataFrame({
'emp_id': [1, 2, 3, 4, 5],
'name': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'],
'dept_id': [101, 102, 101, 103, 102],
})
departments = pd.DataFrame({
'dept_id': [101, 102, 104],
'dept_name': ['Engineering', 'Sales', 'Marketing'],
})
# Inner join (default) - only matching rows
result = pd.merge(employees, departments, on='dept_id')
# Explicit how parameter
result = pd.merge(employees, departments, on='dept_id', how='inner')
```
### Join Types
```python
# Inner join - only matching rows from both
inner = pd.merge(employees, departments, on='dept_id', how='inner')
# Result: 4 rows (emp_id 4 has dept_id 103 which doesn't exist in departments)
# Left join - all rows from left, matching from right
left = pd.merge(employees, departments, on='dept_id', how='left')
# Result: 5 rows (Diana has NaN for dept_name)
# Right join - all rows from right, matching from left
right = pd.merge(employees, departments, on='dept_id', how='right')
# Result: 4 rows (Marketing has no employees, but is included)
# Outer join - all rows from both
outer = pd.merge(employees, departments, on='dept_id', how='outer')
# Result: 6 rows (includes unmatched from both sides)
# Cross join - cartesian product
cross = pd.merge(employees, departments, how='cross')
# Result: 15 rows (5 employees x 3 departments)
```
### Merging on Different Column Names
```python
employees = pd.DataFrame({
'emp_id': [1, 2, 3],
'name': ['Alice', 'Bob', 'Charlie'],
'department': [101, 102, 101],
})
departments = pd.DataFrame({
'id': [101, 102],
'dept_name': ['Engineering', 'Sales'],
})
# Different column names
result = pd.merge(
employees,
departments,
left_on='department',
right_on='id'
)
# Drop duplicate column after merge
result = result.drop('id', axis=1)
```
### Merging on Multiple Columns
```python
sales = pd.DataFrame({
'region': ['East', 'East', 'West', 'West'],
'product': ['A', 'B', 'A', 'B'],
'sales': [100, 150, 120, 180],
})
targets = pd.DataFrame({
'region': ['East', 'East', 'West'],
'product': ['A', 'B', 'A'],
'target': [90, 140, 110],
})
# Merge on multiple columns
result = pd.merge(sales, targets, on=['region', 'product'], how='left')
```
### Merging on Index
```python
# Set index before merge
employees_idx = employees.set_index('emp_id')
salaries = pd.DataFrame({
'emp_id': [1, 2, 3, 4],
'salary': [80000, 75000, 70000, 65000],
}).set_index('emp_id')
# Merge on index
result = pd.merge(employees_idx, salaries, left_index=True, right_index=True)
# Mix of column and index
result = pd.merge(
employees,
salaries,
left_on='emp_id',
right_index=True
)
```
---
## Handling Duplicate Columns
### Suffixes
```python
df1 = pd.DataFrame({
'id': [1, 2, 3],
'value': [10, 20, 30],
'date': ['2024-01-01', '2024-01-02', '2024-01-03'],
})
df2 = pd.DataFrame({
'id': [1, 2, 3],
'value': [100, 200, 300],
'date': ['2024-02-01', '2024-02-02', '2024-02-03'],
})
# Default suffixes
result = pd.merge(df1, df2, on='id')
# Columns: id, value_x, date_x, value_y, date_y
# Custom suffixes
result = pd.merge(df1, df2, on='id', suffixes=('_jan', '_feb'))
# Columns: id, value_jan, date_jan, value_feb, date_feb
```
### Validate Merge Cardinality
```python
# Validate merge relationships (pandas 2.0+)
# Raises MergeError if validation fails
# One-to-one: each key appears at most once in both DataFrames
result = pd.merge(df1, df2, on='id', validate='one_to_one') # or '1:1'
# One-to-many: keys unique in left only
result = pd.merge(employees, salaries, on='emp_id', validate='one_to_many') # or '1:m'
# Many-to-one: keys unique in right only
result = pd.merge(salaries, employees, on='emp_id', validate='many_to_one') # or 'm:1'
# Many-to-many: no uniqueness requirement (default)
result = pd.merge(df1, df2, on='id', validate='many_to_many') # or 'm:m'
```
### Indicator Column
```python
# Add indicator column showing source of each row
result = pd.merge(
employees,
departments,
on='dept_id',
how='outer',
indicator=True
)
# _merge column values: 'left_only', 'right_only', 'both'
# Custom indicator name
result = pd.merge(
employees,
departments,
on='dept_id',
how='outer',
indicator='source'
)
# Filter by indicator
left_only = result[result['_merge'] == 'left_only']
both = result[result['_merge'] == 'both']
```
---
## Join (Index-Based)
### DataFrame.join()
```python
# join() is for index-based joining (simpler syntax)
employees = pd.DataFrame({
'name': ['Alice', 'Bob', 'Charlie'],
'dept_id': [101, 102, 101],
}, index=[1, 2, 3])
salaries = pd.DataFrame({
'salary': [80000, 75000, 70000],
'bonus': [5000, 4000, 3500],
}, index=[1, 2, 3])
# Join on index
result = employees.join(salaries)
# Join types (same as merge)
result = employees.join(salaries, how='left')
result = employees.join(salaries, how='outer')
```
### Join on Column to Index
```python
employees = pd.DataFrame({
'name': ['Alice', 'Bob', 'Charlie'],
'dept_id': [101, 102, 101],
})
departments = pd.DataFrame({
'dept_name': ['Engineering', 'Sales'],
}, index=[101, 102])
# Join left column to right index
result = employees.join(departments, on='dept_id')
```
### Join Multiple DataFrames
```python
df1 = pd.DataFrame({'a': [1, 2]}, index=['x', 'y'])
df2 = pd.DataFrame({'b': [3, 4]}, index=['x', 'y'])
df3 = pd.DataFrame({'c': [5, 6]}, index=['x', 'y'])
# Join multiple at once
result = df1.join([df2, df3])
# With suffixes for duplicate columns
result = df1.join([df2, df3], lsuffix='_1', rsuffix='_2')
```
---
## Concat (Stacking DataFrames)
### Vertical Concatenation (Row-wise)
```python
# Stack DataFrames vertically
df1 = pd.DataFrame({
'name': ['Alice', 'Bob'],
'age': [25, 30],
})
df2 = pd.DataFrame({
'name': ['Charlie', 'Diana'],
'age': [35, 28],
})
# Basic concat (axis=0 is default)
result = pd.concat([df1, df2])
# Reset index
result = pd.concat([df1, df2], ignore_index=True)
# Keep track of source
result = pd.concat([df1, df2], keys=['source1', 'source2'])
# Creates MultiIndex
```
### Horizontal Concatenation (Column-wise)
```python
names = pd.DataFrame({'name': ['Alice', 'Bob', 'Charlie']})
ages = pd.DataFrame({'age': [25, 30, 35]})
salaries = pd.DataFrame({'salary': [50000, 60000, 70000]})
# Concat columns (axis=1)
result = pd.concat([names, ages, salaries], axis=1)
```
### Handling Mismatched Columns
```python
df1 = pd.DataFrame({
'name': ['Alice', 'Bob'],
'age': [25, 30],
})
df2 = pd.DataFrame({
'name': ['Charlie', 'Diana'],
'salary': [70000, 65000],
})
# Outer join (default) - include all columns
result = pd.concat([df1, df2])
# age and salary columns have NaN where not present
# Inner join - only common columns
result = pd.concat([df1, df2], join='inner')
# Only 'name' column
```
### Concat with Verification
```python
# Verify no index overlap
try:
result = pd.concat([df1, df2], verify_integrity=True)
except ValueError as e:
print(f"Index overlap detected: {e}")
# Alternative: use ignore_index
result = pd.concat([df1, df2], ignore_index=True)
```
---
## Combine and Update
### combine_first() - Fill Gaps
```python
# Fill NaN values from another DataFrame
df1 = pd.DataFrame({
'A': [1, np.nan, 3],
'B': [np.nan, 2, 3],
}, index=['a', 'b', 'c'])
df2 = pd.DataFrame({
'A': [10, 20, 30],
'B': [10, 20, 30],
}, index=['a', 'b', 'c'])
# Fill NaN in df1 with values from df2
result = df1.combine_first(df2)
# A: [1, 20, 3], B: [10, 2, 3]
```
### update() - In-Place Update
```python
df1 = pd.DataFrame({
'A': [1, 2, 3],
'B': [4, 5, 6],
}, index=['a', 'b', 'c'])
df2 = pd.DataFrame({
'A': [10, 20],
'B': [40, 50],
}, index=['a', 'b'])
# Update df1 with values from df2 (in-place)
df1.update(df2)
# df1 now has A: [10, 20, 3], B: [40, 50, 6]
# Only update where df2 has non-NaN
df1.update(df2, overwrite=False) # Don't overwrite existing values
```
---
## Advanced Merge Patterns
### Merge with Aggregation
```python
# Merge and aggregate in one operation
orders = pd.DataFrame({
'order_id': [1, 2, 3, 4],
'customer_id': [101, 102, 101, 103],
'amount': [100, 200, 150, 300],
})
customers = pd.DataFrame({
'customer_id': [101, 102, 103],
'name': ['Alice', 'Bob', 'Charlie'],
})
# Get customer summary
customer_summary = orders.groupby('customer_id').agg(
total_orders=('order_id', 'count'),
total_amount=('amount', 'sum'),
).reset_index()
# Merge with customer info
result = pd.merge(customers, customer_summary, on='customer_id')
```
### Merge Asof (Nearest Match)
```python
# Merge on nearest key (useful for time series)
trades = pd.DataFrame({
'time': pd.to_datetime(['2024-01-01 10:00:01', '2024-01-01 10:00:03', '2024-01-01 10:00:05']),
'ticker': ['AAPL', 'AAPL', 'AAPL'],
'price': [150.0, 151.0, 150.5],
})
quotes = pd.DataFrame({
'time': pd.to_datetime(['2024-01-01 10:00:00', '2024-01-01 10:00:02', '2024-01-01 10:00:04']),
'ticker': ['AAPL', 'AAPL', 'AAPL'],
'bid': [149.5, 150.5, 150.0],
'ask': [150.5, 151.5, 151.0],
})
# Merge asof - find nearest quote for each trade
result = pd.merge_asof(
trades.sort_values('time'),
quotes.sort_values('time'),
on='time',
by='ticker',
direction='backward' # Use most recent quote
)
```
### Conditional Merge
```python
# Merge with conditions beyond key equality
# First merge, then filter
products = pd.DataFrame({
'product_id': [1, 2, 3],
'name': ['Widget', 'Gadget', 'Gizmo'],
'category': ['A', 'B', 'A'],
})
discounts = pd.DataFrame({
'category': ['A', 'A', 'B'],
'min_qty': [10, 50, 20],
'discount': [0.05, 0.10, 0.08],
})
# Cross merge then filter
merged = pd.merge(products, discounts, on='category')
# Then apply quantity-based filtering as needed
```
---
## Performance Considerations
### Pre-sorting for Merge
```python
# Sort keys before merge for better performance
df1 = df1.sort_values('key')
df2 = df2.sort_values('key')
# Merge sorted DataFrames
result = pd.merge(df1, df2, on='key')
```
### Index Alignment
```python
# Using index for merge is often faster than columns
df1 = df1.set_index('key')
df2 = df2.set_index('key')
# Join on index
result = df1.join(df2)
```
### Memory-Efficient Merge
```python
# For large DataFrames, reduce memory before merge
# Convert to appropriate types
df1['key'] = df1['key'].astype('int32') # Instead of int64
df1['category'] = df1['category'].astype('category')
# Select only needed columns
cols_needed = ['key', 'value1', 'value2']
result = pd.merge(df1[cols_needed], df2[cols_needed], on='key')
```
---
## Common Merge Patterns
### Left Join with Null Check
```python
# Find unmatched rows after left join
result = pd.merge(employees, departments, on='dept_id', how='left')
unmatched = result[result['dept_name'].isna()]
```
### Anti-Join (Rows Not in Other)
```python
# Find employees NOT in a specific department list
dept_list = [101, 102]
# Method 1: Using isin
not_in_depts = employees[~employees['dept_id'].isin(dept_list)]
# Method 2: Using merge with indicator
merged = pd.merge(
employees,
pd.DataFrame({'dept_id': dept_list}),
on='dept_id',
how='left',
indicator=True
)
not_in_depts = merged[merged['_merge'] == 'left_only']
```
### Self-Join
```python
# Find pairs within same department
employees = pd.DataFrame({
'emp_id': [1, 2, 3, 4],
'name': ['Alice', 'Bob', 'Charlie', 'Diana'],
'dept_id': [101, 101, 102, 101],
})
# Self-join to find pairs
pairs = pd.merge(
employees,
employees,
on='dept_id',
suffixes=('_1', '_2')
)
# Remove self-pairs and duplicates
pairs = pairs[pairs['emp_id_1'] < pairs['emp_id_2']]
```
---
## Best Practices Summary
1. **Choose the right join type** - Default inner may drop data
2. **Validate cardinality** - Use `validate` parameter
3. **Use indicator** - Debug unexpected results
4. **Handle duplicates** - Use meaningful suffixes
5. **Pre-sort for performance** - Especially for large DataFrames
6. **Reset index after operations** - Keep DataFrames usable
7. **Check for NaN after join** - Understand unmatched rows
---
## Anti-Patterns to Avoid
```python
# BAD: Merge without understanding cardinality
result = pd.merge(df1, df2, on='key') # May explode row count
# GOOD: Validate relationship
result = pd.merge(df1, df2, on='key', validate='one_to_one')
# BAD: Repeated merges
result = pd.merge(df1, df2, on='key')
result = pd.merge(result, df3, on='key')
result = pd.merge(result, df4, on='key')
# GOOD: Chain or use reduce
from functools import reduce
dfs = [df1, df2, df3, df4]
result = reduce(lambda left, right: pd.merge(left, right, on='key'), dfs)
# BAD: Ignoring merge indicators
result = pd.merge(df1, df2, on='key', how='outer')
# GOOD: Check merge results
result = pd.merge(df1, df2, on='key', how='outer', indicator=True)
print(result['_merge'].value_counts())
```
---
## Related References
- `dataframe-operations.md` - Filter before/after merge
- `aggregation-groupby.md` - Aggregate before merging
- `performance-optimization.md` - Optimize large merges
.agents/skills/pandas-pro/references/performance-optimization.md
+597
View File
@@ -0,0 +1,597 @@
# Performance Optimization
---
## Overview
Optimizing pandas performance is critical for production workflows. This reference covers memory optimization, vectorization, chunking, and profiling with pandas 2.0+.
---
## Memory Analysis
### Checking Memory Usage
```python
import pandas as pd
import numpy as np
df = pd.DataFrame({
'id': range(1_000_000),
'name': ['user_' + str(i) for i in range(1_000_000)],
'category': np.random.choice(['A', 'B', 'C', 'D'], 1_000_000),
'value': np.random.randn(1_000_000),
'count': np.random.randint(0, 100, 1_000_000),
})
# Basic memory info
print(df.info(memory_usage='deep'))
# Detailed memory by column
memory_usage = df.memory_usage(deep=True)
print(memory_usage)
print(f"Total: {memory_usage.sum() / 1e6:.2f} MB")
# Memory as percentage of total
memory_pct = (memory_usage / memory_usage.sum() * 100).round(2)
print(memory_pct)
```
### Memory Profiling Function
```python
def memory_profile(df: pd.DataFrame) -> pd.DataFrame:
"""Profile memory usage by column with optimization suggestions."""
memory_bytes = df.memory_usage(deep=True)
profile = pd.DataFrame({
'dtype': df.dtypes,
'non_null': df.count(),
'null_count': df.isna().sum(),
'unique': df.nunique(),
'memory_mb': (memory_bytes / 1e6).round(3),
})
# Add optimization suggestions
suggestions = []
for col in df.columns:
dtype = df[col].dtype
nunique = df[col].nunique()
if dtype == 'object':
if nunique / len(df) < 0.5: # Less than 50% unique
suggestions.append(f"Convert to category (only {nunique} unique)")
else:
suggestions.append("Consider string dtype")
elif dtype == 'int64':
if df[col].max() < 2**31 and df[col].min() >= -2**31:
suggestions.append("Downcast to int32")
if df[col].max() < 2**15 and df[col].min() >= -2**15:
suggestions.append("Downcast to int16")
elif dtype == 'float64':
suggestions.append("Consider float32 if precision allows")
else:
suggestions.append("OK")
profile['suggestion'] = suggestions
return profile
print(memory_profile(df))
```
---
## Memory Optimization Techniques
### Downcasting Numeric Types
```python
# Automatic downcasting for integers
df['count'] = pd.to_numeric(df['count'], downcast='integer')
# Automatic downcasting for floats
df['value'] = pd.to_numeric(df['value'], downcast='float')
# Manual downcasting function
def downcast_dtypes(df: pd.DataFrame) -> pd.DataFrame:
"""Reduce memory by downcasting numeric types."""
df = df.copy()
for col in df.select_dtypes(include=['int']).columns:
df[col] = pd.to_numeric(df[col], downcast='integer')
for col in df.select_dtypes(include=['float']).columns:
df[col] = pd.to_numeric(df[col], downcast='float')
return df
df_optimized = downcast_dtypes(df)
print(f"Before: {df.memory_usage(deep=True).sum() / 1e6:.2f} MB")
print(f"After: {df_optimized.memory_usage(deep=True).sum() / 1e6:.2f} MB")
```
### Using Categorical Type
```python
# Convert low-cardinality string columns to category
# Especially effective when unique values << total rows
# Before
print(f"Object dtype: {df['category'].memory_usage(deep=True) / 1e6:.2f} MB")
# After
df['category'] = df['category'].astype('category')
print(f"Category dtype: {df['category'].memory_usage(deep=True) / 1e6:.2f} MB")
# Automatic conversion for low-cardinality columns
def optimize_categories(df: pd.DataFrame, threshold: float = 0.5) -> pd.DataFrame:
"""Convert object columns to category if unique ratio < threshold."""
df = df.copy()
for col in df.select_dtypes(include=['object']).columns:
unique_ratio = df[col].nunique() / len(df)
if unique_ratio < threshold:
df[col] = df[col].astype('category')
return df
```
### Sparse Data Types
```python
# For data with many repeated values (especially zeros/NaN)
sparse_series = pd.arrays.SparseArray([0, 0, 1, 0, 0, 0, 2, 0, 0, 0])
# Create sparse DataFrame
df_sparse = pd.DataFrame({
'sparse_col': pd.arrays.SparseArray([0] * 9000 + [1] * 1000),
'dense_col': [0] * 9000 + [1] * 1000,
})
print(f"Sparse: {df_sparse['sparse_col'].memory_usage() / 1e6:.4f} MB")
print(f"Dense: {df_sparse['dense_col'].memory_usage() / 1e6:.4f} MB")
```
### Nullable Types (pandas 2.0+)
```python
# Use nullable types for proper NA handling with memory efficiency
df = df.astype({
'id': 'Int32', # Nullable int32
'count': 'Int16', # Nullable int16
'value': 'Float32', # Nullable float32
'name': 'string', # Nullable string (more memory efficient)
'category': 'category', # Categorical
})
# Arrow-backed types for even better memory (pandas 2.0+)
df['name'] = df['name'].astype('string[pyarrow]')
df['category'] = df['category'].astype('category')
```
---
## Vectorization
### Replace Loops with Vectorized Operations
```python
# BAD: Row iteration (extremely slow)
result = []
for idx, row in df.iterrows():
if row['value'] > 0:
result.append(row['value'] * 2)
else:
result.append(0)
df['result'] = result
# GOOD: Vectorized with np.where
df['result'] = np.where(df['value'] > 0, df['value'] * 2, 0)
# GOOD: Vectorized with boolean indexing
df['result'] = 0
df.loc[df['value'] > 0, 'result'] = df.loc[df['value'] > 0, 'value'] * 2
```
### Multiple Conditions with np.select
```python
# BAD: Nested if-else in apply
def categorize(row):
if row['value'] < -1:
return 'very_low'
elif row['value'] < 0:
return 'low'
elif row['value'] < 1:
return 'medium'
else:
return 'high'
df['category'] = df.apply(categorize, axis=1) # SLOW!
# GOOD: Vectorized with np.select
conditions = [
df['value'] < -1,
df['value'] < 0,
df['value'] < 1,
]
choices = ['very_low', 'low', 'medium']
df['category'] = np.select(conditions, choices, default='high')
```
### String Operations - Vectorized
```python
# BAD: Apply for string operations
df['upper_name'] = df['name'].apply(lambda x: x.upper())
# GOOD: Vectorized string methods
df['upper_name'] = df['name'].str.upper()
# Combine multiple string operations
df['processed'] = (
df['name']
.str.strip()
.str.lower()
.str.replace(r'\s+', '_', regex=True)
)
```
### Avoid apply() When Possible
```python
# BAD: apply for row-wise calculation
df['total'] = df.apply(lambda row: row['a'] + row['b'] + row['c'], axis=1)
# GOOD: Direct vectorized operation
df['total'] = df['a'] + df['b'] + df['c']
# BAD: apply for element-wise operation
df['squared'] = df['value'].apply(lambda x: x ** 2)
# GOOD: Vectorized
df['squared'] = df['value'] ** 2
# When apply IS appropriate: complex custom logic
def complex_calculation(row):
# Multiple dependencies and conditional logic
if row['type'] == 'A':
return row['value'] * row['multiplier'] + row['offset']
else:
return row['value'] / row['divisor'] - row['adjustment']
# Consider rewriting as vectorized if performance critical
```
---
## Chunked Processing
### Reading Large Files in Chunks
```python
# Read CSV in chunks
chunk_size = 100_000
chunks = []
for chunk in pd.read_csv('large_file.csv', chunksize=chunk_size):
# Process each chunk
processed = chunk[chunk['value'] > 0] # Filter
processed = processed.groupby('category')['value'].sum() # Aggregate
chunks.append(processed)
# Combine results
result = pd.concat(chunks).groupby(level=0).sum()
```
### Chunked Processing Function
```python
def process_large_csv(
filepath: str,
chunk_size: int = 100_000,
filter_func=None,
agg_func=None,
) -> pd.DataFrame:
"""Process large CSV files in chunks."""
results = []
for chunk in pd.read_csv(filepath, chunksize=chunk_size):
# Apply filter if provided
if filter_func:
chunk = filter_func(chunk)
# Apply aggregation if provided
if agg_func:
chunk = agg_func(chunk)
results.append(chunk)
# Combine results
combined = pd.concat(results, ignore_index=True)
# Re-aggregate if needed
if agg_func:
combined = agg_func(combined)
return combined
# Usage
result = process_large_csv(
'large_file.csv',
chunk_size=50_000,
filter_func=lambda df: df[df['value'] > 0],
agg_func=lambda df: df.groupby('category').agg({'value': 'sum'}),
)
```
### Memory-Efficient Iteration
```python
# When you must iterate, use itertuples (not iterrows)
# itertuples is 10-100x faster than iterrows
# BAD: iterrows
for idx, row in df.iterrows():
process(row['name'], row['value'])
# BETTER: itertuples
for row in df.itertuples():
process(row.name, row.value) # Access as attributes
# BEST: Vectorized operations (avoid iteration entirely)
```
---
## Query Optimization
### Efficient Filtering
```python
# Order matters - filter early, compute late
# BAD: Compute on all rows, then filter
df['expensive_calc'] = df['a'] * df['b'] + np.sin(df['c'])
result = df[df['category'] == 'A']
# GOOD: Filter first, compute on subset
mask = df['category'] == 'A'
result = df[mask].copy()
result['expensive_calc'] = result['a'] * result['b'] + np.sin(result['c'])
```
### Using query() for Performance
```python
# query() can be faster for large DataFrames (uses numexpr)
# Traditional boolean indexing
result = df[(df['value'] > 0) & (df['category'] == 'A')]
# query() syntax (faster for large data)
result = df.query('value > 0 and category == "A"')
# With variables
threshold = 0
cat = 'A'
result = df.query('value > @threshold and category == @cat')
```
### eval() for Complex Expressions
```python
# eval() uses numexpr for faster computation
# Standard pandas
df['result'] = df['a'] + df['b'] * df['c'] - df['d']
# Using eval (faster for large DataFrames)
df['result'] = pd.eval('df.a + df.b * df.c - df.d')
# In-place with inplace parameter
df.eval('result = a + b * c - d', inplace=True)
```
---
## GroupBy Optimization
### Pre-sort for Faster GroupBy
```python
# Sort by groupby column first
df = df.sort_values('category')
# Use sort=False since already sorted
result = df.groupby('category', sort=False)['value'].mean()
```
### Use Built-in Aggregations
```python
# BAD: Custom function via apply
result = df.groupby('category')['value'].apply(lambda x: x.mean())
# GOOD: Built-in aggregation
result = df.groupby('category')['value'].mean()
# Built-in aggregations available:
# sum, mean, median, min, max, std, var, count, first, last, nth
# size, sem, prod, cumsum, cummax, cummin, cumprod
```
### Observed Categories
```python
# For categorical columns, use observed=True (pandas 2.0+ default)
df['category'] = df['category'].astype('category')
# Avoid computing for unobserved categories
result = df.groupby('category', observed=True)['value'].mean()
```
---
## I/O Optimization
### Efficient File Formats
```python
# Parquet - best for analytical workloads
df.to_parquet('data.parquet', compression='snappy')
df = pd.read_parquet('data.parquet')
# Feather - best for pandas interchange
df.to_feather('data.feather')
df = pd.read_feather('data.feather')
# CSV with optimizations
df.to_csv('data.csv', index=False)
df = pd.read_csv(
'data.csv',
dtype={'category': 'category', 'count': 'int32'},
usecols=['id', 'category', 'value'], # Only needed columns
nrows=10000, # Limit rows for testing
)
```
### Specify dtypes When Reading
```python
# Specify dtypes upfront to avoid inference overhead
dtypes = {
'id': 'int32',
'name': 'string',
'category': 'category',
'value': 'float32',
'count': 'int16',
}
df = pd.read_csv('data.csv', dtype=dtypes)
# Parse dates efficiently
df = pd.read_csv(
'data.csv',
dtype=dtypes,
parse_dates=['date_column'],
date_format='%Y-%m-%d', # Explicit format is faster
)
```
---
## Profiling and Benchmarking
### Timing Operations
```python
import time
# Simple timing
start = time.time()
result = df.groupby('category')['value'].mean()
elapsed = time.time() - start
print(f"Elapsed: {elapsed:.4f} seconds")
# Using %%timeit in Jupyter
# %%timeit
# df.groupby('category')['value'].mean()
```
### Memory Profiling
```python
# Track memory before/after
import tracemalloc
tracemalloc.start()
# Your operation
df_result = df.groupby('category').agg({'value': 'sum'})
current, peak = tracemalloc.get_traced_memory()
print(f"Current memory: {current / 1e6:.2f} MB")
print(f"Peak memory: {peak / 1e6:.2f} MB")
tracemalloc.stop()
```
### Comparison Template
```python
def benchmark_operations(df: pd.DataFrame, operations: dict, n_runs: int = 5):
"""Benchmark multiple operations."""
results = {}
for name, func in operations.items():
times = []
for _ in range(n_runs):
start = time.time()
func(df)
times.append(time.time() - start)
results[name] = {
'mean': np.mean(times),
'std': np.std(times),
'min': np.min(times),
}
return pd.DataFrame(results).T
# Usage
operations = {
'iterrows': lambda df: [row['value'] for _, row in df.iterrows()],
'itertuples': lambda df: [row.value for row in df.itertuples()],
'vectorized': lambda df: df['value'].tolist(),
}
benchmark_results = benchmark_operations(df.head(10000), operations)
print(benchmark_results)
```
---
## Best Practices Summary
1. **Profile first** - Identify actual bottlenecks before optimizing
2. **Use appropriate dtypes** - int32/float32/category save memory
3. **Vectorize everything** - Avoid loops and apply when possible
4. **Filter early** - Reduce data before expensive operations
5. **Chunk large files** - Process in manageable pieces
6. **Use efficient file formats** - Parquet/Feather over CSV
7. **Leverage built-in methods** - Faster than custom functions
---
## Performance Checklist
Before deploying pandas code:
- [ ] Memory profiled with `memory_usage(deep=True)`
- [ ] Dtypes optimized (downcast, categorical)
- [ ] No iterrows/itertuples in hot paths
- [ ] GroupBy uses built-in aggregations
- [ ] Large files processed in chunks
- [ ] Filters applied before computations
- [ ] Appropriate file format used
- [ ] Benchmarked with representative data size
---
## Anti-Patterns Summary
| Anti-Pattern | Alternative |
|--------------|-------------|
| `iterrows()` for computation | Vectorized operations |
| `apply(lambda)` for simple ops | Built-in methods |
| Loading entire large file | Chunked reading |
| String columns with low cardinality | Category dtype |
| int64 for small integers | int32/int16 |
| Multiple separate filters | Combined boolean mask |
| Repeated groupby calls | Single groupby with multiple aggs |
---
## Related References
- `dataframe-operations.md` - Efficient indexing and filtering
- `aggregation-groupby.md` - Optimized aggregation patterns
- `merging-joining.md` - Efficient merge strategies
.agents/skills/python-executor/SKILL.md
+185
View File
@@ -0,0 +1,185 @@
---
name: python-executor
description: "Execute Python code in a safe sandboxed environment via [inference.sh](https://inference.sh). Pre-installed: NumPy, Pandas, Matplotlib, requests, BeautifulSoup, Selenium, Playwright, MoviePy, Pillow, OpenCV, trimesh, and 100+ more libraries. Use for: data processing, web scraping, image manipulation, video creation, 3D model processing, PDF generation, API calls, automation scripts. Triggers: python, execute code, run script, web scraping, data analysis, image processing, video editing, 3D models, automation, pandas, matplotlib"
allowed-tools: Bash(belt *)
---
# Python Code Executor
Execute Python code in a safe, sandboxed environment with 100+ pre-installed libraries.
![Python Code Executor](https://cloud.inference.sh/u/33sqbmzt3mrg2xxphnhw5g5ear/01k8d8b4mckh6z89dhtxh72dsz.png)
## Quick Start
> Requires inference.sh CLI (`belt`). [Install instructions](https://raw.githubusercontent.com/inference-sh/skills/refs/heads/main/cli-install.md)
```bash
belt login
# Run Python code
belt app run infsh/python-executor --input '{
"code": "import pandas as pd\nprint(pd.__version__)"
}'
```
## App Details
| Property | Value |
|----------|-------|
| App ID | `infsh/python-executor` |
| Environment | Python 3.10, CPU-only |
| RAM | 8GB (default) / 16GB (high_memory) |
| Timeout | 1-300 seconds (default: 30) |
## Input Schema
```json
{
"code": "print('Hello World!')",
"timeout": 30,
"capture_output": true,
"working_dir": null
}
```
## Pre-installed Libraries
### Web Scraping & HTTP
- `requests`, `httpx`, `aiohttp` - HTTP clients
- `beautifulsoup4`, `lxml` - HTML/XML parsing
- `selenium`, `playwright` - Browser automation
- `scrapy` - Web scraping framework
### Data Processing
- `numpy`, `pandas`, `scipy` - Numerical computing
- `matplotlib`, `seaborn`, `plotly` - Visualization
### Image Processing
- `pillow`, `opencv-python-headless` - Image manipulation
- `scikit-image`, `imageio` - Image algorithms
### Video & Audio
- `moviepy` - Video editing
- `av` (PyAV), `ffmpeg-python` - Video processing
- `pydub` - Audio manipulation
### 3D Processing
- `trimesh`, `open3d` - 3D mesh processing
- `numpy-stl`, `meshio`, `pyvista` - 3D file formats
### Documents & Graphics
- `svgwrite`, `cairosvg` - SVG creation
- `reportlab`, `pypdf2` - PDF generation
## Examples
### Web Scraping
```bash
belt app run infsh/python-executor --input '{
"code": "import requests\nfrom bs4 import BeautifulSoup\n\nresponse = requests.get(\"https://example.com\")\nsoup = BeautifulSoup(response.content, \"html.parser\")\nprint(soup.find(\"title\").text)"
}'
```
### Data Analysis with Visualization
```bash
belt app run infsh/python-executor --input '{
"code": "import pandas as pd\nimport matplotlib.pyplot as plt\n\ndata = {\"name\": [\"Alice\", \"Bob\"], \"sales\": [100, 150]}\ndf = pd.DataFrame(data)\n\nplt.bar(df[\"name\"], df[\"sales\"])\nplt.savefig(\"outputs/chart.png\")\nprint(\"Chart saved!\")"
}'
```
### Image Processing
```bash
belt app run infsh/python-executor --input '{
"code": "from PIL import Image\nimport numpy as np\n\n# Create gradient image\narr = np.linspace(0, 255, 256*256, dtype=np.uint8).reshape(256, 256)\nimg = Image.fromarray(arr, mode=\"L\")\nimg.save(\"outputs/gradient.png\")\nprint(\"Image created!\")"
}'
```
### Video Creation
```bash
belt app run infsh/python-executor --input '{
"code": "from moviepy.editor import ColorClip, TextClip, CompositeVideoClip\n\nclip = ColorClip(size=(640, 480), color=(0, 100, 200), duration=3)\ntxt = TextClip(\"Hello!\", fontsize=70, color=\"white\").set_position(\"center\").set_duration(3)\nvideo = CompositeVideoClip([clip, txt])\nvideo.write_videofile(\"outputs/hello.mp4\", fps=24)\nprint(\"Video created!\")",
"timeout": 120
}'
```
### 3D Model Processing
```bash
belt app run infsh/python-executor --input '{
"code": "import trimesh\n\nsphere = trimesh.creation.icosphere(subdivisions=3, radius=1.0)\nsphere.export(\"outputs/sphere.stl\")\nprint(f\"Created sphere with {len(sphere.vertices)} vertices\")"
}'
```
### API Calls
```bash
belt app run infsh/python-executor --input '{
"code": "import requests\nimport json\n\nresponse = requests.get(\"https://api.github.com/users/octocat\")\ndata = response.json()\nprint(json.dumps(data, indent=2))"
}'
```
## File Output
Files saved to `outputs/` are automatically returned:
```python
# These files will be in the response
plt.savefig('outputs/chart.png')
df.to_csv('outputs/data.csv')
video.write_videofile('outputs/video.mp4')
mesh.export('outputs/model.stl')
```
## Variants
```bash
# Default (8GB RAM)
belt app run infsh/python-executor --input input.json
# High memory (16GB RAM) for large datasets
belt app run infsh/python-executor@high_memory --input input.json
```
## Use Cases
- **Web scraping** - Extract data from websites
- **Data analysis** - Process and visualize datasets
- **Image manipulation** - Resize, crop, composite images
- **Video creation** - Generate videos with text overlays
- **3D processing** - Load, transform, export 3D models
- **API integration** - Call external APIs
- **PDF generation** - Create reports and documents
- **Automation** - Run any Python script
## Important Notes
- **CPU-only** - No GPU/ML libraries (use dedicated AI apps for that)
- **Safe execution** - Runs in isolated subprocess
- **Non-interactive** - Use `plt.savefig()` not `plt.show()`
- **File detection** - Output files are auto-detected and returned
## Related Skills
```bash
# AI image generation (for ML-based images)
npx skills add inference-sh/skills@ai-image-generation
# AI video generation (for ML-based videos)
npx skills add inference-sh/skills@ai-video-generation
# LLM models (for text generation)
npx skills add inference-sh/skills@llm-models
```
## Documentation
- [Running Apps](https://inference.sh/docs/apps/running) - How to run apps via CLI
- [App Code](https://inference.sh/docs/extend/app-code) - Understanding app execution
- [Sandboxed Code Execution](https://inference.sh/blog/tools/sandboxed-execution) - Safe code execution for agents
.agents/skills/python-testing-patterns/SKILL.md
+622
View File
@@ -0,0 +1,622 @@
---
name: python-testing-patterns
description: Implement comprehensive testing strategies with pytest, fixtures, mocking, and test-driven development. Use when writing Python tests, setting up test suites, or implementing testing best practices.
---
# Python Testing Patterns
Comprehensive guide to implementing robust testing strategies in Python using pytest, fixtures, mocking, parameterization, and test-driven development practices.
## When to Use This Skill
- Writing unit tests for Python code
- Setting up test suites and test infrastructure
- Implementing test-driven development (TDD)
- Creating integration tests for APIs and services
- Mocking external dependencies and services
- Testing async code and concurrent operations
- Setting up continuous testing in CI/CD
- Implementing property-based testing
- Testing database operations
- Debugging failing tests
## Core Concepts
### 1. Test Types
- **Unit Tests**: Test individual functions/classes in isolation
- **Integration Tests**: Test interaction between components
- **Functional Tests**: Test complete features end-to-end
- **Performance Tests**: Measure speed and resource usage
### 2. Test Structure (AAA Pattern)
- **Arrange**: Set up test data and preconditions
- **Act**: Execute the code under test
- **Assert**: Verify the results
### 3. Test Coverage
- Measure what code is exercised by tests
- Identify untested code paths
- Aim for meaningful coverage, not just high percentages
### 4. Test Isolation
- Tests should be independent
- No shared state between tests
- Each test should clean up after itself
## Quick Start
```python
# test_example.py
def add(a, b):
return a + b
def test_add():
"""Basic test example."""
result = add(2, 3)
assert result == 5
def test_add_negative():
"""Test with negative numbers."""
assert add(-1, 1) == 0
# Run with: pytest test_example.py
```
## Fundamental Patterns
### Pattern 1: Basic pytest Tests
```python
# test_calculator.py
import pytest
class Calculator:
"""Simple calculator for testing."""
def add(self, a: float, b: float) -> float:
return a + b
def subtract(self, a: float, b: float) -> float:
return a - b
def multiply(self, a: float, b: float) -> float:
return a * b
def divide(self, a: float, b: float) -> float:
if b == 0:
raise ValueError("Cannot divide by zero")
return a / b
def test_addition():
"""Test addition."""
calc = Calculator()
assert calc.add(2, 3) == 5
assert calc.add(-1, 1) == 0
assert calc.add(0, 0) == 0
def test_subtraction():
"""Test subtraction."""
calc = Calculator()
assert calc.subtract(5, 3) == 2
assert calc.subtract(0, 5) == -5
def test_multiplication():
"""Test multiplication."""
calc = Calculator()
assert calc.multiply(3, 4) == 12
assert calc.multiply(0, 5) == 0
def test_division():
"""Test division."""
calc = Calculator()
assert calc.divide(6, 3) == 2
assert calc.divide(5, 2) == 2.5
def test_division_by_zero():
"""Test division by zero raises error."""
calc = Calculator()
with pytest.raises(ValueError, match="Cannot divide by zero"):
calc.divide(5, 0)
```
### Pattern 2: Fixtures for Setup and Teardown
```python
# test_database.py
import pytest
from typing import Generator
class Database:
"""Simple database class."""
def __init__(self, connection_string: str):
self.connection_string = connection_string
self.connected = False
def connect(self):
"""Connect to database."""
self.connected = True
def disconnect(self):
"""Disconnect from database."""
self.connected = False
def query(self, sql: str) -> list:
"""Execute query."""
if not self.connected:
raise RuntimeError("Not connected")
return [{"id": 1, "name": "Test"}]
@pytest.fixture
def db() -> Generator[Database, None, None]:
"""Fixture that provides connected database."""
# Setup
database = Database("sqlite:///:memory:")
database.connect()
# Provide to test
yield database
# Teardown
database.disconnect()
def test_database_query(db):
"""Test database query with fixture."""
results = db.query("SELECT * FROM users")
assert len(results) == 1
assert results[0]["name"] == "Test"
@pytest.fixture(scope="session")
def app_config():
"""Session-scoped fixture - created once per test session."""
return {
"database_url": "postgresql://localhost/test",
"api_key": "test-key",
"debug": True
}
@pytest.fixture(scope="module")
def api_client(app_config):
"""Module-scoped fixture - created once per test module."""
# Setup expensive resource
client = {"config": app_config, "session": "active"}
yield client
# Cleanup
client["session"] = "closed"
def test_api_client(api_client):
"""Test using api client fixture."""
assert api_client["session"] == "active"
assert api_client["config"]["debug"] is True
```
### Pattern 3: Parameterized Tests
```python
# test_validation.py
import pytest
def is_valid_email(email: str) -> bool:
"""Check if email is valid."""
return "@" in email and "." in email.split("@")[1]
@pytest.mark.parametrize("email,expected", [
("user@example.com", True),
("test.user@domain.co.uk", True),
("invalid.email", False),
("@example.com", False),
("user@domain", False),
("", False),
])
def test_email_validation(email, expected):
"""Test email validation with various inputs."""
assert is_valid_email(email) == expected
@pytest.mark.parametrize("a,b,expected", [
(2, 3, 5),
(0, 0, 0),
(-1, 1, 0),
(100, 200, 300),
(-5, -5, -10),
])
def test_addition_parameterized(a, b, expected):
"""Test addition with multiple parameter sets."""
from test_calculator import Calculator
calc = Calculator()
assert calc.add(a, b) == expected
# Using pytest.param for special cases
@pytest.mark.parametrize("value,expected", [
pytest.param(1, True, id="positive"),
pytest.param(0, False, id="zero"),
pytest.param(-1, False, id="negative"),
])
def test_is_positive(value, expected):
"""Test with custom test IDs."""
assert (value > 0) == expected
```
### Pattern 4: Mocking with unittest.mock
```python
# test_api_client.py
import pytest
from unittest.mock import Mock, patch, MagicMock
import requests
class APIClient:
"""Simple API client."""
def __init__(self, base_url: str):
self.base_url = base_url
def get_user(self, user_id: int) -> dict:
"""Fetch user from API."""
response = requests.get(f"{self.base_url}/users/{user_id}")
response.raise_for_status()
return response.json()
def create_user(self, data: dict) -> dict:
"""Create new user."""
response = requests.post(f"{self.base_url}/users", json=data)
response.raise_for_status()
return response.json()
def test_get_user_success():
"""Test successful API call with mock."""
client = APIClient("https://api.example.com")
mock_response = Mock()
mock_response.json.return_value = {"id": 1, "name": "John Doe"}
mock_response.raise_for_status.return_value = None
with patch("requests.get", return_value=mock_response) as mock_get:
user = client.get_user(1)
assert user["id"] == 1
assert user["name"] == "John Doe"
mock_get.assert_called_once_with("https://api.example.com/users/1")
def test_get_user_not_found():
"""Test API call with 404 error."""
client = APIClient("https://api.example.com")
mock_response = Mock()
mock_response.raise_for_status.side_effect = requests.HTTPError("404 Not Found")
with patch("requests.get", return_value=mock_response):
with pytest.raises(requests.HTTPError):
client.get_user(999)
@patch("requests.post")
def test_create_user(mock_post):
"""Test user creation with decorator syntax."""
client = APIClient("https://api.example.com")
mock_post.return_value.json.return_value = {"id": 2, "name": "Jane Doe"}
mock_post.return_value.raise_for_status.return_value = None
user_data = {"name": "Jane Doe", "email": "jane@example.com"}
result = client.create_user(user_data)
assert result["id"] == 2
mock_post.assert_called_once()
call_args = mock_post.call_args
assert call_args.kwargs["json"] == user_data
```
### Pattern 5: Testing Exceptions
```python
# test_exceptions.py
import pytest
def divide(a: float, b: float) -> float:
"""Divide a by b."""
if b == 0:
raise ZeroDivisionError("Division by zero")
if not isinstance(a, (int, float)) or not isinstance(b, (int, float)):
raise TypeError("Arguments must be numbers")
return a / b
def test_zero_division():
"""Test exception is raised for division by zero."""
with pytest.raises(ZeroDivisionError):
divide(10, 0)
def test_zero_division_with_message():
"""Test exception message."""
with pytest.raises(ZeroDivisionError, match="Division by zero"):
divide(5, 0)
def test_type_error():
"""Test type error exception."""
with pytest.raises(TypeError, match="must be numbers"):
divide("10", 5)
def test_exception_info():
"""Test accessing exception info."""
with pytest.raises(ValueError) as exc_info:
int("not a number")
assert "invalid literal" in str(exc_info.value)
```
For advanced patterns including async testing, monkeypatching, temporary files, conftest setup, property-based testing, database testing, CI/CD integration, and configuration files, see [references/advanced-patterns.md](references/advanced-patterns.md)
## Test Design Principles
### One Behavior Per Test
Each test should verify exactly one behavior. This makes failures easy to diagnose and tests easy to maintain.
```python
# BAD - testing multiple behaviors
def test_user_service():
user = service.create_user(data)
assert user.id is not None
assert user.email == data["email"]
updated = service.update_user(user.id, {"name": "New"})
assert updated.name == "New"
# GOOD - focused tests
def test_create_user_assigns_id():
user = service.create_user(data)
assert user.id is not None
def test_create_user_stores_email():
user = service.create_user(data)
assert user.email == data["email"]
def test_update_user_changes_name():
user = service.create_user(data)
updated = service.update_user(user.id, {"name": "New"})
assert updated.name == "New"
```
### Test Error Paths
Always test failure cases, not just happy paths.
```python
def test_get_user_raises_not_found():
with pytest.raises(UserNotFoundError) as exc_info:
service.get_user("nonexistent-id")
assert "nonexistent-id" in str(exc_info.value)
def test_create_user_rejects_invalid_email():
with pytest.raises(ValueError, match="Invalid email format"):
service.create_user({"email": "not-an-email"})
```
## Testing Best Practices
### Test Organization
```python
# tests/
# __init__.py
# conftest.py # Shared fixtures
# test_unit/ # Unit tests
# test_models.py
# test_utils.py
# test_integration/ # Integration tests
# test_api.py
# test_database.py
# test_e2e/ # End-to-end tests
# test_workflows.py
```
### Test Naming Convention
A common pattern: `test_<unit>_<scenario>_<expected_outcome>`. Adapt to your team's preferences.
```python
# Pattern: test_<unit>_<scenario>_<expected>
def test_create_user_with_valid_data_returns_user():
...
def test_create_user_with_duplicate_email_raises_conflict():
...
def test_get_user_with_unknown_id_returns_none():
...
# Good test names - clear and descriptive
def test_user_creation_with_valid_data():
"""Clear name describes what is being tested."""
pass
def test_login_fails_with_invalid_password():
"""Name describes expected behavior."""
pass
def test_api_returns_404_for_missing_resource():
"""Specific about inputs and expected outcomes."""
pass
# Bad test names - avoid these
def test_1(): # Not descriptive
pass
def test_user(): # Too vague
pass
def test_function(): # Doesn't explain what's tested
pass
```
### Testing Retry Behavior
Verify that retry logic works correctly using mock side effects.
```python
from unittest.mock import Mock
def test_retries_on_transient_error():
"""Test that service retries on transient failures."""
client = Mock()
# Fail twice, then succeed
client.request.side_effect = [
ConnectionError("Failed"),
ConnectionError("Failed"),
{"status": "ok"},
]
service = ServiceWithRetry(client, max_retries=3)
result = service.fetch()
assert result == {"status": "ok"}
assert client.request.call_count == 3
def test_gives_up_after_max_retries():
"""Test that service stops retrying after max attempts."""
client = Mock()
client.request.side_effect = ConnectionError("Failed")
service = ServiceWithRetry(client, max_retries=3)
with pytest.raises(ConnectionError):
service.fetch()
assert client.request.call_count == 3
def test_does_not_retry_on_permanent_error():
"""Test that permanent errors are not retried."""
client = Mock()
client.request.side_effect = ValueError("Invalid input")
service = ServiceWithRetry(client, max_retries=3)
with pytest.raises(ValueError):
service.fetch()
# Only called once - no retry for ValueError
assert client.request.call_count == 1
```
### Mocking Time with Freezegun
Use freezegun to control time in tests for predictable time-dependent behavior.
```python
from freezegun import freeze_time
from datetime import datetime, timedelta
@freeze_time("2026-01-15 10:00:00")
def test_token_expiry():
"""Test token expires at correct time."""
token = create_token(expires_in_seconds=3600)
assert token.expires_at == datetime(2026, 1, 15, 11, 0, 0)
@freeze_time("2026-01-15 10:00:00")
def test_is_expired_returns_false_before_expiry():
"""Test token is not expired when within validity period."""
token = create_token(expires_in_seconds=3600)
assert not token.is_expired()
@freeze_time("2026-01-15 12:00:00")
def test_is_expired_returns_true_after_expiry():
"""Test token is expired after validity period."""
token = Token(expires_at=datetime(2026, 1, 15, 11, 30, 0))
assert token.is_expired()
def test_with_time_travel():
"""Test behavior across time using freeze_time context."""
with freeze_time("2026-01-01") as frozen_time:
item = create_item()
assert item.created_at == datetime(2026, 1, 1)
# Move forward in time
frozen_time.move_to("2026-01-15")
assert item.age_days == 14
```
### Test Markers
```python
# test_markers.py
import pytest
@pytest.mark.slow
def test_slow_operation():
"""Mark slow tests."""
import time
time.sleep(2)
@pytest.mark.integration
def test_database_integration():
"""Mark integration tests."""
pass
@pytest.mark.skip(reason="Feature not implemented yet")
def test_future_feature():
"""Skip tests temporarily."""
pass
@pytest.mark.skipif(os.name == "nt", reason="Unix only test")
def test_unix_specific():
"""Conditional skip."""
pass
@pytest.mark.xfail(reason="Known bug #123")
def test_known_bug():
"""Mark expected failures."""
assert False
# Run with:
# pytest -m slow # Run only slow tests
# pytest -m "not slow" # Skip slow tests
# pytest -m integration # Run integration tests
```
### Coverage Reporting
```bash
# Install coverage
pip install pytest-cov
# Run tests with coverage
pytest --cov=myapp tests/
# Generate HTML report
pytest --cov=myapp --cov-report=html tests/
# Fail if coverage below threshold
pytest --cov=myapp --cov-fail-under=80 tests/
# Show missing lines
pytest --cov=myapp --cov-report=term-missing tests/
```
For advanced patterns (async testing, monkeypatching, property-based testing, database testing, CI/CD integration, and configuration), see [references/advanced-patterns.md](references/advanced-patterns.md)
.agents/skills/python-testing-patterns/references/advanced-patterns.md
+411
View File
@@ -0,0 +1,411 @@
# Python Testing Patterns — Advanced Reference
Advanced testing patterns including async code, monkeypatching, temporary files, conftest setup, property-based testing, database testing, CI/CD integration, and configuration.
## Pattern 6: Testing Async Code
```python
# test_async.py
import pytest
import asyncio
async def fetch_data(url: str) -> dict:
"""Fetch data asynchronously."""
await asyncio.sleep(0.1)
return {"url": url, "data": "result"}
@pytest.mark.asyncio
async def test_fetch_data():
"""Test async function."""
result = await fetch_data("https://api.example.com")
assert result["url"] == "https://api.example.com"
assert "data" in result
@pytest.mark.asyncio
async def test_concurrent_fetches():
"""Test concurrent async operations."""
urls = ["url1", "url2", "url3"]
tasks = [fetch_data(url) for url in urls]
results = await asyncio.gather(*tasks)
assert len(results) == 3
assert all("data" in r for r in results)
@pytest.fixture
async def async_client():
"""Async fixture."""
client = {"connected": True}
yield client
client["connected"] = False
@pytest.mark.asyncio
async def test_with_async_fixture(async_client):
"""Test using async fixture."""
assert async_client["connected"] is True
```
## Pattern 7: Monkeypatch for Testing
```python
# test_environment.py
import os
import pytest
def get_database_url() -> str:
"""Get database URL from environment."""
return os.environ.get("DATABASE_URL", "sqlite:///:memory:")
def test_database_url_default():
"""Test default database URL."""
# Will use actual environment variable if set
url = get_database_url()
assert url
def test_database_url_custom(monkeypatch):
"""Test custom database URL with monkeypatch."""
monkeypatch.setenv("DATABASE_URL", "postgresql://localhost/test")
assert get_database_url() == "postgresql://localhost/test"
def test_database_url_not_set(monkeypatch):
"""Test when env var is not set."""
monkeypatch.delenv("DATABASE_URL", raising=False)
assert get_database_url() == "sqlite:///:memory:"
class Config:
"""Configuration class."""
def __init__(self):
self.api_key = "production-key"
def get_api_key(self):
return self.api_key
def test_monkeypatch_attribute(monkeypatch):
"""Test monkeypatching object attributes."""
config = Config()
monkeypatch.setattr(config, "api_key", "test-key")
assert config.get_api_key() == "test-key"
```
## Pattern 8: Temporary Files and Directories
```python
# test_file_operations.py
import pytest
from pathlib import Path
def save_data(filepath: Path, data: str):
"""Save data to file."""
filepath.write_text(data)
def load_data(filepath: Path) -> str:
"""Load data from file."""
return filepath.read_text()
def test_file_operations(tmp_path):
"""Test file operations with temporary directory."""
# tmp_path is a pathlib.Path object
test_file = tmp_path / "test_data.txt"
# Save data
save_data(test_file, "Hello, World!")
# Verify file exists
assert test_file.exists()
# Load and verify data
data = load_data(test_file)
assert data == "Hello, World!"
def test_multiple_files(tmp_path):
"""Test with multiple temporary files."""
files = {
"file1.txt": "Content 1",
"file2.txt": "Content 2",
"file3.txt": "Content 3"
}
for filename, content in files.items():
filepath = tmp_path / filename
save_data(filepath, content)
# Verify all files created
assert len(list(tmp_path.iterdir())) == 3
# Verify contents
for filename, expected_content in files.items():
filepath = tmp_path / filename
assert load_data(filepath) == expected_content
```
## Pattern 9: Custom Fixtures and Conftest
```python
# conftest.py
"""Shared fixtures for all tests."""
import pytest
@pytest.fixture(scope="session")
def database_url():
"""Provide database URL for all tests."""
return "postgresql://localhost/test_db"
@pytest.fixture(autouse=True)
def reset_database(database_url):
"""Auto-use fixture that runs before each test."""
# Setup: Clear database
print(f"Clearing database: {database_url}")
yield
# Teardown: Clean up
print("Test completed")
@pytest.fixture
def sample_user():
"""Provide sample user data."""
return {
"id": 1,
"name": "Test User",
"email": "test@example.com"
}
@pytest.fixture
def sample_users():
"""Provide list of sample users."""
return [
{"id": 1, "name": "User 1"},
{"id": 2, "name": "User 2"},
{"id": 3, "name": "User 3"},
]
# Parametrized fixture
@pytest.fixture(params=["sqlite", "postgresql", "mysql"])
def db_backend(request):
"""Fixture that runs tests with different database backends."""
return request.param
def test_with_db_backend(db_backend):
"""This test will run 3 times with different backends."""
print(f"Testing with {db_backend}")
assert db_backend in ["sqlite", "postgresql", "mysql"]
```
## Pattern 10: Property-Based Testing
```python
# test_properties.py
from hypothesis import given, strategies as st
import pytest
def reverse_string(s: str) -> str:
"""Reverse a string."""
return s[::-1]
@given(st.text())
def test_reverse_twice_is_original(s):
"""Property: reversing twice returns original."""
assert reverse_string(reverse_string(s)) == s
@given(st.text())
def test_reverse_length(s):
"""Property: reversed string has same length."""
assert len(reverse_string(s)) == len(s)
@given(st.integers(), st.integers())
def test_addition_commutative(a, b):
"""Property: addition is commutative."""
assert a + b == b + a
@given(st.lists(st.integers()))
def test_sorted_list_properties(lst):
"""Property: sorted list is ordered."""
sorted_lst = sorted(lst)
# Same length
assert len(sorted_lst) == len(lst)
# All elements present
assert set(sorted_lst) == set(lst)
# Is ordered
for i in range(len(sorted_lst) - 1):
assert sorted_lst[i] <= sorted_lst[i + 1]
```
## Testing Database Code
```python
# test_database_models.py
import pytest
from sqlalchemy import create_engine, Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker, Session
Base = declarative_base()
class User(Base):
"""User model."""
__tablename__ = "users"
id = Column(Integer, primary_key=True)
name = Column(String(50))
email = Column(String(100), unique=True)
@pytest.fixture(scope="function")
def db_session() -> Session:
"""Create in-memory database for testing."""
engine = create_engine("sqlite:///:memory:")
Base.metadata.create_all(engine)
SessionLocal = sessionmaker(bind=engine)
session = SessionLocal()
yield session
session.close()
def test_create_user(db_session):
"""Test creating a user."""
user = User(name="Test User", email="test@example.com")
db_session.add(user)
db_session.commit()
assert user.id is not None
assert user.name == "Test User"
def test_query_user(db_session):
"""Test querying users."""
user1 = User(name="User 1", email="user1@example.com")
user2 = User(name="User 2", email="user2@example.com")
db_session.add_all([user1, user2])
db_session.commit()
users = db_session.query(User).all()
assert len(users) == 2
def test_unique_email_constraint(db_session):
"""Test unique email constraint."""
from sqlalchemy.exc import IntegrityError
user1 = User(name="User 1", email="same@example.com")
user2 = User(name="User 2", email="same@example.com")
db_session.add(user1)
db_session.commit()
db_session.add(user2)
with pytest.raises(IntegrityError):
db_session.commit()
```
## CI/CD Integration
```yaml
# .github/workflows/test.yml
name: Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12"]
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
pip install -e ".[dev]"
pip install pytest pytest-cov
- name: Run tests
run: |
pytest --cov=myapp --cov-report=xml
- name: Upload coverage
uses: codecov/codecov-action@v3
with:
file: ./coverage.xml
```
## Configuration Files
```ini
# pytest.ini
[pytest]
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*
addopts =
-v
--strict-markers
--tb=short
--cov=myapp
--cov-report=term-missing
markers =
slow: marks tests as slow
integration: marks integration tests
unit: marks unit tests
e2e: marks end-to-end tests
```
```toml
# pyproject.toml
[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
addopts = [
"-v",
"--cov=myapp",
"--cov-report=term-missing",
]
[tool.coverage.run]
source = ["myapp"]
omit = ["*/tests/*", "*/migrations/*"]
[tool.coverage.report]
exclude_lines = [
"pragma: no cover",
"def __repr__",
"raise AssertionError",
"raise NotImplementedError",
]
```
.agents/skills/seo/SKILL.md
+513
View File
@@ -0,0 +1,513 @@
---
name: seo
description: Optimize for search engine visibility and ranking. Use when asked to "improve SEO", "optimize for search", "fix meta tags", "add structured data", "sitemap optimization", or "search engine optimization".
license: MIT
metadata:
author: web-quality-skills
version: "1.0"
---
# SEO optimization
Search engine optimization based on Lighthouse SEO audits and Google Search guidelines. Focus on technical SEO, on-page optimization, and structured data.
## SEO fundamentals
Search ranking factors (approximate influence):
| Factor | Influence | This Skill |
|--------|-----------|------------|
| Content quality & relevance | ~40% | Partial (structure) |
| Backlinks & authority | ~25% | ✗ |
| Technical SEO | ~15% | ✓ |
| Page experience (Core Web Vitals) | ~10% | See [Core Web Vitals](../core-web-vitals/SKILL.md) |
| On-page SEO | ~10% | ✓ |
---
## Technical SEO
### Crawlability
**robots.txt:**
```text
# /robots.txt
User-agent: *
Allow: /
# Block admin/private areas
Disallow: /admin/
Disallow: /api/
Disallow: /private/
# Don't block resources needed for rendering
# ❌ Disallow: /static/
Sitemap: https://example.com/sitemap.xml
```
**Meta robots:**
```html
<!-- Default: indexable, followable -->
<meta name="robots" content="index, follow">
<!-- Noindex specific pages -->
<meta name="robots" content="noindex, nofollow">
<!-- Indexable but don't follow links -->
<meta name="robots" content="index, nofollow">
<!-- Control snippets -->
<meta name="robots" content="max-snippet:150, max-image-preview:large">
```
**Canonical URLs:**
```html
<!-- Prevent duplicate content issues -->
<link rel="canonical" href="https://example.com/page">
<!-- Self-referencing canonical (recommended) -->
<link rel="canonical" href="https://example.com/current-page">
<!-- For paginated content -->
<link rel="canonical" href="https://example.com/products">
<!-- Or use rel="prev" / rel="next" for explicit pagination -->
```
### XML sitemap
```xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/</loc>
<lastmod>2024-01-15</lastmod>
<changefreq>daily</changefreq>
<priority>1.0</priority>
</url>
<url>
<loc>https://example.com/products</loc>
<lastmod>2024-01-14</lastmod>
<changefreq>weekly</changefreq>
<priority>0.8</priority>
</url>
</urlset>
```
**Sitemap best practices:**
- Maximum 50,000 URLs or 50MB per sitemap
- Use sitemap index for larger sites
- Include only canonical, indexable URLs
- Update `lastmod` when content changes
- Submit to Google Search Console
### URL structure
```
✅ Good URLs:
https://example.com/products/blue-widget
https://example.com/blog/how-to-use-widgets
❌ Poor URLs:
https://example.com/p?id=12345
https://example.com/products/item/category/subcategory/blue-widget-2024-sale-discount
```
**URL guidelines:**
- Use hyphens, not underscores
- Lowercase only
- Keep short (< 75 characters)
- Include target keywords naturally
- Avoid parameters when possible
- Use HTTPS always
### HTTPS & security
```html
<!-- Ensure all resources use HTTPS -->
<img src="https://example.com/image.jpg">
<!-- Not: -->
<img src="http://example.com/image.jpg">
```
**Security headers for SEO trust signals:**
```
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
```
---
## On-page SEO
### Title tags
```html
<!-- ❌ Missing or generic -->
<title>Page</title>
<title>Home</title>
<!-- ✅ Descriptive with primary keyword -->
<title>Blue Widgets for Sale | Premium Quality | Example Store</title>
```
**Title tag guidelines:**
- 50-60 characters (Google truncates ~60)
- Primary keyword near the beginning
- Unique for every page
- Brand name at end (unless homepage)
- Action-oriented when appropriate
### Meta descriptions
```html
<!-- ❌ Missing or duplicate -->
<meta name="description" content="">
<!-- ✅ Compelling and unique -->
<meta name="description" content="Shop premium blue widgets with free shipping. 30-day returns. Rated 4.9/5 by 10,000+ customers. Order today and save 20%.">
```
**Meta description guidelines:**
- 150-160 characters
- Include primary keyword naturally
- Compelling call-to-action
- Unique for every page
- Matches page content
### Heading structure
```html
<!-- ❌ Poor structure -->
<h2>Welcome to Our Store</h2>
<h4>Products</h4>
<h1>Contact Us</h1>
<!-- ✅ Proper hierarchy -->
<h1>Blue Widgets - Premium Quality</h1>
<h2>Product Features</h2>
<h3>Durability</h3>
<h3>Design</h3>
<h2>Customer Reviews</h2>
<h2>Pricing</h2>
```
**Heading guidelines:**
- Single `<h1>` per page (the main topic)
- Logical hierarchy (don't skip levels)
- Include keywords naturally
- Descriptive, not generic
### Image SEO
```html
<!-- ❌ Poor image SEO -->
<img src="IMG_12345.jpg">
<!-- ✅ Optimized image -->
<img src="blue-widget-product-photo.webp"
alt="Blue widget with chrome finish, side view showing control panel"
width="800"
height="600"
loading="lazy">
```
**Image guidelines:**
- Descriptive filenames with keywords
- Alt text describes the image content
- Compressed and properly sized
- WebP/AVIF with fallbacks
- Lazy load below-fold images
### Internal linking
```html
<!-- ❌ Non-descriptive -->
<a href="/products">Click here</a>
<a href="/widgets">Read more</a>
<!-- ✅ Descriptive anchor text -->
<a href="/products/blue-widgets">Browse our blue widget collection</a>
<a href="/guides/widget-maintenance">Learn how to maintain your widgets</a>
```
**Linking guidelines:**
- Descriptive anchor text with keywords
- Link to relevant internal pages
- Reasonable number of links per page
- Fix broken links promptly
- Use breadcrumbs for hierarchy
---
## Structured data (JSON-LD)
### Organization
```html
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Organization",
"name": "Example Company",
"url": "https://example.com",
"logo": "https://example.com/logo.png",
"sameAs": [
"https://twitter.com/example",
"https://linkedin.com/company/example"
],
"contactPoint": {
"@type": "ContactPoint",
"telephone": "+1-555-123-4567",
"contactType": "customer service"
}
}
</script>
```
### Article
```html
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "How to Choose the Right Widget",
"description": "Complete guide to selecting widgets for your needs.",
"image": "https://example.com/article-image.jpg",
"author": {
"@type": "Person",
"name": "Jane Smith",
"url": "https://example.com/authors/jane-smith"
},
"publisher": {
"@type": "Organization",
"name": "Example Blog",
"logo": {
"@type": "ImageObject",
"url": "https://example.com/logo.png"
}
},
"datePublished": "2024-01-15",
"dateModified": "2024-01-20"
}
</script>
```
### Product
```html
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Product",
"name": "Blue Widget Pro",
"image": "https://example.com/blue-widget.jpg",
"description": "Premium blue widget with advanced features.",
"brand": {
"@type": "Brand",
"name": "WidgetCo"
},
"offers": {
"@type": "Offer",
"price": "49.99",
"priceCurrency": "USD",
"availability": "https://schema.org/InStock",
"url": "https://example.com/products/blue-widget"
},
"aggregateRating": {
"@type": "AggregateRating",
"ratingValue": "4.8",
"reviewCount": "1250"
}
}
</script>
```
### FAQ
```html
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "What colors are available?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Our widgets come in blue, red, and green."
}
},
{
"@type": "Question",
"name": "What is the warranty?",
"acceptedAnswer": {
"@type": "Answer",
"text": "All widgets include a 2-year warranty."
}
}
]
}
</script>
```
### Breadcrumbs
```html
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "BreadcrumbList",
"itemListElement": [
{
"@type": "ListItem",
"position": 1,
"name": "Home",
"item": "https://example.com"
},
{
"@type": "ListItem",
"position": 2,
"name": "Products",
"item": "https://example.com/products"
},
{
"@type": "ListItem",
"position": 3,
"name": "Blue Widgets",
"item": "https://example.com/products/blue-widgets"
}
]
}
</script>
```
### Validation
Test structured data at:
- [Google Rich Results Test](https://search.google.com/test/rich-results)
- [Schema.org Validator](https://validator.schema.org/)
---
## Mobile SEO
### Responsive design
```html
<!-- ❌ Not mobile-friendly -->
<meta name="viewport" content="width=1024">
<!-- ✅ Responsive viewport -->
<meta name="viewport" content="width=device-width, initial-scale=1">
```
### Tap targets
```css
/* ❌ Too small for mobile */
.small-link {
padding: 4px;
font-size: 12px;
}
/* ✅ Adequate tap target */
.mobile-friendly-link {
padding: 12px;
font-size: 16px;
min-height: 48px;
min-width: 48px;
}
```
### Font sizes
```css
/* ❌ Too small on mobile */
body {
font-size: 10px;
}
/* ✅ Readable without zooming */
body {
font-size: 16px;
line-height: 1.5;
}
```
---
## International SEO
### Hreflang tags
```html
<!-- For multi-language sites -->
<link rel="alternate" hreflang="en" href="https://example.com/page">
<link rel="alternate" hreflang="es" href="https://example.com/es/page">
<link rel="alternate" hreflang="fr" href="https://example.com/fr/page">
<link rel="alternate" hreflang="x-default" href="https://example.com/page">
```
### Language declaration
```html
<html lang="en">
<!-- or -->
<html lang="es-MX">
```
---
## SEO audit checklist
### Critical
- [ ] HTTPS enabled
- [ ] robots.txt allows crawling
- [ ] No `noindex` on important pages
- [ ] Title tags present and unique
- [ ] Single `<h1>` per page
### High priority
- [ ] Meta descriptions present
- [ ] Sitemap submitted
- [ ] Canonical URLs set
- [ ] Mobile-responsive
- [ ] Core Web Vitals passing
### Medium priority
- [ ] Structured data implemented
- [ ] Internal linking strategy
- [ ] Image alt text
- [ ] Descriptive URLs
- [ ] Breadcrumb navigation
### Ongoing
- [ ] Fix crawl errors in Search Console
- [ ] Update sitemap when content changes
- [ ] Monitor ranking changes
- [ ] Check for broken links
- [ ] Review Search Console insights
---
## Tools
| Tool | Use |
|------|-----|
| Google Search Console | Monitor indexing, fix issues |
| Google PageSpeed Insights | Performance + Core Web Vitals |
| Rich Results Test | Validate structured data |
| Lighthouse | Full SEO audit |
| Screaming Frog | Crawl analysis |
## References
- [Google Search Central](https://developers.google.com/search)
- [Schema.org](https://schema.org/)
- [Core Web Vitals](../core-web-vitals/SKILL.md)
- [Web Quality Audit](../web-quality-audit/SKILL.md)
.claude/skills/accessibility
Symlink
+1
View File
@@ -0,0 +1 @@
../../.agents/skills/accessibility
.claude/skills/frontend-design
Symlink
+1
View File
@@ -0,0 +1 @@
../../.agents/skills/frontend-design
.claude/skills/pandas-data-analysis
Symlink
+1
View File
@@ -0,0 +1 @@
../../.agents/skills/pandas-data-analysis
.claude/skills/pandas-pro
Symlink
+1
View File
@@ -0,0 +1 @@
../../.agents/skills/pandas-pro
.claude/skills/python-executor
Symlink
+1
View File
@@ -0,0 +1 @@
../../.agents/skills/python-executor
.claude/skills/python-testing-patterns
Symlink
+1
View File
@@ -0,0 +1 @@
../../.agents/skills/python-testing-patterns
.claude/skills/seo
Symlink
+1
View File
@@ -0,0 +1 @@
../../.agents/skills/seo
skills-lock.json
+40
View File
@@ -0,0 +1,40 @@
{
"version": 1,
"skills": {
"accessibility": {
"source": "addyosmani/web-quality-skills",
"sourceType": "autoskills-registry",
"computedHash": "bffe3d08cfe92ebad63699f74ce29e35c19850ebfbf474c1463183cfe34d6a09"
},
"frontend-design": {
"source": "anthropics/skills",
"sourceType": "autoskills-registry",
"computedHash": "82fb11a63fb1e35ee2469516ed02d54695f783115b1540c0e783197af4240a3a"
},
"pandas-data-analysis": {
"source": "pluginagentmarketplace/custom-plugin-python",
"sourceType": "autoskills-registry",
"computedHash": "612eb33d227296b4bdd0ca62785b2e31124a92adf5f40b730b0cf3713adc6f04"
},
"pandas-pro": {
"source": "jeffallan/claude-skills",
"sourceType": "autoskills-registry",
"computedHash": "61f79ae3c9a29fd452e5cf8dee9891f1d0ab7da72fa80aa0366784bcd2b46d2b"
},
"python-executor": {
"source": "inferen-sh/skills",
"sourceType": "autoskills-registry",
"computedHash": "bd2d874c27788964fadf03c0840049de0409407f06f883ee90886164cb22ef69"
},
"python-testing-patterns": {
"source": "wshobson/agents",
"sourceType": "autoskills-registry",
"computedHash": "07b87d62993c0b6159a91d18fc8723b7f4c13d5000c0984266c207493ce641ff"
},
"seo": {
"source": "addyosmani/web-quality-skills",
"sourceType": "autoskills-registry",
"computedHash": "c184da724d1c61ad077f27418ea8e7e88fd54bcdf98165e18be7e4681cbd5e20"
}
}
}