Parse, transform, and analyze CSV files with advanced data manipulation capabilities.

CSV Processor Skill Parse, transform, and analyze CSV files with advanced data manipulation capabilities. Instructions You are a CSV processing expert. When invoked: 1. Parse CSV Files : Auto detect delimiters (comma, tab, semicolon, pipe) Handle different encodings (UTF 8, Latin 1, Windows 1252) Process quoted fields and escaped characters Handle multi line fields correctly Detect and use header rows 2. Transform Data : Filter rows based on conditions Select specific columns Sort and group data Merge multiple CSV files Split large files into smaller chunks Pivot and unpivot data 3. Clean Data : Remove duplicates Handle missing values Trim whitespace Normalize data formats Fix encoding issues Validate data types 4. Analyze Data : Generate statistics (sum, average, min, max, count) Identify data quality issues Detect outliers Profile column data types Calculate distributions Usage Examples Basic CSV Operations Reading CSV Files Python (pandas) JavaScript (csv parser) Python (csv module) Writing CSV Files Python (pandas) JavaScript (csv writer) Data Transformation Patterns Filtering Rows Python (pandas) JavaScript Selecting Columns Python (pandas) JavaScript Sorting Data Python (pandas) JavaScript Grouping and Aggregation Python (pandas) JavaScript (lodash) Merging CSV Files Python (pandas) JavaScript Data Cleaning Operations Remove Duplicates Python (pandas) Handle Missing Values Python (pandas) JavaScript Data Validation Python (pandas) Data Normalization Python (pandas) Data Analysis Operations Statistical Summary Python (pandas) Data Profiling Python (pandas) Generate Report Advanced Operations Splitting Large CSV Files Pivot and Unpivot Data Type Conversion Performance Optimization Reading Large Files Efficiently Writing Large Files Command Line Tools Using csvkit Using awk Best Practices 1. Always validate data before processing 2. Use appropriate data types to save memory 3. Handle encoding issues early in the process 4. Profile data first to understand structure 5. Use chunks for large files 6. Back up original files before transformations 7. Document transformations for reproducibility 8. Validate output after processing 9. Use version control for CSV processing scripts 10. Test with sample data before processing full datasets Common Issues and Solutions Issue: Encoding Errors Issue: Delimiter Detection Issue: Memory Errors Notes Always inspect CSV structure before processing Test transformations on a small sample first Consider using databases for very large datasets Document column meanings and data types Use consistent date and number formats Validate data quality regularly Keep processing scripts version controlled

csv-processor