--- title: "Search and Replace Lines with Sed: Master Advanced Techniques" description: "Master sed's search and replace functionality with practical examples - find, substitute, and transform text efficiently." date: 2025-07-10 categories: ["vps"] tags: ["sed","linux-commands"] --- Need to find and replace text in files? Sed's `s` command handles this in one line. Good for config updates, data cleaning, or quick edits without opening an editor. ## What is `sed`? **sed** (Stream Editor) processes text line by line without loading the whole file into memory. Good for: - Automation and scripts - Large files - Piping with other Unix tools - Batch processing Other sed guides: - [Delete lines](https://www.bitdoze.com/sed-delete-lines/) - [Insert or append text](https://www.bitdoze.com/sed-insert-append-text/) - [Transform text case](https://www.bitdoze.com/sed-change-case/) ## Basic Syntax ```sh sed 's/old/new/' file.txt # Replace first occurrence per line sed 's/old/new/g' file.txt # Replace all occurrences (global) sed 's/old/new/i' file.txt # Ignore case ``` ### Delimiters Use different delimiters when your pattern contains slashes: ```sh sed 's|/old/path|/new/path|g' config.txt sed 's#http://old#https://new#g' urls.txt ``` ### Key Options | Option | Function | Example | |--------|----------|---------| | `-i` | Edit files in-place | `sed -i 's/old/new/g' file.txt` | | `-i.bak` | Edit in-place with backup | `sed -i.bak 's/old/new/g' file.txt` | | `-n` | Suppress default output | `sed -n 's/old/new/p' file.txt` | | `-e` | Multiple commands | `sed -e 's/old/new/' -e 's/foo/bar/' file.txt` | **Important**: Test without `-i` first. Sed outputs to stdout by default. ## Common Examples **Config files:** ```sh sed 's/localhost/production-server/g' config.ini sed 's/port=8080/port=80/g' server.conf ``` **Code refactoring:** ```sh sed 's/oldFunction/newFunction/g' script.js sed 's/var /let /g' legacy.js ``` **Data cleanup:** ```sh sed 's/,/;/g' data.csv # Change delimiter sed 's/ / /g' document.txt # Fix double spaces sed 's/\t/ /g' file.txt # Tabs to spaces ``` ### File Operations ```sh sed 's/old/new/g' file.txt > newfile.txt # Save to new file sed -i 's/old/new/g' file.txt # Edit in place sed -i.bak 's/old/new/g' file.txt # Edit with backup ``` ### Multiple Replacements ```sh sed -e 's/old1/new1/g' -e 's/old2/new2/g' file.txt sed 's/old1/new1/g; s/old2/new2/g' file.txt ``` ### Safety Tips 1. Test without `-i` first 2. Use `-i.bak` for backups 3. Verify with `diff original.txt modified.txt` ## Advanced Pattern Matching with Regular Expressions Regular expressions unlock sed's full potential for complex search and replace operations. Master these patterns to handle sophisticated text transformations. ### Basic Regular Expression Elements **Wildcard character (`.`):** ```sh sed 's/t.st/test/g' filename # Matches "test", "tast", "t3st", etc. sed 's/c.t/cat/g' pets.txt # Matches "cat", "cut", "cot", etc. ``` **Character classes:** ```sh sed 's/[aeiou]/X/g' filename # Replace any vowel with X sed 's/[0-9]/N/g' filename # Replace any digit with N sed 's/[A-Z]/L/g' filename # Replace uppercase letters with L ``` **Predefined character classes:** ```sh sed 's/[[:digit:]]/N/g' filename # Replace digits (same as [0-9]) sed 's/[[:alpha:]]/L/g' filename # Replace letters sed 's/[[:space:]]/X/g' filename # Replace whitespace characters ``` ### Quantifiers **Zero or more (`*`):** ```sh sed 's/ab*c/X/g' filename # Matches "ac", "abc", "abbc", "abbbc" sed 's/[0-9]*/NUM/g' filename # Matches empty string or any digits ``` **One or more (`\+`):** ```sh sed 's/[0-9]\+/NUM/g' filename # Matches one or more digits sed 's/a\+/A/g' filename # Matches "a", "aa", "aaa", etc. ``` **Exact occurrences (`\{n\}`):** ```sh sed 's/[0-9]\{3\}/XXX/g' filename # Matches exactly 3 digits sed 's/a\{2,4\}/A/g' filename # Matches 2 to 4 'a' characters ``` ### Anchors and Boundaries **Line anchors:** ```sh sed 's/^Error/WARNING/' filename # Replace "Error" at line start sed 's/end$/END/' filename # Replace "end" at line end sed 's/^$/EMPTY/' filename # Replace empty lines ``` **Word boundaries:** ```sh sed 's/\bcat\b/dog/g' filename # Replace whole word "cat" only sed 's/\btest\b/exam/g' filename # Avoids matching "testing" or "retest" ``` ### Practical Advanced Examples **Phone number formatting:** ```sh # Transform (123) 456-7890 to 123-456-7890 sed 's/(\([0-9]\{3\}\)) \([0-9]\{3\}\)-\([0-9]\{4\}\)/\1-\2-\3/g' contacts.txt ``` **Email extraction and masking:** ```sh # Replace email addresses with [EMAIL] sed 's/[a-zA-Z0-9._%+-]\+@[a-zA-Z0-9.-]\+\.[a-zA-Z]\{2,\}/[EMAIL]/g' data.txt ``` **Date format conversion:** ```sh # Convert MM/DD/YYYY to YYYY-MM-DD sed 's/\([0-9]\{2\}\)\/\([0-9]\{2\}\)\/\([0-9]\{4\}\)/\3-\1-\2/g' dates.txt ``` **URL protocol updates:** ```sh # Change HTTP to HTTPS sed 's/http:\/\/\([^[:space:]]*\)/https:\/\/\1/g' urls.txt ``` ### Grouping and Back-references **Capture groups with `\(\)` and back-references with `\1`, `\2`:** ```sh # Swap first and last names sed 's/\([A-Za-z]*\) \([A-Za-z]*\)/\2, \1/g' names.txt # Duplicate words detection and removal sed 's/\b\([a-zA-Z]\+\) \1\b/\1/g' text.txt # Extract filename from path sed 's/.*\/\([^\/]*\)$/\1/' paths.txt ``` ### Complex Pattern Examples **Log processing:** ```sh # Extract timestamp from log entries sed 's/^\[\([0-9-: ]*\)\] .*/\1/' server.log # Replace IP addresses with [IP] sed 's/\([0-9]\{1,3\}\.\)\{3\}[0-9]\{1,3\}/[IP]/g' access.log ``` **Code refactoring:** ```sh # Update function calls: oldFunc(param) -> newFunc(param) sed 's/oldFunc(\([^)]*\))/newFunc(\1)/g' code.js # Convert single quotes to double quotes in strings sed "s/'\([^']*\)\"/\"\1\"/g" script.js ``` ### Extended Regular Expressions **Using `-E` flag for enhanced patterns:** ```sh # Multiple alternatives with | sed -E 's/(cat|dog|bird)/animal/g' pets.txt # Simplified quantifiers (no escaping needed) sed -E 's/[0-9]{3}-[0-9]{2}-[0-9]{4}/XXX-XX-XXXX/g' ssn.txt # Non-capturing groups sed -E 's/(http|https)://[^[:space:]]*/[URL]/g' text.txt ``` ### Testing and Debugging Regular Expressions **Preview matches before replacement:** ```sh # Show what would be matched grep 'pattern' filename # Show line numbers with matches grep -n 'pattern' filename # Test with sed's print flag sed -n 's/pattern/replacement/p' filename ``` **Build patterns incrementally:** ```sh # Start simple sed 's/[0-9]/X/' filename # Add complexity gradually sed 's/[0-9]\+/NUM/' filename # Final complex pattern sed 's/[0-9]\{3\}-[0-9]\{2\}-[0-9]\{4\}/XXX-XX-XXXX/' filename ``` **Pro tip**: Regular expressions can be tricky. Always test your patterns thoroughly on sample data before applying to important files. ## Advanced sed Techniques and Best Practices Master these advanced techniques to make your sed operations more efficient, precise, and safe. ### Targeted Line Processing **Limit operations to specific line ranges:** ```sh sed '1,10s/old/new/g' file.txt # Replace only in lines 1-10 sed '5,$s/old/new/g' file.txt # Replace from line 5 to end sed '10s/old/new/g' file.txt # Replace only on line 10 ``` **Target lines by pattern:** ```sh sed '/pattern/s/old/new/g' file.txt # Replace only in lines containing "pattern" sed '/^#/s/old/new/g' file.txt # Replace only in comment lines sed '/ERROR/s/old/new/g' log.txt # Replace only in error lines ``` ### Preview and Testing Techniques **Preview changes before applying:** ```sh sed -n 's/old/new/p' file.txt # Show only changed lines sed 's/old/new/g' file.txt | head -20 # Preview first 20 lines sed 's/old/new/g' file.txt | diff file.txt - # Show differences ``` **Test with line numbers:** ```sh nl file.txt | sed 's/old/new/g' # Show line numbers for context ``` ### Working with Special Characters **Escape literal characters:** ```sh sed 's/\./DOT/g' file.txt # Escape literal dots sed 's/\*/STAR/g' file.txt # Escape literal asterisks sed 's/\$/DOLLAR/g' file.txt # Escape literal dollar signs sed 's/\//SLASH/g' file.txt # Escape literal forward slashes ``` **Use alternative delimiters:** ```sh sed 's|/old/path|/new/path|g' file.txt # Use | for paths sed 's#http://old#https://new#g' file.txt # Use # for URLs sed 's@old@new@g' file.txt # Use @ as delimiter ``` ### Batch Processing Multiple Files **Process all files of a type:** ```sh find . -name "*.txt" -exec sed -i 's/old/new/g' {} \; find . -name "*.conf" -exec sed -i.bak 's/old/new/g' {} \; ``` **Using xargs for efficiency:** ```sh find . -name "*.txt" | xargs sed -i 's/old/new/g' find . -name "*.js" -print0 | xargs -0 sed -i 's/console.log/logger.debug/g' ``` **Loop through files:** ```sh for file in *.txt; do sed -i.backup 's/old/new/g' "$file" echo "Processed: $file" done ``` ### Advanced Pattern Techniques **Using back-references for complex replacements:** ```sh # Swap two words sed 's/\(foo\) \(bar\)/\2 \1/g' file.txt # Duplicate text sed 's/\(important\)/\1 \1/g' file.txt # Rearrange data fields sed 's/\([^,]*\),\([^,]*\),\([^,]*\)/\3,\1,\2/' data.csv ``` **Multiple operations in sequence:** ```sh sed -e 's/old1/new1/g' -e 's/old2/new2/g' -e 's/old3/new3/g' file.txt ``` **Conditional replacements:** ```sh # Replace only if line contains specific pattern sed '/contains_this/{s/old/new/g;}' file.txt # Replace in specific sections sed '/START/,/END/{s/old/new/g;}' file.txt ``` ### Performance Optimization **Process large files efficiently:** ```sh # Stop after first match per line (faster) sed 's/old/new/' file.txt # Use specific patterns to reduce processing sed '/pattern/s/old/new/g' file.txt ``` **Combine operations to reduce passes:** ```sh # Instead of multiple sed calls sed 's/old1/new1/g; s/old2/new2/g; s/old3/new3/g' file.txt ``` ### Safety and Backup Strategies **Always backup important files:** ```sh cp original.txt original.txt.backup sed -i.$(date +%Y%m%d) 's/old/new/g' original.txt ``` **Test on sample data first:** ```sh head -100 largefile.txt > sample.txt sed 's/old/new/g' sample.txt # Test your pattern # If good, apply to original: sed -i.backup 's/old/new/g' largefile.txt ``` **Use version control:** ```sh git add file.txt # Stage current version sed -i 's/old/new/g' file.txt # Make changes git diff # Review changes ``` ### Practical Workflow Examples **Configuration file updates:** ```sh # Update server configuration across multiple files find /etc/nginx -name "*.conf" -exec sed -i.backup \ -e 's/old-server.com/new-server.com/g' \ -e 's/port 8080/port 80/g' {} \; ``` **Code refactoring:** ```sh # Update function names in JavaScript files find ./src -name "*.js" -exec sed -i \ 's/\boldFunction\b/newFunction/g' {} \; ``` **Log processing:** ```sh # Clean and standardize log files sed -e 's/DEBUG/[DEBUG]/g' \ -e 's/ERROR/[ERROR]/g' \ -e 's/INFO/[INFO]/g' app.log > standardized.log ``` ### Common Pitfalls to Avoid 1. **Forgetting to escape special characters** 2. **Not testing patterns before applying to important files** 3. **Using global replacement when you only want first occurrence** 4. **Not backing up files before in-place editing** 5. **Making patterns too broad (matching unintended text)** ### Pro Tips - Start with simple patterns and add complexity gradually - Use `grep` to test your patterns before using in `sed` - Keep a collection of tested sed patterns for reuse - Document complex regular expressions for future reference - Consider using `awk` or `perl` for very complex text processing **Remember**: The key to mastering sed is practice and patience. Build your skills incrementally and always prioritize data safety. ## Conclusion Mastering `sed` for searching and replacing text is a game-changer for anyone who works with text files regularly. The tricks I've shared are just the start to harnessing the full potential of this powerful stream editor. Remember to craft your commands with precision and always double-check your patterns. Whether you're tweaking a single file or tackling multiple files at once, `sed` can be your best ally—just be sure to back up your data before you dive in. With these strategies in your toolkit, you're well on your way to becoming a `sed` command wizard. Happy editing!