Step-by-Step Guide to Managing and Transferring Large NGS Data

Next-Generation Sequencing (NGS) generates massive amounts of data, often in the form of FASTQ files, which can be several gigabytes in size. Efficient storage and transfer of these files are critical for downstream analysis. This guide provides strategies for managing and transferring large NGS data, including compression techniques, storage solutions, and transfer protocols.

Step 1: Data Compression

1.1 Why Compress NGS Data?

• Reduced Storage Requirements: Compressed files take up less disk space.
• Faster Transfers: Smaller files transfer more quickly over networks.
• Cost Efficiency: Lower storage and bandwidth costs.

1.2 Compression Tools

1.2.1 gzip

• Command:
  bash

  Copy

  gzip my_data.fastq

• Decompression:
  bash

  Copy

  gunzip my_data.fastq.gz

1.2.2 bzip2

• Command:
  bash

  Copy

  bzip2 my_data.fastq

• Decompression:
  bash

  Copy

  bunzip2 my_data.fastq.bz2

1.2.3 DSRC (DNA Sequence Reads Compressor)

• Command:
  bash

  Copy

  dsrc e my_data.fastq my_data.fastq.dsrc

• Decompression:
  bash

  Copy

  dsrc d my_data.fastq.dsrc my_data.fastq

1.2.4 BAM Format

• Convert FASTQ to BAM using Picard:
  bash

  Copy

  java -Xms2048m -jar FastqToSam.jar \
    FASTQ=my_data.fastq \
    QUALITY_FORMAT=Standard \
    OUTPUT=my_data.fastq.bam \
    READ_GROUP_NAME=RGN_test \
    SAMPLE_NAME=SampName \
    PLATFORM=Illumina

Step 2: Storage Solutions

2.1 Local Storage

• External Hard Drives: Use eSATA or USB 3.0 for fast transfers.
• Network-Attached Storage (NAS): Ideal for small to medium-sized labs.
• Storage Area Network (SAN): Suitable for large-scale operations but expensive.

2.2 Cloud Storage

• Amazon S3: Scalable and cost-effective.
• Google Cloud Storage: Integrated with Google’s data analysis tools.
• Microsoft Azure: Offers hybrid cloud solutions.

2.3 Tape Backup

• Long-Term Archiving: Cost-effective for long-term storage.
• LTO Tapes: High capacity and durability.

Step 3: Data Transfer Protocols

3.1 LAN Transfer

• Gigabit Ethernet: Provides speeds up to 1 Gbps.
• 10 Gigabit Ethernet: Offers speeds up to 10 Gbps for faster transfers.

3.2 High-Speed Transfer Tools

3.2.1 Aspera

• Command:
  bash

  Copy

  ascp -i aspera_key -l 1000M -k 1 user@host:/path/to/file /local/path

3.2.2 FDT (Fast Data Transfer)

• Command:
  bash

  Copy

  java -jar fdt.jar -c host -d /local/path -p 12345

3.2.3 Tsunami

• Command:
  bash

  Copy

  tsunami -c host -p 12345 -f /local/path

3.3 rsync

• Command:
  bash

  Copy

  rsync -avz --progress user@host:/path/to/file /local/path

Step 4: Data Management Strategies

4.1 Metadata Tracking

• Use BAM headers to store metadata.
• Example:
  bash

  Copy

  @RG ID:1 SM:SampName PL:Illumina LB:Lib1 PU:Run1

4.2 Data Archiving

• Retention Policy: Define what data to keep and for how long.
• Compression: Archive old data in compressed formats.

4.3 Data Cleaning

• Remove Redundant Data: Delete raw images and intermediate files.
• Automate Cleanup: Use cron jobs or scripts to automate data cleaning.

Step 5: Grid Computing and Parallel Processing

5.1 Grid Computing

• HTCondor: Manages job scheduling across distributed resources.
• Sun Grid Engine (SGE): Handles job queuing and resource allocation.

5.2 Parallel Processing

• Multicore Alignment: Use tools like BWA-MEM or Bowtie2 with multiple threads.
  bash

  Copy

  bwa mem -t 8 reference.fa reads.fq > aligned.sam

• Cluster Computing: Distribute tasks across multiple nodes using MPI.

Step 6: Example Workflow

6.1 Data Compression and Transfer

1. Compress FASTQ files using DSRC:
   bash

   Copy

   dsrc e my_data.fastq my_data.fastq.dsrc

2. Transfer compressed files using Aspera:
   bash

   Copy

   ascp -i aspera_key -l 1000M -k 1 user@host:/path/to/my_data.fastq.dsrc /local/path

6.2 Data Analysis

1. Decompress files:
   bash

   Copy

   dsrc d my_data.fastq.dsrc my_data.fastq

2. Align reads using BWA:
   bash

   Copy

   bwa mem -t 8 reference.fa my_data.fastq > aligned.sam

Conclusion

Efficiently managing and transferring large NGS data involves a combination of compression techniques, robust storage solutions, and high-speed transfer protocols. By following this guide, you can optimize your data handling processes, ensuring that your NGS data is stored securely and transferred quickly for downstream analysis. Whether you are working in a small lab or a large-scale sequencing facility, these strategies will help you manage your data effectively.