Step-by-Step Manual: How to Manage Your Files & Directories for Bioinformatics Projects

1. Define a Clear Directory Structure

Create a standardized hierarchy for your projects to avoid confusion and ensure consistency. Here’s a suggested structure:

PROJECT_NAME/
├── README.txt                # Project overview and instructions
├── planning/                 # Initial project planning files
├── data/                     # Raw and processed data
│   ├── raw/                  # Raw data (e.g., FASTQ, BAM)
│   ├── processed/            # Processed data (e.g., filtered, normalized)
│   └── metadata/             # Metadata files (e.g., sample information)
├── code/                     # Scripts and source code
│   ├── src/                  # Source code for analysis
│   ├── scripts/              # Utility scripts
│   └── pipelines/            # Workflow definitions (e.g., Makefiles, Snakemake)
├── results/                  # Analysis results
│   ├── tables/               # Tabular results (e.g., CSV, TSV)
│   ├── plots/                # Figures and visualizations
│   └── logs/                 # Log files from analyses
├── manuscript/               # Manuscript-related files
│   ├── figures/              # Final figures for publication
│   ├── tables/               # Final tables for publication
│   └── drafts/               # Drafts of the manuscript
└── archive/                  # Archived files (e.g., old versions, unused data)

2. Use Descriptive File and Folder Names

• Avoid vague names like file1.txt or results_final_v2.xlsx.
• Use consistent naming conventions, such as:
  • PROJECTNAME_DATE_DESCRIPTION_EXT (e.g., RNAseq_20231012_raw_reads.fastq.gz).
  • Include metadata in filenames (e.g., sampleID_condition_replicate).

3. Document Everything

• README Files: Include a README.txt in every directory to describe its contents, purpose, and any relevant instructions.
• Metadata: Store metadata files (e.g., sample information, experimental conditions) in a dedicated folder.
• Version Control: Use tools like Git to track changes in code and documentation.

4. Separate Raw Data from Processed Data

• Keep raw data immutable and store it in a dedicated raw/ folder.
• Processed data should be stored separately in a processed/ folder with clear documentation on how it was generated.

5. Use Version Control for Code

• Store all scripts and code in a code/ or src/ directory.
• Use Git or another version control system to track changes and collaborate with others.
• Include a requirements.txt or environment.yml file to document dependencies.

6. Automate Workflows

• Use workflow managers like Snakemake, Nextflow, or Makefiles to automate repetitive tasks.
• Store workflow definitions in a pipelines/ directory.

7. Organize Results

• Store results in a results/ directory, subdivided into tables/, plots/, and logs/.
• Use descriptive names for result files to make them easy to locate and interpret.

8. Archive Old Files

• Move completed projects or outdated files to an archive/ directory.
• Compress large files to save space (e.g., .tar.gz or .zip).

9. Use Tools for File Management

• Bash History: Use directory-specific bash history to track commands (e.g., dieter.plaetinck.be/per_directory_bash_history).
• File Comments: Use tools like Total Commander to add comments to files.
• Cloud Storage: Use services like Dropbox or Sparkleshare for file sharing and synchronization.

10. Ensure Reproducibility

• Document all steps in your analysis pipeline.
• Use tools like Sweave (for R) or Jupyter Notebooks (for Python) to combine code, results, and documentation.
• Store all parameters and configurations in a parameters/ directory.

11. Backup Your Data

• Regularly back up your data to external drives or cloud storage.
• Use automated backup tools to ensure no data is lost.

12. Collaborate Effectively

• Use shared drives or cloud-based platforms (e.g., Google Drive, Dropbox) for team collaboration.
• Maintain a central wiki or documentation system to track file locations and project progress.

13. Adopt Best Practices from Literature

• Refer to William Stafford Noble’s article: A Quick Guide to Organizing Computational Biology Projects.
• Explore tools like Sumatra for tracking computational experiments.

14. Regularly Review and Clean Up

• Periodically review your directory structure and remove unnecessary files.
• Ensure all team members follow the same organizational standards.

Example Workflow

1. Start a New Project:
   • Create a PROJECT_NAME/ directory with subfolders (data/, code/, results/, etc.).
   • Add a README.txt to describe the project.
2. Store Raw Data:
   • Place raw data in data/raw/ and document its source in data/metadata/.
3. Write and Store Code:
   • Store scripts in code/src/ and use Git for version control.
4. Run Analyses:
   • Use workflow managers to automate analyses and store results in results/.
5. Document Results:
   • Add descriptions of results in README.txt or a dedicated results/logs/ file.
6. Archive Completed Work:
   • Move finished projects to archive/ and compress large files.

By following these steps, you can create a well-organized and reproducible bioinformatics project that is easy to manage and share with collaborators.