Documentation
Research
HiPerGator
Standard Operating Procedure: HiPerGator

Standard Operating Procedure: HiPerGator

This SOP outlines the procedures for accessing and utilizing the HiPerGator supercomputer resources by members of the Biomedical Informatics Lab under Dr. Mei Liu. Applicable to all lab members, including postdocs, doctoral students, data scientists, master’s students, and temporary research assistants.

Account Overview

  • Group Faculty Sponsor: Dr. Yonghui Wu
  • Lab PI: Dr. Mei Liu
  • Access: Each member is granted individual access credentials which must be used in accordance with university policies, this SOP, and the UF HOBI HiPerGator Usage Guidelines.

Directory Structure and Usage

For detailed information on the directory structure and usage, please refer to the HiPerGator Storage page and the HiPerGator QOS page.

  • Home Directory (/home/user_name)

    • Purpose: For storing software builds, conda environments, configuration files, and scripts.
    • Access: Private to each member.
    • Space Limitation: Storage space in this directory is limited. Do not store datasets here.

  • Blue Directory (/yonghui.wu/blue/user_name)

    • Purpose: For active research projects and fast-access data storage needs.
    • Access: Shared among all group members by default.
    • Usage: Store actively used data and project files in this directory. Ideal for data requiring high-speed access.

  • Orange Directory (/yonghui.wu/orange/user_name)

    • Purpose: For backups and less frequently accessed data.
    • Access: Shared among all group members by default.
    • Usage: Use this directory for backing up important files and storing large data not regularly accessed. Do not use it for data-intensive operations.

Access and Security

For detailed information on how to set up the HiPerGator account, please refer to the HiPerGator Setup Guide.

  • Requesting Access: New account access can be requested through the Request HiPerGator Account link. Additionally, send an email to the lab admin to coordinate with the faculty sponsor for approval of your request.
  • Credentials: Use your UF credentials to access HiPerGator. Do not share your login information with others.
  • Data Security: All restricted data must be stored on HiPerGator. Members should not download non-aggregated data onto personal or work devices to ensure data security. Please refer to the UF Data Classification Guidelines to determine the sensitivity level of the data.

Data Management

  • File Organization: Maintain a clear directory structure and naming convention for files and folders to ensure easy access and collaboration. For more details, see the Coding Guidelines.
  • Data Transfer: Use secure data transfer methods such as SFTP or SSH for transferring data to and from HiPerGator. For more details, see the HiPerGator Setup Guide.
  • Regular Backups: Schedule regular backups for important data—whether raw data or data that need to be “frozen” as a reference point—to the Orange Directory to prevent data loss.

Resource Utilization

  • Job Scheduling: Utilize the SLURM scheduler for running computational tasks. Ensure that jobs are correctly prioritized and do not unnecessarily consume resources. For jobs that require extra resources, follow the departmental coordination policies detailed in the HiPerGator QOS page. For more information on SLURM, please refer to the HiPerGator SLURM Guide.
  • Monitoring Usage: Regularly monitor your data storage and compute usage. Stay within assigned quotas for our group (see HiPerGator QOS for allocated quotas) unless additional resources are approved. Methods for managing usage can be found on the HiPerGator QOS page.

Compliance and Etiquette

Training and Support

  • Training Sessions:

    • External Training: The UFRC provides training courses on HiPerGator usage. You can access these courses at HiPerGator Training.
    • Internal Training: Additional training sessions are available upon request from our data scientists. These sessions offer tailored guidance and answers to specific queries.

  • Support: If you encounter issues that require more direct or immediate assistance, please reach out to the UFIT RC Support System.

Integrating GitHub with HiPerGator

  • Code Management: Members should refer to the GitHub SOP for detailed instructions on using Git on HiPerGator and processes related to sharing code with our lab’s GitHub account.
  • Collaborative Workflow: For a comprehensive guide on integrating HiPerGator, personal GitHub accounts, and the lab GitHub account, refer to our Git Workflow Guidelines.

Review and Amendments

  • SOP Review:The SOP will be reviewed annually or as necessary to adapt to changes in technology, university policies, or lab requirements.
  • Feedback and Changes: We encourage all team members to provide feedback on the SOP. All proposed amendments will be reviewed by the admins, and changes will be implemented based on consensus and alignment with lab goals and policies.