Information for Principal Investigators

Recent changes

During Q4 2014, NSC replaced the storage system that provides the shared file systems (/home, /nobackup/global, /software) to our Triolith, Kappa and Matter systems.

The new system has also replaced the Vagn storage system.

The most important change that you need to be aware of is that all storage is now connected to a project. Personal storage (i.e /nobackup/global) is no longer available1.

As the Principal Investigator (PI) of a project, you are responsible for providing enough storage to your project members, and to decide how files and directories should be stored within the project directory.

The day-to-day work of organizing data, discussing storage needs with NSC does not neccesarily have to be done by the PI. Please contact NSC Support if you want to delegate these tasks to someone else within your project.

What do you need to do as PI to manage storage?

  • Keep track of and manage your project's disk usage. Initially we will provide a command-line tool (snicquota) that you can run on the cluster that will report who is using how much of your project's disk space. Later we will make this information available in NSC Express. A notification system is in place that will email you when the project storage quota is exceeded (the emails can also be sent to all project members or a list of addresses if you prefer that).
  • Apply for more storage space when needed. See this page for more information on this.
  • Plan for what to do with your project's data when the project ends. If you're granted a new project that will continue using the same data, you can choose to keep your old project storage directory. A storage directory that is no longer connected to an active project becomes read-only after 30 days, and is deleted after another 30 days!

How much data can be stored in your project directory? (quota)

  • The total volume of data that can be stored in your project directory is limited.
  • The limit/allocation is different for each project.
  • New projects will be given a default allocation of 500 GiB. If you need more than that, please read this page for information on how to apply for more space.
  • Do not hesitate to apply for more space if that is needed to work efficiently in your project! The default allocation (500 GiB) is intentionally set fairly low. Most projects with many members or handling large data sets will need to apply for more space.
  • There is also a limit of how many files can be stored (because storing many small files has a grater "cost" in terms of system performance than storing few large files). If you need to store more files than the default limit of one million files per project, please contact NSC Support.
  • You can see how much space is allocated to your project in NSC Express, under "Projects" you will find it as a "Resource Allocation" of the resource "Centre Storage".

Limiting how much data each user can store

Most projects seems to get along fine without limiting how much each user can store inside the project directory.

However, it is possible for one user to fill up the entire project storage directory and cause problems (e.g failed jobs) for other users.

If you find that you need more control over who gets to use how much storage space, we can place limits on individual users or groups of users within a certain project directory. If you're interested in using such limits, please contact NSC Support.

Who can store data in the project directory?

By default, only members of your project will be able to read and write files in the project directory. It is possible to allow others to e.g read files from the project directory or parts of it. Please contact NSC Support if you want to open up your project directory to non-members.

Ownership of data and the rights to use project storage

NSC's position is that computing time and project storage are allocated to the project by SNAC, and that SNAC has authorized the PI to decide who gets to be a member in the project, and how much computing and storage resources each member may use.

In order to manage the project storage, the project PI (or someone designated by the PI) can therefore get assistance from NSC with the following:

  • See how much storage space is used by each project member.
  • Read the contents of any directories/files within the project storage directory (e.g to determine what type of data is stored if it should be kept).
  • Delete any data within the project storage directory (e.g to free up space used by unwanted files owned by inactive users).
  • Change ownership and file permissions of any data within the project directory (e.g to take ownership of still-relevant files from inactive users).

We strongly encourage PI:s to talk to project members before deleting or changing the ownership of their data.

Please note that NSC considers this to be unrelated to the intellectual property rights of the actual file contents. This just concerns the rights to store and organize bits and bytes on disk.

When a user is removed from a project, files in the project directory owned by that user are not automatically removed. It is up to the project PI to decide if the files should be kept on disk or not.

A user's home directory is considered to be personal and not connected to any project. NSC will not allow PI:s to delete or view contents from the home directory without permission from the user. Home directories of users no longer active at NSC will eventually be deleted.

What to do if your data does not fit into the one project - one directory model?

If the default model of "one project directory per project" does not match your workflow (e.g if your research group has several SNAC projects that need to share the same data), please contact NSC Support to discuss your options for data storage.

It is possible both to split an a storage allocation into two or more smaller project directories, or merge the allocations of two or more projects into one larger project directory.

The following conditions must be met for a merge or split to be approved:

  • The new storage project(s) must each have a named responsible person ("storage PI"). This person must have a good overview of the data stored in the project directory (what is stored? how much? why? by whom? ...) and be able to discuss the project's storage needs with NSC. This is especially important if the project requests an increased storage allocation, the responsible person must be able to explain to NSC in sufficient detail why more storage space is needed.
  • The merged storage project must not be so large that it becomes impossible for a single responsible person to have this overview. For example, NSC would not approve "/proj/uppsala_university".

  1. Exception: All users will have a small home directory for settings, small personal files etc. All large files and all files that should be shared with others should be put in the project storage directories under /proj.