Handing Folders for Consistent Management of Data and Modlels



  • Concept of the storage folder
  • Create/delete/navigate a storage folder
  • Automount storage folder

Backend.AI supports dedicated storage to preserve user’s files. Since the files and directories of a compute session are deleted upon session termination, it is recommended to save them in a storage folder. List of storage folders can be found by selecting the Storage on the left sidebar. You can see the information such as the folder name and ID, the NFS host name where the folder is located (Location), and folder access rights (Permission).

Folder list in Storage page

There are two types of storage folders. User type folders can be created by normal users, and you can check that ther is one user icon in the Type column. On the other hand, Group folders can be recognized by an icon with multiple users in the column. Group folders are created by domain admins, and normal users can only see group folders which are created in a group where user belongs.

Create storage folder

You can create a storage folder with the desired name by clicking the NEW FOLDER button. Enter the name of the folder to be created in Folder name, and select one of User / Group for Type. (Depending on the server settings, only one of User or Group may be selectable.) When creating a group folder, the Group field must be set. The group folder will be bind to the group specified in the Group field, and only users belonging to the group can mount and use the group folder. After setting the values as desired, you can create a folder by clicking the CREATE button.

Folder creation dialog

The meaning of each fields that can be selected in the creation dialog is as follows.

  • Folder name: The name of the folder. You can enter up to 64 characters.
  • Host: NFS host to create folder. You can choose if you have multiple NFS hosts.
  • Type: Determines the type of folder to be created. It can be set as User or Group. The User folder is a folder that users can create and can use alone, and the Group folder can be created by admin and shared by users in the group.
  • Usage Mode: You can set the purpose of the folder. There are three types: General, Data, and Model. It is classified for the development of functions exclusively for data and model stores in the future, and currently there is no difference in UI depending on the purpose.
  • Group: When creating a group folder, designate the group to which the folder belongs. Group folders must belong to a group. However, it does not play any role when creating a user folder.

Explore folder

You can click the folder icon in the Control column to bring up a file explorer where you can view the contents of that folder.

Controls in folder item

You can see that directories and files inside the folder will be listed, if exists. Click a directory name in the Name column to move to the directory. You can click the download button or delete button in the Actions column to download it or delete it entirely from the directory. You can rename a file/directory as well. For more detailed file operations, you can mount this folder when creating a compute session, and then use a service like Terminal or Jupyter Notebook to do it.

File explorer of a storage folder

You can create a new directory on the current path with the NEW FOLDER button (in the file explorer), or upload a local file with the UPLOAD FILES button. All of these file operations can also be performed using the above-described method of mounting folders into a compute session.

The maximum length of file or directory inside a folder may depends on the host file system. But, it usually cannot exceed 255 characters.

To close file explorer, click the X button in the upper right.

Rename folder

If you have permission to rename the storage folder, you can rename it by clicking the edit icon in the Control column. When you click the icon button, a rename dialog will appears. Write new folder name and then click RENAME button.

Delete folder

If you have permission to delete the storage folder, you can delete it by clicking the trash can icon in the Control column. When you click the Delete button, a confirm dialog appears. To prevent accidental deletion, you have to enter the name of the folder to be deleted, explicitly.

Folder deletion dialog

The folders created here can be mounted when creating a compute session. Folders are mounted under the user’s default working directory, /home/work/, and the files stored in this directory will not be deleted when the compute session is terminated. (However, if you delete the folder itself, it will be gone).

Automount folder

Storage & Folders page has an Automount Folders tab. Click this tab to see a list of folders whose names prefixed with a dot (.). When you create a folder, if you specify a name that starts with a dot (.), it is added to the Automount Folders tab, not the Folders tab. Automount Folders are special folders that are automatically mounted in your home directory even if you do not mount them manually when creating a compute session. By using this feature, creating and using Storage folders such as .local, .linuxbrew, .pyenv, and etc., you can configure a certain user packages or environments that do not change with different kinds of compute session.

For more detailed information on the usage of Automount folders, refer to examples of using automount folders.

Automount folders