Create a Compute Session with Mounted Folders

Note

Objectives

  • Mount persistent storage folders when creating a compute session
  • Data can be managed independently from the life cycle of the compute session

When you start a compute session, the user has access to the /home/work/ directory, and the normal directores and files created under /home/work/ will disappear when the compute session is destroyed. This is because compute sessions are dynamically created and deleted based on the container. To preserve the data inside the container independent of the life cycle of the container, a separate host folder must be mounted into the container, and then files must be created within the mounted folder. Backend.AI provides a function to mount storage folders when creating a compute session.

Let’s go to the Sessions page and click the START button to create a new compute session. In the session create dialog, click Folders to mount to see a list of storage folders that a user can mount. Among them, you can add them by clicking the folder you want to mount. You can also mount multiple folders simultaneously by clicking multiple items. In this example, we will mount two folders, user1-ml-test and user2-vfolder, and then create a compute session.

Launch a compute session with storage folders

Now, open the terminal by clicking the terminal icon in the created session. If you run ls command in the terminal, you can see that the user1-ml-test and user2-vfolder folders are mounted under the home directory. Let’s create a test_file under user2-vfolder to see if the file can be preserved after the compute session is destroyed. The contents of this file will filled with “file inside user2-vfolder”.

Mounted folders in terminal

If you run ls command against user2-vfolder, you can see that the file was created successfully. Also note the contents of the file with the cat command.

Now delete the compute session and go to the Storage page. Locate the user2-vfolder folder, open a file explorer and check that the file test_file exists. Click the file download button in Actions to download the file to the local machine and open it in an editor to confirm that the contents of the file are “file inside user2-vfolder”.

Download icon in the folder explorer

Like this, when creating a compute session, you can mount storage folders and perform any file operations on those mounted folders to save data even after the compute session termination.

Configuring a compute session environment using an automount folder

Sometimes you need a new program or library that is not pre-installed in a compute session. In that case, you can install packages and configure a certain environment regardless of the type of compute session by using the Storage folder, which persists independent of session lifecycle, and the automount folder.

Install Python packages via pip

Creating a folder named .local allows a user to install Python user packages in that folder. This is because installing a package with the –-user option appended to pip installs the package in the .local folder under the user’s home folder (note that automount folder is mounted under user’s home folder). So, if you want to install and keep the Python package tqdm regardless of the type of computing environment, you can issue the following command in your terminal:

pip install --user tqdm

After that, when a new compute session is created, the .local folder where the tqdm package is installed is automatically mounted, so you can use the tqdm package without reinstalling.

Warning

If you spawn multiple sessions that uses multiple Python versions, there may be compatibility issues with the packages. This can be circumvented by branching PYTHONPATH environment variable via the .bashrc. This is because the user’s pip package is installed in the path specified in the PYTHONPATH.

Install packages via Homebrew

Package managers like Ubuntu’s apt or CentOS’s yum usually require sudo permissions. For security, sudo and root accesses are blocked by default in Backend.AI’s compute session (it may allowed depending on the configuration), so we recommend to use Homebrew on Linux which does not require sudo. Homebrew can be installed and used in the following ways:

  • Create .linuxbrew folder in Data & Storage page
  • Create a compute session (.linuxbrew folder is automatically mounted)
  • Install Homebrew on Linux
sh -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)"
export PATH=/home/work/.linuxbrew/bin:$PATH
brew
  • Install package
brew install fortune
fortune

You can manage various settings using the automount folder in the same way as above. More details can be found on the Backend.AI wiki.