Chat Page#

Chat page enables users to conveniently compare and interact with different LLM models all in one location. This allows users to experience the services offered by Backend.AI as well as a variety of large language models (LLMs).

Selecting models#

Users can select the deployment and model from the top left corner of each chat card on the Chat page. Clicking the Deployment field (indicated by a deployment icon prefix) opens a dropdown showing available deployments under a "Deployment" header. Once a deployment is selected, the model dropdown header updates to show "{deployment name}'s Models", listing the models associated with that deployment.

If models cannot be loaded for the selected deployment, a configuration panel appears with the following alerts:

• LLM models not found (warning): The model list could not be retrieved from the deployment's endpoint. Adjust the base path or token in the panel below and click Refresh Model Information to retry.
• Desired replica count is 0 (warning): The selected deployment has its replica count set to 0 and cannot respond. Set the desired replica count to 1 or more in the Deployment settings to enable it.

The following alerts may also appear in the chat card:

• Endpoint URL is not valid (error): The endpoint URL for the selected deployment could not be resolved. Verify that the deployment is properly configured and has a reachable URL.
• Streaming error (error): An error occurred while communicating with the model. The message describes the cause. Dismiss the alert and retry your message after resolving the issue.

Refer to the description below for the necessary inputs to configure custom model settings:

• Base Path (optional): The path suffix appended to the deployment's endpoint URL when sending requests. Make sure to include the version information. For instance, when utilizing the OpenAI API, enter v1.
• Token (optional): An authentication key to access the model service. Tokens can be generated from various services, not just Backend.AI. The format and generation process may vary depending on the service. Always refer to the specific service's guide for details. For instance, when using the service generated by Backend.AI, please refer to the Generating Tokens section for instructions on how to generate tokens.

Add or remove comparison chat cards#

To add new comparison chat cards, click the comparison icon button located in the top right corner.

To remove a chat session, click on the more button located in the upper right corner of the chat card. Then a dropdown menu will appear, and users can select Delete Chat to remove a chat session. Please be cautious as this will delete all entered content.

Clear chat history#

Clicking the more button will reveal the Clear chat option. By selecting this, users will erase all chat history associated with the card, although the card session itself will remain active.

Synchronize input#

The Sync Input button, located at the top right, enables the synchronization of input across chat cards where the option is enabled. Enabling 'Sync input' means that pressing Enter or clicking the Send button on any card will submit the input from the card users are currently working on. This functionality is beneficial for comparing the outputs of various models using identical input data.

Parameter adjustment#

Click the parameter button in the top-right corner to adjust the parameters for each model. Users can set various values such as Max Tokens, Temperature, Top P, and Top K. Using the synchronize feature, users can apply different parameters to the same model and then compare the results.

Chat history#

To start new chat, click the + button located in the top right corner.

All chat history is stored in local storage, and users can access previous chats by clicking the history button in the top-right corner.