Customs AI

Status: Done

Description: A game made using Unity and OpenAI ChatGPT with the purpose of showing the potential of AI as a training tool for border control officers.

Year:2024

Software: Unity, OpenAI API

Download Link

Design Process

The purpose of this game was to integrate ChatGPT into Unity and make a training tool the the Marechaussee

The main activities you can do in the game are:

-Interrogate Travelers

You are able to talk with the NPCs and ask them questions to discern if they are just travelers or people with bad intentions

-Observe behavior

NPC’s display different emotions when talking this behavior is different for bad NPCs and Good NPCs

Development

Integration of ChatGPT into Unity:

  • The OpenAI-Unity repository was used to simplify communication between Unity and the OpenAI API.

  • An OpenAI account was created, and an API secret key was generated.

  • Authentication data, including the secret key and organization name, were stored in a .json file in the C:\Users\UserName.openAI directory.

  1. Script Development for Communication:

    • A main script named ChatGPT handles communication with the ChatGPT API.

    • The SendRequest() method sends messages to the API and retrieves responses.

    • Two key scriptable objects were created:

      • npcBehaviour (ChatPrompt): Stores NPC-specific prompts and behaviors.

      • conversationData (ConversationData): Stores the entire conversation between the player and the NPCs.

  2. NPC Initialization and Context Setup:

    • NPCs receive their starting prompts based on their names via a dictionary in the Start() method.

    • Certain NPCs, designated as "bad guys," intentionally omit current time data to make their responses suspicious.

  3. NPC Management:

    • An NPCManager script controls NPC logic, including:

      • Shuffling NPCs: Uses the Fisher-Yates algorithm to randomize the order of NPCs.

      • Moving NPCs: Moves NPCs to specified positions using a coroutine for smooth transitions.

      • Updating NPC Data: Updates UI elements with NPC-specific information (e.g., name, nationality, date of birth).

  4. Addressing NPC Character Consistency:

    • Initial attempts at prompt engineering to keep NPCs in character were unsuccessful due to the default assistant nature of the ChatGPT 3.5 model.

    • Custom models were trained using 30 dialogues per character (7 NPCs total) to maintain character consistency.

    • Dialogues included common questions and tailored responses to prevent NPCs from breaking character.

  5. Example Training Dialog:

    • For instance, Gabriella Cornelisse is characterized as vibrant and outgoing, often using "darling" in her responses.

    • Training involved crafting potential player questions and specific AI responses to guide the AI's behavior during interactions.

Scripts

using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.EventSystems;
using UnityEngine.UI;

namespace OpenAI
{
    public class ChatGPT : MonoBehaviour
    {
        [Header("UI Elements")]
        [SerializeField] private InputField inputField;
        [SerializeField] private ScrollRect scroll;
        [SerializeField] private RectTransform sent;
        [SerializeField] private RectTransform received;
        [SerializeField] private Button buttonSend;

        [Header("NPC Settings")]
        [SerializeField] private ChatPrompt npcBehaviour;
        [SerializeField] private ConversationData conversationData;
        [SerializeField] private Animator npcAnimator;
        [SerializeField] private Animator talkCloud;
        [SerializeField] private float typingSpeed;

        [Header("Sound Settings")]
        [SerializeField] private AudioClip[] voiceSounds; // Array of different typing sounds
        [SerializeField] private AudioSource typeSoundSource;

        [Header("Timer Settings")]
        [SerializeField] private int emotionCheckPoint;

        [Header("OpenAI Settings")]
        public string model;
        public float temperature;

        private float height;
        private OpenAIApi openai = new OpenAIApi();
        private List<ChatMessage> messages = new List<ChatMessage>();
        private SpriteRenderer spriteRenderer;
        private bool userMessageSending;
        private bool npcMessageSending;
        private bool typing;
        private bool isTimerRunning;
        private float conversationStartTime;
        private float elapsedTime;
        private int chatType;

        public virtual void Start()
        {
            if (conversationData == null)
            {
                Debug.LogError("ConversationData is not assigned!");
                return;
            }
            // Clear conversation data at the start of each session
            conversationData.ClearConversation();
            spriteRenderer = GetComponent<SpriteRenderer>();

            // Get the current date and time
            DateTime currentDateTime = DateTime.Now;

            // Format the date and time as a string with year, month, day, and hour
            string formattedDateTime = currentDateTime.ToString("yyyy-MM-dd HH");
            string dayOfWeekString = currentDateTime.DayOfWeek.ToString();

            // Dictionary to map NPC names to their behaviors
            Dictionary<string, Func<string>> npcBehaviors = new Dictionary<string, Func<string>>()
            {
                { "Mohamed Atta", () => npcBehaviour.NpcBehaviour },
                { "Elena Gilbert", () => npcBehaviour.NpcBehaviour + formattedDateTime + dayOfWeekString },
                { "Frank Abagnale", () => npcBehaviour.NpcBehaviour },
                { "Timothy McVeigh", () => npcBehaviour.NpcBehaviour },
                { "Gabriella Cornelisse", () => npcBehaviour.NpcBehaviour + formattedDateTime + dayOfWeekString },
                { "Kenji Tanaka", () => npcBehaviour.NpcBehaviour + formattedDateTime + dayOfWeekString },
                { "Robby Naish", () => npcBehaviour.NpcBehaviour + formattedDateTime + dayOfWeekString }
            };

            if (npcBehaviors.ContainsKey(npcBehaviour.name))
            {
                // Add system prompt to messages list
                var systemPrompt = new ChatMessage()
                {
                    Role = "system",
                    Content = npcBehaviors[npcBehaviour.name]()
                };
                messages.Add(systemPrompt);
            }
        }

        private void AppendUserMessage(ChatMessage message)
        {
            StartCoroutine(TypeUserMessage(message));
        }

        private void AppendNPCMessage(ChatMessage message)
        {
            StartCoroutine(TypeNPCMessage(message));
        }

        private IEnumerator TypeUserMessage(ChatMessage message)
        {
            if (!isTimerRunning)
            {
                StartTimer();
            }
            userMessageSending = true;
            scroll.content.SetSizeWithCurrentAnchors(RectTransform.Axis.Vertical, 0);
            var item = Instantiate(sent, scroll.content);
            var textComponent = item.GetChild(0).GetChild(0).GetComponent<Text>();

            string fullText = message.Content;
            message.Content = ">";
            textComponent.text = message.Content;

            LayoutRebuilder.ForceRebuildLayoutImmediate(item);

            float firstMessageHeight = item.sizeDelta.y;
            float currentHeight = height - firstMessageHeight;

            for (int i = 0; i < fullText.Length; i++)
            {
                if (!typing)
                {
                    typing = true;
                }
               
                message.Content += fullText[i];
                textComponent.text = message.Content;

                LayoutRebuilder.ForceRebuildLayoutImmediate(item);

                float newHeight = height + item.sizeDelta.y;
                scroll.content.SetSizeWithCurrentAnchors(RectTransform.Axis.Vertical, newHeight);

                item.anchoredPosition = new Vector2(0, -currentHeight);
                currentHeight = newHeight;
                ScrollToBottom();
                yield return new WaitForSeconds(0.05f);
            }
            typing = false;
            height = currentHeight;
            userMessageSending = false;

        }

        private IEnumerator TypeNPCMessage(ChatMessage message)
        {
            // Wait until user message is sent
            while (userMessageSending)
            {
                yield return null;
            }
            npcMessageSending = true;
            var item = Instantiate(received, scroll.content);
            var textComponent = item.GetChild(0).GetChild(0).GetComponent<Text>();

            string fullText = message.Content;
            message.Content = "";
            textComponent.text = message.Content;

            float firstMessageHeight = item.sizeDelta.y;
            float currentHeight = height;

            npcAnimator.SetBool("npcChatting"+chatType, true);
            talkCloud.SetBool("talkCloud", true);
            yield return new WaitForSeconds(1);
            for (int i = 0; i < fullText.Length; i++)
            {
                if (!typing)
                {
                    typing = true;
                }
                // Play a random typing sound
                PlayRandomTypeSound();
                item.anchoredPosition = new Vector2(0, -height);
                message.Content += fullText[i];
                textComponent.text = message.Content;

                float newHeight = height + item.sizeDelta.y;
                scroll.content.SetSizeWithCurrentAnchors(RectTransform.Axis.Vertical, newHeight);
                currentHeight = newHeight;

                ScrollToBottom();
                yield return new WaitForSeconds(typingSpeed);
            }

            typing = false;
            height = currentHeight;
            scroll.verticalNormalizedPosition = 0;
            npcAnimator.SetBool("npcChatting2", false);
            npcAnimator.SetBool("npcChatting1", false);
            talkCloud.SetBool("talkCloud", false);
            npcMessageSending = false;

        }

        private async void SendReply()
        {
            var userMessage = inputField.text;

            var userChatEntry = new ConversationData.ChatEntry()
            {
                role = "Officer",
                content = userMessage
            };
            var division = new ConversationData.ChatEntry()
            {
                role = "",
                content = ""
            };

            conversationData.chatEntries.Add(userChatEntry);
            conversationData.chatEntries.Add(division);

            var newMessage = new ChatMessage()
            {
                Role = "user",
                Content = userMessage
            };

            AppendUserMessage(newMessage);

            messages.Add(newMessage);

            if (inputField != null)
            {
                inputField.text = "";
            }

            var completionResponse = await openai.CreateChatCompletion(new CreateChatCompletionRequest()
            {
                Model = model,
                Messages = messages,
                Temperature = temperature,
            });

            if (completionResponse.Choices != null && completionResponse.Choices.Count > 0)
            {
                var message = completionResponse.Choices[0].Message;
                message.Content = message.Content.Trim();

                var aiChatEntry = new ConversationData.ChatEntry()
                {
                    role = npcBehaviour.name,
                    content = message.Content
                };
                var proceds = new ConversationData.ChatEntry()
                {
                    role = "",
                    content = ""
                };
                conversationData.chatEntries.Add(aiChatEntry);
                conversationData.chatEntries.Add(proceds);

                messages.Add(message);
                AppendNPCMessage(message);
            }
            else
            {
                Debug.LogWarning("No text was generated from this prompt.");
            }
        }

        // Update method to set the flag when user message is being sent
        public virtual void Update()
        {
           
            if (inputField != null)
            {
                if (!string.IsNullOrWhiteSpace(inputField.text) && Input.GetKeyDown(KeyCode.Return))
                {
                    userMessageSending = true; // Set flag when user message is being sent
                    SendReply();
                }
            }
            if (isTimerRunning)
            {
                elapsedTime = Time.time - conversationStartTime;

                if (elapsedTime < emotionCheckPoint)
                {
                    chatType = 1;
                }
                else
                {
                    chatType = 2;
                }
            }
        }

        public void ButtonSendReply()
        {
            SendReply();
            EventSystem.current.SetSelectedGameObject(null);
        }

        private void ScrollToBottom()
        {
            // Scroll to the bottom of the content
            scroll.normalizedPosition = new Vector2(0, 0);
        }

        public void StartTimer()
        {
            conversationStartTime = Time.time;
            isTimerRunning = true;
        }

        public float StopTimer()
        {
            if (isTimerRunning)
            {
                elapsedTime = Time.time - conversationStartTime;
                isTimerRunning = false;
            }
            return elapsedTime;
        }
        private void PlayRandomTypeSound()
        {
            if (voiceSounds != null && voiceSounds.Length > 0)
            {
                int randomIndex = UnityEngine.Random.Range(0, voiceSounds.Length);
                typeSoundSource.clip = voiceSounds[randomIndex];
                typeSoundSource.Play();
            }
        }
    }
    
}