A Generative Al Engineer has developed an LLM application to answer questions about internal company policies. The Generative AI Engineer must ensure that the application doesn't hallucinate or leak confidential data.
Which approach should NOT be used to mitigate hallucination or confidential data leakage?

• A. Fine-tune the model on your data, hoping it will learn what is appropriate and not
• B. Use a strong system prompt to ensure the model aligns with your needs.
• C. Limit the data available based on the user's access level
• D. Add guardrails to filter outputs from the LLM before it is shown to the user

A Generative Al Engineer is creating an LLM system that will retrieve news articles from the year 1918 and related to a user ' s query and summarize them. The engineer has noticed that the summaries are generated well but often also include an explanation of how the summary was generated, which is undesirable.
Which change could the Generative Al Engineer perform to mitigate this issue?

• A. Split the LLM output by newline characters to truncate away the summarization explanation.
• B. Provide few shot examples of desired output format to the system and/or user prompt.
• C. Revisit their document ingestion logic, ensuring that the news articles are being ingested properly.
• D. Tune the chunk size of news articles or experiment with different embedding models.

A Generative Al Engineer needs to design an LLM pipeline to conduct multi-stage reasoning that leverages external tools. To be effective at this, the LLM will need to plan and adapt actions while performing complex reasoning tasks.
Which approach will do this?

• A. Encourage the LLM to make multiple API calls in sequence without planning or structuring the calls, allowing the LLM to decide when and how to use external tools spontaneously.
• B. Tram the LLM to generate a single, comprehensive response without interacting with any external tools, relying solely on its pre-trained knowledge.
• C. Use a Chain-of-Thought (CoT) prompting technique to guide the LLM through a series of reasoning steps, then manually input the results from external tools for the final answer.
• D. Implement a framework like ReAct which allows the LLM to generate reasoning traces and perform task-specific actions that leverage external tools if necessary.

A Generative Al Engineer is tasked with developing an application that is based on an open source large language model (LLM). They need a foundation LLM with a large context window.
Which model fits this need?

• A. Llama2-70B
• B. MPT-30B
• C. DBRX
• D. DistilBERT

A Generative AI Engineer has created a RAG application which can help employees retrieve answers from an internal knowledge base, such as Confluence pages or Google Drive. The prototype application is now working with some positive feedback from internal company testers. Now the Generative Al Engineer wants to formally evaluate the system's performance and understand where to focus their efforts to further improve the system.
How should the Generative AI Engineer evaluate the system?

• A. Use cosine similarity score to comprehensively evaluate the quality of the final generated answers.
• B. Curate a dataset that can test the retrieval and generation components of the system separately. Use MLflow's built in evaluation metrics to perform the evaluation on the retrieval and generation components.
• C. Benchmark multiple LLMs with the same data and pick the best LLM for the job.
• D. Use an LLM-as-a-judge to evaluate the quality of the final answers generated.