Gemma 4

14 of 14 prompts answered, 2 marked as failures.

1. Instruction following

   The prompt

   List three practical ways to reduce GPU memory use when running a neural network for inference. Answer with exactly three bullet points, each under 15 words, and nothing else.

   A correct answer: Exactly three bullet points, each under 15 words, with no text before or after them. Each way is real, such as lower precision, a smaller batch or a shorter input.

   * Use mixed-precision inference with FP16 or BF16 data types.
   * Apply weight quantization to convert weights to INT8 or INT4.
   * Implement model pruning to remove redundant neurons and connections.

2. Summarizing

   FAIL · Doesn't say the conditions are written down before observing, and says the reference stars account for atmospheric clarity, which the text doesn't
   The prompt

   Summarize the paragraph below in exactly two sentences. A small observatory on a hill keeps a logbook for every night it opens. Before anyone looks through the telescope, they write down the cloud cover, the temperature and how steady the air is, because a blurry night and a sharp night can make the same star look very different. The staff also measure the same three reference stars at the start and end of each session. If those reference readings drift, every other number from that night is marked as uncertain rather than thrown away, so that later readers can decide for themselves how much to trust it.

   A correct answer: Exactly two sentences. It mentions writing down the conditions before observing, and using reference stars to mark a night's numbers as uncertain. It adds nothing that the paragraph doesn't say.

   The observatory records environmental conditions and reference star measurements to account for variations in atmospheric clarity. If these reference readings drift, the data is marked as uncertain to allow future researchers to judge its reliability.

3. Extracting data

   The prompt

   Extract these fields from the note below and answer with a JSON object only, without code fences: customer (string), date (YYYY-MM-DD), total_eur (number), items (array of strings). Note: Order for Mara Lindqvist, placed on 3 March 2026. She bought a desk lamp for 34.90 euros, two notebooks at 4.50 euros each, and a USB cable for 9.99 euros.

   A correct answer: Valid JSON with exactly these keys and nothing around it: customer "Mara Lindqvist", date "2026-03-03", total_eur 53.89, and items naming the lamp, the notebooks and the cable.

   {
     "customer": "Mara Lindqvist",
     "date": "2026-03-03",
     "total_eur": 53.89,
     "items": [
       "desk lamp",
       "notebook",
       "notebook",
       "USB cable"
     ]
   }

4. Arithmetic

   The prompt

   A train leaves at 09:40 and travels 210 km at an average speed of 84 km/h. At what time does it arrive? Explain in at most three short steps, then give the final answer on its own line as "Answer: HH:MM".

   A correct answer: The last line is "Answer: 12:10". The trip takes 210 / 84 = 2.5 hours.

   1. Calculate the travel time by dividing the distance by the speed: $210 \text{ km} \div 84 \text{ km/h} = 2.5 \text{ hours}$.
   2. Convert the travel time into hours and minutes: $2.5 \text{ hours} = 2 \text{ hours and } 30 \text{ minutes}$.
   3. Add the travel time to the departure time: $09:40 + 2 \text{ hours } 30 \text{ minutes} = 12:10$.
   
   Answer: 12:10

5. Logic

   The prompt

   Ana, Ben and Chloe each own exactly one pet: a cat, a dog or a parrot, and no two of them own the same pet. Ana does not own the dog. Chloe does not own the cat. Ben owns neither the cat nor the parrot. Who owns which pet? Answer in the form "Ana: ..., Ben: ..., Chloe: ...".

   A correct answer: Ana: cat, Ben: dog, Chloe: parrot.

   To solve this, let's look at the clues provided:
   
   1.  **Ben owns neither the cat nor the parrot.** Since there are only three pets (cat, dog, parrot), Ben must own the **dog**.
   2.  **Ana does not own the dog.** Since Ben owns the dog, this is already consistent.
   3.  **Chloe does not own the cat.** Since Ben owns the dog, Chloe must own either the cat or the parrot. Because she doesn't own the cat, Chloe must own the **parrot**.
   4.  **Ana** is the only person left, and the only pet left is the **cat**.
   
   Ana: cat, Ben: dog, Chloe: parrot

6. Writing code

   The prompt

   Write a Python function is_palindrome(s: str) -> bool that ignores case, spaces and punctuation. After the function, show two example calls with their expected results as comments.

   A correct answer: A correct function: it keeps only letters and digits, lowercased, and compares them with their reverse. The examples are right, such as "A man, a plan, a canal: Panama" giving True.

   Here is the Python function. It uses the `str.isalnum()` method to filter out any characters that are not letters or numbers, and `str.lower()` to ensure the check is case-insensitive.
   
   ```python
   import string
   
   def is_palindrome(s: str) -> bool:
       # Filter the string: keep only alphanumeric characters and convert to lowercase
       cleaned_chars = [char.lower() for char in s if char.isalnum()]
       
       # Join the list into a string (optional, but good for clarity)
       filtered_string = "".join(cleaned_chars)
       
       # Check if the string is equal to its reverse
       return filtered_string == filtered_string[::-1]
   
   # Example 1: A phrase with spaces, punctuation, and mixed casing
   # Expected Result: True
   print(is_palindrome("A man, a plan, a canal: Panama")) 
   
   # Example 2: A standard non-palindrome
   # Expected Result: False
   print(is_palindrome("Hello World"))
   ```

7. Fixing code

   The prompt

   This Python function should return the average of a list of numbers, but it gives wrong results. Explain the bug in one sentence, then give the corrected function. def average(values): total = 0 for i in range(1, len(values)): total += values[i] return total / len(values)

   A correct answer: It says the loop skips the first number, because the range starts at 1. The fix sums every number, for example sum(values) / len(values). Handling an empty list is a bonus, not required.

   The bug is that the loop starts at index 1 instead of index 0, causing the function to skip the first element of the list.
   
   ```python
   def average(values):
       total = 0
       for i in range(len(values)):
           total += values[i]
       return total / len(values)
   ```

8. Answering from a text

   The prompt

   Answer two questions using only the text below. If the text doesn't say, answer "Not stated." 1. What happens to a night's numbers when the reference readings drift? 2. In what year did the observatory open? Text: A small observatory on a hill keeps a logbook for every night it opens. Before anyone looks through the telescope, they write down the cloud cover, the temperature and how steady the air is, because a blurry night and a sharp night can make the same star look very different. The staff also measure the same three reference stars at the start and end of each session. If those reference readings drift, every other number from that night is marked as uncertain rather than thrown away, so that later readers can decide for themselves how much to trust it.

   A correct answer: Question 1: they are marked as uncertain, not thrown away. Question 2: "Not stated." Any year it gives is invented, and a failure.

   1. Every other number from that night is marked as uncertain rather than thrown away.
   2. Not stated.

9. Translating

   FAIL · Translates "measurements" as 치수, which means physical dimensions, and gives two options with romanization instead of one message
   The prompt

   Translate into Korean, polite and natural, as you would write to a colleague: "Thanks for sending the measurements. Could you run the larger model again tomorrow morning, when the machine is idle?"

   A correct answer: Natural, polite Korean that keeps every part: thanks for the measurements, run the larger model again, tomorrow morning, when the machine is idle.

   Depending on your relationship with the colleague, here are a few ways to say this. All of these are polite and natural for a workplace setting.
   
   ### Option 1: Standard Polite (Most common/Recommended)
   This is the most natural way to speak to a colleague you work with regularly. It is polite but not overly formal.
   
   > **"치수 보내주셔서 감사합니다. 내일 오전 중에 기계가 비었을 때, 큰 모델 다시 한번만 돌려주실 수 있을까요?"**
   > *(Chisu bonae-jusyeoseo gamsahamnida. Naeil ojeon junge gigyega bieot-eul ttae, keun model dasi hanbeonman dollyeo-jusil su isseulkkayo?)*
   
   ### Option 2: Slightly More Formal
   Use this if the colleague is senior to you or if you want to be extra respectful.
   
   > **"치수 보내주셔서 감사합니다. 내일 오전 기계가 한가할 때 큰 모델 다시 한번 실행 부탁드려도 될까요?"**
   > *(Chisu bonae-jusyeoseo gamsahamnida. Naeil ojeon gigyega hang-gahal ttae keun model dasi hanbeon silhaeng butak-deuryeodo doelkkayo?)*
   
   ### Key Vocabulary Used:
   *   **치수 (Chisu):** Measurements
   *   **기계가 비었을 때 (Gigyega bieot-eul ttae):** When the machine is empty/idle (Very natural in a factory/lab setting).
   *   **기계가 한가할 때 (Gigyega hang-gahal ttae):** When the machine is free/not busy (Slightly more formal).
   *   **돌려주실 수 있을까요? (Dollyeo-jusil su isseulkkayo?):** Could you run it? (In Korean, "run" a machine/process is often expressed as "돌리다").

10. Rewriting

    The prompt

    Rewrite this message to be polite and clear, in under 60 words, keeping every fact: "your model page is wrong. the license says apache but the weights are non commercial. fix it"

    A correct answer: Under 60 words and polite. It still says the page lists Apache while the weights are non-commercial, and asks for a correction.

    "I noticed a discrepancy on your model page: the license is listed as Apache, but the weights are marked as non-commercial. Could you please update the page to reflect the correct licensing information? Thank you!"

11. Classifying

    The prompt

    Label each review as positive, negative or mixed. Answer with a numbered list of labels only. 1. Setup took five minutes and it has worked perfectly ever since. 2. The battery died after two days and support never replied. 3. Great screen, but the speakers are tinny and quiet. 4. Honestly the best purchase I have made this year.

    A correct answer: 1 positive, 2 negative, 3 mixed, 4 positive, and nothing else.

    1. Positive
    2. Negative
    3. Mixed
    4. Positive

12. Explaining

    The prompt

    Explain to a high-school student what VRAM is and why it limits which AI models can run on a graphics card. Use at most 80 words.

    A correct answer: At most 80 words, and correct: VRAM is the graphics card's own memory, and a model's weights and working data must fit in it for the card to run the model.

    Think of VRAM (Video RAM) as the "workspace" for your graphics card. When you run an AI model, the computer must load the model's data into this memory to perform calculations quickly.
    
    If a model is too large to fit into your VRAM, it won't run—or it will run incredibly slowly by using your much slower system RAM. Essentially, VRAM acts as a physical size limit: if the model is bigger than your memory, your hardware can't "hold" it.

13. Command line

    The prompt

    Give one Linux shell command that lists the five largest files under the current directory, searching subdirectories too, with human-readable sizes. Answer with the command only.

    A correct answer: One command that works, such as find . -type f -exec du -h {} + | sort -rh | head -n 5. A command that lists directories instead of files is a failure.

    find . -type f -exec du -h {} + | sort -rh | head -n 5

14. Knowledge

    The prompt

    In one sentence: what is the boiling point of water at sea level in degrees Fahrenheit, and who proposed the Fahrenheit scale?

    A correct answer: 212 °F, and Daniel Gabriel Fahrenheit.

    The boiling point of water at sea level is 212 degrees Fahrenheit, and the scale was proposed by Daniel Gabriel Fahrenheit.

14 of 14 prompts answered, 2 marked as failures.

1. Instruction following

   The prompt

   List three practical ways to reduce GPU memory use when running a neural network for inference. Answer with exactly three bullet points, each under 15 words, and nothing else.

   A correct answer: Exactly three bullet points, each under 15 words, with no text before or after them. Each way is real, such as lower precision, a smaller batch or a shorter input.

   * Use half-precision (FP16) or quantized (INT8) weights.
   * Reduce the input batch size.
   * Disable gradient calculation using `torch.no_grad()`.

2. Summarizing

   FAIL · Doesn't say the conditions are written down before observing, and adds that this ensures data accuracy
   The prompt

   Summarize the paragraph below in exactly two sentences. A small observatory on a hill keeps a logbook for every night it opens. Before anyone looks through the telescope, they write down the cloud cover, the temperature and how steady the air is, because a blurry night and a sharp night can make the same star look very different. The staff also measure the same three reference stars at the start and end of each session. If those reference readings drift, every other number from that night is marked as uncertain rather than thrown away, so that later readers can decide for themselves how much to trust it.

   A correct answer: Exactly two sentences. It mentions writing down the conditions before observing, and using reference stars to mark a night's numbers as uncertain. It adds nothing that the paragraph doesn't say.

   An observatory records atmospheric conditions and reference star measurements to ensure data accuracy. If these readings drift, the night's data is marked as uncertain so future researchers can evaluate its reliability.

3. Extracting data

   The prompt

   Extract these fields from the note below and answer with a JSON object only, without code fences: customer (string), date (YYYY-MM-DD), total_eur (number), items (array of strings). Note: Order for Mara Lindqvist, placed on 3 March 2026. She bought a desk lamp for 34.90 euros, two notebooks at 4.50 euros each, and a USB cable for 9.99 euros.

   A correct answer: Valid JSON with exactly these keys and nothing around it: customer "Mara Lindqvist", date "2026-03-03", total_eur 53.89, and items naming the lamp, the notebooks and the cable.

   {
     "customer": "Mara Lindqvist",
     "date": "2026-03-03",
     "total_eur": 53.89,
     "items": [
       "desk lamp",
       "notebook",
       "notebook",
       "USB cable"
     ]
   }

4. Arithmetic

   The prompt

   A train leaves at 09:40 and travels 210 km at an average speed of 84 km/h. At what time does it arrive? Explain in at most three short steps, then give the final answer on its own line as "Answer: HH:MM".

   A correct answer: The last line is "Answer: 12:10". The trip takes 210 / 84 = 2.5 hours.

   1. Calculate travel time: 210 km ÷ 84 km/h = 2.5 hours.
   2. Convert 0.5 hours to minutes: 0.5 × 60 = 30 minutes.
   3. Add 2 hours and 30 minutes to 09:40.
   
   Answer: 12:10

5. Logic

   The prompt

   Ana, Ben and Chloe each own exactly one pet: a cat, a dog or a parrot, and no two of them own the same pet. Ana does not own the dog. Chloe does not own the cat. Ben owns neither the cat nor the parrot. Who owns which pet? Answer in the form "Ana: ..., Ben: ..., Chloe: ...".

   A correct answer: Ana: cat, Ben: dog, Chloe: parrot.

   Ben owns neither the cat nor the parrot, so Ben must own the dog. Since Ben owns the dog, Ana and Chloe must own the cat and the parrot. Chloe does not own the cat, so Chloe must own the parrot. This leaves the cat for Ana.
   
   Ana: cat, Ben: dog, Chloe: parrot

6. Writing code

   The prompt

   Write a Python function is_palindrome(s: str) -> bool that ignores case, spaces and punctuation. After the function, show two example calls with their expected results as comments.

   A correct answer: A correct function: it keeps only letters and digits, lowercased, and compares them with their reverse. The examples are right, such as "A man, a plan, a canal: Panama" giving True.

   Here is the Python function. It uses the `.isalnum()` method to filter out any characters that are not letters or numbers.
   
   ```python
   import string
   
   def is_palindrome(s: str) -> bool:
       # Filter the string: keep only alphanumeric characters and convert to lowercase
       cleaned = "".join(char.lower() for char in s if char.isalnum())
       
       # Compare the cleaned string with its reverse
       return cleaned == cleaned[::-1]
   
   # Example calls:
   print(is_palindrome("A man, a plan, a canal: Panama")) # Expected: True
   print(is_palindrome("race a car"))                     # Expected: False
   ```

7. Fixing code

   The prompt

   This Python function should return the average of a list of numbers, but it gives wrong results. Explain the bug in one sentence, then give the corrected function. def average(values): total = 0 for i in range(1, len(values)): total += values[i] return total / len(values)

   A correct answer: It says the loop skips the first number, because the range starts at 1. The fix sums every number, for example sum(values) / len(values). Handling an empty list is a bonus, not required.

   The loop starts at index 1 instead of 0, causing the first element of the list to be skipped.
   
   ```python
   def average(values):
       total = 0
       for i in range(0, len(values)):
           total += values[i]
       return total / len(values)
   ```
   
   *(Note: In Python, you can simplify this further by using `sum(values) / len(values)`)*

8. Answering from a text

   The prompt

   Answer two questions using only the text below. If the text doesn't say, answer "Not stated." 1. What happens to a night's numbers when the reference readings drift? 2. In what year did the observatory open? Text: A small observatory on a hill keeps a logbook for every night it opens. Before anyone looks through the telescope, they write down the cloud cover, the temperature and how steady the air is, because a blurry night and a sharp night can make the same star look very different. The staff also measure the same three reference stars at the start and end of each session. If those reference readings drift, every other number from that night is marked as uncertain rather than thrown away, so that later readers can decide for themselves how much to trust it.

   A correct answer: Question 1: they are marked as uncertain, not thrown away. Question 2: "Not stated." Any year it gives is invented, and a failure.

   1. Every other number from that night is marked as uncertain rather than thrown away.
   2. Not stated.

9. Translating

   FAIL · Translates "measurements" as 치수, which means physical dimensions, and gives two options with romanization instead of one message
   The prompt

   Translate into Korean, polite and natural, as you would write to a colleague: "Thanks for sending the measurements. Could you run the larger model again tomorrow morning, when the machine is idle?"

   A correct answer: Natural, polite Korean that keeps every part: thanks for the measurements, run the larger model again, tomorrow morning, when the machine is idle.

   Depending on how close you are with this colleague, you can choose between a **standard polite** version (most common) or a **slightly more formal** version.
   
   ### Option 1: Standard Polite (Recommended)
   This is the most natural way to speak to a colleague you work with regularly. It is professional yet friendly.
   
   > **"치수 보내주셔서 감사합니다. 내일 아침에 장비가 비어 있을 때, 큰 모델로 다시 한번 돌려주실 수 있을까요?"**
   > *(Chisu bonaejusyeoseo gamsahamnida. Naeil achime jangbiga bieo isseul ttae, keun model-lo dasi hanbeon dollyeojusil su isseulkkayo?)*
   
   ### Option 2: Slightly More Formal
   Use this if the colleague is senior to you or if you are not very close yet.
   
   > **"치수 송부해주셔서 감사합니다. 내일 오전 중 장비 사용 가능할 때, 큰 모델로 다시 한번 구동 부탁드려도 될까요?"**
   > *(Chisu songbuhaejusyeoseo gamsahamnida. Naeil ojeon jung jangbi sayong ganeunghal ttae, keun model-lo dasi hanbeon gudong butakdeuryeodo doelkkayo?)*
   
   ---
   
   **Key Vocabulary used:**
   *   **치수 (Chisu):** Measurements
   *   **장비가 비어 있다 (Jangbiga bieo itda):** The machine is idle/empty
   *   **돌리다 (Dollida):** To "run" (a model/program/machine) — *very common industry slang.*
   *   **구동하다 (Gudong-hada):** To "run/operate" — *more formal/technical term.*

10. Rewriting

    The prompt

    Rewrite this message to be polite and clear, in under 60 words, keeping every fact: "your model page is wrong. the license says apache but the weights are non commercial. fix it"

    A correct answer: Under 60 words and polite. It still says the page lists Apache while the weights are non-commercial, and asks for a correction.

    "Hello, there is a discrepancy on your model page. The license is listed as Apache, but the weights are marked as non-commercial. Could you please correct this? Thank you!"

11. Classifying

    The prompt

    Label each review as positive, negative or mixed. Answer with a numbered list of labels only. 1. Setup took five minutes and it has worked perfectly ever since. 2. The battery died after two days and support never replied. 3. Great screen, but the speakers are tinny and quiet. 4. Honestly the best purchase I have made this year.

    A correct answer: 1 positive, 2 negative, 3 mixed, 4 positive, and nothing else.

    1. Positive
    2. Negative
    3. Mixed
    4. Positive

12. Explaining

    The prompt

    Explain to a high-school student what VRAM is and why it limits which AI models can run on a graphics card. Use at most 80 words.

    A correct answer: At most 80 words, and correct: VRAM is the graphics card's own memory, and a model's weights and working data must fit in it for the card to run the model.

    VRAM (Video RAM) is high-speed memory built directly into your graphics card. Think of it as a "workbench" where the GPU stores the AI model's data for instant access.
    
    AI models consist of billions of parameters (weights) that must fit entirely within this memory to function. If a model is larger than your available VRAM, it simply won't fit on the "workbench," causing the program to crash or run painfully slow.

13. Command line

    The prompt

    Give one Linux shell command that lists the five largest files under the current directory, searching subdirectories too, with human-readable sizes. Answer with the command only.

    A correct answer: One command that works, such as find . -type f -exec du -h {} + | sort -rh | head -n 5. A command that lists directories instead of files is a failure.

    find . -type f -exec du -h {} + | sort -rh | head -n 5

14. Knowledge

    The prompt

    In one sentence: what is the boiling point of water at sea level in degrees Fahrenheit, and who proposed the Fahrenheit scale?

    A correct answer: 212 °F, and Daniel Gabriel Fahrenheit.

    The boiling point of water at sea level is 212 degrees Fahrenheit, a scale proposed by Daniel Gabriel Fahrenheit.