← OverviewNext scene →

Scene 15 of 26 · Boston

8 questions · nuScenes, DriveLM test split

scene c5b36e5275cc4a2285d095e87347c49a · frame 04de5a1d4f8b45a3a62d6bf0ee07c715

Six camera views around the car: front-left, front, front-right on top; back-left, back, back-right below.c2c4c3c1
Six cameras around the car: front-left, front, front-right (top row); back-left, back, back-right (bottom row). Tap the image to open it full size. Pink tags mark the ground-truth objects referenced in the questions.

Reading object references: <c2,CAM_FRONT_LEFT,1045.0,596.3> means object c2 in the front-left camera at pixel (1045.0, 596.3) of the original 1600×900 image. Models must produce these references themselves, so wrong coordinates mean the model did not find the object.

Which models are compared?

Score = DriveLM final score (0–1, higher is better) on our test split of 3,340 questions. It combines answer accuracy, a GPT judge, language overlap and object matching. Mini-DA is an external checkpoint released by OpenGVLab; all other fine-tuned models are ours.

Showing 3 of 8 models
Q1perception

Identify all the traffic elements in the front view, categorize them, determine their status, and predict the bounding box around each one. The output should be a list formatted as (c, s, x1, y1, x2, y2), where c represents the category, s denotes the status, and x1, y1, x2, y2 are the offsets of the top-left and bottom-right corners of the box relative to the center point.

Ground truth
There are many traffic elements in the front view. The information of these traffic elements is [(traffic light, unknown, 1565.73, 158.82, 1600.0, 234.23), (traffic light, unknown, 1418.3, 211.19, 1450.94, 280.43), (traffic light, unknown, 1234.88, 239.97, 1290.78, 303.59), (traffic light, unknown, 927.82, 256.62, 950.71, 306.49), (traffic light, unknown, 892.7, 233.39, 916.41, 286.53), (traffic light, green, 830.99, 203.87, 854.29, 280.31), (traffic light, green, 831.41, 203.83, 854.65, 280.98), (traffic light, green, 927.22, 342.2, 953.3, 399.3), (traffic light, green, 927.52, 342.18, 952.52, 398.93), (traffic light, red, 967.57, 410.36, 982.4, 421.42)].
Best modelours
There are two traffic elements in the front view: a car and a truck. The car is a traffic sign, while the truck is a traffic sign. The car is a traffic sign, while the truck is a traffic sign. The bounding boxes are as follows: (0, 0, 1080, 430), (1080, 0, 1200, 430), (1080, 430, 1200, 530), (1080, 530, 1200, 630).
Oracleours · upper bound
There are many traffic elements in the front view. The information of these traffic elements is [(traffic light, green, 747.19, 0.58, 779.48, 99.31), (traffic light, green, 1045.42, 75.0, 1073.79, 159.26), (traffic light, unknown, 1045.77, 75.0, 1073.37, 159.26), (traffic light, green, 1544.31, 179.98, 1592.63, 293.31), (traffic light, unknown, 1544.31, 179.98, 1592.63, 293.31), (traffic light, green, 1453.51, 300.83, 1478.7, 383.44), (traffic light, unknown, 1453.51, 300.83,
InternVL2-2B (no fine-tuning)zero-shot
To identify and categorize the traffic elements in the front view, we'll break down the image into distinct categories and provide the bounding boxes for each: 1. **White SUV (CAM_FRONT_LEFT)**: - Category: Vehicle - Status: In motion - Bounding Box: (0, 0, 0, 0) 2. **White SUV (CAM_FRONT)**: - Category: Vehicle - Status: In motion - Bounding Box: (0, 0, 0, 0) 3. **White SUV (CAM_FRONT_RIGHT)**: - Category: Vehicle - Status: In motion - Bounding Box: (0, 0, 0, 0) 4. **White Truck (CAM_FRONT_RIGHT)**: - Category: Vehicle - Status: In motion - Bounding Box: (0, 0, 0, 0) 5. **White Car (CAM_FRONT)**: - Category: Vehicle - Status: In motion - Bounding Box: (0, 0, 0, 0) 6. **White Car (CAM_FRONT_LEFT)**: - Category: Vehicle - Status: In motion - Bounding Box: (0, 0, 0, 0) 7. **White Car (CAM_FRONT)**: - Category: Vehicle - Status: In motion - Bounding Box: (0, 0, 0, 0) 8. **White Car (CAM_FRONT_RIGHT)**: - Category: Vehicle - Status: In motion - Bounding Box: (0, 0, 0, 0) 9. **White Car (CAM_FRONT)**: - Category: Vehicle - Status: In motion - Bounding Box: (0, 0, 0, 0) 10. **White Car (CAM_FRONT_LEFT)**: - Category: Vehicle - Status: In motion - Bounding Box: (0, 0, 0, 0) 11. **White Car (CAM_FRONT_RIGHT)**: - Category: Vehicle - Status: In motion - Bounding Box: (0, 0, 0, 0) 12. **White Car (CAM_FRONT)**: - Category: Vehicle - Status: In motion
Fine-tuned 25kours
There is one traffic element in the front view, which is a white SUV to the front of the ego vehicle, so the category is "S", the status is "S", the offsets of the top-left corner are (1080, 310), the bottom-right corner is (1080, 480), and the offsets of the top-left corner are (1080, 480), (1080, 310).
Fine-tuned 300kours
There are many traffic elements in the front view. The information of these traffic elements is [(traffic light, green, 1015.0, 310.0, 1045.0, 340.0), (traffic light, green, 1045.0, 310.0, 1055.0, 340.0), (traffic light, green, 1055.0, 310.0, 1065.0, 340.0), (traffic light, green, 1065.0, 310.0, 1075.0, 340.0), (traffic light, green, 1075.0, 310.0, 1085.0, 340.0), (traffic light, green, 1085.0, 310.0, 1095.0, 340.0), (traffic light, green, 1095.0, 310.0, 1105.0, 340.0), (traffic light, green, 1115.0, 310.0, 1125.0, 340.0), (traffic light, green, 1125.0, 310.0, 1135.0, 340.0), (traffic light, green, 1135.0, 310.0, 1145.0, 340.0), (traffic light, green, 1145.0, 310.0, 1155.0, 340.0), (traffic light, green, 1155.0, 310.0, 1165.0, 340.0), (traffic light, green, 1165.0, 310.0, 1175.0, 340.0), (traffic light, green, 1175.0, 310.0, 1185.0, 340.0), (traffic light, green, 1185.0, 310.0, 1195.0, 340.0), (traffic light, green, 1195.0, 310.0, 1205.0, 340.0), (traffic light, green, 1205.0, 310.0, 1115.0, 340.0), (traffic light, green, 1115.0, 310.0, 11
OpenGVLab Mini-DAexternal reference
There are many traffic elements in the front view. The information of these traffic elements is [(traffic light[[506, 113, 511, 150]], green), (traffic light[[536, 113, 541, 150]], green), (traffic light[[557, 115, 563, 150]], green), (traffic light[[613, 110, 622, 150]], red), (traffic light[[613, 110, 622, 150]], green), (traffic light[[613, 110, 622, 150]], red), (traffic light[[613, 110, 622, 150]], green), (traffic light[[613, 110, 622, 150]], red), (traffic light[[613, 110, 622, 150]], green), (traffic light[[613, 110, 622, 150]], red), (traffic light[[613, 110, 622, 150]], green), (traffic light[[613, 110, 622, 150]], red), (traffic light[[613, 110, 622, 150]], green), (traffic light[[613, 110, 622, 150]], red), (traffic light[[613, 110, 622, 150]], green), (traffic light[[613, 110, 622, 150]], red), (traffic light[[613, 110, 622, 150]], green), (traffic light[[613, 110, 622, 150]], red), (traffic light[[613, 110, 622, 150]], green), (traffic light[[613, 110, 622, 150]], red), (traffic light[[613, 110, 622, 150]], green), (traffic light[[613, 110, 622, 150]], red), (traffic light[[613, 110
LLaVA-1.6 7Bzero-shot
The image provided is a collage of several different scenes, each depicting a different traffic element. Due to the nature of the collage, it is not possible to provide a single list of all the traffic elements in the front view, as each scene has its own set of elements. However, I can describe the traffic elements in each scene and provide a list for each individual scene. 1. Top Left: - Category: Car - Status: Moving - Bounding Box: (0.156,0.169,0.456,0.487) 2. Top Right: - Category: Car - Status: Moving - Bounding Box: (0.500,0.169,0.734,0.487) 3. Bottom Left: - Category: Car - Status: Moving - Bounding Box: (0.000,0.508,0.208,0.988) 4. Bottom Right: - Category: Car - Status: Moving - Bounding Box: (0.500,0.508,0.734,0.988) 5. Middle Left: - Category: Car - Status: Moving - Bounding Box: (0.156,0.169,0.456,0.487) 6. Middle Right: - Category: Car - Status: Moving - Bounding Box: (0.500,0.169,0.734,0.487) 7. Bottom Left: - Category: Car - Status: Moving - Bounding Box: (0.000,0.508,0.208,0.988) 8. Bottom Right: - Category: Car - Status: Moving - Bounding Box: (0.500,0.508,0.734,0.988) Please note that
LLaMA-Adapter v2 7Bzero-shot
The traffic elements in the front view are: 1. Car (c = 1) 2. Truck (c = 2) 3. Bus (c = 3) 4. Traffic light (c = 4) 5. Street sign (c = 5) 6. Building (c = 6) The status of each element is: 1. Car: moving (s = 1) 2. Truck: moving (s = 1) 3. Bus: moving (s = 1) 4. Traffic light: green (s = 1) 5. Street sign: visible (s = 1) 6. Building: visible (s = 1)
Q2perceptionmultiple choice

What is the moving status of object <c2,CAM_FRONT_LEFT,1045.0,596.3>? Please select the correct answer from the following options: A. Stopped. B. Drive backward. C. Going ahead. D. Turn right.

Ground truth
C
Best modelours✓ correct
C
Oracleours · upper bound✓ correct
C
InternVL2-2B (no fine-tuning)zero-shotno direct answer
The moving status of object <c2,CAM_FRONT_LEFT,1045.0,596.3> is C. Going ahead.
Fine-tuned 25kours✓ correct
C
Fine-tuned 300koursno direct answer
C. Going ahead.
OpenGVLab Mini-DAexternal reference✓ correct
C
LLaVA-1.6 7Bzero-shotno direct answer
A. Stopped
LLaMA-Adapter v2 7Bzero-shotno direct answer
The moving status of object <c2,CAM_FRONT_LEFT,1045.0,596.3> is "Going ahead."
Q3prediction

What object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?

Ground truth
Firstly, notice that <c4,CAM_FRONT,840.4,239.7>. The object is a traffic sign, so the ego vehicle should keep going ahead at the same speed. Secondly, notice that <c2,CAM_FRONT_LEFT,1045.0,596.3>. The object is going ahead, so the ego vehicle should keep going ahead at the same speed. Thirdly, notice that <c3,CAM_BACK_LEFT,352.3,402.8>. The object is stationary, so the ego vehicle should keep going ahead at the same speed.
Best modelours
Firstly, notice <c1, CAM_FRONT, 1200.0, 600.0>. The object is approaching, so the ego should slow down. Secondly, notice <c2, CAM_FRONT, 1200.0, 600.0>. The object is approaching, so the ego should slow down. Thirdly, notice <c3, CAM_FRONT, 1200.0, 600.0>. The object is approaching, so the ego should slow down.
Oracleours · upper bound
Firstly, notice that <c2,CAM_FRONT_LEFT,1045.0,596.3>. The object is going ahead, so the ego vehicle should keep going ahead at the same speed. Secondly, notice that <c4,CAM_FRONT,840.4,239.7>. The object is a traffic sign, so the ego vehicle should keep going ahead at the same speed. Thirdly, notice that <c3,CAM_BACK_LEFT,352.3,402.8>. The object is going ahead, so the ego vehicle should keep going ahead at the same speed.
InternVL2-2B (no fine-tuning)zero-shot
When the ego vehicle is getting to the next possible location, it should first notice the traffic light at the intersection. The state of the traffic light is green, indicating that vehicles can proceed. The ego vehicle should then proceed through the intersection.
Fine-tuned 25kours
Firstly, notice that <c3,CAM_FRONT,1080.8,525.8>. The object is going ahead, so the ego vehicle should keep going ahead at the same speed. Secondly, notice that <c2,CAM_FRONT,1080.8,505.8>. The object is going ahead, so the ego vehicle should keep going ahead at the same speed. Thirdly, notice that <c1,CAM_FRONT,1080.8,505.8>. The object is going ahead, so the ego vehicle should keep going ahead at the same speed.
Fine-tuned 300kours
Firstly, notice that <c3,CAM_FRONT,1080.0,520.0>. The object is going ahead, so the ego vehicle should keep going ahead at the same speed. Secondly, notice that <c1,CAM_FRONT,1080.0,505.8>. The object is going ahead, so the ego vehicle should keep going ahead at the same speed. Thirdly, notice that <c2,CAM_BACK,1010.0,555.8>. The object is going ahead, so the ego vehicle should keep going ahead at the same speed.
OpenGVLab Mini-DAexternal reference
Firstly, notice that the object in front of the ego vehicle is a traffic sign. The ego vehicle should continue going ahead at the same speed. Secondly, notice that the object in front of the ego vehicle is stationary. The ego vehicle should continue going ahead at the same speed. Thirdly, notice that the object in front of the ego vehicle is a traffic sign. The ego vehicle should continue going ahead at the same speed.
LLaVA-1.6 7Bzero-shot
The image you've provided appears to be a collage of different street scenes, each with various objects that the ego vehicle should notice when approaching the next possible location. Here's a breakdown of what the ego vehicle should notice and what actions it should take: 1. **First Object**: The first object the ego vehicle should notice is the traffic light. The state of the traffic light is red, indicating that the ego vehicle should stop. 2. **Second Object**: The second object the ego vehicle should notice is the pedestrian crossing. The state of the pedestrian crossing is clear, suggesting that the ego vehicle should proceed with caution, checking for pedestrians before moving forward. 3. **Third Object**: The third object the ego vehicle should notice is the truck. The truck is parked, so the ego vehicle should be aware of its presence and proceed with caution, especially if there is limited visibility around the truck. 4. **Fourth Object**: The fourth object the ego vehicle should notice is the potted plant. The plant is on the sidewalk, and the ego vehicle should be aware of it to avoid any potential damage to the plant or the sidewalk. 5. **Fifth Object**: The fifth object the ego vehicle should notice is the building. The building is in the background, and the ego vehicle should be aware of its presence to navigate the street correctly. 6. **Sixth Object**: The sixth object the ego vehicle should notice is the car. The car is parked, and the ego vehicle should be aware of its presence to avoid any potential damage to the car or the sidewalk. 7. **Seventh Object**: The seventh object the ego vehicle should notice is the bench. The bench is on the sidewalk, and the ego vehicle should be aware of its presence to avoid any potential damage to the bench or the sidewalk. 8. **Eighth Object**: The eighth object the ego vehicle should notice is the truck. The truck is parked, and the ego vehicle should be aware of its presence to avoid any potential damage to the truck or the sidewalk. 9. **Ninth Object**: The ninth object the ego vehicle should notice is the potted plant. The plant is on the sidewalk, and the ego vehicle should be aware of its presence to avoid any potential damage to the plant or the sidewalk. 10. **Tenth Object**: The tenth object the ego vehicle should notice is
LLaMA-Adapter v2 7Bzero-shot
When the ego vehicle is getting to the next possible location, it should first notice the truck in front of it. The ego vehicle should then take appropriate action, such as slowing down or changing lanes if necessary, to maintain a safe distance from the truck. Next, the ego vehicle should notice the car on the right side of the road. The ego vehicle should then take appropriate action, such as checking its blind spot or changing lanes if necessary, to ensure safe navigation. Finally, the ego vehicle should notice the traffic light. The ego vehicle should then come to a complete stop if the traffic light is red, and proceed with caution if the traffic light is green.
Q4predictionyes / no

Is <c1,CAM_BACK,920.0,645.0> a traffic sign or a road barrier?

Ground truth
No.
Best modelours✓ correct
No.
Oracleours · upper bound✓ correct
No.
InternVL2-2B (no fine-tuning)zero-shotno direct answer
To determine if the coordinates <c1,CAM_BACK,920.0,645.0> represent a traffic sign or a road barrier, let's analyze the image in detail: 1. **Location and Orientation**: - The coordinates are given in the format (x, y, z), where x, y, and z are the coordinates of the point in the image. - The point is located at (920.0, 645.0). 2. **Visual Inspection**: - **Traffic Sign**: A traffic sign typically has a distinct shape, color, or text that indicates a specific rule or regulation. In this image, there is no such distinct feature. - **Road Barrier**: A road barrier is typically a physical structure that separates the road from the sidewalk or other areas. In this image, there is no visible barrier. 3. **Contextual Analysis**: - The coordinates are in the bottom right corner of the image, which is consistent with the typical location of a traffic sign or road barrier. - The surrounding area does not show any signs of a road barrier or sidewalk, which would typically be present in such coordinates. 4. **Conclusion**: - Given the coordinates and the visual inspection, it is reasonable to conclude that the coordinates <c1,CAM_BACK,920.0,645.0> represent a road barrier. In conclusion, the coordinates <c1,CAM_BACK,920.0,645.0> are most likely a road barrier, as there is no visible traffic sign or other typical road barrier features in the image.
Fine-tuned 25kours✓ correct
No.
Fine-tuned 300kours✓ correct
No.
OpenGVLab Mini-DAexternal reference✓ correct
No.
LLaVA-1.6 7Bzero-shotno direct answer
The image you've provided appears to be a collage of different street scenes, and it includes a traffic sign. The sign in question is a "CAM_BACK" sign, which is typically used to indicate that a camera is monitoring the area for traffic enforcement purposes. It is not a physical barrier but rather a warning to drivers that they are being recorded.
LLaMA-Adapter v2 7Bzero-shotno direct answer
The image does not provide enough information to confidently determine whether the <c1,CAM_BACK,920.0,645.0> is a traffic sign or a road barrier. It could be either, depending on the context and the specific location within the image.
Q5planning

What actions could the ego vehicle take based on <c1,CAM_BACK,920.0,645.0>? Why take this action and what's the probability?

Ground truth
The action is to keep going at the same speed. The reason is that there is no safety issue, and the probability is high.
Best modelours
The action is to keep going at the same speed. The reason is that there is no safety issue. The probability is high.
Oracleours · upper bound
The action is to keep going at the same speed. The reason is that there is no safety issue. The probability is high.
InternVL2-2B (no fine-tuning)zero-shot
The ego vehicle, represented by the white SUV in the image, is currently positioned at a crosswalk. Based on the given information, the actions it could take are: 1. **Stop**: The vehicle could potentially stop at the crosswalk to ensure the safety of pedestrians and other vehicles. 2. **Wait**: The vehicle could also wait at the crosswalk, possibly to avoid any potential hazards or to wait for a signal to proceed. 3. **Go**: The vehicle could potentially proceed through the crosswalk, although this action is not explicitly indicated by the given information. The probability of taking this action can be inferred from the context provided by the image. The presence of pedestrians and the fact that the vehicle is at a crosswalk suggest that stopping or waiting is more likely. However, without additional context or information about the surrounding environment, it is not possible to determine the exact probability with certainty. The image alone does not provide enough information to determine the likelihood of proceeding through the crosswalk.
Fine-tuned 25kours
The action is to keep going at the same speed. The reason is that there is no safety issue, and the probability is high.
Fine-tuned 300kours
The action is to keep going at the same speed. The reason is that there is no safety issue. The probability is high.
OpenGVLab Mini-DAexternal reference
The action is to keep going at the same speed. The reason is that there is no safety issue. The probability is high.
LLaVA-1.6 7Bzero-shot
The image you've provided appears to be a sequence of images showing a vehicle's perspective at different points in time, likely from a camera mounted on the back of the vehicle. The sequence is labeled "CAM_BACK," which suggests that the camera is facing backward, capturing the view of the road behind the vehicle. Based on the label "CAM_BACK,920.0,645.0," it seems to indicate that the vehicle is at a specific location (920.0,645.0) and is facing a certain direction (back). Given that the vehicle is at a stoplight, the most likely action the ego vehicle could take is to proceed forward when the traffic light turns green. The probability of this action depends on several factors: 1. The traffic light's current state: If the traffic light is currently red, the vehicle would need to wait until it turns green before proceeding. If it's yellow, the driver might choose to proceed with caution, and if it's green, the vehicle would proceed forward. 2. The presence of other vehicles and pedestrians: The driver must be aware of the traffic conditions behind and in front of the vehicle. If there are other vehicles or pedestrians that have the right of way, the driver must yield to them. 3. The driver's decision-making: The driver must consider the traffic conditions, the road ahead, and any potential hazards before proceeding. 4. The vehicle's mechanical condition: The driver must ensure that the vehicle is in good working condition and is safe to operate. 5. Traffic rules and regulations: The driver must follow the local traffic laws and regulations. The probability of the vehicle proceeding forward when the traffic light turns green would be high, assuming that the driver is following the rules of the road and the vehicle is in good working condition. However, without more context or information about the current state of the traffic light and the surrounding environment, it's impossible to provide a precise probability.
LLaMA-Adapter v2 7Bzero-shot
Based on the given input, the ego vehicle could take the action of turning right, as indicated by the <c1,CAM_BACK,920.0,645.0> command. This action is likely taken to navigate the vehicle around the corner or to change lanes. The probability of the vehicle taking this action is high, as the vehicle is already in motion and approaching the corner, making it necessary to turn right to continue its path.
Q6planning

What actions taken by the ego vehicle can lead to a collision with <c1,CAM_BACK,920.0,645.0>?

Ground truth
Brake suddenly.
Best modelours
No such action will lead to a collision.
Oracleours · upper bound
Back up.
InternVL2-2B (no fine-tuning)zero-shot
The ego vehicle, a white SUV, is approaching a pedestrian crossing. Here are the actions that could lead to a collision with the vehicle in the image: 1. **Speeding**: The SUV is moving at a high speed, which could cause it to cross the pedestrian crossing in a fraction of a second, potentially leading to a collision. 2. **Lane Departure**: The SUV might attempt to change lanes without yielding to the pedestrian, which could result in a collision if the pedestrian is not prepared to move. 3. **Driving Carelessly**: The SUV could be driving recklessly, such as swerving or making sudden turns, which could lead to a collision with the pedestrian. 4. **Not Stopping**: The SUV might not stop at the pedestrian crossing, failing to yield to the pedestrian, which could result in a collision. 5. **Not Braking**: The SUV might not apply the brakes in time to avoid a collision with the pedestrian, especially if it is not paying attention to the pedestrian's presence. These actions, if executed, could lead to a collision with the pedestrian crossing, as depicted in the image.
Fine-tuned 25kours
Back up.
Fine-tuned 300kours
Back up.
OpenGVLab Mini-DAexternal reference
No such action will lead to a collision.
LLaVA-1.6 7Bzero-shot
The image you've provided shows a sequence of actions that can lead to a collision with the vehicle in front, which is identified as <c1,CAM_BACK,920.0,645.0>. Here's a breakdown of the actions shown in the sequence: 1. **CAM_BACK_LEFT**: The ego vehicle is moving to the left, which is a safe maneuver to change lanes or make a turn. 2. **CAM_BACK_RIGHT**: The vehicle continues to move to the right, which is also a safe action to maintain the lane change or turn. 3. **CAM_BACK_LEFT**: The vehicle is now moving back to the left, which is a safe action to return to the original lane or to correct a lane change. 4. **CAM_BACK_RIGHT**: The vehicle is moving back to the right, which is a safe action to return to the original lane or to correct a lane change. 5. **CAM_BACK_LEFT**: The vehicle is moving back to the left, which is a safe action to return to the original lane or to correct a lane change. 6. **CAM_BACK_RIGHT**: The vehicle is moving back to the right, which is a safe action to return to the original lane or to correct a lane change. 7. **CAM_BACK_LEFT**: The vehicle is moving back to the left, which is a safe action to return to the original lane or to correct a lane change. 8. **CAM_BACK_RIGHT**: The vehicle is moving back to the right, which is a safe action to return to the original lane or to correct a lane change. 9. **CAM_BACK_LEFT**: The vehicle is moving back to the left, which is a safe action to return to the original lane or to correct a lane change. 10. **CAM_BACK_RIGHT**: The vehicle is moving back to the right, which is a safe action to return to the original lane or to correct a lane change. 11. **CAM_BACK_LEFT**: The vehicle is moving back to the left, which is a safe action to return to the original lane or to correct a lane change. 12. **CAM_BACK_RIGHT**: The vehicle is moving back to the right, which is a safe action to return to the original lane or to correct a lane
LLaMA-Adapter v2 7Bzero-shot
The ego vehicle is driving down the street, and it is approaching a large truck. The actions taken by the ego vehicle, such as changing lanes, making a sudden turn, or not maintaining a safe distance from the truck, can lead to a collision with the truck. In the image, the ego vehicle is seen changing lanes, which could potentially put it in the path of the truck. To avoid a collision, the ego vehicle should maintain a safe distance from the truck, follow traffic rules, and be aware of the truck's movements and blind spots.
Q7planning

In this scenario, what are safe actions to take for the ego vehicle?

Ground truth
Keep going at the same speed.
Best modelours
Keep going at the same speed, decelerate gradually without braking.
Oracleours · upper bound
Keep going at the same speed, decelerate gradually without braking.
InternVL2-2B (no fine-tuning)zero-shot
Based on the image, the safe actions for the ego vehicle would include maintaining a safe following distance from the truck, ensuring that the truck is in the correct lane, and not overtaking the truck. The ego vehicle should also be prepared to stop if necessary, as the truck is in the right lane and the ego vehicle is in the left lane. Additionally, the ego vehicle should be aware of the truck's presence and any potential hazards it may pose, such as the truck's size and the possibility of the truck changing lanes.
Fine-tuned 25kours
Keep going at the same speed, accelerate, and proceed.
Fine-tuned 300kours
Keep going at the same speed, decelerate gradually without braking.
OpenGVLab Mini-DAexternal reference
Slightly offset to the right, slightly offset to the left, and keep going at the same speed are safe actions to take for the ego vehicle in this scenario.
LLaVA-1.6 7Bzero-shot
In the given scenario, the ego vehicle is a white SUV driving on a city street. To ensure safe actions, the driver should: 1. **Maintain a Safe Following Distance**: Keep a safe distance from the vehicle in front to allow for adequate reaction time in case of sudden braking or changes in traffic flow. 2. **Obey Traffic Signs and Signals**: The driver should follow all traffic signs and signals, including speed limits, stop signs, and traffic lights. 3. **Be Aware of Pedestrians**: The driver should be cautious around crosswalks and pedestrian areas, yielding to pedestrians when required. 4. **Watch for Cyclists**: The driver should be vigilant for cyclists, especially when turning or changing lanes. 5. **Use Turn Signals**: When making turns, the driver should use the appropriate turn signals to communicate their intentions to other road users. 6. **Stay in the Lane**: The driver should stay within the marked lanes and avoid drifting into other lanes or onto the sidewalk. 7. **Be Prepared for Weather Conditions**: If the weather is rainy or foggy, as suggested by the images, the driver should reduce speed, increase following distance, and use headlights appropriately. 8. **Be Alert for Other Vehicles**: The driver should be aware of other vehicles on the road, including trucks, which may have blind spots. 9. **Use Mirrors and Blind Spot Checks**: Regularly check mirrors and blind spots to ensure no other vehicles are in the blind spots or too close. 10. **Stay Focused**: The driver should avoid distractions, such as using a mobile phone, and stay focused on the road. By following these safety guidelines, the driver can navigate the city streets safely and responsibly.
LLaMA-Adapter v2 7Bzero-shot
In the image, there are multiple vehicles, including a truck and a car, driving down a street with buildings in the background. To ensure safety, the ego vehicle should follow these actions:1. Maintain a safe distance from other vehicles, especially the truck, as it is larger and may have a longer stopping distance.2. Be cautious of the truck's blind spots, as it may not be able to see the ego vehicle in certain areas.3. Keep an eye on the truck's movements, as it may make sudden turns or lane changes.4. Be prepared for the truck to change lanes or make a turn, as it may need more space to maneuver.5. Be aware of the truck's braking distance, as it may take longer to come to a complete stop.6. Keep an eye on the traffic lights and follow traffic rules to avoid accidents and ensure smooth traffic flow.By following these safe actions, the ego vehicle can minimize the risk of accidents and ensure a smooth driving experience.
Q8behaviormultiple choice

Predict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the right. The ego vehicle is driving fast. B. The ego vehicle is going straight. The ego vehicle is not moving. C. The ego vehicle is slightly steering to the right. The ego vehicle is driving slowly. D. The ego vehicle is slightly steering to the left. The ego vehicle is driving with normal speed.

Ground truth
B
Best modelours✓ correct
B
Oracleours · upper bound✓ correct
B
InternVL2-2B (no fine-tuning)zero-shotno direct answer
Based on the image, the ego vehicle is driving slowly. This can be inferred from the positioning of the vehicle in the middle of the intersection, with no immediate signs of acceleration or braking. The other options are not supported by the visual evidence in the image.
Fine-tuned 25kours✓ correct
B
Fine-tuned 300koursno direct answer
C. The ego vehicle is slightly steering to the right. The ego vehicle is driving slowly.
OpenGVLab Mini-DAexternal reference✓ correct
B
LLaVA-1.6 7Bzero-shotno direct answer
A. The ego vehicle is slightly steering to the right. The ego vehicle is driving fast.
LLaMA-Adapter v2 7Bzero-shotno direct answer
The correct answer is option B: The ego vehicle is driving fast.