← OverviewNext scene →

Scene 24 of 26 · Boston

8 questions · nuScenes, DriveLM test split

scene f7d01e23508040c2aec430e8da13fb76 · frame 8e2553b1b1184746a85bf2381c050f12

Six camera views around the car: front-left, front, front-right on top; back-left, back, back-right below.c1c2c3c4c5
Six cameras around the car: front-left, front, front-right (top row); back-left, back, back-right (bottom row). Tap the image to open it full size. Pink tags mark the ground-truth objects referenced in the questions.

Reading object references: <c1,CAM_FRONT_RIGHT,1072.1,717.4> means object c1 in the front-right camera at pixel (1072.1, 717.4) of the original 1600×900 image. Models must produce these references themselves, so wrong coordinates mean the model did not find the object.

Which models are compared?

Score = DriveLM final score (0–1, higher is better) on our test split of 3,340 questions. It combines answer accuracy, a GPT judge, language overlap and object matching. Mini-DA is an external checkpoint released by OpenGVLab; all other fine-tuned models are ours.

Showing 3 of 8 models
Q1perception

What are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.

Ground truth
There is a black sedan car to the front right of the ego vehicle, a black sedan car to the front of the ego vehicle, a pedestrian wearing orange pants to the front right of the ego vehicle, a black small car to the back of the ego vehicle, and a white small sedan car to the back of the ego vehicle. The IDs of these objects are <c1,CAM_FRONT_RIGHT,1072.1,717.4>, <c2,CAM_FRONT,1302.0,550.8>, <c3,CAM_FRONT_RIGHT,373.3,530.8>, <c4,CAM_BACK,892.5,515.0>, and <c5,CAM_BACK,500.8,555.8>.
Best modelours
There is a black car to the front of the ego vehicle, a black sedan to the front of the ego vehicle, a black sedan to the front of the ego vehicle, a black sedan to the front of the ego vehicle, a black sedan to the front of the ego vehicle, and a black sedan to the front of the ego vehicle. The IDs of these objects are <c1,CAM_FRONT,1010.0,525.0>, <c2,CAM_FRONT,1010.0,525.0>, <c3,CAM_FRONT,1010.0,525.0>, <c4,CAM_FRONT,1010.0,525.0>, <c5,CAM_FRONT,1010.0,525.0>, and <c6,CAM_FRONT,1010.0,525.0>.
Oracleours · upper bound
There is a black car to the front right of the ego vehicle, a black sedan to the front of the ego vehicle, a pedestrian wearing white clothes to the front right of the ego vehicle, a white sedan to the back of the ego vehicle, and a white sedan to the back of the ego vehicle. The IDs of these objects are <c1,CAM_FRONT_RIGHT,1072.1,717.4>, <c2,CAM_FRONT,1302.5,530.8>, <c3,CAM_FRONT_RIGHT,373.3,530.8>, <c4,CAM_BACK,892.5,515.0>, and <c5,CAM_BACK,500.8,555.8>.
InternVL2-2B (no fine-tuning)zero-shot
In the current scene, the important objects include: 1. **Road**: The primary road is the central focus of the image, providing a clear path for vehicles to travel. 2. **Vehicles**: Multiple cars are visible, including a black sedan, a silver sedan, and a black SUV. These vehicles are essential for understanding traffic flow and road conditions. 3. **Pedestrians**: There are a few pedestrians visible, including a person walking on the sidewalk and another person standing near the storefront. Pedestrians are crucial for understanding the level of activity and potential hazards on the road. 4. **Buildings**: The buildings along the road include a commercial establishment with a storefront and a building with a sign that reads "Svedka." These buildings provide context for the location and help in understanding the urban environment. 5. **Traffic Signs**: There are traffic signs visible, such as a no-parking sign and a stop sign. These signs are important for understanding traffic rules and regulations. 6. **Street Furniture**: The street lamps and traffic signals are essential for managing traffic flow and ensuring safety. These objects collectively provide a comprehensive view of the current scene, allowing for informed decision-making regarding driving and navigation.
Fine-tuned 25kours
There is a black car to the front of the ego vehicle, a black sedan to the front of the ego vehicle, a black sedan to the front of the ego vehicle, a black sedan to the front of the ego vehicle, and a white sedan to the front of the ego vehicle. The IDs of these objects are <c1,CAM_FRONT,1080.0,495.8>, <c2,CAM_FRONT,1080.0,495.8>, <c3,CAM_FRONT,1080.0,495.8>, <c4,CAM_FRONT,1080.0,495.8>, and <c5,CAM_FRONT,1080.0,495.8>.
Fine-tuned 300kours
There is a black car to the front of the ego vehicle, a black sedan to the front of the ego vehicle, a white sedan to the back of the ego vehicle, a white sedan to the front of the ego vehicle, a black sedan to the front right of the ego vehicle, a black sedan to the back right of the ego vehicle, and a black sedan to the back of the ego vehicle. The IDs of these objects are <c1,CAM_FRONT,1080.0,520.0>, <c2,CAM_FRONT,1080.0,505.8>, <c3,CAM_BACK,1010.8,520.0>, <c4,CAM_FRONT,1010.8,505.8>, <c5,CAM_FRONT_RIGHT,1010.8,530.0>, <c6,CAM_BACK_RIGHT,1010.8,530.0>, and <c7,CAM_BACK,1010.8,520.0>.
OpenGVLab Mini-DAexternal reference
There is a black car to the front right of the ego vehicle, a black sedan to the back of the ego vehicle, a black sedan to the front of the ego vehicle, a white sedan to the back of the ego vehicle, a black sedan to the back of the ego vehicle, and a person wearing a white shirt to the front of the ego vehicle. The IDs of these objects are 1, 2, 3, 4, 5, and 6.
LLaVA-1.6 7Bzero-shot
In the current scene, the important objects to consider for future reasoning and driving decisions include: 1. Vehicles: There are several cars parked along the street and one car in motion. 2. Pedestrians: There are people walking on the sidewalks. 3. Traffic lights: There are traffic lights at the intersection. 4. Street signs: There are street signs indicating the direction of traffic and possibly the names of streets. 5. Buildings: The buildings along the street provide context for the urban environment. 6. Trees: There are trees lining the street, which can affect visibility and road conditions. 7. Parking meters: There are parking meters along the street, which indicate where parking is allowed. 8. Road markings: The road has clear lane markings. These objects are important for understanding the current traffic situation, anticipating potential hazards, and making informed driving decisions.
LLaMA-Adapter v2 7Bzero-shot
In the current scene, the important objects to consider for future reasoning and driving decisions include the cars, traffic lights, and pedestrians. The cars are driving down the street, and the traffic lights are present at various locations, indicating the need for drivers to pay attention to their signals. Pedestrians are also present, which means drivers should be cautious and aware of their surroundings to ensure the safety of both pedestrians and other road users.
Q2perceptionmultiple choice

What is the moving status of object <c3,CAM_FRONT_RIGHT,373.3,530.8>? Please select the correct answer from the following options: A. Going ahead. B. Turn right. C. Reverse parking. D. Back up.

Ground truth
A
Best modelours✓ correct
A
Oracleours · upper bound✓ correct
A
InternVL2-2B (no fine-tuning)zero-shotno direct answer
The moving status of object <c3,CAM_FRONT_RIGHT,373.3,530.8> is D. Back up.
Fine-tuned 25kours✓ correct
A
Fine-tuned 300koursno direct answer
A. Going ahead.
OpenGVLab Mini-DAexternal reference✓ correct
A
LLaVA-1.6 7Bzero-shotno direct answer
A. Going ahead
LLaMA-Adapter v2 7Bzero-shotno direct answer
The moving status of object <c3,CAM_FRONT_RIGHT,373.3,530.8> is "Going ahead."
Q3prediction

What object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?

Ground truth
Firstly, notice that <c1,CAM_FRONT_RIGHT,1072.1,717.4>. The object is stationary, so the ego vehicle should continue going ahead at the same speed. Secondly, notice that <c3,CAM_FRONT_RIGHT,373.3,530.8>. The object is going ahead, so the ego vehicle should continue going ahead at the same speed. Thirdly, notice that <c2,CAM_FRONT,1302.0,550.8>. The object is stationary, so the ego vehicle should keep going ahead at the same speed.
Best modelours
Firstly, notice <c1, CAM_FRONT, 1400.0, 530.0>. The object is approaching, so the ego should slow down. Secondly, notice <c2, CAM_FRONT, 1400.0, 530.0>. The object is approaching, so the ego should slow down. Thirdly, notice <c3, CAM_FRONT, 1400.0, 530.0>. The object is approaching, so the ego should slow down.
Oracleours · upper bound
Firstly, notice that <c1,CAM_FRONT_RIGHT,1072.1,717.4>. The object is stationary, so the ego vehicle should continue going ahead at the same speed. Secondly, notice that <c3,CAM_FRONT_RIGHT,373.3,530.8>. The object is going ahead, so the ego vehicle should continue going ahead at the same speed. Thirdly, notice that <c2,CAM_FRONT,1302.5,530.8>. The object is stationary, so the ego vehicle should keep going ahead at the same speed.
InternVL2-2B (no fine-tuning)zero-shot
When the ego vehicle is getting to the next possible location, it should notice the "Svedka" store first. The state of the object perceived by the ego vehicle as the first object is that the store is open. The action the ego vehicle should take is to enter the store. The object that the ego vehicle should notice second is the "Dales' Pale Ale" store. The state of the object perceived by the ego vehicle as the second object is that the store is open. The action the ego vehicle should take is to enter the store. The object that the ego vehicle should notice third is the "Support Wine" store. The state of the object perceived by the ego vehicle as the third object is that the store is open. The action the ego vehicle should take is to enter the store.
Fine-tuned 25kours
Firstly, notice that <c3,CAM_FRONT_RIGHT,1080.8,525.0>. The object is going ahead, so the ego vehicle should keep going ahead at the same speed. Secondly, notice that <c2,CAM_FRONT,1080.8,500.8>. The object is going ahead, so the ego vehicle should keep going ahead at the same speed. Thirdly, notice that <c1,CAM_FRONT,1080.8,500.8>. The object is going ahead, so the ego vehicle should keep going ahead at the same speed.
Fine-tuned 300kours
Firstly, notice that <c3,CAM_FRONT,1080.0,520.0>. The object is stationary, so the ego vehicle should continue ahead at the same speed. Secondly, notice that <c2,CAM_FRONT,1080.0,505.8>. The object is stationary, so the ego vehicle should continue ahead at the same speed. Thirdly, notice that <c1,CAM_BACK,1035.0,520.0>. The object is going ahead, so the ego vehicle should continue ahead at the same speed.
OpenGVLab Mini-DAexternal reference
Firstly, notice that the object in front of the ego vehicle is stationary. The ego vehicle should continue ahead at the same speed. Secondly, notice that the object in front of the ego vehicle is stationary. The ego vehicle should continue ahead at the same speed. Thirdly, notice that the object in front of the ego vehicle is stationary. The ego vehicle should continue ahead at the same speed.
LLaVA-1.6 7Bzero-shot
The image you've provided appears to be a collage of different street scenes, each with various objects that the ego vehicle should be aware of when approaching the next possible location. Here's a breakdown of what the ego vehicle should notice and what actions it should take: 1. First object: The ego vehicle should notice the traffic lights first. The state of the traffic lights is red, indicating that the ego vehicle must stop. The action the ego vehicle should take is to come to a complete stop at the red light. 2. Second object: The second object the ego vehicle should notice is the pedestrian crossing the street. The pedestrian appears to be in the crosswalk, which means the ego vehicle should yield to the pedestrian and wait until they have safely crossed the street before proceeding. 3. Third object: The third object the ego vehicle should notice is the car in the left lane. The car is in motion, so the ego vehicle should be cautious and maintain a safe following distance. 4. Fourth object: The fourth object is the car in the right lane. The car is also in motion, so the ego vehicle should be aware of its presence and maintain a safe distance. 5. Fifth object: The fifth object is the storefront with the sign "CAM BACK RIGHT." The ego vehicle should be aware of the storefront and any potential parking or stopping for the store. 6. Sixth object: The sixth object is the car in the right lane. The car is in motion, so the ego vehicle should be aware of its presence and maintain a safe distance. 7. Seventh object: The seventh object is the car in the left lane. The car is in motion, so the ego vehicle should be aware of its presence and maintain a safe distance. 8. Eighth object: The eighth object is the car in the right lane. The car is in motion, so the ego vehicle should be aware of its presence and maintain a safe distance. 9. Ninth object: The ninth object is the car in the left lane. The car is in motion, so the ego vehicle should be aware of its presence and maintain a safe distance. 10. Tenth object: The tenth object is the car in the right lane. The car is in motion, so the ego vehicle should be aware of its presence and maintain a safe distance. In each of these scenarios
LLaMA-Adapter v2 7Bzero-shot
When the ego vehicle is getting to the next possible location, it should first notice the traffic light, which is in the state of red. The ego vehicle should come to a complete stop, as per traffic rules and road safety. Next, the ego vehicle should notice the parked cars on the side of the road. The ego vehicle should proceed with caution, as it may need to maneuver around the parked cars. Finally, the ego vehicle should notice the pedestrian crossing the street. The ego vehicle should yield to the pedestrian and wait for them to safely cross the road before proceeding.
Q4predictionyes / no

Is <c1,CAM_FRONT_RIGHT,1072.1,717.4> a traffic sign or a road barrier?

Ground truth
No.
Best modelours✓ correct
No.
Oracleours · upper bound✓ correct
No.
InternVL2-2B (no fine-tuning)zero-shotno direct answer
To determine if the coordinates <c1,CAM_FRONT_RIGHT,1072.1,717.4> represent a traffic sign or a road barrier, let's break down the coordinates and analyze the context. 1. **Coordinates Analysis**: - **c1**: This is the location of the camera. - **CAM_FRONT_RIGHT**: This is the direction the camera is facing. - **1072.1, 717.4**: These are the coordinates of the point in the image. 2. **Contextual Analysis**: - **Traffic Signs**: Typically, traffic signs are rectangular or square in shape and have specific dimensions. The coordinates provided are not typical for a traffic sign. - **Road Barriers**: Road barriers are usually rectangular or square and have specific dimensions. The coordinates provided are not typical for a road barrier either. 3. **Conclusion**: - Given the coordinates and typical dimensions of traffic signs and road barriers, it is highly unlikely that the coordinates <c1,CAM_FRONT_RIGHT,1072.1,717.4> represent a traffic sign or a road barrier. The coordinates are too large and irregular for these types of objects. Therefore, the coordinates <c1,CAM_FRONT_RIGHT,1072.1,717.4> most likely represent a point in the image, not a traffic sign or road barrier.
Fine-tuned 25kours✓ correct
No.
Fine-tuned 300kours✓ correct
No.
OpenGVLab Mini-DAexternal reference✓ correct
No.
LLaVA-1.6 7Bzero-shotno direct answer
The image you've provided appears to be a collage of different street scenes, and it includes a traffic sign. The sign in question is a "CAM_FRONT_RIGHT" sign, which is likely a type of camera monitoring system. It's not a physical traffic sign, but rather a digital or electronic sign that indicates the presence of a camera facing the right side of the road.
LLaMA-Adapter v2 7Bzero-shotno direct answer
The image does not provide enough information to confidently determine whether the <c1,CAM_FRONT_RIGHT,1072.1,717.4> is a traffic sign or a road barrier. It could be either, depending on the context and the specific location within the image.
Q5planning

What actions could the ego vehicle take based on <c1,CAM_FRONT_RIGHT,1072.1,717.4>? Why take this action and what's the probability?

Ground truth
The action is to keep going at the same speed. The reason is that there is no safety issue. The probability is high.
Best modelours
The action is to keep going at the same speed. The reason is that there is no safety issue, and the probability is high.
Oracleours · upper bound
The action is to keep going at the same speed. The reason is that there is no safety issue, and the probability is high.
InternVL2-2B (no fine-tuning)zero-shot
The ego vehicle, indicated by the camera's position, is currently positioned at the intersection of Cam Front Right and Cam Back Right. Based on the given coordinates, the ego vehicle can take several actions: 1. **Stop**: The ego vehicle can stop at the intersection to ensure safety and avoid any potential collisions. 2. **Slow Down**: The ego vehicle can slow down to ensure it doesn't cross the intersection or hit any pedestrians or vehicles. 3. **Turn Left**: The ego vehicle can turn left to proceed in the direction it intends to go. 4. **Turn Right**: The ego vehicle can turn right to continue on its intended path. The probability of taking these actions can be calculated based on the current state of the intersection and the surrounding environment. For example, if the ego vehicle is currently in the middle of a lane and there are no other vehicles or pedestrians in the immediate vicinity, the probability of stopping is high. However, if there are pedestrians or other vehicles approaching the intersection, the probability of stopping might be lower. In this case, the ego vehicle is currently positioned at the intersection of Cam Front Right and Cam Back Right. Given the current state, the ego vehicle has a high probability of stopping at the intersection to ensure safety.
Fine-tuned 25kours
The action is to keep going at the same speed. The reason is that there is no safety issue, and the probability is high.
Fine-tuned 300kours
The action is to keep going at the same speed. The reason is that there is no safety issue, and the probability is high.
OpenGVLab Mini-DAexternal reference
The action is to keep going at the same speed. The reason is that there is no safety issue, and the probability is high.
LLaVA-1.6 7Bzero-shot
The image you've provided appears to be a sequence of images showing a vehicle's perspective at different points in time, likely from a camera mounted on the front right of the vehicle. The sequence seems to be capturing the vehicle's movement through an intersection. Based on the first image, the ego vehicle is currently at the intersection, waiting for the traffic light to change. The vehicle is likely to proceed forward when the light turns green. The probability of this action is high, as it is the standard and expected behavior for a vehicle at a traffic light. In the second image, the vehicle is still at the intersection, but the traffic light has turned green. The vehicle should proceed forward, as it is safe to do so and it is the correct action based on the traffic signal. The probability of this action is very high, as it is the standard and expected behavior for a vehicle when the light turns green. In the third image, the vehicle is still at the intersection, but the traffic light has turned red. The vehicle should stop, as it is not safe to proceed forward with a red light. The probability of this action is high, as it is the standard and expected behavior for a vehicle when the light turns red. In the fourth image, the vehicle is still at the intersection, but the traffic light has turned yellow. The vehicle should prepare to stop, as the light is about to turn red. The probability of this action is moderate, as the driver should be prepared to stop but has not yet done so. In the fifth image, the vehicle is still at the intersection, but the traffic light has turned green again. The vehicle should proceed forward, as it is safe to do so and it is the correct action based on the traffic signal. The probability of this action is very high, as it is the standard and expected behavior for a vehicle when the light turns green. In the sixth image, the vehicle is still at the intersection, but the traffic light has turned red again. The vehicle should stop, as it is not safe to proceed forward with a red light. The probability of this action is high, as it is the standard and expected behavior for a vehicle when the light turns red. In the seventh image, the vehicle is still at the intersection, but the traffic light has turned yellow again. The vehicle should prepare to stop, as the light is about to turn red. The probability of this action is moderate, as the driver should
LLaMA-Adapter v2 7Bzero-shot
Based on the image coordinates <c1,CAM_FRONT_RIGHT,1072.1,717.4>, the ego vehicle is likely in the left lane of a multi-lane road. The vehicle could take several actions, such as changing lanes to the right, merging into the right lane, or staying in the left lane. The probability of the vehicle changing lanes to the right is 50% (since the coordinates are in the left lane), merging into the right lane is 25% (since the coordinates are in the middle lane), and staying in the left lane is 25% (since the coordinates are in the left lane). The probability of the vehicle changing lanes to the right is the highest, as it is in the left lane, while the probability of staying in the left lane is the lowest, as it is in the left lane.
Q6planning

What actions taken by the ego vehicle can lead to a collision with <c1,CAM_FRONT_RIGHT,1072.1,717.4>?

Ground truth
Sharp right turn.
Best modelours
Moderate right turn.
Oracleours · upper bound
Sharp right turn.
InternVL2-2B (no fine-tuning)zero-shot
The ego vehicle, which is the car in the foreground, is driving straight down the road. The car in the background, labeled as "CAM_FRONT_RIGHT," is positioned directly in front of the ego vehicle. The distance between the two cars is approximately 1072.1 meters (717.4 feet) as indicated by the coordinates provided in the image. ### Chain of Thought (CoT) Analysis: 1. **Position of the Ego Vehicle:** The ego vehicle is driving straight down the road. 2. **Position of the Background Car:** The background car is positioned directly in front of the ego vehicle. 3. **Distance Calculation:** The distance between the two cars is calculated as: \[ \text{Distance} = \text{Length of road} - \text{Length of ego vehicle} - \text{Length of background car} \] - **Length of road:** 1072.1 meters (717.4 feet) - **Length of ego vehicle:** 0 meters (0 feet) - **Length of background car:** 1072.1 meters (717.4 feet) 4. **Calculation:** \[ \text{Distance} = 1072.1 \, \text{meters} - 0 \, \text{meters} - 717.4 \, \text{meters} = 1072.1 \, \text{meters} \] 5. **Conclusion:** The ego vehicle is approaching the background car, and the calculated distance suggests that there is a potential collision risk. The ego vehicle should be cautious and slow down to avoid a collision with the background car.
Fine-tuned 25kours
Sharp right turn.
Fine-tuned 300kours
Sharp right turn.
OpenGVLab Mini-DAexternal reference
Sharp right turn.
LLaVA-1.6 7Bzero-shot
The image you've provided shows a sequence of photos that depict a vehicle's movement through an intersection. The actions that could lead to a collision with the vehicle in the front right corner (<c1,CAM_FRONT_RIGHT,1072.1,717.4>) include: 1. **Failure to Yield**: If the ego vehicle does not yield to the vehicle in the front right corner when required, it could lead to a collision. 2. **Running a Red Light**: If the ego vehicle runs a red light, it could collide with the vehicle in the front right corner. 3. **Making a Left Turn Without Signaling**: If the ego vehicle makes a left turn without signaling, it could be unexpected by the vehicle in the front right corner, leading to a collision. 4. **Not Checking Blind Spots**: If the ego vehicle does not check its blind spots before making a turn or changing lanes, it could fail to see the vehicle in the front right corner, resulting in a collision. 5. **Speeding**: If the ego vehicle is speeding and cannot stop in time to avoid a collision with the vehicle in the front right corner. 6. **Distracted Driving**: If the ego vehicle's driver is distracted and not paying attention to the road, they might not notice the vehicle in the front right corner, leading to a collision. 7. **Not Following Traffic Rules**: If the ego vehicle does not follow the traffic rules, such as stopping at a stop sign or not making a turn when it's not allowed, it could lead to a collision with the vehicle in the front right corner. 8. **Not Using Turn Signals**: If the ego vehicle does not use turn signals to indicate its intentions, the vehicle in the front right corner might not anticipate the turn, leading to a collision. 9. **Not Adjusting for Road Conditions**: If the ego vehicle does not adjust its speed or following distance based on road conditions, it could lead to a collision with the vehicle in the front right corner. 10. **Not Checking Mirrors**: If the ego vehicle's driver does not check their mirrors before making a turn or changing lanes, they might not see the vehicle in the front right corner, leading to a collision. It's important to note that
LLaMA-Adapter v2 7Bzero-shot
The ego vehicle is driving on a city street with other cars, and it is approaching a red light. If the ego vehicle continues driving without stopping at the red light, it may collide with the car that is currently stopped at the red light. This is because the ego vehicle is moving in the same direction as the stopped car, and if it does not stop, it will likely hit the stopped car. The ego vehicle should come to a complete stop at the red light to avoid a collision with the stopped car.
Q7planning

In this scenario, what are safe actions to take for the ego vehicle?

Ground truth
Keep going at the same speed, decelerate gradually without braking.
Best modelours
Keep going at the same speed, decelerate gradually without braking.
Oracleours · upper bound
Keep going at the same speed, decelerate gradually without braking.
InternVL2-2B (no fine-tuning)zero-shot
Based on the image, the safe actions for the ego vehicle would include maintaining a safe following distance from the car in front of it, as indicated by the "CAM_FRONT" text. The ego vehicle should also be prepared to stop if necessary, as suggested by the "CAM_FRONT" text. Additionally, the ego vehicle should be aware of the pedestrian crossing the street, as indicated by the "CAM_FRONT" text, and should be cautious of the pedestrian's movement. The ego vehicle should also be aware of the traffic signals and adjust its speed accordingly to ensure it does not cross the intersection.
Fine-tuned 25kours
Keep going at the same speed, accelerate, and proceed.
Fine-tuned 300kours
Keep going at the same speed, decelerate gradually without braking.
OpenGVLab Mini-DAexternal reference
Keep going at the same speed, decelerate gradually without braking.
LLaVA-1.6 7Bzero-shot
In the given scenario, the ego vehicle is a car driving down a city street with various other vehicles, pedestrians, and traffic signs. To ensure safe driving, the following actions should be taken: 1. **Maintain a Safe Following Distance**: Keep a safe distance from the vehicle in front to allow for adequate reaction time in case of sudden braking or changes in traffic flow. 2. **Obey Traffic Signs and Signals**: Pay attention to traffic signs and signals, such as stop signs, traffic lights, and pedestrian crossings. 3. **Be Aware of Pedestrians**: Watch for pedestrians crossing the street or walking along the sidewalk. Yield to them when required. 4. **Use Turn Signals**: When making turns, use the appropriate turn signals to indicate your intentions to other road users. 5. **Stay in the Correct Lane**: Keep to the right side of the road, as indicated by the lane markings. 6. **Be Prepared for Parked Cars**: Be cautious of parked cars, especially when turning or changing lanes, as they may obstruct your view or the view of other drivers. 7. **Watch for Cyclists**: Be aware of cyclists who may be sharing the road with you. 8. **Slow Down in Low-Visibility Conditions**: If the weather is poor, such as during rain or fog, reduce your speed to ensure you have enough time to react to any potential hazards. 9. **Use Your Headlights**: If you're driving in low-light conditions, turn on your headlights to increase your visibility to other road users. 10. **Stay Alert**: Always stay focused on the road and avoid distractions such as using your phone or eating while driving. By following these safety guidelines, the ego vehicle can navigate the city street safely and responsibly.
LLaMA-Adapter v2 7Bzero-shot
In the image, there are multiple cars and a truck on the street, with some of them stopped at a red light. To ensure safety, the ego vehicle should follow these actions:1. Obey traffic signals: The ego vehicle should come to a complete stop at the red light, as seen in the image.2. Be cautious of other vehicles: The ego vehicle should be aware of other cars and trucks on the road, maintaining a safe distance and being prepared for sudden movements.3. Look out for pedestrians: The ego vehicle should be vigilant for pedestrians crossing the street or walking near the road, as there are a few people visible in the image.4. Be prepared for changes in traffic flow: The ego vehicle should be ready to react to any changes in traffic flow, such as other vehicles moving or changing lanes, and adjust its speed and position accordingly.By following these safe actions, the ego vehicle can navigate the busy street and minimize the risk of accidents or traffic violations.
Q8behaviormultiple choice

Predict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is going straight. The ego vehicle is driving fast. B. The ego vehicle is going straight. The ego vehicle is driving with normal speed. C. The ego vehicle is slightly steering to the left. The ego vehicle is driving slowly. D. The ego vehicle is slightly steering to the right. The ego vehicle is driving slowly.

Ground truth
A
Best modelours✗ wrong
B
Oracleours · upper bound✓ correct
A
InternVL2-2B (no fine-tuning)zero-shotno direct answer
Based on the image, the ego vehicle is driving straight. The image shows the ego vehicle in the center of the road with no visible signs of deviation or steering to the left or right. The other vehicles are also driving straight ahead, indicating that the ego vehicle is following the same path.
Fine-tuned 25kours✓ correct
A
Fine-tuned 300koursno direct answer
B. The ego vehicle is going straight. The ego vehicle is driving with normal speed.
OpenGVLab Mini-DAexternal reference✓ correct
A
LLaVA-1.6 7Bzero-shotno direct answer
Based on the image provided, the ego vehicle is going straight and driving with normal speed. The vehicle is not steering to the left or right, and there is no indication of the vehicle driving fast or slowly.
LLaMA-Adapter v2 7Bzero-shotno direct answer
The ego vehicle is driving with normal speed, going straight down the street.