scene 59530445fcc04c2090bfaf63f342f794 · frame 79025431db8a4c8083c4e381d9704e62
c1c2c3c4c5Six cameras around the car: front-left, front, front-right (top row); back-left, back, back-right (bottom row). Tap the image to open it full size. Pink tags mark the ground-truth objects referenced in the questions.
Reading object references:<c1,CAM_BACK,1182.5,488.3> means object c1 in the back camera at pixel (1182.5, 488.3) of the original 1600×900 image. Models must produce these references themselves, so wrong coordinates mean the model did not find the object.
Upper bound: our fine-tuned model, tested with ground-truth object markers and their coordinate labels drawn on the images (no retraining). Not deployable; measures the grounding bottleneck.
Released checkpoint of InternVL2-2B, referenced as Mini-InternVL2-2B-DA by OpenGVLab. Fully fine-tuned on the original DriveLM split (~378k questions, ~10× our data); under 0.9% of its training data comes from our test scenes.
General-purpose vision-language model, no driving fine-tuning.
OpenGVLab/llama_adapter_v2_multimodal7b
Score = DriveLM final score (0–1, higher is better) on our test split of 3,340 questions. It combines answer accuracy, a GPT judge, language overlap and object matching. Mini-DA is an external checkpoint released by OpenGVLab; all other fine-tuned models are ours.
Showing 3 of 8 models
Q1perception
What are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.
Ground truth
There is a white truck to the back of the ego vehicle, a person wearing yellow clothes and gray pants to the front of the ego vehicle, a white van to the back of the ego vehicle, a white truck to the front of the ego vehicle, and a person wearing a blue shirt and gray pants to the back of the ego vehicle. The IDs of these objects are <c1,CAM_BACK,1182.5,488.3>, <c2,CAM_FRONT,311.7,504.2>, <c3,CAM_BACK,833.3,487.5>, <c4,CAM_FRONT,1132.5,414.2>, and <c5,CAM_BACK,716.7,504.2>.
Best modelours
There is a white truck to the front of the ego vehicle, a white sedan to the front of the ego vehicle, a white sedan to the front of the ego vehicle, a white sedan to the front of the ego vehicle, and a white sedan to the front of the ego vehicle. The IDs of these objects are <c1,CAM_FRONT,1080.0,525.0>, <c2,CAM_FRONT,1080.0,505.0>, <c3,CAM_FRONT,1080.0,505.0>, <c4,CAM_FRONT,1080.0,505.0>, and <c5,CAM_FRONT,1080.0,505.0>.
Oracleours · upper bound
There is a white truck to the back of the ego vehicle, a pedestrian wearing a blue shirt to the front of the ego vehicle, a white truck to the front of the ego vehicle, and a white truck to the back of the ego vehicle. The IDs of these objects are <c1,CAM_BACK,882.5,488.3>, <c2,CAM_FRONT,311.7,504.2>, <c3,CAM_FRONT,1234.2,435.0>, and <c4,CAM_FRONT,1132.5,414.2>.
InternVL2-2B (no fine-tuning)zero-shot
In the current scene, the important objects include:
1. **Truck**: The white truck with the "XPO Logistics" logo is a critical object. It is likely involved in some form of logistics or transportation, which is relevant to the context of the image.
2. **Road**: The road is the primary setting for the scene. It is the medium through which the truck is traveling, and it is also the area where the pedestrian is crossing.
3. **Pedestrian**: The pedestrian is crossing the road, which is a critical element of the scene. The pedestrian's presence indicates that the road is in use and that the area is accessible.
4. **Buildings**: The multi-story buildings in the background are significant as they provide context and scale to the scene. They also suggest that the location is urban or semi-urban.
5. **Trees and Sidewalk**: The trees and sidewalk are important for providing context and setting the scene. They help to define the environment and indicate that the area is designed for pedestrian and vehicular traffic.
6. **Traffic Signs**: The traffic signs, including the "P" sign, are important for indicating the direction of traffic and ensuring safety. They also provide information about the road's layout and regulations.
These objects collectively help to create a coherent and functional scene, which is essential for understanding the context and making informed driving decisions.
Fine-tuned 25kours
There is a white truck to the front of the ego vehicle, a white truck to the front right of the ego vehicle, a white truck to the front of the ego vehicle, a white truck to the front of the ego vehicle, and a white truck to the front of the ego vehicle. The IDs of these objects are <c1,CAM_FRONT,1080.0,495.8>, <c2,CAM_FRONT_RIGHT,1080.0,495.8>, <c3,CAM_FRONT,1080.0,495.8>, <c4,CAM_FRONT,1080.0,495.8>, and <c5,CAM_FRONT,1080.0,495.8>.
Fine-tuned 300kours
There is a white truck to the front of the ego vehicle, a white truck to the front of the ego vehicle, a white truck to the front of the ego vehicle, a white truck to the front of the ego vehicle, a white truck to the front of the ego vehicle, and a white truck to the front of the ego vehicle. The IDs of these objects are <c1,CAM_FRONT,1080.0,480.0>, <c2,CAM_FRONT,1080.0,490.0>, <c3,CAM_FRONT,1080.0,490.0>, <c4,CAM_FRONT,1080.0,490.0>, <c5,CAM_FRONT,1080.0,490.0>, and <c6,CAM_FRONT,1080.0,490.0>.
OpenGVLab Mini-DAexternal reference
There is a white truck to the front of the ego vehicle, a white truck to the back of the ego vehicle, a white truck to the front of the ego vehicle, and a pedestrian wearing a yellow top and black pants to the front of the ego vehicle. The IDs of these objects are 1, 2, 3, and 4.
LLaVA-1.6 7Bzero-shot
In the current scene, the important objects to consider for future reasoning and driving decisions include:
1. The large truck in the middle of the image, which is a significant obstacle and requires attention to its size and potential blind spots.
2. The pedestrian crossing sign on the right side of the image, indicating a pedestrian crossing area that drivers should be cautious of.
3. The traffic lights, which are essential for regulating traffic flow and ensuring safety at intersections.
4. The parked cars on the right side of the image, which could obstruct the view of oncoming traffic or pedestrians.
5. The trees and grass on the right side of the image, which could limit visibility around corners or at intersections.
6. The buildings in the background, which might indicate a commercial or residential area, and could influence traffic patterns and pedestrian activity.
7. The road markings, which provide information about lanes, turns, and other road features that drivers need to be aware of.
These objects are crucial for drivers to navigate safely and efficiently in the current scene.
LLaMA-Adapter v2 7Bzero-shot
In the current scene, the important objects to consider for future reasoning and driving decisions include the parked cars, the truck, the traffic light, and the street signs. The parked cars and truck are stationary, and the traffic light is currently green, indicating that it is safe to proceed. The street signs provide information about the road conditions, speed limits, or upcoming turns, which are essential for drivers to make informed decisions while driving. By taking these objects into account, drivers can anticipate potential hazards, maintain safe speeds, and follow traffic rules to ensure a safe driving experience.
Q2perceptionmultiple choice
What is the moving status of object <c1,CAM_BACK,1182.5,488.3>? Please select the correct answer from the following options: A. Turn right. B. Going ahead. C. Reverse parking. D. Stopped.
Ground truth
B
Best modelours✓ correct
B
Oracleours · upper bound✓ correct
B
InternVL2-2B (no fine-tuning)zero-shotno direct answer
The moving status of object <c1,CAM_BACK,1182.5,488.3> is D. Stopped.
Fine-tuned 25kours✓ correct
B
Fine-tuned 300koursno direct answer
D. Stopped.
OpenGVLab Mini-DAexternal reference✓ correct
B
LLaVA-1.6 7Bzero-shotno direct answer
D. Stopped
LLaMA-Adapter v2 7Bzero-shotno direct answer
The moving status of object <c1,CAM_BACK,1182.5,488.3> is "Reverse parking."
Q3prediction
What object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?
Ground truth
Firstly, notice that <c4,CAM_FRONT,1132.5,414.2>. The object is going ahead, so the ego vehicle should turn right. Secondly, notice that <c2,CAM_FRONT,311.7,504.2>. The object is going ahead, so the ego vehicle should turn right. Thirdly, notice that <c5,CAM_BACK,716.7,504.2>. The object is going ahead, so the ego vehicle should keep going ahead at the same speed.
Best modelours
Firstly, notice <c1, CAM_FRONT, 1400.0, 600.0>. The object is approaching, so the ego should slow down. Secondly, notice <c2, CAM_FRONT, 1400.0, 600.0>. The object is approaching, so the ego should slow down. Thirdly, notice <c3, CAM_FRONT, 1400.0, 600.0>. The object is approaching, so the ego should slow down.
Oracleours · upper bound
Firstly, notice that <c4,CAM_FRONT,1132.5,414.2>. The object is stationary, so the ego vehicle should continue ahead at the same speed. Secondly, notice that <c2,CAM_FRONT,311.7,504.2>. The object is stationary, so the ego vehicle should continue ahead at the same speed. Thirdly, notice that <c3,CAM_BACK,1080.8,488.3>. The object is going ahead, so the ego vehicle should continue ahead at the same speed.
InternVL2-2B (no fine-tuning)zero-shot
When the ego vehicle is getting to the next possible location, it should notice the white FedEx truck first. The state of the object perceived by the ego vehicle as the first object is the white FedEx truck. The action the ego vehicle should take is to continue driving straight ahead. The ego vehicle should notice the white FedEx truck second when it is getting to the next possible location. The state of the object perceived by the ego vehicle as the second object is the white FedEx truck. The action the ego vehicle should take is to continue driving straight ahead. The ego vehicle should notice the white FedEx truck third when it is getting to the next possible location. The state of the object perceived by the ego vehicle as the third object is the white FedEx truck. The action the ego vehicle should take is to continue driving straight ahead.
Fine-tuned 25kours
Firstly, notice that <c3,CAM_FRONT,1080.8,525.0>. The object is going ahead, so the ego vehicle should keep going ahead at the same speed. Secondly, notice that <c2,CAM_FRONT,1080.8,505.0>. The object is going ahead, so the ego vehicle should keep going ahead at the same speed. Thirdly, notice that <c1,CAM_FRONT,1080.8,485.0>. The object is going ahead, so the ego vehicle should keep going ahead at the same speed.
Fine-tuned 300kours
Firstly, notice that <c3,CAM_FRONT,1080.0,490.8>. The object is going ahead, so the ego vehicle should keep going ahead at the same speed. Secondly, notice that <c1,CAM_FRONT,1080.0,490.8>. The object is going ahead, so the ego vehicle should keep going ahead at the same speed. Thirdly, notice that <c2,CAM_FRONT,1080.0,490.8>. The object is going ahead, so the ego vehicle should keep going ahead at the same speed.
OpenGVLab Mini-DAexternal reference
Firstly, notice that the object in front of the ego vehicle is going ahead. The ego vehicle should turn right. Secondly, notice that the object in front of the ego vehicle is going ahead. The ego vehicle should turn right. Thirdly, notice that the object in front of the ego vehicle is going ahead. The ego vehicle should turn right.
LLaVA-1.6 7Bzero-shot
The image you've provided appears to be a collage of different street scenes, each with its own set of objects and actions. Here's a description of the objects and actions that the ego vehicle should consider when approaching each scene:
1. **CAM_BACK_LEFT**: The ego vehicle should first notice the construction area in the background. The state of the construction area is active, with construction equipment and materials visible. The ego vehicle should slow down, be prepared for potential lane closures, and follow any posted signs or directions provided by construction personnel.
2. **CAM_BACK_RIGHT**: The ego vehicle should notice the large truck in the middle of the road. The state of the truck is stationary, possibly waiting for a signal or for other traffic to clear. The ego vehicle should maintain a safe distance from the truck and be prepared to stop if necessary.
3. **CAM_FRONT_LEFT**: The ego vehicle should notice the pedestrian crossing the street. The state of the pedestrian is in motion, crossing the street. The ego vehicle should yield to the pedestrian and wait for them to cross safely.
4. **CAM_FRONT_RIGHT**: The ego vehicle should notice the parked cars on the side of the road. The state of the cars is stationary. The ego vehicle should be cautious when passing the parked cars, as there might be pedestrians entering or exiting the vehicles.
5. **CAM_BACK_LEFT**: The ego vehicle should notice the traffic light in the background. The state of the traffic light is not visible in the image, but the ego vehicle should be prepared to stop or proceed depending on the color of the light.
6. **CAM_BACK_RIGHT**: The ego vehicle should notice the parked cars on the side of the road. The state of the cars is stationary. The ego vehicle should be cautious when passing the parked cars, as there might be pedestrians entering or exiting the vehicles.
7. **CAM_FRONT_LEFT**: The ego vehicle should notice the pedestrian crossing the street. The state of the pedestrian is in motion, crossing the street. The ego vehicle should yield to the pedestrian and wait for them to cross safely.
8. **CAM_FRONT_Right**: The ego vehicle should notice the parked cars on the side of the road. The state of the cars is stationary. The ego vehicle should be
LLaMA-Adapter v2 7Bzero-shot
When the ego vehicle is getting to the next possible location, it should first notice the traffic light, which is in the state of "red." The ego vehicle should then take the appropriate action, which is to stop at the red light. Next, the ego vehicle should notice the truck, which is in the state of "moving." The ego vehicle should then take the appropriate action, which is to proceed with caution, as the truck may be moving at a slower speed or may be turning. Finally, the ego vehicle should notice the bus, which is in the state of "moving." The ego vehicle should then take the appropriate action, which is to proceed with caution, as the bus may be moving at a slower speed or may be turning.
Q4predictionyes / no
Is <c1,CAM_BACK,1182.5,488.3> a traffic sign or a road barrier?
Ground truth
No.
Best modelours✓ correct
No.
Oracleours · upper bound✓ correct
No.
InternVL2-2B (no fine-tuning)zero-shotno direct answer
To determine if the coordinates <c1,CAM_BACK,1182.5,488.3> represent a traffic sign or a road barrier, let's analyze the image closely.
1. **Location and Orientation**:
- The coordinates are given in the format (x, y, z), where x, y, and z are the coordinates of the point in the image.
- The coordinates provided are (1182.5, 488.3).
2. **Visual Inspection**:
- **Traffic Sign**: A traffic sign typically has a distinct shape, color, or text that clearly indicates a specific rule or regulation.
- **Road Barrier**: A road barrier is usually a physical structure that obstructs or restricts the flow of traffic.
3. **Contextual Analysis**:
- The coordinates are located in the middle of a road, which is a typical place for traffic signs.
- There is no visible text or shape that would indicate a traffic sign.
4. **Conclusion**:
- Given the coordinates are in the middle of a road, it is highly likely that this point represents a road barrier rather than a traffic sign.
Therefore, the coordinates <c1,CAM_BACK,1182.5,488.3> are most likely a road barrier.
Fine-tuned 25kours✓ correct
No.
Fine-tuned 300kours✓ correct
No.
OpenGVLab Mini-DAexternal reference✓ correct
No.
LLaVA-1.6 7Bzero-shotno direct answer
The image you've provided appears to show a traffic sign rather than a road barrier. The sign is labeled "CAM_BACK_LEFT," which suggests it is a camera positioned to capture vehicles from the left side of the road as they pass by. This type of camera is often used for traffic monitoring and enforcement purposes.
LLaMA-Adapter v2 7Bzero-shotno direct answer
The image does not provide enough information to confidently determine whether the <c1,CAM_BACK,1182.5,488.3> is a traffic sign or a road barrier. It could be either, depending on the context and the specific location within the image.
Q5planning
What actions could the ego vehicle take based on <c1,CAM_BACK,1182.5,488.3>? Why take this action and what's the probability?
Ground truth
The action is to turn right. The reason is that there is no safety issue, which is high.
Best modelours
The action is to keep going at the same speed. The reason is that there is no safety issue. The probability is high.
Oracleours · upper bound
The action is to keep going at the same speed. The reason is that there is no safety issue. The probability is high.
InternVL2-2B (no fine-tuning)zero-shot
The ego vehicle, represented by the white truck in the center of the image, is currently positioned at the intersection of a two-way street. Based on the given information, the ego vehicle could take the following actions:
1. **Turn Left**: The ego vehicle is currently facing the intersection, and the direction indicated by the arrow on the road sign is to the left. Therefore, the ego vehicle could turn left to continue on the leftmost lane.
2. **Turn Right**: The ego vehicle is currently facing the intersection, and the direction indicated by the arrow on the road sign is to the right. Therefore, the ego vehicle could turn right to continue on the rightmost lane.
3. **Go Straight**: The ego vehicle is currently facing the intersection, and there are no visible arrows or signs indicating a straight path. Therefore, the ego vehicle could proceed straight through the intersection.
**Probability**:
- **Turn Left**: The probability of the ego vehicle turning left is high, as the road sign indicates a left turn. The ego vehicle is likely to follow the direction of the arrow on the road sign.
- **Turn Right**: The probability of the ego vehicle turning right is also high, as the road sign indicates a right turn. The ego vehicle is likely to follow the direction of the arrow on the road sign.
- **Go Straight**: The probability of the ego vehicle going straight is low, as there are no visible arrows or signs indicating a straight path. The ego vehicle is likely to proceed straight through the intersection.
In summary, the ego vehicle should take the action of turning left or right based on the road signs and the current direction it is facing.
Fine-tuned 25kours
The action is to keep going at the same speed. The reason is that there is no safety issue, and the probability is high.
Fine-tuned 300kours
The action is to keep going at the same speed. The reason is that there is no safety issue. The probability is high.
OpenGVLab Mini-DAexternal reference
The action is to keep going at the same speed. The reason is that there is no safety issue. The probability is high.
LLaVA-1.6 7Bzero-shot
The image you've provided appears to be a sequence of images showing a vehicle's perspective as it moves through an intersection. The sequence is labeled with different camera angles:
1. Front Left
2. Back Left
3. Front Right
4. Back Right
5. Front Left
6. Back Left
Based on the label <c1,CAM_BACK,1182.5,488.3>, it seems to be referring to the first camera angle (Front Left) and the specific frame number (1182.5) within that angle.
The action that the ego vehicle could take based on this label is to proceed straight through the intersection, as indicated by the "Front Left" camera angle. This is a common action for vehicles at an intersection, as it allows them to continue on their intended path.
The probability of this action being the correct one depends on several factors:
1. Traffic rules: If the traffic rules at the intersection allow for straight-through movement, then the probability is high that this is the correct action.
2. Road markings: If the road markings indicate that vehicles can proceed straight through the intersection, then the probability is high that this is the correct action.
3. Traffic signals: If there are no traffic signals or if the traffic signals indicate that vehicles can proceed straight through the intersection, then the probability is high that this is the correct action.
4. Road signs: If there are no road signs indicating a need to turn or stop, then the probability is high that this is the correct action.
5. Vehicle position: If the vehicle is in a position where it is safe and legal to proceed straight through the intersection, then the probability is high that this is the correct action.
6. Other road users: If there are no other road users (such as pedestrians, cyclists, or vehicles) that would be affected by the vehicle proceeding straight through the intersection, then the probability is high that this is the correct action.
If all these factors align, the probability of the correct action being to proceed straight through the intersection is high. However, if any of these factors indicate that a different action is required, then the probability of the correct action being to proceed straight through the intersection is low.
LLaMA-Adapter v2 7Bzero-shot
Based on the given coordinates, the ego vehicle is at the intersection of a street and a sidewalk. The vehicle could take several actions, such as turning right, turning left, or continuing straight. The probability of the vehicle taking each action depends on the specific traffic rules and regulations in the area. For example, if the vehicle is in a city with right-hand traffic, it is more likely to turn right or continue straight. However, if the vehicle is in a city with left-hand traffic, it is more likely to turn left. The exact probability of each action would depend on the specific traffic rules and regulations in the area.
Q6planning
What actions taken by the ego vehicle can lead to a collision with <c2,CAM_FRONT,311.7,504.2>?
Ground truth
Slight left turn.
Best modelours
Accelerating and going straight.
Oracleours · upper bound
Moderate left turn.
InternVL2-2B (no fine-tuning)zero-shot
The ego vehicle, which is the car in the image, has several actions that could potentially lead to a collision with the vehicle in the center of the image, labeled as `<c2,CAM_FRONT,311.7,504.2>`.
1. **Speeding:** The ego vehicle is currently traveling at a moderate speed. If the ego vehicle were to exceed this speed, it could potentially collide with the vehicle in the center due to the lack of sufficient distance between them.
2. **Lane Change:** The ego vehicle is currently in the left lane. If the ego vehicle were to change lanes to the right, it could potentially collide with the vehicle in the center if it were to enter the same lane.
3. **Speeding Too Fast:** The ego vehicle is currently traveling at a moderate speed. If the ego vehicle were to exceed this speed, it could potentially collide with the vehicle in the center due to the lack of sufficient distance between them.
4. **Driving Carelessly:** The ego vehicle is currently driving in a straight path. If the ego vehicle were to suddenly make a turn or change lanes recklessly, it could potentially collide with the vehicle in the center.
5. **Speeding Too Fast:** The ego vehicle is currently traveling at a moderate speed. If the ego vehicle were to exceed this speed, it could potentially collide with the vehicle in the center due to the lack of sufficient distance between them.
6. **Driving Carelessly:** The ego vehicle is currently driving in a straight path. If the ego vehicle were to suddenly make a turn or change lanes recklessly, it could potentially collide with the vehicle in the center.
7. **Speeding Too Fast:** The ego vehicle is currently traveling at a moderate speed. If the ego vehicle were to exceed this speed, it could potentially collide with the vehicle in the center due to the lack of sufficient distance between them.
8. **Driving Carelessly:** The ego vehicle is currently driving in a straight path. If the ego vehicle were to suddenly make a turn or change lanes recklessly, it could potentially collide with the vehicle in the center.
9. **Speeding Too Fast:** The ego vehicle is currently traveling at a moderate speed. If the ego vehicle were to exceed this speed, it could potentially collide with the vehicle in the center due to the lack of sufficient distance between them.
10. **Driving Carelessly:** The ego vehicle is currently driving
Fine-tuned 25kours
Accelerating and going straight.
Fine-tuned 300kours
Accelerating and going straight.
OpenGVLab Mini-DAexternal reference
Accelerating and going straight.
LLaVA-1.6 7Bzero-shot
The image you've provided shows a sequence of actions that can lead to a collision with the vehicle in the front lane, as indicated by the "CAM_FRONT" label. Here's a breakdown of the actions shown in the sequence:
1. **CAM_BACK_LEFT**: The ego vehicle is in the left lane, and it appears to be changing lanes to the right.
2. **CAM_BACK_RIGHT**: The ego vehicle is now in the right lane, and it seems to be maintaining its position.
3. **CAM_FRONT_LEFT**: The ego vehicle is now in the left lane, and it appears to be maintaining its position.
4. **CAM_FRONT_RIGHT**: The ego vehicle is now in the right lane, and it seems to be maintaining its position.
5. **CAM_FRONT_LEFT**: The ego vehicle is now in the left lane, and it appears to be maintaining its position.
6. **CAM_FRONT_RIGHT**: The ego vehicle is now in the right lane, and it seems to be maintaining its position.
7. **CAM_FRONT_LEFT**: The ego vehicle is now in the left lane, and it appears to be maintaining its position.
8. **CAM_FRONT_RIGHT**: The ego vehicle is now in the right lane, and it seems to be maintaining its position.
9. **CAM_FRONT_LEFT**: The ego vehicle is now in the left lane, and it appears to be maintaining its position.
10. **CAM_FRONT_RIGHT**: The ego vehicle is now in the right lane, and it seems to be maintaining its position.
11. **CAM_FRONT_LEFT**: The ego vehicle is now in the left lane, and it appears to be maintaining its position.
12. **CAM_FRONT_RIGHT**: The ego vehicle is now in the right lane, and it seems to be maintaining its position.
13. **CAM_FRONT_LEFT**: The ego vehicle is now in the left lane, and it appears to be maintaining its position.
14. **CAM_FRONT_RIGHT**: The ego vehicle is now in the right lane, and it seems to be maintaining its position.
15. **CAM_FRONT_LEFT**: The ego vehicle is now in the left lane,
LLaMA-Adapter v2 7Bzero-shot
The ego vehicle can collide with <c2,CAM_FRONT,311.7,504.2> if it is moving in the wrong direction, such as turning left when it should be turning right, or if it is not paying attention to the traffic light and continues to move forward when the light is red. Additionally, if the ego vehicle is too close to the other vehicle, it may not have enough time or space to react and avoid a collision.
Q7planning
In this scenario, what are safe actions to take for the ego vehicle?
Ground truth
Turn right, brake suddenly.
Best modelours
Keep going at the same speed, decelerate gradually without braking.
Oracleours · upper bound
Keep going at the same speed, decelerate gradually without braking.
InternVL2-2B (no fine-tuning)zero-shot
Based on the image, the safe actions for the ego vehicle would include maintaining a safe following distance from the moving truck, ensuring that the truck is within the lane and not crossing the lane lines, and being aware of the truck's speed and direction. The vehicle should also be prepared to stop if necessary to avoid a collision. Additionally, the driver should be vigilant for any pedestrians or other vehicles that may be approaching the intersection.
Fine-tuned 25kours
Keep going at the same speed, accelerate, and proceed.
Fine-tuned 300kours
Keep going at the same speed, decelerate gradually without braking.
OpenGVLab Mini-DAexternal reference
Keep going at the same speed, decelerate gradually without braking.
LLaVA-1.6 7Bzero-shot
In the given scenario, the ego vehicle is a truck driving on a city street. Here are some safe actions the driver should take:
1. **Follow Traffic Rules**: The driver should obey all traffic signs and signals, including speed limits, stop signs, and traffic lights.
2. **Maintain a Safe Distance**: The driver should keep a safe distance from other vehicles, especially the car in front, to allow for adequate reaction time in case of sudden stops or changes in traffic flow.
3. **Be Aware of Surroundings**: The driver should be aware of pedestrians, cyclists, and other vehicles in the vicinity. This includes checking mirrors regularly and using turn signals when changing lanes or making turns.
4. **Use Indicators**: When turning or changing lanes, the driver should use the appropriate turn signals to communicate their intentions to other road users.
5. **Stay in the Lane**: The driver should stay within the marked lanes and avoid drifting into other lanes or onto the shoulder.
6. **Watch for Road Markings**: The driver should pay attention to road markings, such as lane dividers, to ensure they are in the correct lane.
7. **Slow Down in Construction Zones**: If the truck is in a construction zone, the driver should reduce speed as required by the posted speed limits.
8. **Be Prepared for Unexpected Events**: The driver should be prepared for unexpected events, such as sudden stops or changes in traffic flow, by maintaining a safe speed and being ready to react.
9. **Use Caution at Intersections**: The driver should be cautious when approaching intersections, especially if there are pedestrians or cyclists present.
10. **Regular Vehicle Maintenance**: The driver should ensure that the truck is well-maintained, including checking the brakes, tires, and lights regularly.
By following these safety guidelines, the driver can help ensure a safe journey for themselves and others on the road.
LLaMA-Adapter v2 7Bzero-shot
In the image, there are multiple vehicles on the road, including a truck and a car. The ego vehicle should take the following safe actions to ensure a smooth and secure journey:1. Maintain a safe distance from other vehicles, especially the truck, to avoid collisions and give enough space for the truck to maneuver.2. Be cautious when changing lanes or making turns, as there are other vehicles on the road that may not be visible or may be in the blind spot.3. Pay attention to traffic signals and signs, such as the stop sign in the image, and follow the rules to avoid accidents and ensure the safety of all road users.4. Keep an eye on the truck's movements, as it may have limited visibility or longer stopping distances due to its size.5. Be prepared for sudden stops or changes in traffic flow, as the truck may need more time to react to traffic conditions.By following these safe actions, the ego vehicle can navigate the road safely and responsibly, reducing the risk of accidents and ensuring a smooth journey for all road users.
Q8behaviormultiple choice
Predict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the right. The ego vehicle is driving slowly. B. The ego vehicle is steering to the left. The ego vehicle is driving very fast. C. The ego vehicle is slightly steering to the right. The ego vehicle is driving with normal speed. D. The ego vehicle is slightly steering to the left. The ego vehicle is driving fast.
Ground truth
C
Best modelours✓ correct
C
Oracleours · upper bound✓ correct
C
InternVL2-2B (no fine-tuning)zero-shotno direct answer
Based on the image, the ego vehicle is driving slowly. This can be inferred from the positioning of the vehicle in the center of the image, with no visible signs of acceleration or sudden steering changes. The surrounding environment, including the presence of other vehicles and the overall calm atmosphere, also supports the conclusion that the ego vehicle is driving at a moderate speed.
Fine-tuned 25kours✓ correct
C
Fine-tuned 300koursno direct answer
C. The ego vehicle is slightly steering to the right. The ego vehicle is driving with normal speed.
OpenGVLab Mini-DAexternal reference✓ correct
C
LLaVA-1.6 7Bzero-shotno direct answer
A. The ego vehicle is slightly steering to the right. The ego vehicle is driving slowly.
LLaMA-Adapter v2 7Bzero-shotno direct answer
The ego vehicle is driving with normal speed, as it is steering to the right and not driving very fast.