| 0 | Questionperception | What are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision. |
| GT Answer | There is a blue sedan to the back of the ego vehicle, a black sedan to the front right of the ego vehicle, a black SUV to the back of the ego vehicle, a black SUV to the front of the ego vehicle, a white truck to the front of the ego vehicle, and a green light to the front of the ego vehicle. The IDs of these objects are <c1,CAM_BACK,791.2,537.2>, <c2,CAM_FRONT_RIGHT,1005.0,650.8>, <c3,CAM_BACK,375.8,560.8>, <c4,CAM_FRONT,715.0,498.3>, <c5,CAM_FRONT,440.8,468.1>, and <c6,CAM_FRONT,764.8,422.1>. |
| InternVL2-2B | In the current scene, the important objects include:
1. **Truck**: The large truck is the most prominent object. It is positioned in the center of the image, indicating it is likely a key vehicle on the road.
2. **Cars**: There are several cars visible, including a black sedan, a dark-colored SUV, and a white car. These cars are positioned on the road, suggesting they are either in motion or waiting to move.
3. **Buildings**: The buildings on either side of the road are also important objects. They provide context and help define the urban environment.
4. **Street and Road**: The road itself is a crucial element, as it is the primary means of transportation in this scene. The presence of lane markings and traffic lights further indicates that it is a well-organized roadway.
5. **Traffic Lights**: The traffic lights are essential for regulating traffic flow and ensuring safety. They are positioned at the intersection, indicating that the road is a busy thoroughfare.
6. **Pedestrian Crossing**: The sidewalks and crosswalks are important for pedestrian safety. They provide a designated area for people to cross the road safely.
7. **Street Signs and Signage**: The street signs and signage are crucial for navigation and traffic regulation. They help drivers and pedestrians understand the rules and directions of the road.
These objects collectively contribute to the overall functionality and safety of the urban environment depicted in the image. |
| Mini-InternVL2-2B-DA-DriveLM | There is a black car to the front right of the ego vehicle, a black SUV to the back of the ego vehicle, a black sedan to the front of the ego vehicle, a black sedan to the front of the ego vehicle, and a green light to the front of the ego vehicle. The IDs of these objects are 0, 1, 2, 3, and 4. |
| llama_adapter_v2_multimodal7b | In the current scene, the important objects to consider for future reasoning and driving decisions include the cars, traffic lights, and pedestrians. The cars are driving down the street, and there are multiple traffic lights at different positions along the road. The pedestrians are also present, indicating that the area is likely a busy urban environment. Drivers should be aware of these objects and their positions to ensure safe and smooth traffic flow, as well as to follow traffic rules and regulations. |
| InternVL2-2b-LoRA-25k-drivelm | There is a black car to the back of the ego vehicle, a black SUV to the back of the ego vehicle, a black sedan to the front of the ego vehicle, a blue sedan to the front of the ego vehicle, and a white truck to the front of the ego vehicle. The IDs of these objects are <c1,CAM_BACK,1080.0,525.8>, <c2,CAM_BACK,1080.0,525.8>, <c3,CAM_FRONT,1080.0,500.8>, <c4,CAM_FRONT,1080.0,500.8>, and <c5,CAM_FRONT,1080.0,500.8>. |
| InternVL2-2b-LoRA-25k-drivelm-offline-redcircle-ctag-bkgd | There is a black car to the front of the ego vehicle, a black car to the front right of the ego vehicle, a black sedan to the back of the ego vehicle, a black car to the front of the ego vehicle, a white truck to the front of the ego vehicle, and a green light to the front of the ego vehicle. The IDs of these objects are <c1,CAM_FRONT,775.0,498.3>, <c2,CAM_FRONT_RIGHT,1005.0,650.8>, <c3,CAM_BACK,375.8,560.8>, <c4,CAM_FRONT,840.8,496.7>, <c5,CAM_FRONT,440.8,468.1>, and <c6,CAM_FRONT,764.8,422.1>. |
| InternVL2-2b-LoRA-25k_plus_DL-PL-10pct | There is a black car to the front of the ego vehicle, a black sedan to the front of the ego vehicle, a black sedan to the front of the ego vehicle, a black sedan to the front of the ego vehicle, and a black sedan to the front of the ego vehicle. The IDs of these objects are <c1,CAM_FRONT,1010.0,525.0>, <c2,CAM_FRONT,1010.0,525.0>, <c3,CAM_FRONT,1010.0,525.0>, <c4,CAM_FRONT,1010.0,525.0>, and <c5,CAM_FRONT,1010.0,525.0>. |
| InternVL2-2b-LoRA-300k-drivelm | There is a black car to the front right of the ego vehicle, a black car to the back of the ego vehicle, a white truck to the front of the ego vehicle, and a black sedan to the front of the ego vehicle. The IDs of these objects are <c1,CAM_FRONT_RIGHT,10.8,675.8>, <c2,CAM_BACK,10.8,555.8>, <c3,CAM_FRONT,10.8,475.8>, and <c4,CAM_FRONT,10.8,495.0>. |
| llava-v1.6-mistral-7b-hf | In the current scene, the important objects to consider for future reasoning and driving decisions include:
1. Vehicles: There are several cars and a truck visible in the image. These vehicles are important for understanding traffic flow and potential obstacles.
2. Pedestrians: There are a few pedestrians visible, which indicates that there might be crosswalks or pedestrian areas that drivers need to be aware of.
3. Traffic lights: There are traffic lights at various points in the image, which are crucial for understanding when it is safe to proceed or when to stop.
4. Parking meters: There are parking meters along the street, which suggest that there might be parking restrictions or designated parking areas.
5. Buildings: The presence of buildings indicates that this is an urban area, which means drivers should be aware of potential blind spots, pedestrian crossings, and other urban driving considerations.
6. Street signs: There are street signs visible, which can provide information about speed limits, directions, or other important information.
7. Bicycles: There is a bicycle visible, which could indicate a bike lane or the presence of cyclists.
8. Trees and greenery: The presence of trees and greenery can affect visibility and may require drivers to be cautious around curves or intersections.
9. Parking spaces: There are parking spaces visible, which can be useful for drivers looking for a place to park.
10. Road markings: The yellow lines on the road indicate no-passing zones, which are important for understanding where it is safe to overtake or where you must stay in your lane.
These objects, along with the overall layout of the street, will be important for making driving decisions in the future. |
| 1 | Questionperception | What is the moving status of object <c1,CAM_BACK,791.2,537.2>? Please select the correct answer from the following options: A. Reverse parking. B. Drive backward. C. Going ahead. D. Turn right. |
| GT Answer | C |
| InternVL2-2B | The moving status of object <c1,CAM_BACK,791.2,537.2> is C. Going ahead. |
| Mini-InternVL2-2B-DA-DriveLM | C |
| llama_adapter_v2_multimodal7b | The moving status of object <c1,CAM_BACK,791.2,537.2> is "Going ahead." |
| InternVL2-2b-LoRA-25k-drivelm | C |
| InternVL2-2b-LoRA-25k-drivelm-offline-redcircle-ctag-bkgd | C |
| InternVL2-2b-LoRA-25k_plus_DL-PL-10pct | C |
| InternVL2-2b-LoRA-300k-drivelm | C. Going ahead. |
| llava-v1.6-mistral-7b-hf | C. Going ahead |
| 2 | Questionprediction | In this scenario, what object is most likely to consider <c6,CAM_FRONT,764.8,422.1>? |
| GT Answer | <c4,CAM_FRONT,715.0,498.3>. |
| InternVL2-2B | In this scenario, the object most likely to consider `<c6,CAM_FRONT,764.8,422.1>` is the black car in the bottom right corner of the image. This is because the black car is positioned in the foreground of the image, and the coordinates provided (764.8, 422.1) are within the range of the car's position in the frame. |
| Mini-InternVL2-2B-DA-DriveLM | The ego vehicle. |
| llama_adapter_v2_multimodal7b | In this scenario, the most likely object to consider is the car, as it is the primary subject of the image and is captured from various angles and perspectives. |
| InternVL2-2b-LoRA-25k-drivelm | <c3,CAM_FRONT,1080.0,505.0>. |
| InternVL2-2b-LoRA-25k-drivelm-offline-redcircle-ctag-bkgd | <c4,CAM_FRONT,717.5,498.3>. |
| InternVL2-2b-LoRA-25k_plus_DL-PL-10pct | <c3,CAM_FRONT,1080.0,510.0>. |
| InternVL2-2b-LoRA-300k-drivelm | The ego vehicle. |
| llava-v1.6-mistral-7b-hf | The object most likely to consider <c6,CAM_FRONT,764.8,422.1> in this scenario is the car in the front-right image. This is because the car is positioned at the front of the image, and the label "CAM_FRONT" suggests that the camera is facing forward from the perspective of the car. |
| 3 | Questionprediction | Is <c1,CAM_BACK,791.2,537.2> a traffic sign or a road barrier? |
| GT Answer | No. |
| InternVL2-2B | To determine if the coordinates <c1,CAM_BACK,791.2,537.2> represent a traffic sign or a road barrier, let's analyze the image in detail:
1. **Location and Orientation**:
- The coordinates are given in a format that typically represents a point on a map or a point in a 2D image.
- The coordinates are specified as (c1, CAM_BACK, 791.2, 537.2), where c1 is the x-coordinate, CAM_BACK is the y-coordinate, and 791.2 and 537.2 are the respective values.
2. **Contextual Analysis**:
- **Traffic Sign**: A traffic sign typically has a specific shape, color, or text that indicates a particular rule or regulation. In this image, there is no visible text or shape that would suggest it is a traffic sign.
- **Road Barrier**: A road barrier is usually a physical structure that separates the road from the surrounding area, such as a guardrail, barrier wall, or a sign indicating a no-entry zone. In the image, there is no visible barrier or sign that would indicate a road barrier.
3. **Image Content**:
- The image shows a street scene with cars and buildings, but there is no indication of any barriers or signs that would suggest a road barrier.
- The coordinates are within the visible area of the image, and there is no indication of any additional context that would suggest a barrier.
4. **Conclusion**:
- Based on the analysis, the coordinates <c1,CAM_BACK,791.2,537.2> do not represent a traffic sign or a road barrier. They are likely a point on the map or a specific location within the image. |
| Mini-InternVL2-2B-DA-DriveLM | No. |
| llama_adapter_v2_multimodal7b | The traffic sign in the image is a "Can Back" sign, which is placed on the road to indicate that vehicles should not proceed further back than a specific point. It is not a road barrier. |
| InternVL2-2b-LoRA-25k-drivelm | No. |
| InternVL2-2b-LoRA-25k-drivelm-offline-redcircle-ctag-bkgd | No. |
| InternVL2-2b-LoRA-25k_plus_DL-PL-10pct | No. |
| InternVL2-2b-LoRA-300k-drivelm | No. |
| llava-v1.6-mistral-7b-hf | The image you've provided appears to be a collage of different street scenes, and it's not clear from this image alone whether the object labeled as <c1,CAM_BACK,791.2,537.2> is a traffic sign or a road barrier. Traffic signs are typically mounted on poles and are designed to convey information to road users, while road barriers are physical structures that block or restrict the passage of vehicles.
Without more context or a clearer image, it's not possible to definitively determine the nature of the object in question. If you have more information or a clearer image, I could provide a more accurate answer. |
| 4 | Questionplanning | What actions could the ego vehicle take based on <c1,CAM_BACK,791.2,537.2>? Why take this action and what's the probability? |
| GT Answer | The action is to keep going at the same speed. The reason is to avoid a collision, which is high probability. |
| InternVL2-2B | The ego vehicle, indicated by the camera's perspective, is currently positioned at the intersection of a two-way street. The action it could take is to proceed straight or turn left.
### Chain of Thought (CoT) Analysis:
1. **Current Position**: The ego vehicle is at the intersection of a two-way street.
2. **Direction**: The camera is facing forward, and the ego vehicle is facing the intersection.
3. **Intersection**: The intersection is marked with a pedestrian crossing sign, indicating a potential hazard.
4. **Proceeding Straight or Turning Left**:
- **Proceeding Straight**: The ego vehicle could continue straight to avoid the pedestrian crossing.
- **Turning Left**: The ego vehicle could turn left to avoid the pedestrian crossing and continue on the same road.
### Probability:
- **Proceeding Straight**:
- **Probability**: 0.5 (50% chance of proceeding straight)
- **Reasoning**: The intersection is clearly marked, and the ego vehicle is facing it directly. The pedestrian crossing sign is visible, indicating a potential hazard. The ego vehicle should proceed straight to avoid the hazard.
- **Turning Left**:
- **Probability**: 0.5 (50% chance of turning left)
- **Reasoning**: The ego vehicle is facing the intersection, and the pedestrian crossing sign is visible. The ego vehicle should turn left to avoid the hazard and continue on the same road.
### Conclusion:
The ego vehicle should either proceed straight or turn left to avoid the pedestrian crossing at the intersection. The probability of each action is 50%, with the ego vehicle having a 50% chance of proceeding straight and a 50% chance of turning left. |
| Mini-InternVL2-2B-DA-DriveLM | The action is to keep going at the same speed. The reason is that there is no safety issue. The probability is high. |
| llama_adapter_v2_multimodal7b | Based on the given input, the ego vehicle is likely to take the action of turning right at the intersection. This action is indicated by the <c1,CAM_BACK,791.2,537.2> command, which represents the vehicle's current position and orientation. The vehicle is currently in the right lane, and the camera is facing backward, which suggests that the vehicle is approaching the intersection. The probability of the vehicle turning right at the intersection is high, as it is following the right lane and the camera is facing backward, indicating that the vehicle is approaching the intersection. |
| InternVL2-2b-LoRA-25k-drivelm | The action is to keep going at the same speed. The reason is that there is no safety issue, and the probability is high. |
| InternVL2-2b-LoRA-25k-drivelm-offline-redcircle-ctag-bkgd | The action is to keep going at the same speed. The reason is that there is no safety issue. The probability is high. |
| InternVL2-2b-LoRA-25k_plus_DL-PL-10pct | The action is to keep going at the same speed. The reason is that there is no safety issue. The probability is high. |
| InternVL2-2b-LoRA-300k-drivelm | The action is to keep going at the same speed. The reason is that there is no safety issue, and the probability is high. |
| llava-v1.6-mistral-7b-hf | The image you've provided appears to be a sequence of images showing a vehicle's perspective as it moves through an intersection. The sequence is labeled with different camera angles:
1. CAM_FRONT_LEFT
2. CAM_BACK_LEFT
3. CAM_BACK_RIGHT
4. CAM_FRONT_RIGHT
Based on the label <c1,CAM_BACK,791.2,537.2>, it seems to be referring to a specific camera angle and its position relative to the vehicle. The camera angle is CAM_BACK, which means the camera is positioned behind the vehicle. The position is 791.2 meters to the left of the vehicle, and the angle is 537.2 degrees.
Given this information, the ego vehicle could take the following actions:
1. **Slow down or stop**: If the vehicle is approaching the intersection and the traffic light is red, the driver should slow down or stop to comply with traffic regulations.
2. **Yield to pedestrians**: If there are pedestrians crossing the intersection, the driver should yield to them, ensuring their safety.
3. **Follow traffic signals**: The driver should follow the traffic signals at the intersection, which may include stopping, yielding, or proceeding depending on the signal.
4. **Be aware of other vehicles**: The driver should be aware of other vehicles in the intersection, especially those coming from the left and right, as indicated by the CAM_BACK_LEFT and CAM_BACK_RIGHT labels.
The probability of the vehicle needing to take these actions depends on the current traffic conditions at the intersection. If the traffic light is red, the probability of the vehicle needing to stop or slow down is high. If there are pedestrians crossing, the probability of the vehicle needing to yield is also high. If the traffic light is green and there are no pedestrians, the probability of the vehicle proceeding is high.
It's important to note that these actions are based on the information provided and the actual traffic conditions at the intersection could be different. The driver should always prioritize safety and follow traffic rules. |
| 5 | Questionplanning | What actions taken by the ego vehicle can lead to a collision with <c1,CAM_BACK,791.2,537.2>? |
| GT Answer | Back up. |
| InternVL2-2B | The ego vehicle, which is the car in the foreground, is driving straight down the road. The vehicle is positioned directly in the lane marked with a dashed line, which is the lane intended for vehicles turning left. The car in the background is also in the same lane, heading in the same direction.
Given the following sequence of events:
1. The ego vehicle is in the rightmost lane.
2. The car in the background is in the leftmost lane.
3. The ego vehicle is approaching the car in the background.
The ego vehicle must be cautious and ensure it does not cross the dashed line, which is the lane for vehicles turning left. The car in the background is also approaching the dashed line, and the ego vehicle must be prepared to react to any potential collision.
To avoid a collision, the ego vehicle should:
- Slow down and maintain a safe distance from the car in the background.
- Look for any potential obstacles or changes in traffic ahead.
- Adjust its speed and direction to ensure it does not cross the dashed line.
By following these actions, the ego vehicle can reduce the risk of a collision with the car in the background. |
| Mini-InternVL2-2B-DA-DriveLM | No such action will lead to a collision. |
| llama_adapter_v2_multimodal7b | The ego vehicle is driving down the street, and it is approaching a red light at an intersection. If the ego vehicle continues driving without stopping, it may collide with the <c1,CAM_BACK,791.2,537.2> object, which is likely another vehicle or a pedestrian. The driver should come to a complete stop at the red light and check for any oncoming traffic or pedestrians before proceeding. This action would help avoid a collision and ensure the safety of all road users. |
| InternVL2-2b-LoRA-25k-drivelm | Back up. |
| InternVL2-2b-LoRA-25k-drivelm-offline-redcircle-ctag-bkgd | Brake suddenly. |
| InternVL2-2b-LoRA-25k_plus_DL-PL-10pct | No such action will lead to a collision. |
| InternVL2-2b-LoRA-300k-drivelm | Back up. |
| llava-v1.6-mistral-7b-hf | The image you've provided appears to be a series of snapshots from a vehicle's camera system, showing different views of the road ahead. To avoid a collision with the vehicle labeled as <c1,CAM_BACK,791.2,537.2>, the ego vehicle should take the following actions:
1. **Maintain a Safe Following Distance**: The ego vehicle should ensure it maintains a safe distance from the vehicle in front of it, as indicated by the "CAM_BACK" label. This is important to allow enough time to react to any sudden changes in traffic conditions.
2. **Stay in the Lane**: The ego vehicle should stay within its lane, as indicated by the "CAM_BACK" label, to avoid drifting into the path of the vehicle ahead.
3. **Be Aware of Surroundings**: The ego vehicle should be aware of its surroundings, including the presence of other vehicles, pedestrians, and potential obstacles. This is particularly important when changing lanes or making turns.
4. **Use Turn Signals**: If the ego vehicle needs to change lanes or make a turn, it should use its turn signals to communicate its intentions to other road users.
5. **Slow Down When Necessary**: If the ego vehicle detects that the vehicle in front is slowing down or stopping, it should also slow down to avoid a collision.
6. **Be Prepared for Sudden Stops**: The ego vehicle should be prepared for sudden stops or changes in traffic flow. This includes keeping a firm grip on the steering wheel, maintaining a safe speed, and being ready to apply the brakes if necessary.
7. **Use the Rearview Mirrors**: The ego vehicle's driver should regularly check the rearview mirrors to monitor the position and actions of the vehicle in front.
8. **Follow Traffic Rules**: The ego vehicle should follow all traffic rules and regulations, including speed limits, traffic signals, and road signs.
By following these guidelines, the ego vehicle can significantly reduce the risk of a collision with the vehicle labeled as <c1,CAM_BACK,791.2,537.2>. |
| 6 | Questionplanning | In this scenario, what are safe actions to take for the ego vehicle? |
| GT Answer | Keep going at the same speed, decelerate gradually without braking, slightly offset to the left, and change to the left lane. |
| InternVL2-2B | Based on the image, the safe actions for the ego vehicle would include maintaining a safe following distance from the truck, ensuring that the truck is in the correct lane, and not overtaking the truck. The ego vehicle should also be prepared to stop if necessary, as the truck is in the opposite lane, potentially indicating a potential hazard. Additionally, the ego vehicle should be aware of the truck's presence and any potential for a collision, and should proceed with caution. |
| Mini-InternVL2-2B-DA-DriveLM | Keep going at the same speed, decelerate gradually without braking. |
| llama_adapter_v2_multimodal7b | In the image, there are multiple cars on the road, including a blue car and a truck. To ensure safety, the ego vehicle should follow these actions:1. Maintain a safe distance from other vehicles, especially the blue car and the truck, to avoid collisions.2. Be cautious when changing lanes or making turns, as there are other cars on the road.3. Pay attention to traffic signals and signs, such as the stop sign visible in the image, to follow the rules and avoid accidents.4. Keep an eye on the surroundings, including the blue car and the truck, to anticipate their movements and react accordingly.5. Use turn signals when changing lanes or making turns to communicate intentions to other drivers.By following these safe actions, the ego vehicle can navigate the busy street and minimize the risk of accidents. |
| InternVL2-2b-LoRA-25k-drivelm | Keep going at the same speed, accelerate, and proceed. |
| InternVL2-2b-LoRA-25k-drivelm-offline-redcircle-ctag-bkgd | Keep going at the same speed, decelerate gradually without braking. |
| InternVL2-2b-LoRA-25k_plus_DL-PL-10pct | Keep going at the same speed, decelerate gradually without braking. |
| InternVL2-2b-LoRA-300k-drivelm | Keep going at the same speed, decelerate gradually without braking. |
| llava-v1.6-mistral-7b-hf | In the given scenario, the ego vehicle is a car driving down a city street with various traffic conditions. Here are some safe actions the driver should take:
1. **Follow Traffic Rules**: The driver should obey all traffic signals and signs, including speed limits, stop signs, and traffic lights.
2. **Maintain a Safe Distance**: The driver should keep a safe distance from other vehicles, especially when following a car in front. This allows for adequate reaction time in case of sudden stops or changes in traffic flow.
3. **Be Aware of Surroundings**: The driver should be aware of pedestrians, cyclists, and other vehicles in the vicinity. This includes checking mirrors regularly and using turn signals when changing lanes or making turns.
4. **Anticipate Traffic Changes**: The driver should anticipate changes in traffic flow, such as merging lanes, pedestrian crossings, and potential obstacles.
5. **Use Turn Signals**: When turning, the driver should use the appropriate turn signal well in advance to alert other road users of their intended direction.
6. **Stay in the Lane**: The driver should stay in the lane they are in and avoid sudden lane changes that could lead to accidents.
7. **Be Prepared for Emergencies**: The driver should be prepared for emergencies by having a charged cell phone, a first aid kit, and a basic toolkit in the car.
8. **Park Safely**: When parking, the driver should park in designated parking areas, ensuring that the car is not obstructing traffic or pedestrian paths.
9. **Check Blind Spots**: Before changing lanes or making turns, the driver should check their blind spots to ensure no other vehicles are in their path.
10. **Stay Alert**: The driver should stay alert and avoid distractions such as using a mobile phone while driving.
By following these safety measures, the driver can significantly reduce the risk of accidents and ensure a safe journey. |
| 7 | Questionbehavior | Predict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the right. The ego vehicle is driving very fast. B. The ego vehicle is steering to the left. The ego vehicle is driving slowly. C. The ego vehicle is going straight. The ego vehicle is driving fast. D. The ego vehicle is going straight. The ego vehicle is not moving. |
| GT Answer | C |
| InternVL2-2B | Based on the image, the ego vehicle is driving straight. The image shows the ego vehicle in the center of the road with no visible signs of steering to the left or right, nor any indication of driving very fast or slowly. The vehicle is also not moving, as it is stationary in the center of the road. Therefore, the correct answer is D. The ego vehicle is going straight. The ego vehicle is not moving. |
| Mini-InternVL2-2B-DA-DriveLM | C |
| llama_adapter_v2_multimodal7b | The ego vehicle is driving fast, as it is seen speeding down the street. |
| InternVL2-2b-LoRA-25k-drivelm | C |
| InternVL2-2b-LoRA-25k-drivelm-offline-redcircle-ctag-bkgd | C |
| InternVL2-2b-LoRA-25k_plus_DL-PL-10pct | C |
| InternVL2-2b-LoRA-300k-drivelm | C. The ego vehicle is going straight. The ego vehicle is driving fast. |
| llava-v1.6-mistral-7b-hf | C. The ego vehicle is going straight. The ego vehicle is driving fast. |
