Kache.ai - Annotation Guidelines

The goal of this post is to provide instructions for the new machine labeling policy. These guidelines also serve as a getting started guide to introduce the Scalabel Platform, created by the Berkeley Deep Drive Team.

Scalabel Labeling Web Application

For object detection, Kache uses Scalabel, which provides us a variety of advantages over the legacy systems (VGG, etc). For example, scalabel allow us to tag images with custom attributes. In addition to labeling categories (e.g., cars), we can advantageously encode features from the environment such as the occlusion and truncation parameters. In addition, Scalabel provides a flexible and robust data format that supports labeling pipelines particularly suited for autonomous driving. We will be utilizing Scalabel as the premiere tool for providing human annotations moving forward.

---

Getting Started - Navigating the Web App

To create a project, navigate to the Scalabel Application and simply upload the relevant annotations file for correction. As an example, I preloaded the BDD100k dataset for visualization:

---

BDD100K - PT1	BDD100K - PT2	BDD100K - PT3	BDD100K - PT4
PT1	PT2	PT3	PT4

---

Getting Started - Creating a New Project & Uploading an Annotations File

• Once you see the screen shown below, click Create A New Project
• Next, choose a project name (No spaces allowed). For bounding box corrections, click on Image and 2D Bounding Box from the dropdown menus. Depending on the project, the next uploads will vary depending on the task, but the scalabel examples provide the categories.yml and bbox_attributes.yml configuration files.
• Upload your annotations file into the Item list (see below).
• Select a Task Size this size determines the number of chunks you want to break up your annotations file. For example, if you have an item list of 10,000 images and your task size is 10, the you will have generated 10 task lists; each with 1000 items. The 10 task lists will be assigned to the project name of your choosing.
• The vendor id will be 0 for now, in the future we will use this field to represent different users of a company (for QA purposes).

| Project Creation - Scalabel |

Homepage	Create Project	Upload Configs	Upload Annotations

Generally, the annotations file will be provided by internal scripts or databse which selects images intended for training nets. Upload the given document into the item list.

---

And that's it. Clicking Enter will generate a new task list. Clicking on Go To Project Dashboard on the next page will give you the option to click one the tasks generated and begin annotating. Also from the dashboard, you will have the option to export your results into bdd format for training

---

---

Style Guide:

In this section, we will explore the BDD distibution and define our policy for labeling.

---

Style Guides: Good Techniques

We would like to continue to maintain BDD's granularity and attribute associations in all new images. this will allow us to easily blend our interal data with the BDD distribution. In particular, we will maintain using occlusion and truncation for all pertaining objects. For our purposes, the respective attributes are defined as follows:

• Occlusion: When one object is hidden by another object that passes between it and the observer. The term refers to any situation in which an object in the foreground blocks from view (occults) an object in the background. In our sense, occlusion applies to the visual scene observed from computer-generated imagery when foreground objects obscure distant objects dynamically, as the scene changes over time.

Occlusion Examples	Occlusion Examples

• Truncation: The bounding box of the object specified does not correspond to the full extent of the object e.g. an image of a person from the waist up, or a view of a car extending outside the outside the field of view of the camera/image, such that it is partially shown. It is assumed with truncated images that a subsequent perspective shift will then clarify the bounds in the view of the truncated object.

Truncation Examples	Truncation Examples

---

Style Guides: Mis-Labels / Mis-Classifications / Class-Ambiguities

Overall the BDD dataset is remarkably accurate and the class ambiguities are nuanced, however there are some consistent miscategorizations that we would like fixed in our final production dataset. Among those subtleties, a few features we should aim to correct are as follows:

---

Semi-Truck Attribute Labels

Our definition of a truck is somewhat different from the definition in the BDD dataset. To avoid this confusion, we have set up the Semi attribute to assign to truck labels of vehicles which require a class A license or similar to drive. Examples of trucks which do not require a Semi tag are pickups, tow trucks, RVs, trailers, etc.

Another proper Semi truck type are the Towing Wrecker Vehicles. The example above is a tow truck but should be labeled as a truck without semi for the reasons explained below in the class taxonomy.

Similar to the trucks class, our definition of a bus is more constrained. We will define a bus as only those vehicles which are large enough to have the typical bus-styled swinging door openening mechanism which attached to a turn crank. An example of a truck mis-label is shown above. In this case, we want to classify this vehicle as a bus because it likely has a door that need to swing out to open. As it is a shuttle, it is also likely to make frequent stops.

Car Mis-Labels

---

Example 1	Example 2	Example 3	Example 4

---

With the exceptions to the above, our definition of cars is identical to BDD.. One small difference to note is that we will define the car bounding box to include the car itself, excluding protruding items like the car atenna mounted above the car.

In the other examples, the car classes are either a) mis-labeled altogether, or b) labeled, but missing the proper occlusion or truncation attrubutes. We must clean these instances up.

Mis-Labeled Examples

---

Mis-Labeled Examples

---

Class Taxonomy

Another great feature of Scalabel is that you can easily change the classification template to whatever task suits your need. We intend to leverage these capabilities by customizing our class naming scheme. This guide will serve as the official instructions on which classes to label unless there are special circumstances. As of now, those classes are:

• person
• rider (person atop a motorized vehicle or bicycle)
• car
• truck(special regard to Semi truck attribute)
• bus
• train
• motor
• bike
• traffic sign
• traffic light
• trailer
• construct-cone
• construct-sign
• construct-barrel
• construct-pole
• construct-equipment
• traffic light-red
• traffic light-amber
• traffic light-green
• traffic sign-stop_sign
• traffic sign-slow_sign
• traffic sign-speed_sign

And more to come.

---

Class Taxonomy: Car

This category includes vans, lightweight utility vehicles, SUVs, sedans, hatchbacks, classic cars, sports cars, exotic cars, etc. This category also includes special purpose compact vehicles. i.e., Mini Shuttles, Meter Maids

• Note: Any type of pickup truck will be excluded from this category and labeled as trucks.

Class Taxonomy: Van (Car Sub-Category)

Van Ex. 1	Van Ex. 2

Real-World Van Examples	Real-World Van Examples

---

Class Taxonomy: Mini-Shuttle (Car Sub-Category)

Mini-Shuttle Ex. 1	Mini-Shuttle Ex. 2	Mini-Shuttle Ex. 3	Mini-Shuttle Ex. 4

Real-World Mini-Shuttle Examples	Real-World Mini-Shuttle Examples

---

Class Taxonomy: Truck

This category includes pickup trucks (light to heavy duty), trucks/straight trucks (chassis cab trucks), mail delivery trucks, recreational vehicles (RVs), Motorized Campers, and Semi-Tractors.

---

Class Taxonomy: Pickup Truck (Truck Sub-Category)

Pickup Truck Ex.1	Pickup Truck Ex.2

Real-World Pickup Truck Examples	Real-World Pickup Truck Examples

Class Taxonomy: Postal/Delivery-Truck (Truck Sub-Category)

Postal/Delivery-Truck Ex.1	Postal/Delivery-Truck Ex.2	Postal/Delivery-Truck Ex.3	Postal/Delivery-Truck Ex.4

Real-World Postal/Delivery-Truck Examples	Real-World Postal/Delivery-Truck Examples

---

Class Taxonomy: Recreational Vehicle a.k.a. "RVs" (Truck Sub-Category)

RV Ex.1	RV Ex.2

Real-World RV Examples	Real-World RV Examples

---

Class Taxonomy: Box-Truck (Truck Sub-Category)

This sub-category includes Box Trucks. This category is tricky and commonly mistaken with the Semi category. However, there are several distinctions. For example, the Chassis Cab Box Truck usually has a wider profile, and mostly have roll-up doors. If in doubt just label it as a truck and add the Semi attribute unless you are absolutely sure.

Chassis-Cab Box Truck Explanation

Box-Truck Ex.1	Box-Truck Ex.2	Box-Truck Ex.3	Box-Truck Ex.4

Real-World Box-Truck Examples	Real-World Box-Truck Examples

---

Class Taxonomy: Chassis Cab-Truck (Truck Sub-Category)

This sub-category includes Chassis Cab trucks that do not belong to the Box Truck subcategory.

• Note: Chassis Cab trucks have some trailer or machinery permanently attached to the chassis of the truck. Common examples include garbage trucks, concrete trucks, and box trucks

Chassis Cab-Truck Ex.1	Chassis Cab-Truck Ex.2	Chassis Cab-Truck Ex.3	Chassis Cab-Truck Ex.4

Real-World Chassis Cab-Truck Examples	Real-World Chassis Cab-Truck Examples

---

Class Taxonomy: Semi-Truck (CRUCIAL Truck Sub-Category)

Not to be confused with Chasis Cab Trucks (See above), this sub-category corresponds to what are known as Semi-Trucks. A Semi Truck has a fifth wheel coupling to attach various types of semi trailers. For our labeling purposes, these trucks should be labeled as trucks with the semi attribute whether or not the corresponding semi-trailer is attached to it. Of note, semi trucks are mechanically different from the earlier mentioned chassis cab trucks in that:

1. The trailer component is detachable from the chassis of the car.
2. The trailer component is articulated Many types of trailers (https://en.wikipedia.org/wiki/Semi-trailer)

• Note: These examples should be labeled as truck with the additional Semi attribute to distinguish from all other trucks in the category.

Semi Definition

---

Semi-Truck Examples	Semi-Truck Examples	Semi-Truck Examples	Semi-Truck Examples

Real-World Semi-Truck Examples	Real-World Semi-Truck Examples

---

Class Taxonomy: Bus

This category includes tour buses, shuttle buses, school buses, articulated buses, trolleys, and more. Particular shuttle buses are easy to confuse with vans and some trucks. Buses are utilized for 9+ persons, public transport or long distance transport.

---

Class Taxonomy: Trolley (Bus Sub-Category)

Trolley Ex.1	Trolley Ex.2	Trolley Ex.3	Trolley Ex.4

Real-World Trolley Examples	Real-World Trolley Examples

---

Class Taxonomy: Shuttle Bus (Bus Sub-Category)

Shuttle Bus Ex.1	Shuttle Bus Ex.2	Shuttle Bus Ex.3	Shuttle Bus Ex.4

Real-World Shuttle Bus Examples	Real-World Shuttle Bus Examples

---

Class Taxonomy: Tour Bus (Bus Sub-Category)

Tour Bus Ex.1	Tour Bus Ex.2	Tour Bus Ex.3	Tour Bus Ex.4

Real-World Tour Bus Examples	Real-world Tour Bus Examples

---

Class Taxonomy: Public-Transit Bus (Bus Sub-Category)

Transit Bus Ex.1	Transit Bus Ex.2	Transit Bus Ex.3	Transit Bus Ex.4

Real-World Transit Bus Examples	Real-World Transit Bus Examples

---

Class Taxonomy: Long-Distance Bus (Bus Sub-Category)

Long-Distance Bus Ex.1	Long-Distance Bus Ex.2	Long-Distance Bus Ex.3	Long-Distance Bus Ex.4

Real-World Long-Distance Bus Examples	Real-World Long-Distance Bus Examples

---

Class Taxonomy: Traffic/Control Signs

Sign installed from the state/city authority, usually for information of the driver/cyclist/pedestrian in an everyday traffic scene, e.g. traffic- signs, direction signs - without their poles. No ads/commercial signs. The front side and back side of a sign containing the information. Note that commercial signs attached to buildings become building, attached to poles or standing on their own become billboard

Class Taxonomy: Construction Sign(s) (construct-sign Category)

This category includes roadway signs in the United States, which have increasingly used symbols rather than words to convey their message. Symbols provide instant communication with roadway users, overcome language barriers, and are becoming standard for traffic control devices throughout the world.

• Note: The color of construction signs is an important indicator of the information they contain. The use of red on signs is limited to stop, yield, and prohibition signs.A yellow sign conveys a general warning message; green shows permitted traffic movements or directional guidance; fluorescent yellow/green indicates pedestrian crossings and school zones; orange is used for warning and guidance in roadway work zones

Construction Sign Ex.1	Construction Sign Ex.2	Construction Sign Ex.3	Construction Sign Ex.4

Real-World Construction Sign Examples	Real-World Construction Sign Examples

---

Class Taxonomy: Slow Sign (traffic sign-slow_sign Category)

Slow Sign Ex.1	Slow Sign Ex.2	Slow Sign Ex.3	Slow Sign Ex.4

---

Class Taxonomy: Speed Sign (traffic sign-speed_sign Category)

Speed Sign Ex.1	Speed Sign Ex.2	Speed Sign Ex.3	Speed Sign Ex.4

Real-World Speed Sign Examples	Real-World Speed Sign Examples

---

Class Taxonomy: Stop Sign (traffic sign-stop_sign Category)

Stop Sign Ex.1	Stop Sign Ex.2	Stop Sign Ex.3	Stop Sign Ex.4

Real-World Stop Sign Examples	Real-World Stop Sign Examples

Class Taxonomy: Trailer

This category includes any non-motorized vehicle generally intended to be pulled by a motorized vehicle used primarily for hauling. Note: This category thoughtfully and purpousfully excludes semi-trailers that are attached to Semi-Tractors.

• Examples: camper trailer, cargo trailer, utility trailer, semi trailer (only when fully detached from Semi-tractor)

---

Trailer Ex.1	Trailer Ex.2

---

Class Taxonomy: Construction Equipment

This category includes any machinery that is used in contruction zones and sites.

• Note: Construction Equipment is usually painted yellow or orange according to federal law. In addition, this machinery, if motorized, is not allowed to drive outside the construction zone.

---

Construction Sign Ex.1	Construction Sign Ex.2	Construction Sign Ex.3	Construction Sign Ex.4

Real-World Construction Sign Examples	Real-World Construction Sign Examples

---

Class Taxonomy: Rider

This category includes non-pedstrians detected on the road. Our definition of rider is a human that would use some device to move a distance of at least 1 meter. Includes, riders/drivers of bicycles, motorbikes, scooters, skateboards, horses, roller-blades, wheel-chairs, road cleaning cars, cars without roof. Note that a visible driver of a car with roof can only be seen through the window.

• Note: Since holes are not labeled, the human is included in the car label.

---

Rider Ex.1	Rider Ex.2

Real-World Rider Examples	Real-World Rider Examples

---

Tips & Tricks

1. Objects that are smaller than 7 * 7 pixels can be ignored. The bounding box smaller than 7 * 7 will turn grey and disappear when you finish.
2. Zoom in with your browser to draw the bounding boxes more accurately.
3. Hit "h"(keyboard) to hide category label tags on the bounding boxes, and to show them after hitting "h"(keyboard) again.
4. The remove operation is irreversible.
5. If you refresh the page before submission, all previous history will not be saved.

Developer Appendix:

Here is the Json format which is supported by the training pipeline scripts currently in iPython notebooks See e.g.:

• darkernet.py (python wrapper around darknet.py)
• rails-reactor API
• bdd-formatter ros/atm modules
• data_to_coco

---

Data-Format

Our dataser is compatible with the labels generated by Scalabel. A label json file is a list of frame objects with the fields below:

- name: string
- url: string
- videoName: string (optional)
- attributes:
    - weather: "rainy|snowy|clear|overcast|undefined|partly cloudy|foggy"
    - scene: "tunnel|residential|parking lot|undefined|city street|gas stations|highway|"
    - timeofday: "daytime|night|dawn/dusk|undefined"
- intrinsics
    - focal: [x, y]
    - center: [x, y]
    - nearClip:
- extrinsics
    - location
    - rotation
- timestamp: int64 (epoch time ms)
- index: int (optional, frame index in this video)
- labels [ ]:
    - id: int32
    - category: string (classification)
    - manual: boolean (whether this label is created or modified manually)
    - attributes:
        - semi: boolean
        - occluded: boolean
        - truncated: boolean
        - trafficLightColor: "red|green|yellow|none"
        - areaType: "direct | alternative" (for driving area)
        - laneDirection: "parallel|vertical" (for lanes)
        - laneStyle: "solid | dashed" (for lanes)
        - laneTypes: (for lanes)
    - box2d:
       - x1: float
       - y1: float
       - x2: float
       - y2: float
   - box3d:
       - alpha: (observation angle if there is a 2D view)
       - orientation: (3D orientation of the bounding box, used for 3D point cloud annotation)
       - location: (3D point, x, y, z, center of the box)
       - dimension: (3D point, height, width, length)
   - poly2d: an array of objects, with the structure
       - vertices: [][]float (list of 2-tuples [x, y])
       - types: string (each character corresponds to the type of the vertex with the same index in vertices. ‘L’ for vertex and ‘C’ for control point of a bezier curve.
       - closed: boolean (closed for polygon and otherwise for path)

BDD100K Details

Road object categories:

[
    "bike",
    "bus",
    "car",
    "motor",
    "person",
    "rider",
    "traffic light",
    "traffic sign",
    "train",
    "truck"
]

They are labeld by box2d.

Drivable area category is drivable area. There are two area types areaType:

[
    "alternative",
    "direct"
]

Lane marking category is lane. There are 8 lane styles laneStyle:

[
    "crosswalk",
    "double other",
    "double white",
    "double yellow",
    "road curb",
    "single other",
    "single white",
    "single yellow"
]

Both drivable areas and lane markings are labeled by poly2d. Please check the visulization code show_labels.py for examples of drawing all the labels.

---