Engineering Workflow

This document explains the workflow for anyone working with issues in GitLab Inc. For the workflow that applies to the wider community see the contributing guide.

GitLab Flow

Products at GitLab are built using the GitLab Flow.

We have specific rules around code review.

Reverting a merge request

In line with our values of short toes, making two-way-door decisions and bias for action, anyone can propose to revert a merge request. When deciding whether an MR should be reverted, the following should be true:

• Something broke and there is no acceptable work around. Examples of this include:
  • A feature broke and is categorized as ~severity::1 or ~severity::2. See severity labels
  • CI Health incident
  • There are failing migrations
• There are no dependencies on the change. For example, a database migration has not been run on production.

Reverting merge requests that add non-functional changes and don’t remove any existing capabilities should be avoided in order to prevent designing by committee.

The intent of a revert is never to place blame on the original author. Additionally, it is helpful to inform the original author so they can participate as a DRI on any necessary follow up actions.

Set pipeline::expedited on merge requests that fix a CI Health incident to skip non-essential jobs. Also set ci-health:fix (or ci-health:fix-foss for FOSS-specific failures), which announces the fix in #ci-health-incidents.

CI Health incidents

A CI Health incident (formerly a master-broken incident) is a failure that blocks production deployments or blocks engineers from merging their merge requests.

A failure usually starts with a change merged to the default branch, but it can show up only in merge request pipelines, only on the default branch, or both. Fixing it comes before other development work. The longer an incident stays open, the bigger the impact on production deployments (for example, auto-deploys, monthly releases, and security releases) and the more merge requests it can block (every open merge request has to rebuild on top of the broken default branch through merged results pipelines).

Questions? Ask in #g_development_health or open a thread on the CI Health Incidents feedback issue.

If an incident blocks your pipeline

Check the open CI Health incidents and CI Health Systemic issues. Each one is also posted in #ci-health-incidents or #ci-health-systemic, and for test file incidents, also in the owning group’s channel when the group is known:

• If your failure is listed and someone is assigned, they’re on it. Check the incident and your group’s Slack channel to see if someone is already investigating.
• If your failure is listed and no one is assigned yet, post in the owning group’s Slack channel, then in #development to get help.
• If your failure is listed as a CI Health Systemic issue, it’s an infrastructure problem that’s already tracked.
• If your failure isn’t listed yet, an incident opens only after the same failure has blocked several pipelines, so a new failure can take a few hours to appear. If it never appears, the system may have missed it. Tell us on the feedback issue. We’re working on making failures visible before they become incidents (team issue 869).

Legacy master-broken incidents are still created for every failed master pipeline and posted to #master-broken. You don’t need to act on them.

On weekends and holidays, post in the group channel or #development. The timeline for investigation resumes on the next working day.

For failures on a maintained stable branch, see Broken Stable Branches.

If you’re assigned to an incident

The incident is attributed to the group that owns the failing job or test. If you’re in that group, you’re its DRI. If no group is attributed yet, anyone who is blocked can pick it up.

1. Assign it to yourself

Assign the incident to yourself. Check the incident discussion and your group’s Slack channel to see if someone has already started.

2. Find the cause

Start with Duo: type @duo-developer-gitlab-org Please find out which merge request caused this CI Health incident. in the incident. An automation to do this automatically is planned (team issue 620).

If Duo couldn’t find it, use the sample failed pipelines and Since Detection sections in the incident, or the Grafana link at the top.

To find the causing merge request manually:

• Look at the commit that triggered the failing job.
• Browse project activity and recent merge commits, searching for keywords from the failing spec (for example, a spec in geo/ failing: search “geo” or “shard”).
• Use the History and Blame buttons on any spec file to find recent changes.
• Check stack traces and Capybara screenshots (in artifacts/tmp/capybara).

Once you find the merge request, link it on the incident and assign the incident to the author. If you can’t reach the author, assign the maintainer who approved or merged the MR. If no one is available, find the team’s EM at product categories and ask for help in #development. If you can’t identify a merge request, ask for help in #development.

Flaky test or regression? A flaky test sometimes passes and sometimes fails without any code change: it passes when you retry the CI job or can’t be reproduced locally. A regression always fails after a specific merge request, even on retry, and you can reproduce it locally on the default branch. CI Health incidents usually come from regressions. If you think the failure is flaky, search test failure issues by spec path for an existing issue and add a comment linking this incident.

3. Fix it

The author of the causing merge request usually fixes it. If they’re unavailable, their team picks it up, or anyone who is blocked and able to fix it can self-assign.

Before you start, check whether someone has already opened a revert or a quick fix to avoid parallel efforts. If you want to open the revert early and hand it to the author, you can ping the author or their maintainer. You can also post in #backend_maintainers or #frontend_maintainers to coordinate.

The default approach is to revert:

• Open a revert merge request. Open an issue to reinstate the work and assign it to the author so the original contribution isn’t lost.
• Reverts go straight to maintainer review with 1 maintainer approval. The maintainer can request an additional review if the revert is non-trivial.
• Set pipeline::expedited on the fix MR to skip non-essential jobs. Also set ci-health:fix (or ci-health:fix-foss for FOSS-specific failures), which announces the fix in #ci-health-incidents.
• If the merge request performs a database migration, follow the Deployment blockers process before reverting. If the migration has already run in any environment, tell release managers in #releases. Discuss whether a rollback migration is needed, or disable it with a no-op migration.

If the failure looks flaky rather than a regression, quarantine the test. Use Fast Quarantine to remove it immediately if the flakiness keeps causing incidents. Create a flaky test issue if one doesn’t exist: use the Broken Master - Flaky template from the New issue button on the failing job’s page (in gitlab-org/gitlab), and add the quarantined test label.

If reverting isn’t possible or would introduce more risk, fix forward:

• Treat the fix as priority::1 severity::1.
• Keep the change minimal; refactor as a follow-up.
• Set pipeline::expedited and ci-health:fix (or ci-health:fix-foss) on the fix MR.

If only test-on-gdk jobs are failing, stop them from being added to new pipelines using the test-on-gdk runbook.

Make sure to address all failures in the pipeline. Once the most visible failure is fixed, later jobs can still fail.

4. After the fix is merged

Apply the Pick into auto-deploy label (with severity::1 and priority::1) so the fix gets picked into the next auto-deploy package. See maintained versions and the Stable branches section below for which branches need a backport.

Link the fix merge request on the incident and reply in the #ci-health-incidents Slack thread so others know a fix is on the way.

You don’t need to close the incident. It closes itself 4 hours after the last matching failure.

If you’ve reverted or quarantined but the root cause is still unclear, continue the investigation in the incident.

Merging while an incident is open

If your pipeline fails only because of an open CI Health incident, wait for the fix rather than working around it. Merge trains build on top of the default branch, so an unrelated merge can add new failures and make the incident harder to track down. Don’t bypass the failure unless the situation is urgent.

When the situation is urgent, there is an exception.

Criteria

The merge request:

• will be deployed to GitLab.com to alleviate an ongoing production incident, or
• fixes an open CI Health incident (we can have several open at once).

How to request an urgent merge

1. Make sure your latest pipeline completed less than 2 hours ago (a pipeline that completed earlier may no longer reflect the default branch).
2. Post in #frontend_maintainers or #backend_maintainers.
3. Make it clear this is a merge during a CI Health incident, explain why waiting would make the situation worse, and link your merge request and the CI Health incident (or the production incident).

Instructions for the maintainer

1. React with :eyes: on the Slack request in #frontend_maintainers or #backend_maintainers so other maintainers know it’s being assessed.

2. Assess whether the request qualifies. Ask the requester for details in the MR if you’re unsure.

3. If it doesn’t qualify, tell the requester why in the MR.

4. To qualify, all of the following must be true:

   • The latest pipeline completed less than 2 hours ago.
   • All failures in the pipeline also appear in an open CI Health incident.

5. Add a comment:

   Merging while CI Health incident <INCIDENT_URL> is open.
   
   Failure in <JOB_URL> is tracked in that incident.
   

6. If the Merge button is enabled (unlikely), click it. Otherwise:

   • Unset “Pipelines must succeed” in Settings > Merge requests for gitlab-org/gitlab.
   • Click Merge.
   • Re-enable “Pipelines must succeed” immediately.

7. If the MR is on the merge train, a warning will appear. It’s acceptable to dismiss it.

Mirror pipeline alerts

The #master-broken-mirrors channel is where Release Managers and Developer Experience teams watch alerts from the dev.gitlab.org mirror of gitlab-ee. Flaky failures there don’t need a response. Real code breaks do: follow the steps above.

Broken JiHu validation pipelines

These run in some merge requests and can break at times. See What to do when the validation pipeline failed.

Stable branches

To guarantee the readiness of any GitLab release, it is fundamental that failures on stable branches are addressed with priority, similar to master branch failures. It is the merge request author’s responsibility to backport the following to the maintained stable branches:

• Backport any master broken incident that impacts the maintained versions.
• Backport flaky failure fixes, including quarantined specs.
• Backport catch-up corrective actions that are not bugs (spec fixes, documentation changes, performance improvements, etc).

Follow the engineering runbook to backport changes to stable branches.

Security Issues

Security issues are managed and prioritized by the security team. If you are assigned to work on a security issue in a milestone, you need to follow the Security Release process.

If you find a security issue in GitLab, create a confidential issue mentioning the relevant security and engineering managers, and post about it in #security.

If you accidentally push security commits to gitlab-org/gitlab, we recommend that you:

1. Delete the relevant branch ASAP
2. Inform a release manager in #releases. It may be possible to execute a garbage collection (via the Housekeeping task in the repository settings) to remove the commits.

For more information on how the entire process works for security releases, see the documentation on security releases.

Regressions

A ~regression implies that a previously verified working functionality no longer works. Regressions are a subset of bugs. The ~regression label is used to imply that the defect caused the functionality to regress. The label tells us that something worked before and it needs extra attention from Engineering and Product Managers to schedule/reschedule.

The regression label does not apply to bugs for new features for which functionality was never verified as working. These, by definition, are not regressions.

A regression should always have the ~regression:xx.x label on it to designate when it was introduced. If it’s unclear when it was introduced, the latest released version should be added.

Regressions should be considered high priority issues that should be solved as soon as possible, especially if they have severe impact on users. When identified in time, for example in a SaaS deployment, fixing them within the same milestone avoids their being included with that release.

Use of the ~regression label on MRs

For better efficiency, it’s common for a regression to be fixed in an MR without an issue being created, either through reversion of the original MR or a code change. Regardless of whether there is an issue or not, the MR should also have the ~regression and ~regression:xx.x labels. This allows for trends to be accurately measured.

Basics

1. Start working on an issue you’re assigned to. If you’re not assigned to any issue, find the issue with the highest priority and relevant label you can work on, and assign it to yourself. You can use this query, which sorts by priority for the started milestones, and filter by the label for your team.

2. If you need to schedule something or prioritize it, apply the appropriate labels (see Scheduling issues).

3. If you are working on an issue that touches on areas outside of your expertise, be sure to mention someone in the other group(s) as soon as you start working on it. This allows others to give you early feedback, which should save you time in the long run.

4. If you are working on an issue that requires access to specific features, systems, or groups, open an access request to obtain access on staging and production for testing your changes after they are merged.

5. When you start working on an issue:

   • Add the workflow::in dev label to the issue.
   • Create a merge request (MR) by clicking on the Create merge request button in the issue. This creates a MR with the labels, milestone and title of the issue. It also relates the just created MR to the issue.
   • Assign the MR to yourself.
   • Work on the MR until it is ready, it meets GitLab’s definition of done, and the pipeline succeeds.
   • Edit the description and click on the Remove the Draft: prefix from the title button.
   • Assign it to the suggested reviewer(s) from Reviewer Roulette. If there are reviewers for multiple categories, for example: frontend, backend and database, assign all of them. Alternatively, assign someone who specifically needs to review. When assigning, also @mention them in the comments, requesting a review.
   • (Optionally) Unassign yourself from the MR. Some may find leaving the MR assigned to themselves easier to track the MRs they are responsible for by using the built in MR button/notification icon in the GitLab navigation bar.
   • Change the workflow label of the issue to workflow::in review. If multiple people are working on the issue or multiple workflow labels might apply, consider breaking the issue up. Otherwise, default to the workflow label farthest away from completion.
   • Potentially, a reviewer offers feedback and assigns back to the author.
   • The author addresses the feedback and this goes back and forth until all reviewers approve the MR.
   • After approving, the reviewer in each category unassigns themselves and assigns the suggested maintainer in their category.
   • Maintainer reviews take place with any back and forth as necessary and attempts to resolve any open threads.
   • The last maintainer to approve the MR, follows the Merging a merge request guidelines.
   • (Optionally) Change the workflow label of the issue to workflow::verification, to indicate all the development work for the issue has been done and it is waiting to be deployed and verified. We will use this label in cases where the work was requested to be verified by product OR we determined we need to perform this verification in production.
   • When the change has been verified, change the workflow label to workflow::complete and close the issue.

6. You are responsible for the issues assigned to you. This means it has to ship with the milestone it’s associated with. If you are not able to do this, you have to communicate it early to your manager and other stakeholders (e.g. the product manager, other engineers working on dependent issues). In teams, the team is responsible for this (see Working in Teams). If you are uncertain, err on the side of overcommunication. It’s always better to communicate doubts than to wait.

7. You (and your team, if applicable) are responsible for:

   • Ensuring that your changes apply cleanly to GitLab Enterprise Edition.
   • The testing of a new feature or fix, especially right after it has been merged and packaged.
   • Creating any relevant feature or API documentation
   • Shipping secure code, (see Security is everyone’s responsibility).

8. Once a release candidate has been deployed to the staging environment, please verify that your changes work as intended. We have seen issues where bugs did not appear in development but showed in production (e.g. due to CE-EE merge issues).

Be sure to read general guidelines about issues and merge requests.

Updating Workflow Labels Throughout Development

Team members use labels to track issues throughout development. This gives visibility to other developers, product managers, and designers, so that they can adjust their plans during a monthly iteration. An issue should follow these stages:

• workflow::in dev: A developer indicates they are developing an issue by applying the in dev label.
• workflow::in review: A developer indicates the issue is in code review and UX review by replacing the in dev label with the in review label.
• workflow::verification: A developer indicates that all the development work for the issue has been done and is waiting to be deployed, then verified.
• workflow::complete: A developer indicates the issue has been verified and everything is working by adding the workflow::complete label and closing the issue.

Workflow labels are described in our Development Documentation and Product Development Flow.

Working in Teams

For larger issues or issues that contain many different moving parts, you’ll be likely working in a team. This team will typically consist of a backend engineer, a frontend engineer, a Product Designer and a product manager.

1. Teams have a shared responsibility to ship the issue in the planned release.
   1. If the team suspects that they might not be able to ship something in time, the team should escalate / inform others as soon as possible. A good start is informing your manager.
   2. It’s generally preferable to ship a smaller iteration of an issue, than ship something a release later.
2. Consider starting a Slack channel for a new team, but remember to write all relevant information in the related issue(s). You don’t want to have to read up on two threads, rather than only one, and Slack channels are not open to the greater GitLab community.
3. If an issue entails frontend and backend work, consider separating the frontend and backend code into separate MRs and merge them independently under feature flags. This will ensure frontend/backend engineers can work and deliver independently.
   1. It’s important to note that even though the code is merged behind a feature flag, it should still be production ready and continue to hold our definition of done.
   2. A separate MR containing the integration, documentation (if applicable) and removal of the feature flags should be completed in parallel with the backend and frontend MRs, but should only be merged when both the frontend and backend MRs are on the master branch.

In the spirit of collaboration and efficiency, members of teams should feel free to discuss issues directly with one another while being respectful of others’ time.

Convention over Configuration

Avoid adding configuration values in the application settings or in gitlab.yml. Only add configuration if it is absolutely necessary. If you find yourself adding parameters to tune specific features, stop and consider how this can be avoided. Are the values really necessary? Could constants be used that work across the board? Could values be determined automatically? See Convention over Configuration for more discussion.

Choosing Something to Work On

Start working on things with the highest priority in the current milestone. The priority of items are defined under labels in the repository, but you are able to sort by priority.

After sorting by priority, choose something that you’re able to tackle and falls under your responsibility. That means that if you’re a frontend developer, you work on something with the label frontend.

To filter very precisely, you could filter all issues for:

• Milestone: Started
• Assignee: None (issue is unassigned)
• Label: Your label of choice. For instance CI/CD, Discussion, Quality, frontend, or Platform
• Sort by priority

Use this link to quickly set the above parameters. You’ll still need to filter by the label for your own team.

If you’re in doubt about what to work on, ask your lead. They will be able to tell you.

Triaging and Reviewing Code from the rest of the Community

It’s every developers’ responsibilities to triage and review code contributed by the rest of the community, and work with them to get it ready for production.

Merge requests from the rest of the community should be labeled with the Community contribution label.

When evaluating a merge request from the community, please ensure that a relevant PM is aware of the pending MR by mentioning them.

This should be to be part of your daily routine. For instance, every morning you could triage new merge requests from the rest of the community that are not yet labeled Community contribution and either review them or ask a relevant person to review it.

Make sure to follow our Code Review Guidelines.

Working with GitLab.com

GitLab.com is a very large instance of GitLab Enterprise Edition. It runs release candidates for new releases, and sees a lot of issues because of the amount of traffic it gets. There are several internal tools available for developers at GitLab to get data about what’s happening in the production system:

Performance Data

There is extensive monitoring publicly available for GitLab.com. For more on this and related tools, see the monitoring handbook.

Error Reporting

• Sentry is our error reporting tool
• log.gprd.gitlab.net has production logs
• prometheus.gitlab.com has alerts for the production team

Scheduling Issues

GitLab Inc has to be selective in working on particular issues. We have a limited capacity to work on new things. Therefore, we have to schedule issues carefully.

Product Managers are responsible for scheduling all issues in their respective product areas, including features, bugs, and tech debt. Product managers alone determine the prioritization, but others are encouraged to influence the PMs decisions. The UX Lead and Engineering Leads are responsible for allocating people making sure things are done on time. Product Managers are not responsible for these activities, they are not project managers.

Direction issues are the big, prioritized new features for each release. They are limited to a small number per release so that we have plenty of capacity to work on other important issues, bug fixes, etc.

If you want to schedule an issue with the Seeking community contributions label, please remove the label first.

Any scheduled issue should have a team label assigned, and at least one type label.

Requesting Something to be Scheduled

To request scheduling an issue, ask the responsible product manager

We have many more requests for great features than we have capacity to work on. There is a good chance we’ll not be able to work on something. Make sure the appropriate labels (such as customer) are applied so every issue is given the priority it deserves.

Product Development Timeline

Teams (Product, UX, Development, Quality) continually work on issues according to their respective workflows. There is no specified process whereby a particular person should be working on a set of issues in a given time period. However, there are specific deadlines that should inform team workflows and prioritization.

With the monthly release date being the third Thursday of the release month, the code cut-off is the Friday prior.

The next milestone begins the Saturday after code cut-off.

All other important dates for a milestone are relative to the release date:

• Monday, 19 days before the milestone begins:
  • Draft of the issues that will be included in the next release (released next month).
  • Start capacity and technical discussions with engineering/UX.
  • Assess error budgets to determine feature/reliability balance.
  • Development engineering manager provides product manager with prioritization input for ~type::maintenance issues per cross-functional prioritization
  • Quality provides product manager with prioritization input for ~type::bug issues per cross-functional prioritization
• Monday, 12 days before the milestone begins:
  • Product manager, taking into consideration prioritization input from development EM, Quality, and UX to create a plan of issues for the upcoming milestone.
  • Release scope is finalized. In-scope issues marked with milestone %x.y; label ~deliverable applied.
  • Kickoff document is updated with relevant items to be included.
• Monday, 5 days before the milestone begins:
  • Release scope is finalized. In-scope issues marked with milestone %x.y; label ~deliverable applied.
  • Kickoff document is updated with relevant items to be included.
• Monday immediately after the milestone begins: Kick off! 📣
  • Company Kickoff call live streamed.
  • Development on milestone begins.
• Monday, 9 days after the milestone begins:
  • The development lead for each stage/section coordinates a stage/section level review with the quad cross-functional dashboard review process. After the stages/section level reviews are complete, the VP of Development coordinates a summary review with the CTO, VP of Product, and VP of UX.
• Wednesday, 11 days after the milestone begins:
  • GitLab Bot opens Group Retrospective issue for the current milestone.
• Friday, the day the milestone ends:
  • Issues for milestone completed, with docs, and have been merged into master.
  • Feature flags should be flipped from default off to default on after verification to be in the release. See feature flags.
  • Merging by the milestone code cut-off (Friday) does not guarantee that the feature will be in the release. See release timelines.
  • Individual release post entries merged for all relevant issues.
  • By end of the day, milestone %x.y is expired.
• On or around Wednesday, the day before the release date:
  • Group Retrospective issues are updated with shipped and missed deliverables and team-members are tagged in the discussion.
• Wednesday, the day before the release date:
  • Milestone Cleanup runs on the schedule at Milestone cleanup schedule.
• Third Thursday of the release month: Release Day! 🚀
  • Release shipped to production.
  • Release post published.
• Friday immediately after release day:
  • The patch release process for the milestone starts. This includes regular and security patch releases.
  • All of the milestone’s unfinished issues and merge requests are automatically moved to the next milestone, with the exception of ~security issues.
• On or around the Wednesday immediately following the release day:
  • Product plans are updated to reflect previous and current releases, including category epics and direction pages.
• On or around the second Monday following the release day:
  • Non-critical security patches are released.

Refer to release post content reviews for additional deadlines.

Note that deployments to GitLab.com are more frequent than monthly major/minor releases. See auto deploy transition guidance for details.

Kickoff

At the beginning of each release, we have a kickoff meeting, publicly livestreamed to YouTube. In the call, the Product Development team (PMs, Product Designers, and Engineers) communicate with the rest of the organization which issues are in scope for the upcoming release. The call is structured by product area with each PM leading their part of the call.

The Product Kickoff page is updated each month, which follows the content on the livestream.

Milestone Cleanup

Engineering Managers are responsible for capacity planning and scheduling for their respective teams with guidance from their counterpart Product Managers.

To ensure hygiene across Engineering, we run scheduled pipelines to move unfinished work (open issues and merge requests) with the expired milestone to the next milestone, and label ~"missed:x.y" for the expired milestone. Additionally, label ~"missed-deliverable" whenever ~"Deliverable" is presented.

This is currently implemented as part of our automated triage operations. Additionally, issues with the ~Deliverable label which have a milestone beyond current +1, will have the ~Deliverable label removed.

We keep the milestone open for 3 months after it’s expired, based on the release and maintenance policy.

The milestone cleanup is currently applied to the following groups and projects:

• GitLab, schedule
• GitLab Runner, schedule
• GitLab Gitaly, schedule
• GitLab charts, schedule
• GitLab QA, schedule
• Omnibus GitLab, schedule (only moving milestone for now, not labelling)

Milestones closure is in the remit of the Delivery team. At any point in time a release might need to be created for an active milestone,and once that is no longer the case, the Delivery team closes the milestone.

Milestone cleanup schedule

The milestone cleanup will happen on the milestone due date.

These actions will be applied to open issues:

• Open issues and merge requests will be moved to the next milestone, and labeled with ~"missed:x.y".
• ~"missed-deliverable" will also be added whenever ~"Deliverable" is presented.

Milestones are closed when the Delivery team no longer needs to create a backport release for a specific milestone.

Use Group Labels and Group Milestones

When working in GitLab (and in particular, the GitLab.org group), use group labels and group milestones as much as you can. It is easier to plan issues and merge requests at the group level, and exposes ideas across projects more naturally. If you have a project label, you can promote it to a group milestone. This will merge all project labels with the same name into the one group label. The same is true for promoting group milestones.

Technical debt

We definitely don’t want our technical debt to grow faster than our code base. To prevent this from happening we should consider not only the impact of the technical debt but also consider the impacts spreading like a contagion. How big and how fast is this problem going to be over time? Is it likely a bad piece of code will be copy-pasted for a future feature? In the end, the amount of resources available is always less than amount of technical debt to address.

As we innovate our platform, we will have situations where a strategic decision will be made to incur technical debt in order to preserve a higher feature velocity to meet market and customer demand. This accrual of technical debt presents risks due to the long-term implications for the usability, security, reliability, scalability, accessibility, and/or availability of our product and platform.

Therefore technical debt may be accrued but must be:

1. triaged,
2. assigned a priority and severity, and
3. have a plan to remediate within the assigned SLA for all instances of severities S1 and S2, and
4. assigned a backlog label, ~"backlog::prospective" if it is expected to be worked on in the next 18 months, ~"backlog::no-commitment" otherwise.

If it is to be deferred by more than 6 months, it should be considered for the product and engineering roadmap. Technical debt issues you wish to be closed must not affect the “*abilities” and have a justification included on why it cannot be remediated within the next 18 months (meaning, there is an action plan on the Engineering roadmap).

To help with prioritization and decision-making process here, we recommend thinking about contagion as an interest rate of the technical debt. There is a great comment from the internet about it:

You wouldn’t pay off your $50k student loan before first paying off your $5k credit card and it’s because of the high interest rate. The best debt to pay off first is one that has the highest loan payment to recurring payment reduction ratio, i.e. the one that reduces your overall debt payments the most, and that is usually the loan with the highest interest rate.

Technical debt is prioritized like other technical decisions in product groups by product management.

For technical debt which might span, or fall in gaps between groups they should be brought up for a globally optimized prioritization in retrospectives or directly with the appropriate member of the Product Leadership team. Additional avenues for addressing technical debt outside of product groups are Strategic Priority Codes and working groups.

Deferred UX

Sometimes there is an intentional decision to deviate from the agreed-upon MVC, which sacrifices the user experience. When this occurs, the Product Designer creates a follow-up issue and labels it Deferred UX to address the UX gap in subsequent releases.

For the same reasons as technical debt, we don’t want Deferred UX to grow faster than our code base.

These issues are prioritized like other technical decisions in product groups by product management.

As with technical debt, Deferred UX should be brought up for globally optimized prioritization in retrospectives or directly with the appropriate member of the Product Leadership team.

UI polish

UI polish issues are visual improvements to the existing user interface, touching mainly aesthetic aspects of the UI that are guided by Pajamas foundations. UI polish issues generally capture improvements related to color, typography, iconography, and spacing. We apply the UI polish label to these issues. UI polish issues don’t introduce functionality or behavior changes to a feature.

Examples of UI polish

• Aesthetic improvements (example): removing unnecessary borders from a UI, updating the background color of an element, fixing the font size of a heading element.
• Misalignment of text, buttons, etc (example): although because many times something isn’t broken, these improvements are considered UI polish. These could also be considered a bug.
• Incorrect spacing between UI elements (example): when two interface elements are using inconsistent spacing values, such as 10px instead of 8px. It could also be considered technical debt. Note that if two interface elements have zero space between them, its an obvious bug.
• Visual inconsistencies across different product areas (example): visual inconsistencies could occur when we have a series of buttons on a particular view. For example, when 3/4 of them have been migrated to use the Pajamas component, and 1/4 of them are still using a deprecated button, resulting in a visual inconsistency. This is considered a UI polish.

What is not UI polish

• Functional inconsistency related to the experience: for example, using a manual action to add an assignee automatically shows the assignee in the sidebar but using a manual action to add a weight to an issue does not automatically show the weight in the sidebar. This is not currently considered UI polish. It would be considered a UX issue.
• Improving visibility of system status: status indicator improvements are experience improvements and are not classified as UI polish.
  • Even when updating something that is purely visual, such as a status icon, to improve the meaning the user has of what they are viewing, we are trying to improve the experience of that user.

Monitor Merge Request Trends

Open merge requests sometimes become idle (not updated by a human in more than a month). Once a month, engineering managers will receive an Merge requests requiring attention triage issue that includes all (non-WIP/Draft) MRs for their group and use it to determine if any action should be taken (such as nudging the author/reviewer/maintainer). This assists in getting merge requests merged in a reasonable amount of time which we track with the Open MR Review Time (OMRT) and Open MR Age (OMA) performance indicators in internal Tableau dashboards.

Open merge requests may also have other properties that indicate that the engineering manager should research them and potentially take action to improve efficiency. One key property is the number of threads, which, when high, may indicate a need to update the plan for the MR or that a synchronous discussion should be considered. Another property is the number of pipelines, which, when high, may indicate a need to revisit the plan for the MR. These metrics are not yet included in an automatically created a triage issue.

Security is everyone’s responsibility

Security is our top priority. Our Security Team is raising the bar on security every day to protect users’ data and make GitLab a safe place for everyone to contribute. There are many lines of code, and Security Teams need to scale. That means shifting security left in the Software Development LifeCycle (SDLC). You can find more secure development help in the #security_help Slack channel.

Being able to start the security review process earlier in the software development lifecycle means we will catch vulnerabilities earlier, and mitigate identified vulnerabilities before the code is merged. You should know when and how to proactively seek an Application Security Review. You should also be familiar with our Secure Coding Guidelines.

We are fixing the obvious security issues before every merge, and therefore, scaling the security review process. Our workflow includes a check and validation by the reviewers of every merge request, thereby enabling developers to act on identified vulnerabilities before merging. As part of that process, developers are also encouraged to reach out to the Security Team to discuss the issue at that stage, rather than later on, when mitigating vulnerabilities becomes more expensive. After all, security is everyone’s job. See also our Security Paradigm.

Rapid Engineering Response

From time to time, there are occasions that engineering team must act quickly in response to urgent issues. This section describes how the engineering team handles certain kinds of such issues.

Scope

Not everything is urgent. See below for a non-exclusive list of things that are in-scope and not in-scope. As always, use your experience and judgment, and communicate with others.

• In Scope
  • Last-minute release blocking bug or security patch before an imminent release.
  • High severity (severity::1/priority::1) security issues. Refer to security severity and priority.
  • Highest priority and severity customer issues based on the priority and severity definitions.
• Not In Scope
  • An operational issue of GitLab.com or a self managed customer environment. This falls under the on-call process.
  • Self developed and maintained tools that are not officially supported products by GitLab.
  • Feature request by a specific customer.

Process

1. Person requesting Rapid Engineering Response creates an issue supplying all known information and applies priority and severity (or security severity and priority) to the best of their ability.
2. Person requesting Rapid Engineering Response raises the issue to their own manager and the subject matter domain engineering manager (or the delegation if OOO).
   1. In case a specific group cannot be determined, raise the issue to the Director of Engineering (or the delegation if OOO) of the section.
   2. In case a specific section cannot be determined, raise the issue to the Sr. Director of Development (or the delegation if OOO).
3. The engineering sponsor (subject matter Manager, Director, and/or Sr. Director) invokes all stakeholders of the subject matter as a rapid response task force to determine the best route of resolution:
   1. Engineering manager(s)
   2. Product Management
   3. QE
   4. UX
   5. Docs
   6. Security
   7. Support
   8. Distribution engineering manager
   9. Delivery engineering manager (Release Management)
4. Adjust priority and severity or security severity and priority if necessary, and work collaboratively on the determined resolution.

Performance Refinement

A bi-weekly performance refinement session is held by the Development and QE teams jointly to raise awareness and foster wider collaboration about high-impact performance issues. A high impact issue has a direct measurable impact on GitLab.com service levels or error budgets.

Scope

The Performance Refinement issue board is reviewed in this refinement exercise.

Process

1. To participate in the bi-weekly refinement, ask your engineering director to forward the invite of the Performance Refinement meeting which is at 15:00 UTC every other Thursday. Here is the meeting agenda.
2. To nominate issues to the board:
   1. Assign a performance severity on the issue to help asses the priority assignment for the refinement session.
   2. Ensure that the issue clearly explains the problem, the (potential) impact on GitLab.com’s availability, and ideally, clearly defines a proposed solution to the problem.
   3. Use the label bug::performance.
3. For the issues under the Open column:
   1. An engineering manager will be assigned if either the Milestone or the label workflow::ready for development is missing.
   2. Engineering manager brings assigned issue(s) to the Product Manager for prioritization and planning.
   3. Engineering manager unassigns themselves once the issue is planned for an iteration, i.e. associated with a Milestone and the label workflow::ready for development.
4. To highlight high-impact issues which need additional discussion, please add an agenda item.
5. If a guest attendee would be helpful for collaboration, please forward the invite. For example, a CSM or Support Engineer may have information that would be helpful for an upcoming topic.

Infradev

The infradev process is established to identify issues requiring priority attention in support of SaaS availability and reliability. These escalations are intended to primarily be asyncronous as timely triage and attention is required. In addition to primary management through the Issues, any gaps, concerns, or critical triage is handled in the SaaS Availability weekly standup.

Scope

The SaaS Health Dashboard is the primary focus of this process for the GitLab application. infradev issues also live in the following supported projects, all of which are monitored for SLO compliance by triage-ops automation:

• Production Engineering
• Gitaly
• Runway

Additional projects with open infradev issues are visible in the SaaS Health Dashboard.

infradev issues currently also exist in projects that are not yet covered by triage-ops SLO monitoring. Any new location that will host infradev issues must have triage-ops configured to enforce SLO monitoring before issues are added.

Relationship to corrective actions

Every follow-up item from an incident receives the corrective action label automatically. The infradev label is applied in addition when the item requires urgently prioritized attention to support SaaS availability and reliability. Not every corrective action is an infradev issue, but every infradev issue created from an incident is also a corrective action. See Incident Follow Up Issues for the full definitions and the process for triaging items from an incident.

Roles and Responsibilities

Infrastructure

1. Nominate issues by adding Infradev label.
2. Assess Severity and Priority and apply the corresponding label as appropriate.
3. Provide as much information as possible to assist development engineering troubleshooting.

Development

1. Development directors are responsible for triaging Infradev issues regularly by following the triage process below.
2. Development managers are encouraged to triage issues regularly as well.
3. Development managers collaborate with their counterpart Product Managers to refine, schedule, and resolve Infradev issues.
4. Usually, issues are nominated as Infradev issues by SREs or Managers in the Infrastructure Department. Development engineers/managers are not expected to nominate Infradev issues.
   1. However, when it’s necessary to spin off new issues from an existing Infradev issue, development engineers and managers may also apply Infradev label to the new issues.
   2. When development engineers and managers split off new Infradev issues, they must have a Severity and Priority labels to the new issues. The labels should correspond to the importance of the follow-on work.

Product Management

1. Product Managers perform holistic prioritization of both product roadmap and Infradev issues as one unified backlog.
2. Product Managers collaborate with their counterpart Development Managers to refine, schedule, and resolve Infradev issues.

Triage Process

(To be completed primarily by Development Engineering Management)

Issues are nominated to the board through the inclusion of the label infradev and will appear on the infradev board.

1. Review issues in the Open column. Look for issues within your Stage/Group/Category, but also for those which lack a clear assignment or where the assignment may need correction.
2. Review the severity on the issue to validate appropriate prioritization.
3. Ensure that the issue clearly explains the problem, the (potential) impact on GitLab.com’s availability, and ideally, clearly defines a proposed solution to the problem.
4. Assign a Development Manager and a Product Manager to any issue where the Milestone or the label workflow::ready for development is missing.
   1. Development Manager and Product Manager collaborate on the assigned issue(s) for prioritization and planning.
   2. Development Manager and Product Manager unassign themselves once the issue is planned for an iteration, i.e. associated with a Milestone and the label workflow::ready for development.
5. All Issues should be prioritized into the appropriate workflow stage. It is the intent to maintain no Open (un-triaged) items.

Issues with ~infradev ~severity::1 ~priority::1 ~production request labels applied require immediate resolution.

During triage, teams may request that the infrastructure platforms team remove the infradev label from ~severity::3 and ~severity::4 issues by commenting with their reasoning and pinging the incident lead, if they determine the issue does not meet the bar for infradev prioritization. The infradev label must not be removed from ~severity::1 or ~severity::2 issues.

~infradev issues requiring a ~“breaking change” should not exist. If a current ~infradev issue requires a breaking change then it should split into two issues. The first issue should be the immediate ~infradev work that can be done under current SLOs. The second issue should be ~“breaking change” work that needs to be completed at the next major release in accordance with deprecation guidance. Agreement from development DRI as well as the infrastructure DRI should be documented on the issue.

Infradev issues are also shown in the monthly Error Budget Report.

Reporting

The state of infradev issues is shared at the Operational Excellence Meeting meeting.

Historical metrics

Historical trends and distribution of infradev issues over time are available in the Tableau Infradev Dashboard.

A Guide to Creating Effective Infradev Issues

Triage of infradev Issues is desired to occur asynchronously. These points below with endure that your infradev issues gain maximum traction.

1. Use the InfraDev issue template to create the issue on the gitlab-org/gitlab issue tracker.
2. Clearly state the scope of the problem, and how it affects GitLab SaaS Platforms. Examples could include:
   1. Reliability issues: the problem could cause a widespread outage or degradation on GitLab.com. example
   2. Saturation issues: the problem could leave to increased saturation, latency issues due to resource over-utilization. example
   3. Service-level degradation: the problem is causing our service-level monitoring to degrade, impacting the overall SLA of GitLab.com and potentially leaving to SLA violations. example
   4. Alerts not resulting in incidents: alerts that are triggered but do not result in actionable incidents for the Tier 1 EOC (SRE on call), yet require action by the team owning the service. Examples include noisy alerts, alerts that need tuning, alerts that signal a growing problem that is not yet an incident but may become one, or alerts that no longer need to exist. The team owning the service that is triggering the alert is responsible for determining the correct outcome and taking action (proactive action, tune, or remove). This reduces alerting noise and distraction for the Tier 1 EOC to only include alerts that represent production incidents while ensuring all alerts are prioritized and addressed appropriately.
   5. Problems which extend the time to diagnosis of incidents: for example, issues which degrade the observability of GitLab.com, swallow user-impacting errors or logs, etc. These could lead to incidents taking much longer to clear, and impacting availability. example
   6. Deficiencies in our public APIs which lead to customers compensating by generating substantially more traffic to get the required results. example
3. Quantify the effect of the problem to help ensure that correct prioritization occurs.
   1. Include costs to availability. The Incident Budget Explorer dashboard can help here.
   2. Include the number of times alerts have fired owing to the problem, how much time was spent dealing with the problem, and how many people were involved.
   3. Include screenshots of visualization from Grafana or Kibana.
   4. Always include a permalink to the source of the screenshot so that others can investigate further.
4. Provide a clear, unambiguous, self-contained solution to the problem. Do not add the infradev label to architectural problems, vague solutions, or requests to investigate an unknown root-cause.
5. Ensure scope is limited. Each issue should be able to be owned by a single stage group team and should not need to be broken down further. Single task solutions are best.
6. Ensure a realistic severity is applied: review the availability severity label guidelines and ensure that applied severity matches. Always ensure all issues have a severity, even if you are unsure. The severity of the infradev issue should reflect the ongoing risk if the issue goes unresolved — it is independent of the severity of the incident that generated it.
7. If possible, include ownership labels for more effective triage. The product categories can help determine the appropriate stage group to assign the issue to.
8. Cross-reference links to Production Incidents, PagerDuty Alerts, Slack Alerts and Slack Discussions. To help ensure that the team performing the triage have all the available data.
   1. By adding “Related” links on the infradev issue, the Infradev Status Report will display a count of the number of production incidents related to each infradev issue, for easier and clearer prioritization.
9. Ensure that the issue title is accurate, brief and clear. Change the title over time if you need to keep it accurate.
10. By adding an infradev label to an issue, you are assuming responsibility and becoming the sponsor/champion of the issue.
11. Provide a method for validating that the original issue still exists
    1. Sometimes infradev issues will resolve on their own, or are resolved as a side-effect of an unrelated change.
    2. In the infradev issue description, provide a clear way of checking whether the problem still exists.
    3. Having a way of checking validity can save on a great deal of back-and-forth discussion between Infradev Triage participants including Engineering Managers, Directors and Product Managers and make space for other non-resolved issues to get scheduled sooner.
    4. Ideally, provide a link to a Grafana query or an ELK query and clear instructions on how to interpret the results to determine whether the problem is still occurring. Check the “Verfication” section in this issue as an example of this.
    5. Alternatively, provide clear instructions on how to recreate or validate the problem.
    6. If an issue has been resolved, use the following process:
       1. Reassign the issue back to the author, or an appropriate owner, requesting that they confirm the resolution, and close the issue if they concur. If not, they should follow up with a note and unassign themselves.

View page source - Edit this page - please contribute.