Drone doesn't implment Gitlab OAuth2 token refresh and causes 404 and code change webhook fails

Tried latest drone 2.12.1 and its fixed oauth2 token refresh issue.
but got new issue when doing multi pipeline with clone step on every pipeline.

is your issue related to multi pipeline?
becase when its multi pipeline with only one clone step it work.

in my case, its not make the job failure. in drone ui, it will stuck with forever loading on next pipeline.

log from drone:

WARN[7691] manager: cannot generate netrc.  error="The provided authorization grant is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client."

Everytime a pipeline starts, the go-scm does a force refresh of the OAuth token which cause the existing token expried immedentily even the expired one was refresh half a second ago (multi pipeline). The token will then be distributed to agent for code cloning and will not be updated any more.

Please use the Personal Access token for code clone.

@phiexz Please check my lastest reply for the issue of multiple pipeline stucking.

Thanks @gluxhappy , using DRONE_GIT_USERNAME fix multiple pipeline issue :+1:

Thanks @gluxhappy

Yeah, you are right . I faced the same issue with @phiexz

However, I’ve used the drone_git_username and drone_git_password, I can’t enable a new drone repo anymore.

How could we solve this issue?

Thanks.

You need to ensure the drone_git_username and associated personal access token you provided has the permission for code cloning.
Without setting drone_git_username, the drone uses the OAuth token for code cloning. Once you provided this setting, although the drone still using the OAuth token for API interaction(i.e. list projets), the code cloning utilizes the drone_git_username and the token only.

The problems folks are facing is due to a problem with the GitLab refresh token implementation. They actually tried to make the same change back in 2015 and it caused the same issues, and they ended up rolling it back. See my comment in this thread that describes why this is problematic:
https://github.com/harness/drone/issues/855

This GitLab change has impacted many products, including Hashicorp products, meaning this problem is not isolated to Drone. There is already a P1 issue in the GitLab issue tracker that was created a few weeks back. We recommend folks (politely) vote for this issue to be resolved:

4 Likes

Thank you for the insight. It’s quite frustrating that Gitlab has apparently learned nothing from this change failing in 2015.

It appears from the linked issue that they are moving the milestone to their next release and are generally prepared to drown this in the usual bureaucracy. In the meantime, until this is fixed (if it ever is - and even then, it could easily take them months or years, given their lack of interest in listening to people’s feedback on this breaking change from the start), what workarounds can you recommend for Drone users? There’s still no real consensus in this thread on a 100% solution. At our company, we’ve updated to the version with the aforementioned PR and are using DRONE_GIT_USERNAME and DRONE_GIT_PASSWORD, but once every hundred builds or so, there’s still an auth issue. It’s definitely not as workflow-breaking as before those two workarounds, but still frustrating for developers. Are there any workarounds known to you that we could apply, but were not mentioned in this thread yet? Thank you in advance!

We have been assured the gitlab team will resolve the issue. It is considered a P1 / S1 issue and has been assigned an owner, so hopefully they will make fast progress. I sincerely wish there was a workaround I could propose, however, I am not aware of any workaround to this problem.

I tried the latest Gitlab 15.2 and Drone’s latest version but still I get the issue for “msg”: “cannot parse webhook: Invalid webhook signature”.

The error msg you faced seems not the same?

So is it okay to upgrade to 15.2?

This is unrelated to oauth tokens. This has a simple explanation, and we have helped folks resolve this issue in the past.

The webhook signature is verified using a per-repository secret token stored in the database. The typical root cause of this error is deleting an existing Drone installation and re-installing from scratch, deleting and re-creating a repository, or manually tempering with the webhook in the GitHub user interface. You can resolve this problem by re-activating your repository in the user interface. You may also need to delete any old webhooks in GitHub.

You can find more threads on this topic here:
https://community.harness.io/search?q=Invalid%20webhook%20signature

Please note that if you need help with this issue, we would kindly ask that you start a new thread so that we can keep this discussion focused on GitLab oauth expiration.

@brad Hello Brad.

I’ve upgraded to the latest gitlab 15.2.2.

It looks like the Oauth2 token refresh didn’t fix. I need to login again to renew the token and prevent 404 issue several hours a time.

Could you please check the issue has been fixed yet?

Thanks.

There is a significant difference in how gitlab works with oauth2. The changes they have made so far have not brought its feature in line with how other git providers work. We have raised a ticket to detail the differences in a ticket here oauth2 refreshing of tokens, default behaviour (#372358) · Issues · GitLab.org / GitLab · GitLab

Do you know if there is any progress with this as I looked at Disable OAuth access token reuse feature (#363525) · Issues · GitLab.org / GitLab · GitLab and it seems to be closed and oauth2 refreshing of tokens, default behaviour (#372358) · Issues · GitLab.org / GitLab · GitLab has no assignees. To confirm without DRONE_GIT_USERNAME and DRONE_GIT_PASSWORD defined multiple pipelines fail?

Hi @Nilan_Morjaria-Patel welcome to the Harness community :tada: Right now it is currently blocked.