Pipeline missing npm packages

We face a weird issue recently, some pipelines missing node_moudles that had been installed in previous steps.
I’ve tried to add a temp folder for each step but it doesn’t work.

by the way, after restarting the pipeline this step pass with no issue. thoughts?

/drone/src is a volume that persists between steps in the same pipeline which means the node_modules folder is available to subsequent steps [1]. I created a simple test and was unable to reproduce the issue you are describing [2][3]. I can confirm there have been no changes or regressions that would prevent this from working as expected.

I have attached a screenshot below that demonstrates my successful test. I have also included links below so that you can see our sample code and configuration, and compare with your own.

The only root cause I am aware of for the issue you described would be yaml misconfiguration. We are happy to help advise, but would kindly request a sample yaml and repository that we can use to reproduce the problem (similar to the sample repository and yaml we provided at the links below).

[1] https://docs.drone.io/pipeline/docker/syntax/workspace/
[2] https://github.com/drone/hello-world/blob/dffe7eb820dc98f6682eb915dc2cd9ed09868a77/.drone.yml
[3] https://cloud.drone.io/drone/hello-world/293/1/3

It’s working for 90% of the pipelines, but some of them missing this shared folder I guess. how can I verify this folder is shared between each step?

I am not aware of any instance in which the shared folder would be missing (unless you somehow added a yaml configuration that accidentally conflicted with the shared folder). The shared folder contains the source code cloned from your git repository. If the shared folder was missing, your source code would be missing too.

If the shared folder were missing, and your source code was missing, the npm install would fail due to missing package.json file. This does not seem to be the case based on the error message in your screenshot.

yes, that makes sense. I’ll dig into the yaml configuration. can I share it here just to double-check?

yes, you can share a copy of your yaml in this forum.

this is the problematic pipeline, please let me know if you spot any issue.

kind: pipeline
type: kubernetes
name: regression

node_selector:
  eks.amazonaws.com/nodegroup: automation

resources:
  requests:
    cpu: 7500
    memory: 30Gi

trigger:
  event:
    - pull_request
steps:
  - name: Install dev dependencies
    image: node:14
    commands:
      - npm ci
    environment:
      NODE_ENV: development
      GIT_PAT:
        from_secret: git_pat

  - name: Cancel running regressions
    image: node:14
    commands:
      - node scripts/drone/cancel_running_regression.js
    environment:
      DRONE_TOKEN:
        from_secret: DRONE_TOKEN
    depends_on:
      - Install dev dependencies

  - name: Run lint
    image: node:14
    commands:
      - npm run lint
    environment:
      NODE_ENV: test
    depends_on:
      - Cancel running regressions

  - name: Run regression test
    image: node:14
    failure: ignore
    commands:
      - echo "REGRESSION_STATUS=failure" > /tmp/.env
      - npm run test
      - rm -rf /tmp/.env
      - echo "REGRESSION_STATUS=success" > /tmp/.env
    environment:
      NODE_ENV: test
      TEST_DATABASE_URL: postgres://xxxx:[email protected]:5432/zzzz
    volumes:
      - name: env
        path: /tmp
    depends_on:
      - Run lint

services:
  - name: database
    image: postgres:12-alpine
    commands:
      - |
        docker-entrypoint.sh  -c 'min_wal_size=2GB' -c 'min_wal_size=4GB'  -c 'shared_buffers=4GB' -c 'effective_cache_size=4GB' -c 'wal_buffers=16MB' -c 'maintenance_work_mem=1GB' -c 'fsync=off' -c 'synchronous_commit=off' -c 'full_page_writes=off' -c 'max_connections=150' -c 'client_min_messages=warning'
    environment:
      POSTGRES_USER: xxxx
      POSTGRES_DB: yyyy
      POSTGRES_PASSWORD: zzzz

volumes:
  - name: env
    temp: {}

I’ve just noticed that some pipelines shared the same name, is that might be the issue?

any update here ? @bradrydzewski

@Amit_Baranes what happens if you change npm ci to npm i?

In my head it is a fresh docker container and install each time the pipeline is run

based on the docs npm-ci | npm Docs - there needs to be a package-lock.json, but I don’t think there would be a package-lock.json because it is a fresh install

apart from that is there any way there could be a jest version mismatch - this feels like more of a node dependency mismatch issue than a drone issue

there is a package-lock.json file since we run npm ci for all our pipelines. the thing is it doesn’t happen all the time. it might happen once \ twice a day or 15 times a day. also, it’s not always jest module but others as well.
as for now, I have 2 guesses -

  1. we do something wrong on the pipeline level.
  2. nodes having issues passing this sharing folder between steps.

any update here? @bradrydzewski