Web3JS Socket/Connection Issues

Dealing with Socket/Connection issues in Web3JS

Socket connection issues are encountered while using the `@solana/web3.js` library to communicate with RPCs

Errors

The errors you’ve faced might be like the following

fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11730:11)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at runNextTicks (node:internal/process/task_queues:64:3)
    at process.processImmediate (node:internal/timers:449:9)
    at async ClientBrowser.callServer (/home/alice/project/node-spammer/node_modules/.pnpm/@solana+web3.js@1.91.4/node_modules/@solana/web3.js/lib/index.cjs.js:4926:17) {
  cause: ConnectTimeoutError: Connect Timeout Error
      at onConnectTimeout (node:internal/deps/undici/undici:6869:28)
      at node:internal/deps/undici/undici:6825:50
      at Immediate._onImmediate (node:internal/deps/undici/undici:6857:13)
      at process.processImmediate (node:internal/timers:478:21) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}

TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11730:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async ClientBrowser.callServer (/home/alice/project/node-spammer/node_modules/.pnpm/@solana+web3.js@1.91.4/node_modules/@solana/web3.js/lib/index.cjs.js:4926:17) {
  cause: Error: connect ECONNREFUSED 202.8.10.86:443
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '202.8.10.86',
    port: 443
  }
}

fetch failed
    at node:internal/deps/undici/undici:12345:11
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async ClientBrowser.callServer (/home/alice/project/node_modules/@solana/web3.js/lib/index.cjs.js:4926:17) {
  cause: Error: Client network socket disconnected before secure TLS connection was established
      at connResetException (node:internal/errors:787:14)
      at TLSSocket.onConnectEnd (node:_tls_wrap:1727:19)
      at TLSSocket.emit (node:events:530:35)
      at endReadableNT (node:internal/streams/readable:1696:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    code: 'ECONNRESET',
    path: undefined,
    host: 'api.mainnet-beta.solana.com',
    port: 443,
    localAddress: null
  }
}

fetch failed
    at node:internal/deps/undici/undici:12345:11
    at async ClientBrowser.callServer (/home/alice/project/node_modules/@solana/web3.js/lib/index.cjs.js:4926:17) {
  cause: SocketError: other side closed
      at TLSSocket.onSocketEnd (node:internal/deps/undici/undici:8903:26)
      at TLSSocket.emit (node:events:530:35)
      at endReadableNT (node:internal/streams/readable:1696:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    code: 'UND_ERR_SOCKET',
    socket: {
      localAddress: '202.8.10.86',
      localPort: 52680,
      remoteAddress: undefined,
      remotePort: undefined,
      remoteFamily: undefined,
      timeout: undefined,
      bytesWritten: 5159,
      bytesRead: 0
    }
  }
}

Cause

These errors are caused by NodeJS. It uses too many local ports to communicate with the server. At some point, this hits a limit and existing connections are broken and new ones are not allowed causing the errors. You can check the number of ports used by your Node process via the following command on UNIX based systems.

lsof -i -n -P | grep node | awk '{print $9}' | awk -F '->' '{print $1}' | awk -F ':' '{print $2}' | sort -u | wc -l

These errors originate from the `undici` library; Not by web3.js or your RPC. This library is NodeJS’ implementation of the `fetch` function used to make API calls. Following are some references to this issue w.r.t. `undici`

Links
https://github.com/nodejs/undici/issues/583
https://sg.wantedly.com/companies/rightcode/post_articles/871110
https://ethereum.stackexchange.com/questions/130028/hardhat-deploy-script-on-rsk-throws-und-err-socket
https://github.com/nodejs/undici/issues/2412

We’ve been in touch with Anza Labs and the maintainers of `@solana/web3.js` and have confirmed that this issue is not with web3.js.

Fixes

A fix to these issues is to limit the number of ports NodeJS can use. This can be done easily from your JS code. We've set the number of allowed ports to 100 in our tests and performed over 2 million asynchronous getMultipleAccounts calls with 1000 addresses in each call. All of these calls resolved in ~2mins. You are free to set the number as you see fit.

What is an ideal connection count for you?

Each established connection needs a TLS handshake. More details here. Establishing connection is time a consuming process and in this case is unnecessary since existing connections can be reused. It'll also be comparatively less load on the RPC as well. We recommend setting the connection limit to 50. You can monitor the number of connections using the command mentioned above in the Causesection. And, if you feel like all connections are always in use and your application can benefit from more connections then you can increase the limit.

1. Configuring undici

// npm install undici

import { setGlobalDispatcher, Agent } from "undici";

setGlobalDispatcher(
  new Agent({
    connections: 50,
  })
);

1. Using Axios for API calls instead of `undici`. The `@solana/web3.js` library allows you to specify a custom fetch function in the connection object. You can use Axios there. Here’s an example

   const agent = new https.Agent({
     maxSockets: 50,
   });
   
   const axiosObject = axios.create({
     httpsAgent: agent,
   });

   Here's a full example on using axios with web3.js https://gist.github.com/WilfredAlmeida/9adea27abb5958178c4370c5656e89b7
2. Using the Bun runtime. Bun is a new JS runtime. As per our tests, these issues aren’t encountered at all via Bun. Please note that bun is quite new and may cause other problems.
3. Using the new version of `@solana/web3.js`. Anza Labs is developing a new and improved version of the library which, as of this writing, is in technical preview. It uses the `undici` library and the errors are less frequent in it. If encountered, with the configuration mentioned in [1] they should be resolved. Do note that the library is not compatible with the current version, meaning that it’ll be a ground up rewrite of your codebase.

Other SDK/Libraries

If you're using SDKs like jito-ts which uses node-fetch underneath, or some other libraries, you may encounter errors like ERR_STREAM_PREMATURE_CLOSE which have this same underlying issue and limiting the number of connections for NodeJS has resolved the issue for our customers. The number of connections for libraries like node-fetch and others can be limited by providing an http(s).Agent configuration. For example

new https.Agent({
  maxSockets: 50,
});

We’re continuously working on this issue. Improvements and progress will be added to this document.

We've proposed that the @solana/web3.js library have a default configurable connection limit. You can follow an upvote the discussion on GitHub.