Node Log
This page details some important or frequently asked logs from Klaytn nodes. If the Klaytn log is modified or newly added/deleted, please edit this page as well.
For more detailed information about log types, you can refer to log_modules.go
If you encounter any abnormal situation, please report it to the klaytn team via github, Klaytn Forum, or Discord.
Error Logs
| Log Type | Node Type | Log Message | Description | Suggested Guide |
|---|---|---|---|---|
| Blockchain | CN/PN/EN | ########## BAD BLOCK #########Chain config: %v Number: %vHash: 0x%x%v Error: %v############################## | A bad block occurs when the received receipt and the execution result do not match. If a node stops with bad block log, it could be due to two reasons. - Case 1. The configuration of the node is wrong such, as the binary version. - Case 2. There’s a problem with the code. It is very likely that other nodes will also experience the same problem. | This error is critical, so if you see any bad block, please make an issue or report it to the Klaytn GitHub repository. |
| ConsensusIstanbulCore | CN/PN/EN | Drop an empty message from timeout channel | It means that the round change timer will expire. This error is printed if the timer closes accidentally. | The error may occur when the downloader is started. check next log is also printed: Block synchronisation started. |
| NetworksP2P | CN/PN/EN | Protocol istanbul/64 failed id=04680a827fa1b240 conn=staticdial err="write tcp 10.117.2.105:34396->10.117.2.27:32323: use of closed etwork connection" Protocol istanbul/64 failed err="shutting down" | This log can be printed when the other node is disconnected. It is usually followed by Disconnected a P2P Peer log. | Check if the disconnected peer is reconnected again. If it is not reconnected, check the network status or peer connection admin_peers |
| NodeCN | CN | fail to SendNewBlockHashes err="write tcp 10.117.2.124:24108->10.117.2.108:32323: use of closed network connection" fail to SendNewBlockHashes err="shutting down" | same as Protocol istanbul/64 failed | same as Protocol istanbul/64 failed |
| NodeCN | CN | fail to SendNewBlock peer=d35220eccdb0de7b err="shutting down" | same as Protocol istanbul/64 failed | same as Protocol istanbul/64 failed |
| NetworksRPC | EN (mostly) | FastWebsocketHandler fail to upgrade message error="websocket: version != 13" | Version issue of WebSocket connection | The header of the request should contain Sec-Websocket-Version field with the value set at 13. You may not have used klaytn rpc client. |
Warn Logs
| Log Type | Node Type | Log Message | Description | Suggested Guide |
|---|---|---|---|---|
| Blockchain | CN/PN/EN | Upgrade database version from=N/A to=3 | It is logged at the beginning of the node start-up | You don't need to handle this. |
| ConsensusIstanbulCore | CN | [RC] round= | Round change log is started with [RC] tag. | If the round does not end in one or two rounds and continues to go up, then the network status or peer connection should be analyzed first. peer connection check api: admin_peers |
| ConsensusIstanbulCore | CN | unexpected request address= state="Accept request" seq=312 err="old message" number=311 hash=d960ea…6df6de | A proposer mines a block, but it is turned out unexpected. In most cases, it is too old to be a new block. | You don't need to handle this. |
| Node | CN/PN/EN | Failed doConnTypeHandshake addr=10.117.2.252:28516 conn=inbound conntype=-1 err="read tcp 10.117.2.78:32324->10.117.2.252:28516: i/o timeout | By dialing, the two P2P peers setup a connection. This log is printed if the setup fails. | Check if the disconnected peer is reconnected again. If not, check the network status or peer connection peer connection check api: admin_peers |
| NodeCN | PN/EN | Failed to filter bodies peer=c02e4b4d471c56b9 lenTxs=1 | A node received the unwanted block header of body when fetching. - lenTxs: non-requested number of txs | You don't need to handle this. |
| Work | CN | Transaction aborted due to time limit hash= | The block execution time when mining should not exceed 250ms, so the last transaction can be aborted due to this time limit. | Confirm that the transaction enters the block. |
| Work | CN | Transaction failed, account skipped hash=b1b26c...6b220a err="insufficient balance for transfer" Error(before v1.6.2) Warn(after v1.6.2) | When a transaction cannot be executed during mining due to an insufficient balance in the from account (Theoretically, it occurs when the balance was sufficient at the time when the transaction was created and entered the txpool, but not at the actual execution time.) | Check if the from account is really out of balance. |
Info Logs
Info log contains the additional information for you to know more about the node status, so you don't need to handle Info level log.
| Log Type | Node Type | Log Message | Description |
|---|---|---|---|
| Blockchain | CN/PN/EN | Regenerated local transaction journal transactions=0 accounts=0 | When node is shut down, local txs are journaled to a file (default file name is transactions.rlp). When node is restarted with the journaled file, the local transactions can be regenerated based on the file. - transactions: number of the regenerated local transaction. - accounts: number of the regenerated accounts(==from) |
| Blockchain | CN/PN/EN | Inserted a new block number=14 hash=13cbfc…f007fc txs=0 gas=0 elapsed=793.458µs processTxs=167ns finalize=157.708µs validateState=7.542µs totalWrite=443.417µs trieWrite=256.667µs | If the node is not a proposer at that block, and the consensus is successful, the node have executed(==validates) the block. In other words, a block is inserted. - gas: total gas spent during tx execution. This field is commonly used when testing the network to find the used gas per block. |
| NetworksP2P | CN/PN/EN | [Dial] Add dial candidate from static nodes id=62a08a4b9f091c4b NodeType=0 ip=10.117.2.8 mainPort=32323 port=[32323] | A new P2P peer is connected, and it is a static node. A node added manually by using static-nodes.json or addpeer api is called static node. If it is a multichannel, it uses two ports. ex. [32323, 32324]. - id: dst peer id - NodeType: dst node type(cn,pn,en,bn) - ip: dst ip - mainPort: dst TCP listening port number - port: dst TCP listening port number including both the main port and sub port. |
| NetworksP2P | CN/PN/EN | Added a multichannel P2P Peer id=28a6760472a078fb conn=staticdial peerID=28a6760472a078fb | A new peer is connected as a multichannel peer. - id/peerID: my node’s peer id - conn: type of connection - inbound: somebody connects me - staticdial: static connection, such as static-nodes.json or addPeer - trusteddial: trusted connection, such as trust-nodes.json. it can be always reconnected and established even the number of connections exceeds max limit. |
| NetworksP2P | CN/PN/EN | Disconnected a multichannel P2P Peer id=28a6760472a078fb conn=inbound peerID=28a6760472a078fb peerName=Klaytn/v1.7.3+acae89350c/darwin-arm64/go1.18.1 err=EOF | A multichannel peer is disconnected. - peerName: It shows my node info - err: The reason why the connection is disconnected |
| NetworksP2P | CN/PN/EN | ProtocolManager.processConsensusMsg closed id=28a6760472a078fb conn=inbound PeerName=Klaytn/v1.7.3+acae89350c/darwin-arm64/go1.18.1 | When a P2P node is disconnected, the consensus message channel between them is closed, too. |
| StorageStateDB | CN/PN/EN | Persisted trie from memory database blockNum=23460 updated nodes=4 updated nodes size=499.00B time=539.959µs gcnodes=68 gcsize=10.55kB gctime=226.499µs livenodes=245 livesize=37.80kB | This log is printed to inform you that trie db has been committed. Here, commit means flushing db change to the actual db. A commit is made periodically. - Case 1. If the node is a full node, trie commit is made for every 128 block. - Case 2. If the node is an archive node, trie commit is made for every block. Commit is made at next circumstances, too. - , A commit is made when a node is shut down. - A commit is made when memory size exceeds the cap. Tip. - gc stands for garbage collection. Here, garbage collection means flushing db writes of trie node change. - A full node stores the information of every 128 cycle and the latest 128 blocks. - Archive node stores the information of every block. |
| Work | CN | Commit new mining work number=14 hash=438ef7…68ca7f txs=0 elapsed=605.375µs commitTime=184.708µs finalizeTime=414.375µs | Every CN mines a block in preparation for round change - number: block number - hash: block hash (it is not a final version) - txs: number of transactions in a block - elapsed: total block mining time (commitTime + finalizeTime) - commitTime: transactions execution time in a block - finalizeTime: block finalize time |
| Work | CN | Successfully sealed new block number=14 hash=13cbfc…f007fc | [Only Proposer] Sealing a new block is successful. Sealing contains the next steps. - Ibft consensus process for the block. - Update timestamp and signatures of the block |
| Work | CN | Successfully wrote mined block num=14 hash=13cbfc…f007fc txs=0 elapsed=617.709µs | [Only Proposer] If the node is a proposer, and consensus is succeed, the proposer needs to store the block execution result in the db. This log means the storing was successful. |
| Work | CN | Mining too far in the future wait=1s | In order to maintain 1 second block creation period, the node sleeps for "1s - previous block generation/propagation/execution time". - wait: how much time the node sleeps |
| VM | CN/PN/EN | Returning since the addr is not a program account addr= | Somebody tries to call a non-existent contract. Tip. In Klaytn, program account is equivalent to contract account. |