Fabric Operations Cheatsheet

Quick-reference commands for operating and troubleshooting the CertChain Hyperledger Fabric network on OpenShift. All commands run from the Terminal tab.

Prerequisites

oc login https://:6443 -u admin

# Namespaces
CENTRAL=certchain
TP=certchain-techpulse
DF=certchain-dataforge
NP=certchain-neuralpath

# Channel
CHANNEL=certchannel

Network Health

Block Height (All Peers)

Compare block heights across organizations — all peers should show the same value:

for NS in $TP $DF $NP; do
  ORG=$(echo $NS | sed 's/certchain-//')
  HEIGHT=$(oc exec deployment/peer0 -n $NS -- \
    peer channel getinfo -c $CHANNEL 2>/dev/null | grep -o '{.*}' \
    | python3 -c "import sys,json; print(json.load(sys.stdin)['height'])" 2>/dev/null)
  echo "$ORG: height=$HEIGHT"
done

Sample output

techpulse: height=55
dataforge: height=55
neuralpath: height=55

Block Height (via Prometheus Metrics)

An alternative approach using Prometheus metrics — works even when the peer CLI is unresponsive:

for ORG in techpulse dataforge neuralpath; do
  H=$(oc exec deployment/couchdb -n certchain-$ORG -- \
    curl -s http://peer0:9443/metrics 2>/dev/null \
    | grep "^ledger_blockchain_height" | awk '{print $2}')
  echo "$ORG: height=$H"
done

Sample output

techpulse: height=55
dataforge: height=55
neuralpath: height=55

Gossip Peers (via Metrics)

Verify each peer can see the other organizations via gossip protocol. Each peer should see 2 remote peers (the other 2 orgs):

for ORG in techpulse dataforge neuralpath; do
  PEERS=$(oc exec deployment/couchdb -n certchain-$ORG -- \
    curl -s http://peer0:9443/metrics 2>/dev/null \
    | grep "^gossip_membership_total_peers_known" | awk '{print $2}')
  echo "$ORG: $PEERS gossip peers"
done

Sample output

techpulse: 2 gossip peers
dataforge: 2 gossip peers
neuralpath: 2 gossip peers

Pod Health Across Namespaces

for NS in $CENTRAL $TP $DF $NP; do
  echo "=== $NS ==="
  oc get pods -n $NS --no-headers | grep -v Completed | awk '{print $1, $3}'
done

Sample output

=== certchain ===
cert-portal-6d8b9c7f4-x2k9m Running
fabric-ca-5f8d6c9b7-qm4vn Running
grafana-deployment-7c8d5b6f4-r8j2k Running
keycloak-7d9e8f6c5-t3m7n Running
orderer0-6f7d8e9c5-p4k8m Running
postgres-6c7d8e9f5-w2j6n Running
verify-api-5d6e7f8c9-q9m3k Running
=== certchain-techpulse ===
cert-admin-api-7e8f9d6c5-k2m4n Running
certcontract-6d7e8f9c5-j3n7m Running
couchdb-5c6d7e8f9-p8k2j Running
course-manager-ui-8f9d6e7c5-w4m6n Running
keycloak-9d6e7f8c5-t5n8m Running
orderer-7c8d9e6f5-q2k4j Running
peer0-6e7f8d9c5-r6m9n Running
postgres-5d6e7f8c9-j3k7m Running
...

Orderer Cluster

BFT Consensus Status

Check SmartBFT leader, cluster size, and committed block number across all orderers:

# Uses CouchDB pod (has curl) to reach all orderers via cluster DNS
for PAIR in central/orderer0.certchain techpulse/orderer.certchain-techpulse dataforge/orderer.certchain-dataforge neuralpath/orderer.certchain-neuralpath; do
  ORG="${PAIR%%/*}"
  HOST="${PAIR##*/}"
  oc exec deployment/couchdb -n $TP -- \
    curl -s http://$HOST.svc.cluster.local:8443/metrics 2>/dev/null \
    | grep "^consensus_BFT" \
    | awk -v org="$ORG" '/is_leader/{l=$2} /committed_block_number/{b=$2} /cluster_size/{s=$2} END{print org": leader="l" block="b" cluster_size="s}'
done

Sample output

central: leader=1 block=54 cluster_size=4
techpulse: leader=0 block=54 cluster_size=4
dataforge: leader=0 block=54 cluster_size=4
neuralpath: leader=0 block=54 cluster_size=4

leader=1 indicates the current BFT leader. Exactly one orderer should be the leader at any time. The block value should be equal across all orderers (±1 during active transactions).

Orderer Logs (Recent Blocks)

Watch the most recent block activity on orderer0:

oc logs deployment/orderer0 -n $CENTRAL --tail=100 2>/dev/null \
  | grep -E "Committed block|Start consensus|Delivering" | tail -5

Sample output

... Committed block [53] with 1 transaction(s) ... channel=certchannel
... Committed block [54] with 1 transaction(s) ... channel=certchannel
... Committed block [55] with 0 transaction(s) ... channel=certchannel

If no output appears, the network has been idle and the recent log lines contain only TLS handshake messages (normal — caused by OpenShift Router health checks on passthrough Routes). Increase --tail=500 or issue a certificate to generate new block activity.

Chaincode Operations

Why not use peer CLI directly? You might expect to run commands like peer lifecycle chaincode querycommitted or peer chaincode query by exec-ing into the peer pod. This won’t work because:

The peer pod runs with a peer identity (OU=peer), not an admin identity (OU=admin)
Chaincode queries and lifecycle commands require admin-level MSP credentials
The admin MSP is mounted only in the cert-admin-api pods, not in the peer pods

Instead, use the REST APIs below (which have the admin MSP configured) or query CouchDB directly for raw state inspection.

Query a Certificate (via API)

Read a certificate without authentication through the public verify-api:

curl -sk "https:///api/v1/verify/TP-FSWD-001" | python3 -m json.tool

Sample output

{
    "certID": "TP-FSWD-001",
    "studentName": "Alice Johnson",
    "courseName": "Full-Stack Web Development",
    "issueDate": "2025-09-15",
    "expiryDate": "2028-09-15",
    "status": "VALID",
    "issuerOrg": "TechPulseMSP"
}

Batch Verify Multiple Certificates

curl -sk "https:///api/v1/verify/batch?ids=TP-FSWD-001,DF-DSML-001,NP-CCNA-001" \
  | python3 -m json.tool

Sample output

{
    "results": [
        {"certID": "TP-FSWD-001", "status": "VALID", "issuerOrg": "TechPulseMSP"},
        {"certID": "DF-DSML-001", "status": "VALID", "issuerOrg": "DataForgeMSP"},
        {"certID": "NP-CCNA-001", "status": "VALID", "issuerOrg": "NeuralPathMSP"}
    ],
    "totalVerified": 3
}

List Certificates (via API)

First, obtain an admin JWT token:

TOKEN=$(curl -sk -X POST \
  "https:///realms/techpulse/protocol/openid-connect/token" \
  -d "grant_type=password" \
  -d "client_id=course-manager-ui" \
  -d "username=admin@techpulse.demo" \
  -d "password=admin" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])")

Then list all certificates for TechPulse:

curl -sk "https:///api/v1/certificates?page=0&size=100" \
  -H "Authorization: Bearer $TOKEN" \
  | python3 -c "import sys,json; data=json.load(sys.stdin); certs=data.get('content',data) if isinstance(data,dict) else data; [print(f'  {c[\"certID\"]:20s} {c[\"status\"]}') for c in certs if isinstance(c, dict)]"

Sample output

  TP-FSWD-001          ACTIVE
  TP-FSWD-002          ACTIVE
  TP-UXUI-001          ACTIVE
  TP-UXUI-002          ACTIVE
  ...

Issue a Certificate (via API)

CERT_ID="CHEAT-$(date +%s)"
curl -sk -X POST "https:///api/v1/certificates" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d "{
    \"certID\": \"$CERT_ID\",
    \"studentID\": \"student01@techpulse.demo\",
    \"studentName\": \"Cheatsheet Test\",
    \"courseID\": \"CHEAT-101\",
    \"courseName\": \"Cheatsheet Course\",
    \"issueDate\": \"2026-03-17\",
    \"expiryDate\": \"2027-03-17\"
  }" | python3 -m json.tool

Sample output

{
    "certID": "CHEAT-1742208000",
    "studentID": "student01@techpulse.demo",
    "studentName": "Cheatsheet Test",
    "courseID": "CHEAT-101",
    "courseName": "Cheatsheet Course",
    "issueDate": "2026-03-17",
    "expiryDate": "2027-03-17",
    "status": "ACTIVE",
    "txID": "a1b2c3d4e5f6..."
}

Get Dashboard Stats (via API)

curl -sk "https:///api/v1/dashboard/stats" \
  -H "Authorization: Bearer $TOKEN" | python3 -m json.tool

Sample output

{
    "totalCertificates": 100,
    "activeCertificates": 96,
    "revokedCertificates": 2,
    "expiredCertificates": 2
}

CouchDB State Database

Database Statistics

for NS in $TP $DF $NP; do
  ORG=$(echo $NS | sed 's/certchain-//')
  STATS=$(oc exec deployment/couchdb -n $NS -- \
    curl -s -u admin:adminpw http://localhost:5984/certchannel_certcontract 2>/dev/null)
  DOC_COUNT=$(echo "$STATS" | python3 -c "import sys,json; print(json.load(sys.stdin).get('doc_count','N/A'))" 2>/dev/null)
  echo "$ORG: $DOC_COUNT documents"
done

Sample output

techpulse: 100 documents
dataforge: 100 documents
neuralpath: 100 documents

List All CouchDB Databases

oc exec deployment/couchdb -n $TP -- \
  curl -s -u admin:adminpw http://localhost:5984/_all_dbs 2>/dev/null \
  | python3 -m json.tool

Sample output

[
    "_replicator",
    "_users",
    "certchannel_",
    "certchannel__lifecycle",
    "certchannel__lifecycle$$h_implicit_org_...",
    "certchannel__lifecycle$$p_implicit_org_...",
    "certchannel_certcontract",
    "certchannel_lscc",
    "fabric__internal"
]

The certchannel_certcontract database contains the chaincode world state. The certchannel_lifecycle$$* databases track chaincode lifecycle approvals per organization.

Query a Certificate in CouchDB

Read the raw CouchDB document (includes Fabric metadata like _rev, _attachments):

oc exec deployment/couchdb -n $TP -- \
  curl -s -u admin:adminpw \
    "http://localhost:5984/certchannel_certcontract/TP-FSWD-001" 2>/dev/null \
  | python3 -m json.tool

Sample output

{
    "_id": "TP-FSWD-001",
    "_rev": "1-abc123...",
    "certID": "TP-FSWD-001",
    "studentID": "alice.johnson@techpulse.demo",
    "studentName": "Alice Johnson",
    "courseID": "TP-FSWD",
    "courseName": "Full-Stack Web Development",
    "issueDate": "2025-09-15",
    "expiryDate": "2028-09-15",
    "status": "ACTIVE",
    "issuerOrg": "TechPulseMSP",
    "txID": "7f8e9d...",
    "~version": "CgMBFQ=="
}

The ~version field is Fabric’s MVCC (Multi-Version Concurrency Control) marker used for read-write conflict detection during endorsement.

Compact a CouchDB Database

Reclaim disk space by compacting the state database (safe while peer is running):

oc exec deployment/couchdb -n $TP -- \
  curl -s -u admin:adminpw -X POST \
    -H "Content-Type: application/json" \
    http://localhost:5984/certchannel_certcontract/_compact 2>/dev/null

Sample output

{"ok":true}

CouchDB Active Tasks

Check compaction progress and other background tasks:

oc exec deployment/couchdb -n $TP -- \
  curl -s -u admin:adminpw http://localhost:5984/_active_tasks 2>/dev/null \
  | python3 -m json.tool

Sample output

[]

An empty array means no background tasks are running. During compaction, you’d see entries with type: "database_compaction" and progress percentage.

Identity & Certificates

List Registered Fabric CA Identities

Lists all identities registered with the centralized Fabric CA (peers, orderers, admins):

oc exec deployment/fabric-ca -n $CENTRAL -- sh -c \
  'FABRIC_CA_CLIENT_HOME=/tmp/ca-admin \
   fabric-ca-client enroll -u http://admin:adminpw@localhost:7054 \
     --tls.certfiles /var/hyperledger/fabric-ca-server/ca-cert.pem 2>/dev/null && \
   FABRIC_CA_CLIENT_HOME=/tmp/ca-admin \
   fabric-ca-client identity list \
     --tls.certfiles /var/hyperledger/fabric-ca-server/ca-cert.pem 2>/dev/null'

Sample output

Name: admin, Type: client, Affiliation: , Max Enrollments: -1, Attrs: [...]
Name: peer0-techpulse, Type: peer, Affiliation: techpulse, Max Enrollments: -1
Name: peer0-dataforge, Type: peer, Affiliation: dataforge, Max Enrollments: -1
Name: peer0-neuralpath, Type: peer, Affiliation: neuralpath, Max Enrollments: -1
Name: orderer0, Type: orderer, Affiliation: , Max Enrollments: -1
Name: orderer1-techpulse, Type: orderer, Affiliation: techpulse, Max Enrollments: -1
Name: orderer2-dataforge, Type: orderer, Affiliation: dataforge, Max Enrollments: -1
Name: orderer3-neuralpath, Type: orderer, Affiliation: neuralpath, Max Enrollments: -1
Name: admin-techpulse, Type: client, Affiliation: techpulse, Max Enrollments: -1
Name: admin-dataforge, Type: client, Affiliation: dataforge, Max Enrollments: -1
Name: admin-neuralpath, Type: client, Affiliation: neuralpath, Max Enrollments: -1

Inspect an MSP Certificate

View the X.509 details of an organization’s admin certificate.

oc extract secret/admin-msp -n $TP --keys=signcerts --to=- 2>/dev/null \
  | python3 -c "
import sys, ssl, tempfile, os
pem = sys.stdin.buffer.read()
tmp = tempfile.NamedTemporaryFile(suffix='.pem', delete=False)
tmp.write(pem); tmp.close()
c = ssl._ssl._test_decode_cert(tmp.name)
os.unlink(tmp.name)
subj = ','.join(f'{k}={v}' for t in c['subject'] for k,v in t)
issr = ','.join(f'{k}={v}' for t in c['issuer'] for k,v in t)
print(f'Subject:    {subj}')
print(f'Issuer:     {issr}')
print(f'Not Before: {c[\"notBefore\"]}')
print(f'Not After:  {c[\"notAfter\"]}')
"

Sample output

Subject:    countryName=US,...,organizationalUnitName=admin,commonName=admin-techpulse
Issuer:     countryName=US,...,organizationalUnitName=Fabric,commonName=fabric-ca-server
Not Before: Mar 17 04:37:00 2026 GMT
Not After:  Mar 17 04:44:00 2027 GMT

Check Certificate Expiry (All Orgs)

for NS in $TP $DF $NP; do
  ORG=$(echo $NS | sed 's/certchain-//')
  EXPIRY=$(oc extract secret/admin-msp -n $NS --keys=signcerts --to=- 2>/dev/null \
    | python3 -c "import sys,ssl,tempfile,os; pem=sys.stdin.buffer.read(); tmp=tempfile.NamedTemporaryFile(suffix='.pem',delete=False); tmp.write(pem); tmp.close(); c=ssl._ssl._test_decode_cert(tmp.name); os.unlink(tmp.name); print(c['notAfter'])" 2>/dev/null)
  echo "$ORG admin cert expires: $EXPIRY"
done

Sample output

techpulse admin cert expires: Mar 17 04:44:00 2027 GMT
dataforge admin cert expires: Mar 17 04:44:00 2027 GMT
neuralpath admin cert expires: Mar 17 04:46:00 2027 GMT

Inspect Peer TLS Certificate

oc extract secret/peer0-tls -n $TP --keys=server.crt --to=- 2>/dev/null \
  | python3 -c "
import sys, ssl, tempfile, os
pem = sys.stdin.buffer.read()
tmp = tempfile.NamedTemporaryFile(suffix='.pem', delete=False)
tmp.write(pem); tmp.close()
c = ssl._ssl._test_decode_cert(tmp.name)
os.unlink(tmp.name)
subj = ','.join(f'{k}={v}' for t in c['subject'] for k,v in t)
print(f'Subject:   {subj}')
print(f'Not After: {c[\"notAfter\"]}')
if 'subjectAltName' in c:
    dns = [v for t,v in c['subjectAltName'] if t=='DNS']
    print(f'DNS:       {\", \".join(dns)}')
"

Sample output

Subject:   countryName=US,...,organizationalUnitName=peer,commonName=peer0-techpulse
Not After: Mar 17 04:44:00 2027 GMT
DNS:       peer0, peer0.certchain-techpulse.svc.cluster.local, peer0-certchain-techpulse.apps...

Inspect Orderer TLS Certificate

oc extract secret/orderer0-tls -n $CENTRAL --keys=server.crt --to=- 2>/dev/null \
  | python3 -c "
import sys, ssl, tempfile, os
pem = sys.stdin.buffer.read()
tmp = tempfile.NamedTemporaryFile(suffix='.pem', delete=False)
tmp.write(pem); tmp.close()
c = ssl._ssl._test_decode_cert(tmp.name)
os.unlink(tmp.name)
subj = ','.join(f'{k}={v}' for t in c['subject'] for k,v in t)
print(f'Subject:   {subj}')
print(f'Not After: {c[\"notAfter\"]}')
if 'subjectAltName' in c:
    dns = [v for t,v in c['subjectAltName'] if t=='DNS']
    print(f'DNS:       {\", \".join(dns)}')
"

Sample output

Subject:   countryName=US,...,organizationalUnitName=orderer,commonName=orderer0
Not After: Mar 17 04:43:00 2027 GMT
DNS:       orderer0, orderer0.certchain.svc.cluster.local, orderer0-certchain.apps...

Peer Operations

Peer Channel Membership

oc exec deployment/peer0 -n $TP -- \
  peer channel list 2>/dev/null | grep -v "INFO"

Sample output

Channels peers has joined:
certchannel

Peer Logs (Endorsement Activity)

oc logs deployment/peer0 -n $TP --tail=30 2>/dev/null \
  | grep -E "Committed block|endorseProposal|chaincode"

Sample output

... Committed block [54] to ledger channel=certchannel
... Committed block [55] to ledger channel=certchannel

Peer Metrics Summary

Endorsement success/failure counts and proposal rates:

M=$(oc exec deployment/couchdb -n $TP -- \
  curl -s http://peer0:9443/metrics 2>/dev/null)
echo "Endorsement proposals (successful):"
echo "$M" | grep "^endorser_successful_proposals " | awk '{print "  "$2}'
echo "Endorsement proposals (failed):"
echo "$M" | grep "^endorser_proposal_acl_failures" | awk '{print "  "$2}'
echo "Ledger height:"
echo "$M" | grep "^ledger_blockchain_height" | awk '{print "  "$2}'

Sample output

Endorsement proposals (successful):
  39
Endorsement proposals (failed):
  0
Ledger height:
  55

Rebuild State Databases

If CouchDB is corrupted, rebuild from the immutable ledger (see Resilience for full walkthrough):

# Peer must be stopped first
oc scale deployment peer0 -n $TP --replicas=0
oc rollout status deployment peer0 -n $TP --timeout=30s

# Mark databases for rebuild
oc debug deployment/peer0 -n $TP -- peer node rebuild-dbs

# Restart peer — it replays all blocks into fresh CouchDB
oc scale deployment peer0 -n $TP --replicas=1

This is a destructive operation — it drops all CouchDB databases for this peer and replays them from the block files. The peer will be unavailable until replay completes.

Keycloak & Authentication

Get an Admin JWT Token

TOKEN=$(curl -sk -X POST \
  "https:///realms/techpulse/protocol/openid-connect/token" \
  -d "grant_type=password" \
  -d "client_id=course-manager-ui" \
  -d "username=admin@techpulse.demo" \
  -d "password=admin" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])")
echo "Token: ${TOKEN:0:20}..."

Sample output

Token: eyJhbGciOiJSUzI1Ni...

Decode a JWT Token

echo "$TOKEN" | cut -d. -f2 | base64 -d 2>/dev/null | python3 -m json.tool

Sample output

{
    "sub": "a1b2c3d4-...",
    "realm_access": {
        "roles": ["org-admin", "default-roles-techpulse"]
    },
    "email": "admin@techpulse.demo",
    "preferred_username": "admin@techpulse.demo",
    "given_name": "Admin",
    "family_name": "TechPulse"
}

List Keycloak Users

KC_ADMIN_TOKEN=$(curl -sk -X POST \
  "https:///realms/master/protocol/openid-connect/token" \
  -d "grant_type=password" \
  -d "client_id=admin-cli" \
  -d "username=admin" \
  -d "password=admin" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])")

curl -sk "https:///admin/realms/techpulse/users" \
  -H "Authorization: Bearer $KC_ADMIN_TOKEN" \
  | python3 -c "import sys,json; [print(u['username'] + '  ' + u.get('email','')) for u in json.load(sys.stdin)]"

Sample output

admin@techpulse.demo  admin@techpulse.demo
instructor01@techpulse.demo  instructor01@techpulse.demo
student01@techpulse.demo  student01@techpulse.demo
student02@techpulse.demo  student02@techpulse.demo

ArgoCD Operations

Check All Application Status

oc get applications -n openshift-gitops \
  -o custom-columns='NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status'

Sample output

NAME                   SYNC     HEALTH
certchain-central      Synced   Healthy
certchain-dataforge    Synced   Healthy
certchain-neuralpath   Synced   Healthy
certchain-showroom     Synced   Healthy
certchain-techpulse    Synced   Healthy
field-content          Synced   Healthy

Force Sync an Application

oc annotate application certchain-central -n openshift-gitops \
  argocd.argoproj.io/refresh=hard --overwrite

Sample output

application.argoproj.io/certchain-central annotated

Pause/Resume Auto-Sync

# Pause (for manual changes)
oc patch application certchain-techpulse -n openshift-gitops --type=merge \
  -p '{"spec":{"syncPolicy":{"automated":null}}}'

# Resume
oc patch application certchain-techpulse -n openshift-gitops --type=merge \
  -p '{"spec":{"syncPolicy":{"automated":{"prune":true,"selfHeal":true}}}}'

Sample output

application.argoproj.io/certchain-techpulse patched
application.argoproj.io/certchain-techpulse patched