HollowByte: OpenSSL DoS Vulnerability Exploits Tiny Payloads to Cripple Servers
A newly disclosed denial-of-service (DoS) vulnerability, dubbed HollowByte, allows unauthenticated attackers to exhaust server memory with just an 11-byte malicious payload targeting OpenSSL. The flaw, silently patched by the OpenSSL team, affects widely used versions of the library, which underpins secure communications for web servers, databases, and programming languages.
How HollowByte Works
Researchers from Okta’s Red Team revealed that vulnerable OpenSSL versions allocate memory based on a 4-byte header in TLS handshake messages, which declares the size of incoming data before verifying the actual payload. Attackers exploit this by sending an 11-byte input with a falsified header claiming a much larger message will follow. The server reserves the specified memory, then hangs indefinitely waiting for data that never arrives.
By repeating this across multiple connections, attackers force the server to allocate excessive memory. While OpenSSL frees buffers when connections drop, the GNU C Library (glibc) retains small-to-medium allocations for reuse, leading to heap fragmentation. The server’s Resident Set Size (RSS) a measure of active memory balloons uncontrollably, even after the attack stops. The only recovery method is a full process restart.
Impact and Affected Systems
OpenSSL is embedded in critical software, including:
- Web servers (NGINX, Apache)
- Programming runtimes (Node.js, Python, Ruby, PHP)
- Databases (MySQL, PostgreSQL)
- Most Linux distributions (pre-installed for TLS encryption)
Okta’s tests on NGINX demonstrated that low-resource environments can be completely depleted of memory, while high-capacity servers may lose up to 25% of available RAM all while attack traffic remains below typical security alert thresholds. Though DoS flaws are less severe than remote code execution vulnerabilities, they can cause operational disruptions and reputational harm.
Patch and Mitigation
The OpenSSL team addressed HollowByte as a "hardening fix" rather than a formal security vulnerability, but the issue has been resolved in:
- OpenSSL 4.0.1
- Backported to 3.6.3, 3.5.7, 3.4.6, and 3.0.21
The fix modifies memory allocation to ignore header claims and expand buffers only when data arrives. Organizations are advised to upgrade OpenSSL packages immediately to prevent exploitation.
The Apache Software Foundation cybersecurity rating report: https://www.rankiteo.com/company/the-apache-software-foundation
OpenSSL Corporation cybersecurity rating report: https://www.rankiteo.com/company/openssl-corporation
OpenJS Foundation cybersecurity rating report: https://www.rankiteo.com/company/openjs-foundation
Oracle MySQL cybersecurity rating report: https://www.rankiteo.com/company/mysql
"id": "THEOPEOPEMYS1784312702",
"linkid": "the-apache-software-foundation, openssl-corporation, openjs-foundation, mysql",
"type": "Vulnerability",
"date": "7/2026",
"severity": "100",
"impact": "5",
"explanation": "Attack threatening the organization's existence"{'affected_entities': [{'industry': 'Technology/Internet',
'name': 'NGINX',
'type': 'Web server'},
{'industry': 'Technology/Internet',
'name': 'Apache',
'type': 'Web server'},
{'industry': 'Technology/Software Development',
'name': 'Node.js',
'type': 'Programming runtime'},
{'industry': 'Technology/Software Development',
'name': 'Python',
'type': 'Programming runtime'},
{'industry': 'Technology/Software Development',
'name': 'Ruby',
'type': 'Programming runtime'},
{'industry': 'Technology/Software Development',
'name': 'PHP',
'type': 'Programming runtime'},
{'industry': 'Technology/Data Management',
'name': 'MySQL',
'type': 'Database'},
{'industry': 'Technology/Data Management',
'name': 'PostgreSQL',
'type': 'Database'},
{'industry': 'Technology/IT Infrastructure',
'name': 'Linux distributions',
'type': 'Operating System'}],
'attack_vector': 'Unauthenticated remote exploitation via TLS handshake '
'messages',
'description': 'A newly disclosed denial-of-service (DoS) vulnerability, '
'dubbed HollowByte, allows unauthenticated attackers to '
'exhaust server memory with just an 11-byte malicious payload '
'targeting OpenSSL. The flaw affects widely used versions of '
'the library, which underpins secure communications for web '
'servers, databases, and programming languages. The server '
'allocates memory based on a falsified 4-byte header in TLS '
'handshake messages, leading to indefinite memory exhaustion '
'and heap fragmentation.',
'impact': {'brand_reputation_impact': 'Reputational harm due to service '
'disruptions',
'downtime': 'Requires full process restart for recovery',
'operational_impact': 'Operational disruptions due to memory '
'depletion',
'systems_affected': 'Memory exhaustion leading to server '
'unresponsiveness'},
'lessons_learned': 'Memory allocation mechanisms in critical libraries like '
'OpenSSL must verify payload sizes before reserving '
'resources to prevent DoS attacks. Silent patches may '
'leave organizations vulnerable if not promptly applied.',
'post_incident_analysis': {'corrective_actions': 'OpenSSL patched to ignore '
'header claims and expand '
'buffers only when data '
'arrives. Organizations '
'advised to upgrade '
'immediately.',
'root_causes': 'OpenSSL allocated memory based on '
'a 4-byte header in TLS handshake '
'messages without verifying the '
'actual payload size, allowing '
'attackers to falsify headers and '
'exhaust memory.'},
'recommendations': 'Immediately upgrade OpenSSL to the latest patched '
'versions (4.0.1, 3.6.3, 3.5.7, 3.4.6, or 3.0.21). Monitor '
'for unusual memory usage patterns in TLS-handling '
'services. Consider implementing rate-limiting or '
'behavioral analysis to detect low-volume DoS attacks.',
'references': [{'source': 'Okta’s Red Team'}],
'response': {'containment_measures': 'Upgrade to patched OpenSSL versions '
'(4.0.1, 3.6.3, 3.5.7, 3.4.6, 3.0.21)',
'recovery_measures': 'Full process restart required to recover '
'memory',
'remediation_measures': 'OpenSSL patched to ignore header claims '
'and expand buffers only when data '
'arrives'},
'title': 'HollowByte: OpenSSL DoS Vulnerability Exploits Tiny Payloads to '
'Cripple Servers',
'type': 'Denial-of-Service (DoS)',
'vulnerability_exploited': 'OpenSSL memory allocation flaw (CVE not assigned, '
'silently patched)'}