Introduction
Varnish Cache is a powerful HTTP accelerator that speeds up websites by caching responses in memory. This tutorial covers deployment, VCL (Varnish Configuration Language) programming, cache invalidation strategies, and real-world patterns for dynamic sites like WordPress, Magento, and custom applications.
Understanding Varnish Architecture
Varnish sits between clients and backend web servers. Architecture: child process (handles connections, runs VCL), manager process (configuration compilation, monitoring, CLI). Memory storage (default, LRU eviction) or Malloc (RAM only). Key components: VCL (caching logic), Hit/Miss decision, Grace mode (serve stale content during backend failures), Saint mode (blacklist failing backends). Varnish uses virtual memory, configurable via -s parameter. Limits: -p thread_pools, -p thread_pool_max, -p listen_depth for high concurrency.
Installation and Basic Setup
Ubuntu/Debian: sudo apt install varnish -y. RHEL/CentOS: sudo yum install varnish -y. Varnish 7.0+ recommended. Default port 6081 (cd /lib/systemd/system/varnish.service, change exec start). Move to port 80: edit varnish.service ExecStart line: -a :80. System configuration (/etc/default/varnish): VARNISH_LISTEN_PORT=80. Configuration directory: /etc/varnish/. Default.vcl defines behavior. Start: sudo systemctl enable varnish, sudo systemctl start varnish. Test: curl -I http://localhost. Check with varnishlog, varnishstat, varnishadm.
Backend Configuration
Define backend in default.vcl: backend default { .host = "127.0.0.1"; .port = "8080"; .first_byte_timeout = 300s; .connect_timeout = 5s; .between_bytes_timeout = 2s; .max_connections = 100; }. Multiple backends: backend app1 { .host = "10.0.0.1"; }, backend app2 { .host = "10.0.0.2"; }. Load balancing: import directors; sub vcl_init { new lb = directors.round_robin(); lb.add_backend(app1); lb.add_backend(app2); }. Use lb.backend() in vcl_recv. Health checks: .probe = { .url = "/health"; .timeout = 2s; .interval = 5s; .window = 5; .threshold = 3; }. Failover: use directors.fallback() or manual logic in vcl_backend_fetch.
VCL (Varnish Configuration Language)
VCL subroutines executed at different HTTP phases: vcl_recv (request received, before lookup), vcl_hash (cache key generation), vcl_hit (cache hit), vcl_miss (cache miss), vcl_backend_fetch (fetch from backend), vcl_backend_response (backend response received), vcl_deliver (response to client). Return statements: hash, lookup, deliver, fetch, restart, abandon. Example: sub vcl_recv { if (req.method == "POST") { return (pass); } }. Important headers modification: unset req.http.Cookie to increase cacheability. Use req.http.X-Forwarded-For for proxy chains.
Caching Strategies for Dynamic Sites
WordPress: cache product pages, category pages but bypass cart/checkout/my-account. Remove login cookies: sub vcl_recv { if (req.http.cookie !~ "wordpress_logged_in|comment_author|wp-postpass") { unset req.http.cookie; } }. Cache full page except for logged-in users. TTL for posts: set beresp.ttl = 24h;. Purge on post-update: send PURGE request. Magento: cache category, product, CMS pages. Remove session cookies for guest users. Set grace mode: set beresp.grace = 6h; (serve stale content while refreshing). Drupal: cache nodes, views but not user-specific pages. Use ESI (Edge Side Includes) for user blocks. Custom apps: use Cache-Control headers from backend, Varnish respects them automatically. Override with beresp.ttl for default.
Cache Key Variations (Mobile, Currency, Language)
Customize hash key for variations: sub vcl_hash { hash_data(req.url); if (req.http.Accept-Language) { hash_data(regsub(req.http.Accept-Language, ",[^,]+", "")); } if (req.http.User-Agent ~ "mobile") { hash_data("mobile"); } }. Currency: hash_data(req.http.X-Currency) (provided by backend). A/B testing: hash_data(req.http.X-AB-Test). Device detection: use WURFL or deviceatlas mobile detection in VCL. For dynamic variants, consider ESI or Ajax for non-cacheable blocks rather than exploding cache keys.
Cache Invalidation and Purging
Purge content by exact URL: acl purge_acl { "localhost"; "192.168.1.0/24"; }. sub vcl_recv { if (req.method == "PURGE") { if (!client.ip ~ purge_acl) { return (synth(405, "Not allowed")); } return (purge); } }. Ban content by pattern: sub vcl_recv { if (req.method == "BAN") { ban("obj.http.X-Host ~ " + req.http.host + " && obj.http.X-URL ~ " + req.url); return (synth(200, "Banned")); } }. X-URL header set in vcl_backend_response: set beresp.http.X-URL = bereq.url;. Soft purge (keep stale object while fetching fresh): return (purge) vs return (softpurge) in Varnish 6+. Tag-based purging: assign tags (prod-123, category-5), ban by tag on content update.
Grace Mode and Saint Mode
Grace mode serves stale content during backend issues: sub vcl_backend_response { set beresp.grace = 6h; } (serves object after TTL expiry while fetching fresh). Use in vcl_hit: if (obj.ttl >= 0s) { return (deliver); } if (obj.ttl + obj.grace > 0s) { return (deliver); }. Saint mode blacklists failing backend responses: sub vcl_backend_response { if (beresp.status == 500 || beresp.status == 503) { set beresp.uncacheable = true; return (abandon); } }. Or mark backend sick: if (beresp.status == 500) { set beresp.http.X-Failing = "true"; return (retry); }. Use with backend probes to detect failing servers. Grace + Stale-If-Error header from backend: Cache-Control: max-age=600, stale-if-error=86400.
ESI (Edge Side Includes)
ESI fragments cacheable page parts independently. Enabled by default: sub vcl_backend_response { set beresp.do_esi = true; }. Markers in HTML:
Compression and Streaming
Varnish can handle compressed (gzip) content. Ensure backend sends Vary: Accept-Encoding. Varnish stores both compressed and uncompressed versions. sub vcl_backend_response { if (beresp.http.content-type ~ "text/html|css|js") { set beresp.do_gzip = true; } }. Modifying compressed content: disable do_gzip first. Streaming large responses: set beresp.do_stream = true (prevents memory buffering). For video, large files, API responses with pagination. Use streaming for delivery but caching might be less efficient. Range requests (partial content) handled automatically with streaming.
Security Headers and Browser Caching
Remove sensitive headers: sub vcl_deliver { unset resp.http.X-Powered-By; unset resp.http.Server; }. Add security headers: set resp.http.X-Frame-Options = "DENY"; set resp.http.X-Content-Type-Options = "nosniff"; set resp.http.X-XSS-Protection = "1; mode=block"; }. Browser caching (separate from Varnish caching): set resp.http.Cache-Control = "public, max-age=3600"; set resp.http.Expires = max-age time. For static assets: set resp.http.Cache-Control = "public, max-age=31536000, immutable";. Remove Varnish headers from public responses: unset resp.http.X-Varnish; unset resp.http.Via;.
Performance Tuning and Optimization
Thread pools: -p thread_pools=4 (multicore systems), -p thread_pool_min=200, -p thread_pool_max=5000. Memory: malloc storage 8GB+ for high-traffic sites: -s malloc,8g. Increase workspace: -p workspace_backend=64k, -p workspace_client=64k. Session timeout: -p timeout_idle=15. Tune listen depth: -p listen_depth=1024. LRU eviction: monitor n_lru_nuked (varnishstat). Busy workers: -p auto_restart=on, -p max_restarts=4. Use varnishncsa for logging, varnishlog -g request for debugging. CLI commands: param.show, param.set to adjust runtime. Benchmark with varnishreplay (from VCL). Consider using Varnish High Availability (VHA) for crash recovery.
Integration With Nginx/Apache
Standard flow: client -> Varnish (port 80) -> Nginx/Apache (port 8080). Configure Nginx to trust Varnish for client IP: set_real_ip_from 127.0.0.1; real_ip_header X-Forwarded-For;. Apache: RemoteIPHeader X-Forwarded-For, RemoteIPInternalProxy 127.0.0.1. Varnish adds headers: sub vcl_recv { set req.http.X-Forwarded-For = client.ip; }. Backend-specific health checks: Varnish probe hitting Nginx status endpoint. SSL termination: use Nginx (or Hitch for Varnish native TLS). Configuration: Nginx listens 443, proxy_pass to Varnish:80. Ensure Varnish adds X-Forwarded-Proto for backend to generate correct URLs.
Monitoring and Statistics
varnishstat: key metrics (cache_hit, cache_miss, client_req, n_lru_nuked, threads). varnishlog: real-time request logging. varnishtop: most frequent log entries. varnishhist: latency distribution. varnishncsa: Apache-style access logs. Prometheus exporter: varnish_exporter for integration with Grafana. Hit ratio calculation: cache_hit / (cache_hit + cache_miss). Target hit ratio > 80%. Monitoring: alert when hit ratio drops below 60% or n_lru_nuked spikes. Varnish Agent for remote monitoring (API, dashboard). Varnish Plus (commercial) includes real-time dashboard, advanced analytics.
Troubleshooting Common Issues
Cache not hitting: check req.http.Cookie presence, req.method (only GET/HEAD), Authorization header, pragma: no-cache. Use varnishlog -g request -q "ReqURL ~ '^/product'" to check decisions. VCL syntax errors: varnishd -C -f /etc/varnish/default.vcl (compiles and reports errors). Stale content: check beresp.ttl, grace setting, backend Cache-Control max-age. High memory usage: reduce storage (-s malloc,4g) increase nuke_limit (-p nuke_limit=200). Log analysis: varnishlog -g request -q "BerespStatus >= 500" for backend failures. Client disconnections: -p timeout_finch=1s to clean up quickly. CPU spikes: optimize VCL (remove complex regex), increase thread_pool_max, upgrade RAM allocation.
Real-World Example: WordPress with Varnish
Install Varnish Cache plugin for WordPress. VCL: purge on post_save, clear category/tag pages, handle WooCommerce cart exclusion. WP Rocket or W3 Total Cache integration. CloudFlare APO (similar concept) but self-hosted Varnish cheaper for high traffic. Purge on comment/post/update: plugins send PURGE to Varnish API. For logged-in users: bypass cache or use ESI for user bar. Add "X-Cache: HIT" header to identify cached responses. Use varnish-cli for advanced purging patterns. Monitor hit ratio for different post types.
Conclusion
Varnish Cache dramatically reduces backend load and response times. Start with simple WordPress/magento VCL, add custom cache keys for variations, implement purging on content updates, and monitor hit ratio. Tune memory allocation and thread pools based on traffic patterns for optimal performance.