The UK Developer's Guide to GDPR-Compliant Logging

Your logs are probably the least governed dataset in your stack. They record who signed in, from which address, on which device, and what they changed — then they get shipped to a hosted platform where nobody quite remembers setting a retention period. Under UK GDPR that is personal data processing like any other: it needs a purpose, a lawful basis, a retention rule, and an answer when someone asks for a copy. The encouraging part is that compliant logging is mostly good engineering. Here is how to get there.

Where personal data creeps into logs

Very few teams set out to log personal data. It arrives by accident. A request log captures the full URL, and the URL contains a password reset token and an email address. An error handler writes the whole request body, including a date of birth and a free-text note. A validation failure echoes the value that failed, so a national insurance number sits in an aggregator for a year.

Some of it is genuinely useful. Logs are what you reach for during an incident, and they are often the only record of who changed a permission. But the same richness that helps you helps an attacker. Treat log content as a design decision, not a byproduct.

Pick a lawful basis before you pick a log library

UK GDPR requires a lawful basis under Article 6 for each processing purpose. For most application logs, legitimate interests is the realistic answer: keeping the service secure, detecting abuse, diagnosing faults, planning capacity. Where a sector rule demands an audit trail of specific actions, that stream may rest on legal obligation.

Consent is a poor fit. If someone withdraws it, you must stop processing — which means deleting the security record of the very thing you were worried about. Special category data such as health information or ethnicity sometimes lands in free-text fields, and it needs an Article 9 condition on top of a basis. If it is there without one, fix the code, not the policy document.

Legitimate interests is not a free pass. The ICO expects a three-part assessment: identify the interest, show that the logging is necessary and proportionate, and balance it against the individual's rights. Two paragraphs per log stream is usually enough, and it doubles as your accountability record.

Data minimisation: log events, not people

Data minimisation is the most useful lever you have, because you can enforce it in code. Record what happened, not everything that was present when it happened. None of this should ever reach a log line:

• Passwords, password reset tokens, session identifiers, API keys and JSON Web Tokens.
• Full card numbers, CVVs and bank details — these also drag payment card rules into your log platform.
• Authentication headers and cookies.
• Whole request and response bodies, particularly from forms and webhooks.
• Raw exception messages that quote the input value that caused the failure.
• Environment variables and secrets printed by build and deployment jobs.

Replacements are often better data anyway. Log an internal user ID rather than an email address. Log the route template — /orders/:id — rather than the full URL, so query strings and tokens never reach the aggregator. Log an event name and an outcome (invoice.created, payment.declined) rather than a paragraph of free text. If you truly need a value to debug something, put it in a tracing tool with its own access controls, not in a log line that lives for months.

Redaction and pseudonymisation that holds up

Redact at the source

Pipeline redaction is a losing game: by the time a regex has guessed which strings look like email addresses, the data is already moving and the rules are already stale. Build the constraint into the application. Use structured logging with an explicit schema, serialise only known fields, and let unknown fields fail a test rather than reach production. A formatter that drops keys matching a deny-list (password, token, authorization) is a decent second layer, not a first one. Then assert it: build a log entry from a request containing a password field, and fail the build if the password appears in the output.

Hashing is not anonymisation

A hashed email address is pseudonymous, not anonymous. It is still personal data, and an unsalted hash drawn from a small domain can be reversed by brute force. Where you need to correlate logs to a person, use a keyed hash such as HMAC with the key held in a secrets manager. That gives you a lookup path, because you can hash the identifier you were given and search for it. Where you do not need the correlation, do not store the identifier at all.

Truncating an IP address to a /24 reduces identifiability without removing it: a shortened address plus a timestamp plus a user ID is still personal data in most readings. Keep full addresses only where security genuinely needs them, and shorten retention for that stream.

Retention: decide it, set it, prove it

Storage limitation means picking a period and sticking to it. Work backwards from purpose: verbose application logs rarely earn their keep beyond a couple of weeks; ordinary request logs, a few weeks to a few months; security and audit trails, longer where you can justify it. There is no universal number, and inventing one is worse than documenting your reasoning.

Configure expiry in the platform — object storage lifecycle rules, log service retention settings, index lifecycle policies — so nobody has to remember. Then check the awkward places: snapshots, exported CSV files, support bundles on laptops. A line that survives for two years in a backup has not been deleted. Pick one record and follow it all the way to the end.

Access, subject requests and incidents

Production logs often amount to a behavioural picture of your users, so read access is privileged. Role-based permissions, multi-factor authentication and encryption in transit and at rest are the baseline. Log who reads the logs: insider misuse is a realistic risk, and the audit trail is how you would spot it.

Logs are in scope for subject access requests, and a keyed hash gives you a workable search path. Erasure is harder, because immutable log stores are not built for surgical deletion. Short retention is the honest mitigation, since in most cases the data has already expired. Where a line is genuinely needed for security or legal reasons, say so plainly rather than claiming it is gone. If you rely on a hosted logging service, check where it stores and processes data and confirm you have a data processing agreement in place.

A week-one plan

You do not need a programme of work to move the needle. Start with an inventory: which services write logs, what fields they hold, where the data goes, and how long it lives. Then work through this in order.

1. Write down a purpose and lawful basis for each log stream, with a short legitimate interests assessment where that is the basis.
2. Add deny-by-default redaction in the application, plus a test that fails when a known-sensitive field appears.
3. Swap raw identifiers for internal IDs and raw URLs for route templates.
4. Set retention in the platform, including backups and exports, and verify that one record actually expires.
5. Restrict read access to production logs and enable access logging on the log platform itself.
6. Add "what does this add to the logs?" to code review, so the problem does not quietly return.

None of this is legal advice, and it is no substitute for your data protection officer. Anything touching special category data, employee monitoring or a sector-specific audit requirement deserves a proper review before it ships.

Photo: advogadoaguilar / Pixabay