Identify the Source of Unexpected Requests in an On-Prem Network
The company runs its services on old-school on-prem infrastructure: every service runs on a group of similar long-lived virtual machines, each with a stable hostname and IP address. Machines change rarely and often stay around for weeks or even months.
The inventory service, inventory.corp.internal, is one of them. Its regular clients are well known:
the order processing service, orders.corp.internal, looks up items through the API,
and the monitoring job on monitor.corp.internal polls the /health endpoint once a minute.
Recently, the service started receiving requests from other internal addresses, and nobody in your team can explain where they are coming from.
You're given access to one of the inventory service VMs - inventory-01.
The service runs on it and writes an access log to /var/log/inventory/access.log.
The log records the source IP address of every request.
Find out which internal systems are behind the unexpected requests.
The Machines With a Name
Some of the unexpected requests come from machines that have proper names on the company network.
There are two such machines, and their names suggest that they belong to the same service. Identify them by their hostnames, in any order.
Both the short name (e.g., db-01) and the fully qualified name (e.g., db-01.corp.internal) are accepted.
Hint 1
Every line of the access log starts with the client's IP address, and the regular clients make up most of the lines.
You can use text tools like awk, sort, and uniq to produce the list of distinct addresses with the number of requests per address.
Hint 2
On this network, long-lived machines are normally registered in the company DNS.
The resolver on inventory-01 already uses this server (see /etc/resolv.conf).
Resolving orders.corp.internal and monitor.corp.internal can tell you which source addresses belong to the expected clients.
A service may run on more than one machine, so expect a service name to resolve to several addresses.
Hint 3
DNS works in both directions.
Besides turning a name into an address, a DNS server can also map an address back to a name.
This is called a reverse lookup: dig has the -x option for it, and host, nslookup,
and getent hosts accept an IP address in place of a name.
You can use this trick to find names behind the identified unexpected client IP addresses.
The Machine Without a Name
One unexpected source address does not have a reverse DNS record.
It may belong to an old or unmanaged machine that was never properly registered in the company DNS.
Which address is it?
Hint 4
When the DNS server has no reverse record for an address, dig reports status: NXDOMAIN in the answer header,
and host prints 3(NXDOMAIN).
This means the company DNS has no name registered for that address.